Published on | Prototyping: From UX to Front End — Medium Sacha Alanoca

In the name of service efficiency, both public and private sectors are interested in the rapid analysis and collection of personal data. However, the assumption that citizens rate service efficiency higher than data privacy and fairness is rarely questioned. I will highlight the need to open up the debate to define what kind of data we would like to protect and propose first hand solutions to help us gain data control. These solutions include a traffic light signal for manipulative design patterns, and a nutritional chart to see the main “ingredients” in algorithms.

The intimacy of being at home. ©Mehdi Vilquin

The State of IoT and Data Privacy

I was sitting next to my friend Guillaume at our regular café, when he pointed to something on his phone. I moved closer to have a look but to my surprise could not see anything. His screen was black. Guillaume saw my confusion and laughed, telling me he had forgotten that his privacy screen protector was on — he hates when people look over his phone conversations and searches in the metro, and the screen protector prevents side-views.

This interaction with Guillaume left me contemplative — in this new age of the Internet of Things (IoT) and Artificial Intelligence (AI), I could not figure out if his concern was pragmatic or naive. He could maybe hide the content of his phone conversations from his metro-seat neighbor, but what about the rest of his personal life? The breakthrough of emerging technologies such as as smart home devices, autonomous vehicles, mobile finance and biotech all contribute to blurring the lines between the physical, digital, and biological spheres. With the power of AI and the connectivity of IoT, huge amounts of data were now flowing, making it increasingly possible to track and reconstruct his activities in minute detail. Would a screen phone protector still be enough to reassure him in this context?

Last year, Amazon’s Alexa opened up the market for smart home devices. With more than 20 million virtual assistants sold in 2017 and a web-connected microphone placed inside your home, we can expect just as many personal conversations being recorded and navigating the Cloud (information is not stored on the device’s local server). And this is just the beginning — it is estimated that in 10 years’ time, there will be 150 billion networked measuring sensors, 20 times more than there are people on Earth. This will lead the amount of data to double every 12 hours. With so many connected devices and data leaping, it is hard to picture the future implications on our everyday lives.

Convergence of the Interests of Tech Companies and Government Agencies

Nevertheless, it will undoubtedly lead to huge opportunities for both the public and private sectors. As the prominent tech researcher and writer Evgeny Morozov stresses (2013):

A symptom of the current malaise is that the commercial interests of tech companies and the policy interests of government agencies have converged.

In the name of service efficiency, both are interested in the collection and rapid analysis of user data. And we, as citizen-users, are compelled to give more data — to have access to key digital services that are tailored to our needs and solve problems in the fields of health, insurance, and education. We are compelled to feed algorithms with our personal data — the fuel that is conditional for new advances in machine learning.

Assumption that Service Efficiency is Rated Higher than Data Privacy and Fairness

The convergence of these interests comes from the same underlying assumption — that we, citizen-users, rate service efficiency higher than data privacy and fairness. In fact, this assumption is rarely questioned or even publicly debated. With so much going on in our everyday lives, we often find ourselves running against the clock. As a result, we conveniently push our privacy to the background.

The potential misuse of this personal data can already be observed. As flagged by a report from the Center on Privacy and Technology at Georgetown Law School, advances in face recognition technologies make it possible for the police to scan people in real time on surveillance cameras. And with no updated regulation or public debate, many state and local police departments across the United States are building their own face recognition systems. Many of them are more advanced than that of the FBI and are fed by records of convicted criminals, as well as suspects’ drivers license and ID photos. Never before have police forces been able to tap into such a database, especially with such powerful face-recognition technology. And the benefits are certainly clear: violent criminals and fugitives have already been apprehended. What is less clear is how this system affects people’s attendance at protests and outdoor concerts, and at the very core of this, their civil liberties and right to free speech. What would Guillaume think of all of it? Except maybe in a Black Mirror episode, no privacy-screen face protection could hide his presence.

This is particularly concerning because it comes from public authorities in whom we tend to place our trust without first inspecting their underlying methods. Equally concerning are the new methods being used in the retail sector based on similar face recognition systems. “No queues and no checkout,” has been a long-standing ambition to define the future of retail. It was first envisioned by Amazon, but skipping the debate over data privacy and regulation, Chinese companies have jumped the gun, making the concept a reality. Chinese giant Alibaba introduced its futuristic retail stores Hema last year in Shanghai and Beijing. Thanks to its payment mobile app Alipay and a facial recognition system connected to each customer’s account, Hema allows cashier-less contact. This shopping experience may at first seem fun and is certainly less time-consuming, but once again this service-upgrade comes at a privacy cost. Supermarkets can track your shopping patterns and habits and directly link it to your physical appearance. It is an intrusive exercise in your personal routine that is erased by the fluidity and convenience of this new concept.

In both cases mentioned above, the benefits for service efficiency are undeniable. What is questionable is whether we believe that these benefits currently compensate for the data privacy cost engendered.

Our metrics for service success might not be the right ones — but they will remain unchallenged with no debate on what we value most.

Algorithmic Recommendations Blind Us to a Standardized, Predetermined Behavior

One of the reasons we tend not to question the assumption that we rate service efficiency higher than data privacy and fairness — is because the cost of this efficiency is often hidden from us. In other words, algorithmic recommendations seem so well-aligned that the resulting decisions often feel like our own. As the data privacy jurist Spiros Simitis (1987) warns:

The more is known about us, the less likely our choices are to be free and not predetermined by others […]. Habits, activities and preferences are compiled, registered, and retrieved to facilitate better adjustment, not to improve the individual’s capacity to act and decide. Whatever the original incentive for computerization may have been, processing increasingly appears as the ideal means to adapt an individual to a predetermined, standardized behavior that aims at the highest possible degree of compliance with the model patient, consumer, taxpayer, employee, or citizen.

By sorting people into different categories and assigning them e-scores to determine their insurance policies or job qualifications, for example, we may be increasing service efficiency. However, the opacity of algorithms and the fact that we ignore what is taken into account to evaluate us, and where the information compiled on us comes from, directly affects our free will and right for information self-determination.

E-scores define the Human MarketPlace. © Mehdi Vilquin and Pablo Hnatow

By not being in control of our personal information, and letting someone else determine the personalized ads, search results, or newsfeed articles appearing on our screen, a “filter bubble” slowly emerges and blurs the clarity of our perception — of ourselves and of others. It pushes us to a state of intellectual and social isolation by assuming what we want to see based on our past user activity, and selectively erasing potential contradictory viewpoints (Pariser, 2011).

In 2013, the University of Illinois carried out a survey on the Facebook algorithm. The findings of the researcher Karrie Karahalios were disturbing— about 62% of the people interviewed were unaware that the company’s algorithm twisted their newsfeed. Most of them believed that Facebook shared everything that they posted instantly with their friends. Similarly, a Pew Research Report published in 2016 disclosed that 73% of Americans believed that search results were impartial.

This lack of awareness of the existing algorithmic recommendations by such large samples of the population is worrisome. It means that this “filter bubble” might actually be working, making us more tidy and predictable citizen-users. As Evgeny Morozov highlights, “invisible barbed wire” is growing around our intellectual and social lives and “the more information we reveal about ourselves, the denser but also more invisible this barbed wire becomes.”

Dark Patterns Blind Us Further by Capturing Our Time and Attention

This is just one side story. Algorithmic recommendations highlight what can be done with our personal data to control us without us being aware of it. But how the big tech companies, such as social networks, capture the data that feeds these algorithms is another concern.

In a Medium article that quickly went viral in 2016, former Google employee and Design Ethicist Tristan Harris asked how does technology hijack people’s minds? He compared product designers, who, for example, design the layout of our user interface (UI) and user experience (UX), to magicians — both focus on the blind spots, vulnerabilities and limits of people’s perceptions to influence them without being aware. By exploiting our mind’s weaknesses, product designers can create dark patterns that capture our attention and trick us into doing things that we did not intend to do in the first place. Facebook, Instagram or LinkedIn’s scrolling features are examples of dark patterns. One might want to log onto Facebook just to check the address of their event on a given night, but the design layout does not permit them to do so. Instead, they must first look at their newsfeed and then click on the link for their events.

This design was selected to capture our attention. We end up spending more time on these social networks, clicking, liking and watching more content, and thus providing more of our precious data.

Dark patterns play with our psychological vulnerabilities, even though in the name of service efficiency, once again, tech companies will justify these methods.

Ironically, as Tristan Harris emphasizes, even though free will is a principle strongly rooted in our Western culture and “millions of us fiercely defend our right to make ‘free’ choices, […] we ignore how those choices are manipulated upstream by menus we didn’t choose in the first place.” This fictional idea of having free will rarely leads us to question what’s not on the menu, and the goals of the menu provider.

By shaping the menus we pick from, technology hijacks the way we perceive our choices and replaces them with new ones. But the closer we pay attention to the options we’re given, the more we’ll notice when they don’t actually align with our true needs.

In other words, we may be able to choose between options but the quality of alternatives is defined elsewhere. Again, this is problematic because it threatens our free will and right to self-determination. People need to gain consciousness of the value of their data, and the way some of this data is acquired.

They need to have ownership of their digital selves in order to regain control of their identity and personal development. Privacy is not a luxury but a right. Individuals should be able to freely form and express their opinions, controlling the life-cycle of the data that they are ready to share. If this is not the case, then it is no less than a cornerstone of our constitutional rights and democracy that is endangered.

Algorithmic recommendations and dark patterns blind us. ©Mehdi Vilquin

The Need to Open Up the Debate to Avoid the Myth of 100% Privacy

Not deliberating over data privacy and how much we rate service efficiency polarizes the debate. This is one of the greatest dangers at play here. 100% of data privacy is not only impossible to reach, but also undesirable. There is some data that we are willing to share, and I understand that machine learning advances, at least currently, are conditional on massive data input. The challenge is not to defend full data privacy or service efficiency, but rather to guide people into considering what kind of data they would like to protect, what kind of data they would like to share, and what kind of data is open for negotiation. Different data entails different reactions. Only by recognizing this can we make concrete progress and propose first hand solutions.

What are (Some of) the Potential Solutions?

So — now that we have addressed the need to open up the debate on data privacy and invited people to sort out their data, we can be slightly more optimistic and propose some first-hand solutions.

Different data entails different attitudes. ©Mehdi Vilquin

Creating a Data Currency

One of the solutions that has been most discussed is the creation of a data currency, that would allow user-citizens to assert ownership over their digital selves. This is only part of the solution, as it only applies to the data that we are willing to share against something else. It cannot be applied for all of our data — there is some sensitive and highly-personal information that I would never want to share and that should be respected. Nevertheless, being paid for some of our data could be a first step towards gaining more control over it.

We could, for example, have a certain amount of money transferred to our bank account for each post or picture liked (although not in bitcoin, as I am less of an advocate of the crypto-currency since I realized how much energy it consumes!) Tech companies could even consider, when applicable, providing users with credits that they could spend on their goods and services. Amazon, for instance, could provide its users with the option of payment in money or in credits, which could then be spent on a range of pre-selected books, clothing items, etc.

A challenge with data currency though, is that it is hard to estimate the financial return of each set of our personal information, and thus estimate a fair compensation for it. Even more dangerously, we can easily predict that some data might be rated as more valuable than others — someone with a $60,000 income might be a better target consumer than a recent college graduate. This could lead to market distortion and social discrimination, in which people who are the best off get the highest return on their data whereas those who are the worst off get close to nothing.

Information & Transparency

In the end, this all must start with information and transparency. People have to be aware that data is an asset, and of the different ways that it can be manipulated.

I recently discovered a very useful app called Yuka. It allows its users to scan food products, see their ingredients, and be alerted if there are any dangerous food additives. The application even sorts products into four categories (excellent, good, average, and bad) to help you get a quick snapshot of what you’re putting into your cart. I wish the same existed for algorithms. It would be great if at first glance I could see the main ingredients of their mathematical recipe — if they took into account my age, my weight, the last article I read, or clothing item I bought to target me, and what was the impact of each these ingredient-variables.

I imagine an indicator similar to a nutrition chart in the top left corner of my screen, letting me know the mathematical recipe of these algorithms.

Nutritional chart for Facebook’s newsfeed algorithm ingredients. ©Mehdi Vilquin

Realistically speaking though, it may be hard to oblige private companies to release the precious industrial secrets contained in their algorithms. Public agencies, however, should agree to open their formula up — whether for algorithms used for job or university recommendations, or to define an individual’s recidivism rate, public agencies should be transparent and accountable.

Another helpful feature, that could be applied to both public and private sectors, would be to rate the number of dark patterns being used on a website. How many methods are being used to make us lose more time on our screens and capture more of our data. In a similar fashion than previously, websites could be ranged into different categories — green, yellow or red traffic lights for example — to quickly alert us as to how our data is being taken.

Traffic lights to identify dark patterns. ©Mehdi Vilquin

All of this entails the creation of an independent auditing agency — a sort of algorithmic and dark pattern police — that would investigate how our data is captured and what is done with it. A pretty good way of using public spending in my view. Already, some universities and activists are showing the way, such as Princeton’s Web Transparency and Accountability Project and the London-based Algorithmic Justice League.

Privacy by design

Auditing algorithms and dark patterns is just one part of the equation. Tech companies and public agencies can also take the initiative of making their design more ethical and user-friendly. If the data privacy debate was wide-enough, they probably already would have no other choice.

A very obvious example of this is Facebook’s newsfeed. It captures a lot of our time and attention, and if we had the possibility of turning it on or off, like you already can with the chatting feature, it would give us more control.

A second example of ethical design would be to summarize confidentiality policies. It is no secret that very few of us actually read all of what is written when we create a social media profile or email account. Even though confidentiality policies apply directly to what is done with our personal information and its life-cycle, it seems like everything is done to watermark these facts in a sea of words.

If the confidentiality policy layout were synthesized (with the possibility of unfolding all of it) — it would make things much clearer and more intelligible.

Sending sensitive information through ProtonMail service.

A final example of ethical design would be to have more options of how to protect the content of our exchanges. ProtonMail is an excellent example of what can be made. This free and ergonomic email service, developed by CERN and MIT scientists, allows its users to encrypt the content of their messages and decide for how long an email will be kept (a week, a month, a year, etc). These options allow us to gain control over our digital selves, and reflect on the different attitudes we have in front of different data.

Building a New Generation of Politicians, Data Scientists and UX/UI Designers

Of course, all of these advances are conditional to a real debate about data privacy, and a new generation of politicians, data scientists and UX/UI designers. To enable this change of mindset, education needs to adapt. We need degree programs in colleges and fellowships for people interested in public-interest technology. We should also provide children with a minimum understanding of a responsible and critical use of digital technologies from an early-age — to be aware of these algorithmic recommendations and dark patterns from the very start.

And for all people interested in technology-related careers, ethics should be taken into account through interdisciplinary courses on ethical design or AI and ethics.

The mathematician and data scientist Cathy O’Neal presents original insights as to how to include ethics in technological careers. In her book Weapons of Math Destruction (2016), she mentions the case of Emanuel Derman and Paul Wilmott, two financial engineers who after the 2008 crisis published the Financial Modeler’s Manifesto. Inspired by the Hippocratic Oath that medical students pledge when they become doctors, and by the structure of the Communist Manifesto, Derman and Wilmott developed a list of guiding principles for aspiring traders. These are some of their principles:

“ — I will remember that I didn’t make the world, and it doesn’t satisfy my equations. […]

– I will never sacrifice reality for elegance without explaining why I have done so.”

This provides philosophical grounding and an attempt to plant solid values to encourage self-scrutiny. As Cathy O’Neal suggests, data scientists could pledge to a similar kind of oath. Behind its symbolic reach, this initiative could encourage data scientists to evaluate the fairness of their models, and ask how much of its efficiency they would be willing to sacrifice for fairness. It would invite them to take a step back.

Setting the Right Tempo

The underlying question behind these solutions is whether we as a society are willing to sacrifice a bit of efficiency in the name of data privacy and fairness; whether we agree to leave out certain types of data, to have slightly less performant algorithms and perhaps slower advances in machine learning to protect free will and democracy.

This is not an easy debate, and it will require all parts of society to be highly informed and to have critically reflected on the matter.

It will undoubtedly also require policymakers and regulators to open up and lead the debate. To do so effectively, their role, general cultural knowledge, and the selection criteria through which they are recruited will need to adapt. Both policymakers and regulators need to be in concert with emerging technology breakthroughs, their societal impact, and the ethical implications. This will require continuous training, hybrid academic and professional backgrounds, and selection criteria taking into account tech-related knowledge. It also requires policymakers and regulators to not fear technological changes — especially in the field of AI. This fear too often leads them to simplistic conclusions or to leave AI radically outside the public policy agenda.

Having worked in both a governmental agency and a tech company, there is something I observed when it comes to technology: how hard it is to set the right tempo. People tend to be either passive because they feel any decision will be untimely, or hyperactive because they feel they are missing out and their competitors are leading the game.

With so many working parts moving so rapidly, it seems really difficult to act in time. When in fact, what is most needed, is often just to step back to find the right tempo. Only then will we as a society ask the right questions and find the best solutions.

Written by Sacha Alanoca, in collaboration with designer Mehdi Vilquin.

Contact me if you would like more details on the bibliography or have any questions.

Data Privacy and Free Will : Setting the Right Tempo was originally published in Prototypr on Medium, where people are continuing the conversation by highlighting and responding to this story.

Data Privacy and Free Will : Setting the Right Tempo

Top Stories