Building a more responsible recommending ecosystem

Layered oversight and scrutiny

In charting the path towards a more responsible recommending ecosystem, we first must acknowledge that we are still operating under significant uncertainty: In short, we don’t know what we can’t know. So far, only the platforms themselves know how they’re recommender systems function and how they impact users and society. They alone are in a position to gain a robust understanding of what the causal mechanisms underlying many of the problems associated with recommender systems are and how best to mitigate them. These same platforms have been reluctant to share critical information with the public, researchers and policymakers, with few exceptions. We seek but have no reliable insight into critical questions such as: How exactly and to which degree do recommender systems amplify harmful or illegal content? What is their contribution to social and political polarization in our societies? What biases are built into these systems? And how can they be abused and manipulated by malicious actors?

As publicly accessible information and robust empirical evidence on these and other questions is oftentimes lacking, we all too often are poking around in the dark and rely on makeshift approaches to finding answers. The following recommendations are therefore concerned with rectifying just that by creating systemic transparency, improving the conditions for independent research into recommender systems’ design and operation, and building a layered system of oversight and scrutiny.

Disclose publicly what gets recommended — and why

While we have a high-level understanding of how state-of-the-art recommender systems function, meaningful insights into how the content presented to us online is curated are scarce and hard to find. Recommender systems are composed of a cascade of different machine learning models and rules used to source and filter prospective content, rank it, and ultimately assemble a slate of recommendations. A deeper understanding of a recommender system therefore requires information on its components and overall architecture — yet we know little about what its constituent machine learning models optimize for^[1] and the rules based on which content is filtered out and prioritized. To remedy this, public-facing documentation can be a key enabler of transparency.^[2]

Companies should therefore publicly disclose detailed information on how their recommender systems work and how they are operated. This should comprise information both on the technical design and inputs of the system as well as on rules and policies affecting recommendations (see Box 1 for more detail). Additionally, this information should be as complete as possible instead of providing only select, potentially cherry-picked, examples, for instance of signals used to inform recommendations.

Currently, the little information platforms provide in this regard is often difficult to find — buried on company websites or scattered across occasional blog posts.^[3] Instead, all this information should be easily accessible on a designated information page. Additionally, changes to the information provided on the page, like company policy changes, should be traceable over time via a change log or version history. Further, where relevant, this information should also cover regional or country-specific differences in how recommender systems are operated.

The information provided on platforms’ rules and policies related to recommendations should also cover so-called demotion or reduction policies — often also referred to as de-amplification, shadowbanning, or as relating to “borderline” content.^[4] Such policies are aimed at curbing the distribution of certain content which does not violate (but may come close) platforms’ terms of service or policies. By creating and implementing such policies, many platforms have created an opaque gray area for content that they deem both acceptable and undesirable: within the boundaries of free expression and not harmful enough for removal under platforms’ policies, but too harmful to be distributed widely.^[5] Yet, in contrast to platforms’ rules for content removal or account suspension, there is little information about how this determination is made and how such policies are implemented.

There may be good reasons for platforms to demote certain content that negatively affects users’ experience or the quality of discourse on the platform (for example with regard to clickbait or misleading content) or content that is likely to be harmful but still awaits a removal decision. At the same time, such practices can materially affect users’ ability to reach others and exclude them from public conversations while leaving them unaware or merely suspecting of platforms’ intervention. Such policies are likely to be implemented using the same (or similar) machine learning classifiers used for flagging and automatically removing content that violates platforms’ policies. Yet these tools have well-established limitations^[6] and evidence suggests that wrongful removals disproportionately affect historically marginalized people, for example Black individuals discussing racial justice or LGBTQI+ people (for example, under companies’ nudity policies).^[7] Moreover, self-reported data from users suggests a similar pattern when it comes to content demotion.^[8]

To rectify this transparency deficit, platforms should, where applicable, provide clear information about and criteria for content they demote (including definitions of categories like “borderline” content) and provide a description of how such policies are implemented at a technical level (e.g., is such content merely shown to fewer people or removed from recommendations or search results altogether?). Facebook’s Content Distribution Guidelines^[9] released in late 2021 are a step in this direction, outlining high-level categories of demoted content. However, they still often lack specificity and shed little light on the implementation of the guidelines. At the same time, we are lacking an understanding of the rate of such interventions from platforms. While content removals are reported in platforms’ transparency reports, the incidence and spread of demoted content are indeterminate. As part of existing reporting structures, platforms should therefore disclose aggregate data on the content they demote (see Box 2).

Finally, the personalized nature of platforms’ feeds makes it difficult to discern overall trends. Which post spreads most widely? Which creator gets the most engagement? Generally, we don’t know. After all, such trends emerge from the aggregate of each user’s unique recommendations. To give the public a bird’s-eye view of such trends, platforms should publish information^[10] on and provide tools^[11] to analyze the content, pages, creators, and links that garner the most views and engagement on their service. In the long run, this information might help discern what types of content recommendation systems are particularly disposed to raising to the (public) surface.

Enable more and better research on recommender systems

While there already is important research about online platforms and their recommender systems, it has so far been constrained by a lack of access to data for researchers. Instead, they have to rely on their own tools and methods for data collection, each with their own limitations. As platforms have shielded themselves and their data from public scrutiny, we still know far too little about their inner workings and their impact on users and society at large. Additionally, the research that’s available has disproportionately focused on platforms providing better conditions for research because of this.^[12]

While some platforms have recently moved to open up to researchers^[13] — presumably also in response to upcoming data access obligations under the EU’s Digital Services Act (DSA) — the overall conditions for independent research on platforms and their recommender systems remain insufficient.

Therefore, qualified independent researchers conducting research in the public interest should gain access to non-personal and anonymized or pseudonymized data and documentation and be enabled to audit recommender systems. This type of access should also extend to qualified researchers without academic affiliations, such as investigative journalists or researchers working with civil society organizations, as well as to third-party auditors conducting legally mandated audits (as foreseen by, for example, the DSA) or otherwise working with platforms.^[14] Researchers and auditors should not only be granted access to all publicly accessible content on the platform (in anonymized or pseudonymized form), including content removed for terms of service violations^[15] (see Box 3).^[16] Where justified for the purpose of a research project, they should also receive access to technical documentation related to platforms’ recommender systems and be able to review internal policies and processes related to recommender systems (e.g., on content demotion or promotion). Additionally, researchers should be provided with access to recommender systems (for example via APIs) or other tools allowing them to conduct tests or simulate recommendations and user pathways.

At the same time, data access should be conditional on adhering to data protection rules and principles as well as robust security protocols. The European Digital Media Observatory's proposed code of conduct on researcher access to platform data extensively discusses relevant concerns and safeguards in this regard.^[17] Additionally, data that warrants particularly strong safeguards (such as data posing a risk of re-identifying a user) could, for example, be accessed through so-called "clean rooms"; that is, secured and strictly monitored research environments from which no data may be taken without prior privacy review.^[18] Such an approach has also been put forward in the proposed Platform Accountability and Transparency Act (PATA) in the U.S..^[19]

Beyond enabling data access for researchers and auditors, it is also important to allow for research that does not rely on platforms as a potential gatekeeper. However, researchers who use their own tools to study platforms, create so-called “sock puppet” accounts for research purposes, or crowdsource or scrape data often are at risk of facing legal action from platforms, resulting in chilling effects on such research.^[20] For this reason, a safe harbor should be created for researchers, civil society organizations, and journalists conducting public interest research in compliance with research ethics and data protection rules and principles.^[21] The approach put forward in the proposed Platform Accountability and Transparency Act in the U.S. would be a step in the right direction in this regard. Further, amending existing legislation, like the U.S. Computer Fraud and Abuse Act (CFAA), to include explicit protections for public interest research could provide researchers with more legal certainty.

Audit recommender systems

Our recommendations above would create the foundation for platforms to be held accountable by researchers, journalists, and the public at large. But platforms should not only be held accountable by others — they need to carry out their own due diligence. To this end, platforms should conduct or commission audits of their main recommender systems to systematically identify and manage potential risks and manifested harms to individuals and the public interest that arise from the design, functioning, and use of their recommender systems.^[22] This can help platforms better price into their operations the externalities caused by their services and better take stock not only of business and reputational risk, but also of societal risks and risks to users of their service.

For such an exercise to be meaningful and not a box-ticking exercise, audits should ideally meet two key criteria: first, they should be conducted by sufficiently independent third-party auditors with the necessary access to systems, data, and documentation; second, audit reports should be published — without divulging sensitive information — to allow for scrutiny from experts and the general public. If necessary, such audits could also be mandated by regulators for the largest online platforms. A similar approach has been adopted by the DSA in the EU and early lessons from its application as well as new insights from the emerging field of algorithmic auditing will provide valuable lessons on what an effective framework for audits of platform recommender systems can look like.^[23]

Empowered and informed users

Too often, platforms treat users as passive consumers rather than active participants in their online experience. This paternalistic attitude towards users manifests in different ways: sometimes, users lack the means to exercise meaningful control over their experience on the platform; other times, they lack basic information from platforms needed to make informed choices. But users should not be denied agency. The following recommendations therefore aim at empowering and informing users and enabling them to better shape their experience online.

Give users more control over recommendations

First, platforms should allow users to better customize the recommendations or content displayed to them.^[24] For instance, providing additional controls or choices can enable users to better protect their safety on the platform. To this end, users should generally be able to exclude certain keywords, creators, or pages from their recommendations. Further, they should be able to indicate certain types of content they want to exclude from recommendations or be less exposed to (for example through a “do not recommend” button). While many platforms have already implemented such controls, recent Mozilla research on YouTube user controls has also shown that merely offering such controls is not equivalent to effectively empowering users.^[25] Instead, the research has found that YouTube’s user controls are both confusing and often fail to produce the purported effect, only insufficiently or temporarily suppressing unwanted recommendations.^[26] This example illustrates that controls (and intended outcomes) should be easy to understand and access as well as effective at what they purport to achieve.

Platforms should further enable users to exert better control over which of their data, including personal data, is collected and subsequently used to inform recommendations.^[27] This should also extend to being able to exclude data collected from other related products or services (e.g., Google search data used to inform YouTube recommendations) or from previous engagement with certain content, pages, or users. In all cases, such controls should be designed in a manner that promotes genuine choice rather than using deceptive design practices incentivizing users to choose the platforms’ preferred outcome.

Further, users should be provided with an option to opt out of personalized recommendations or to influence the degree of personalization at any point in their product experience. Personalization in this context means recommendations based on personal or behavioral information related to users that was observed or inferred by platforms (i.e., platform-driven personalization). It should not extend to user-driven personalization, such as following or subscribing to certain accounts or channels. Should users opt out of personalization, they might, for example, receive chronological recommendations, recommendations including the most popular content in a given location, or purely search term-based recommendations instead.

Enable users to make more informed choices

While most platforms do provide some transparency to users into why specific advertisements are presented to them, they do little in this regard when it comes to regular content. But having some understanding of why certain content is displayed can be important for users to better customize their experience on the platform — also given that empirical evidence suggests that many users still lack awareness of the algorithmic curation of their feeds.^[28] Despite limitations of AI explainability, platforms should do their best to provide users with an easily accessible and meaningful explanation of why specific content is displayed to them. This should at a minimum include information on the key data points or signals influencing the recommendation in question. Furthermore, the explanation should point to information on how the recommender system works more generally (see above) and to controls available to users.

Finally, like content and account removals, content demotion can have significant impacts on people’s ability to participate in the public conversation and express opinions. Moreover, it can also cause adverse economic impacts, for example for professional creators or artists who rely on platforms for distribution or monetization. But while users are generally notified by platforms when their content or account is removed, content is demoted silently. As a general rule, and in the spirit of the Santa Clara Principles on Transparency and Accountability in Content Moderation,^[29] platforms should also notify users when their content is demoted as a content moderation intervention.^[30] Notices should include information identifying the content in questions as well as the reasons for which it was demoted (e.g., because the content was considered to be close to violating a platform’s nudity policy). After all, in most cases, this relates to content that is expressly permitted on the platform.