Does this button work?
Investigating YouTube’s ineffective user controls
Powered by 22,722 volunteers, Mozilla scrutinized YouTube to determine how much control people actually have over the platform’s recommendation algorithm. This is what we learned.
Read the Full Report
People feel like they don’t have control over their YouTube recommendations…
Our 2021 investigation into YouTube’s recommender system uncovered a range of problems on the platform: an opaque algorithm, inconsistent oversight, and geographic inequalities. We also learned that people feel they don’t have control over their YouTube experience — particularly the videos that are recommended to them.
YouTube says that people can manage their video recommendations through the feedback tools the platform offers. But do YouTube’s user controls actually work?
and our study shows that they really don’t.
Our work is the largest experimental audit of YouTube by independent researchers, powered by Mozilla’s RegretsReporter.
Using RegretsReporter, an open source tool Mozilla built to study YouTube’s recommendation algorithm, we were able to independently audit the platform’s user controls.
- We combined qualitative and quantitative insights to paint a more complete picture of the effectiveness — or rather, ineffectiveness — of YouTube’s controls.
- We leveraged a massive-scale community to collect our data.
- We applied rigorous and powerful research methods including a randomized controlled experiment and a machine learning model.
22 722
Participants
567 880 195
Videos Analyzed
2 758
People Surveyed
Ultimately, we found:
People feel that using YouTube’s user controls does not change their recommendations at all.
They’re right. YouTube’s user control mechanisms don’t effectively prevent “unwanted” recommendations.
People take a trial-and-error approach to controlling their recommendations.
In the qualitative portion of our study, we learned that people do not feel in control of their experience on YouTube, nor do they have clear information about how to curate their recommendations. Many people take a trial-and-error approach to controlling their recommendations using YouTube’s hodgepodge of options, like “Dislike,” “Not Interested,” and other buttons. It doesn’t seem to work. Said one person:
“Nothing changed. Sometimes I would report things as misleading and spam and the next day it was back in. It almost feels like the more negative feedback I provide to their suggestions the higher bulls**t mountain gets. Even when you block certain sources they eventually return.”
When YouTube’s own controls fail, people take matters into their own hands, relying on privacy tools or even changing how they watch videos on YouTube.
People said:
I avoided clicking a video I would like to watch, only because I was worried that doing so would lead me to get politically extreme recommendations.
When the Superbowl came around…if someone recommended a particular commercial, I used to log out of YouTube, watch the commercial, and then log back in. I have sometimes even gone to a different device just because…I don't want that clouding my recommendations.
Just not clicking or hovering over certain videos in hope things will improve.
But it’s hard to know what really works.
Let’s call on YouTube to empower users with real, meaningful controls
Sign the Petition
The result? People can’t escape bad recommendations.
In the quantitative portion of our study, we ran a randomized controlled experiment across our community of RegretsReporter participants that could directly test the effectiveness of YouTube’s user controls. We found that YouTube’s user controls somewhat influence what is recommended, but this effect is meager and most unwanted videos still slip through.
Percentage of Bad Recommendations
among high-risk video pairs assessed by our research assistants
What is a bad recommendation?
In our experiment, a “bad recommendation” is when YouTube recommends videos to users that are similar to a video they had previously rejected (clicked “Stop Recommending”). We determined whether videos were similar by relying on assessments by our research assistants and our machine learning video similarity model.
Which method was most effective?
None of them really.43
Don’t recommend channel
11
Not interested
12
Dislike
29
Remove from watch history
Percentage of bad recommendations prevented
Bad recommendations resurfaced over time.
“It seems like you routinely have to prune things away or it will keep shoving them in your face until you tell it otherwise.” (Participant 112)
Scroll to explore a real recommendation timeline from our study
A Grandma Ate Cookie Dough For Lunch Every Week. This Is What Happened To Her Bones.
A user sees a video they would rather not see again
Dec 8, 2021
A Grandma Ate Cookie Dough For Lunch Every Week. This Is What Happened To Her Bones.
The user decides to use YouTube’s feedback mechanism, ‘Remove from history’
Dec 8, 2021
A Dad Ate 25 Packs Silica Gel For Breakfast. This Is What Happened To His Stomach.
A Student Ate 108 Gummy Antacids For Breakfast. This Is What Happened To Her Kidneys.
But similar videos keep showing up…
A week later
A Dad Mistakenly Drank A Lava Lamp At Bedtime. This Is What Happened To His Kidneys.
A Student Felt A Sharp Pain In Her Side. This Is How Her Organs Shut Down.
A Mom Drank 3 Gallons Water In 2 Hours. This is What Happened to Her Brain.
A Gamer Drank 12 Energy Drinks In 10 Minutes. This Is What Happened To His Organs.
A Woman Drank "35% Food Grade Hydrogen Peroxide." This Is What Happened To Her Brain.
A Student Drank 2 Bottles Over The Counter Cough Medicine. This Is What Happened To Her Brain.
A Toddler Drank His Mom's Essential Oils. This Is What Happened To His Brain.
…and showing up
A month later
A Student Ate Suspicious Leftovers For Lunch. This Is What Happened To His Limbs.
A Boy Ate 25 Laxative Brownies In 1 Hour. This Is What Happened To His Kidneys.
A TikToker Drank 2 Bottles Benadryl. This Is What Happened To Her Organs.
A Bitcoin Miner Heatstroked In His Sleep. This Is What Happened To His Organs.
A TikToker Drank 1 Liter Cough Syrup. This Is What Happened To His Brain.
A Man Drank 6 Glowsticks For Dinner. This Is What Happened To His Stomach.
A Man Swallowed A Fishbone. This Is What Happened To His Liver.
A Student Ate 5 Day Old Pasta For Lunch. This Is How His Liver Shut Down.
A TikToker Chugged 8 Scoops PreWorkout Supplement. This Is What Happened To His Brain.
A Dad Drank A Snowglobe. This Is What Happened To His Kidneys.
A Grandma Ate Cookie Dough For Lunch Every Week. This Is What Happened To Her Bones.
A Man Ate 2 Pounds Licorice Candy. This Is What Happened To His Organs.
A Grandma Ate 1 Pound Chocolate In 6 Hours. This Is What Happened To Her Brain.
A Man Ate 100 Zinc Vitamin C Gummies Everyday. This Is What Happened To His Spinal Cord.
…and showing up
Two months later
A Boy Ate 150 Gummy Vitamins For Breakfast. This Is What Happened To His Bones.
A Toddler Chewed Lead Paint Off His Toys. This Is What Happened To His Brain.
A Girl Suddenly Grew A Beard. This is What Happened To Her Ovaries.
A Student Felt A Sharp Pain In Her Side. This Is How Her Organs Shut Down.
A Toddler Played With His Cat. This Is What Happened To His Brain.
A Boy Ate Only Chips And French Fries For 10 Years. This Is What Happened To His Eyes.
A Dad Didn't Brush His Teeth For 40 Days. This Is What Happened To His Kidneys.
A Lawyer Couldn't Sleep For 9 Days. This Is What Happened To Her Colon.
A Woman Drank 1 Liter Soy Sauce Colon Cleanse In 2 Hours. This Is What Happened To Her Brain.
A Grandpa Set His Clock Forward 1 Hour. This Is What Happened To His Heart.
A Student Ate Gas Station Sushi For Breakfast. This Is What Happened To His Stomach.
A TikToker Drank 1 Bottle Nutmeg Spice. This Is What Happened To His Brain.
A Man Played Video Games Nonstop For 73 Hours. This Is How His Organs Shut Down.
A Scientist Spilled 2 Drops Organic Mercury On Her Hand. This Is What Happened To Her Brain.
A Dad Drank 50 Beers Everyday For 6 Weeks. This Is What Happened To His Brain.
A Man Was Licked By His Dog. This Is How His Organs Shut Down.
A Chef Ate Gas Station Nachos For Dinner. This Is What Happened To His Limbs.
A Diabetic Mom Caught A Fever. This Is How Her Kidneys Shut Down.
A Man Microdosed Dark Web Bought Mushrooms. This Is What Happened To His Organs.
A Student Felt A Sharp Pain In Her Side. This Is How Her Organs Shut Down.
…and showing up
Three months later
“Eventually it always comes back. The algorithm seems incapable of remembering a lesson for very long.” (Participant 187)
Even the most effective feedback methods prevent less than half of bad recommendations.
Our main recommendation is that YouTube should enable people to shape what they see.
YouTube’s user controls should be easy to understand and access. People should be provided with clear information about the steps they can take to influence their recommendations, and should be empowered to use those tools.
YouTube should design its feedback tools in a way that puts people in the driver’s seat. Feedback tools should enable people to proactively shape their experience, with user feedback given more weight in determining what videos are recommended.
YouTube should enhance its data access tools. YouTube should provide researchers with access to better tools that allow them to assess the signals that impact YouTube’s algorithm.
Policymakers should protect public interest researchers. Policymakers should pass and/or clarify laws that provide legal protections for public interest research.
People who use YouTube can get informed about how YouTube recommendations work and download RegretsReporter to contribute data to future crowdsourced research.
Mozilla will continue to operate RegretsReporter as an independent tool for scrutinizing YouTube’s recommendation algorithm. We’ll run research experiments, analyze data, and publish our findings.