About our Methodology

The goal of Mozilla’s *Privacy Not Included buyer’s guide is to help consumers shop smart—and safe—for products that connect to the Internet. Many companies don’t provide clear information about the privacy and security of the connected devices they sell. Is your personal data being used in ways you may not have anticipated or expected? What are the risks of buying a product that collects your health data? How does the company regularly test for and fix security vulnerabilities?

It’s difficult for consumers to get clear, concrete information from companies about the security and privacy of their connected products. With this guide, we hoped to help consumers navigate this landscape by understanding what questions they should ask before buying a device.

Here is the methodology we used to develop this guide.

Methodology

Product Selection

There are 136 products in the 2020 version of our *Privacy Not Included buyer’s guide. These products fit into seven categories—Toys & Games, Smart Home, Home Office (*new), Entertainment, Wearables, Health & Exercise, and Pets.

The goal was to select connected products that were likely to be popular during the holiday season and beyond. We worked with a research firm to provide us with a list of top products in each of our product categories. From there, we refined the list based on our own research of top selling products that were highly rated across a variety of consumer product websites such as Consumer Reports, Wirecutter, The Toy Insider, PC Magazine, Tech Radar, and Gear Brain.

*Privacy Not Included Warning Labels

This year, we assigned *Privacy Not Included warning labels to products we felt had the most problems or gave us concerns about how the data collected was or could be used. We gave a product a *Privacy Not Included warning label if a product received multiple “mini-warnings” for privacy, had an egregious track record of not protecting privacy, or if we could not confirm if they met our Minimum Security Standards.

Worst-case scenario

What could happen if something goes wrong?

We included this section to help people understand risk scenarios related to their privacy and each particular product. We aimed to identify risks that would feel relevant to consumers. It’s likely nothing bad will happen with most of the products in this guide. However, it’s also good to think through what could happen if something goes wrong. This question looks at potential worst-case scenarios for each product, in some cases for fun, and in some cases, based on things that have already happened with the product.

Minimum Security Requirements

Mozilla has established a set of Minimum Security Standards it has determined should be met by any manufacturer developing connected devices. We emailed each company to ask for more information about their product and how it meets our standards. For the companies who did not respond, we conducted additional research to find the answers.

We evaluated each product on our list against five criteria:

Encryption

A product should use encryption in transit and at rest (where applicable). The product must use encryption for all of its network communications functions and capabilities, ensuring that communications aren’t eavesdropped on or modified in transit. User data should be encrypted when it is stored.

Security updates

The product must support automatic security updates for a reasonable period after sale, and be enabled by default. This ensures that when a vulnerability is known, the vendor can make security updates available for consumers, which are verified (using some form of cryptography) and then installed seamlessly.

Strong passwords

If the product uses passwords for remote authentication, it must require that strong passwords are used, including having password strength requirements. Any non-unique default passwords must also be reset as part of the device’s initial setup. This helps protect the device from vulnerability to guessable password attacks, which could result in a compromised device.

Vulnerability management

The vendor must have a system in place to manage vulnerabilities in the product. This must also include a point of contact for reporting vulnerabilities or a bug bounty program. This ensures that vendors are actively managing vulnerabilities throughout the product’s lifecycle.

Privacy Policy

The product must have a privacy policy and/or another privacy page that applies to the device, app, or service we are evaluating.

Privacy

We evaluated the privacy documentation—which included privacy policies, privacy pages, FAQs—for each company making these products to determine (1) what kind of information is generally being collected (personal, biometric, and social), (2) how data is used, (3) how you can control your data, (4) the known track record of protecting customer data, (5) if the product can be used offline, (6) and whether the privacy policy was user-friendly.

What kind of information is being collected?

We listed the personal information collected for each product, as detailed in the privacy policy. This kind of information includes, but is not limited to name, email address, phone number, address.

Biometric data is data that describes our bodies and distinctive personal characteristics. Many devices collect sensitive data about our heart rate, our sleep patterns, and our menstrual cycle, for instance. Some even use data about our face and voice to identify us.

Social data includes information about your friends and contacts. We detailed which products collect this social information. This did not include the sharing of links or other information on social media through the product itself (e.g. sharing your route for a run on Facebook).

How is data used?

How do companies collect and share customer data with third parties? For this question, we analyzed the privacy documentation to determine how and when personal customer data is shared with third parties for reasons other than expected or unrelated to fulfilling operations.

For instance, if a company can share or sell personal customer data with third parties, or if third parties can use data for commercial purposes, then we noted this.

How can you control your data?

Does the company provide a way for users to request deletion of their data? We looked for language around this in the privacy policy and/or whether the company had an online portal that lets users delete their data quickly.

What is the company’s known track record of protecting users’ data?

We evaluated each company’s history of protecting customer data from January 2018 onward. We conducted research to find data breaches or any other incidents.

Can the product be used offline?

We checked to see if the product could be used offline or if being online was a requirement to use the product effectively.

User-friendly privacy information

How easy is it to find clear information about a device’s privacy policies? How clearly is that information stated? Privacy information should be clear, readable, and communicate basic information to consumers about what happens to their data. We looked for whether the company provides easy-to-read privacy information, either in the privacy policy or in another privacy page.

Permissions

We checked to see whether the device or its app uses a camera, microphone, or tracks your location.

For you to set up your new device, you will probably need to download an app. Both of these (the device and the app) will most likely need data to make things work. The app will typically need to request permissions for it to access your more sensitive data.

In our ratings, we evaluated if the device or the app required access to the camera, microphone, or location information. We evaluated the device based on the product website and we used the Google Play store for Android to check on the permissions requested by each app. (Note: apps may access “approximate” or “network” based location. “Tracks Location” was marked as “Yes” if an app requests any location information, including approximate location.)

Artificial Intelligence

We evaluated whether or not the product uses artificial intelligence. We defined AI as: Changes are made to the product’s technology continually based on your user data. This would cover Alexa changing to better understand what you say, to your fitness wearable making recommendations on exercises to do so that you meet a specific wellness goal, or your security camera deciding not to alert you because it can distinguish a racoon from a burglar.

Does the product use AI?

Based on responses from companies and our own research, we noted whether or not a product used AI, or if we could not determine whether or not the product used AI.

Does the AI use your personal data to make decisions about you?

Based on our research, we noted whether or not decisions were made about you. This included a smart assistant knowing that you are talking to it or a foam roller telling you that you are doing an exercise correctly.

Are you able to get information about how the AI works?

One of the biggest issues surrounding artificial intelligence in our consumer products is having access to essential information about how AI-enabled features work. If we could find academic papers or other documentation about the AI, we marked this as Yes, and we included that information in a product’s page.