About our Methodology

The goal of Mozilla’s *Privacy Not Included buyer’s guide is to help consumers shop smart — and safe — for products that connect to the Internet. Many companies don’t provide clear information about the privacy and security of the connected devices they sell. Is your personal data being used in ways you may not have anticipated or expected? What are the risks of buying a product that collects your health data? How does the company regularly test for and fix security vulnerabilities?

It’s difficult for consumers to get clear, concrete information from companies about the security and privacy of their connected products. With this guide, we hoped to help consumers navigate this landscape by understanding what questions they should ask before buying a device.

Here is the methodology we used to develop this guide.

Methodology

Product Selection

There are 76 products in the 2019 version of our *Privacy Not Included buyer’s guide. These products fit into six categories — Toys & Games, Smart Home, Entertainment, Wearables, Health & Exercise, and Pets.

The goal was to select connected products that were likely to be popular during the holiday season and beyond. We selected products that were top sellers on Amazon Prime Day, products featured in the Target Open House, and products that were highly rated across a variety of consumer product websites such as Wirecutter, The Toy Insider, PC Magazine, Tech Radar, and Gear Brain.

Minimum Security Requirements

Mozilla has established a set of Minimum Security Standards it has determined should be met by any manufacturer developing connected devices. We evaluated each product on our list against five criteria:

Encryption

Device should use encryption in transit and at rest (where applicable). The product must use encryption for all of its network communications functions and capabilities, ensuring that communications aren’t eavesdropped or modified in transit. User data should be encrypted when it is stored.

Security updates

The product must support automatic security updates for a reasonable period after sale, and be enabled by default. This ensures that when a vulnerability is known, the vendor can make security updates available for consumers, which are verified (using some form of cryptography) and then installed seamlessly.

Strong passwords

If the product uses passwords for remote authentication, it must require that strong passwords are used, including having password strength requirements. Any non-unique default passwords must also be reset as part of the device’s initial setup. This helps protect the device from vulnerability to guessable password attacks, which could result in a compromised device.

Vulnerability management

The vendor must have a system in place to manage vulnerabilities in the product. This must also include a point of contact for reporting vulnerabilities or a bug bounty program. This ensures that vendors are actively managing vulnerabilities throughout the product’s lifecycle.

Privacy Policy

The product must have a privacy policy and/or another privacy page that applies to the device, app, or service we are evaluating.

Privacy

We evaluated the privacy documentation — which included privacy policies, privacy pages, FAQs – for each company selling these products to determine (1) how data is shared with third parties, (2) whether or not users could request deletion of their data, and (3) the accessibility/readability of the privacy information. Where relevant, we also looked at whether (4) biometric data is collected, and (5) parental controls are in place.

1) How does it share data?

How do companies collect and share customer data with third parties? For this question, we analyzed the privacy documentation to determine how and when personal customer data is shared with third parties for reasons other than standard vendors.

For instance, if a company can share or sell personal customer data with third parties, use data for targeted advertising, or if third parties can use data for commercial purposes, then we noted this. We also noted where the company is permitted to sell or share aggregated, de-identified data. Even if the data is not personally identifiable, consumers have a right to know how their information may be used.

2) Can you delete your data?

Does the company provide a way for users to request deletion of their data? We looked for language around this in the privacy policy and/or whether the company had an online portal that lets users delete their data quickly.

3) User-friendly privacy information

How easy is it to find clear information about a device’s privacy? How clearly is that information stated? Privacy information should be clear, readable, and communicate basic information to consumers about what happens to their data. We looked for whether the company provides easy-to-read privacy information, either in the privacy policy or in another privacy page.

4) Biometric information

Biometric data is data that describes our bodies and distinctive personal characteristics. Many devices collect sensitive data about our heart rate, our sleep patterns, and our menstrual cycle, for instance. Some even use data about our face and voice to identify us. We noted which products collect biometric data.

5) Parental controls?

For products that are intended for children, can parents change and control the settings of the device?

Permissions

We looked into a set of criteria for each product across five questions.

The device/app uses:

This looks at whether a device or app uses a camera, microphone, or location tracking.

For you to set up your new device, you will probably need to download an app. Both of these (the device and the app) will most likely need data to make things work. The app will typically need to request permissions for it to access your more sensitive data.

In our ratings, we evaluated if the device or the app required access to the camera, microphone, or GPS location information. We evaluated the device based on the product website and we used the Google Play store for Android to check on the permissions requested by each app. (Note: apps may access “approximate” or “network” based location. “Tracks Location” was marked as “Yes” if an app requests any location information, including approximate location.)

Worst-case scenario

What could happen if something goes wrong?

We included this section to help people understand risk scenarios related to their privacy and each particular product. We aimed to identify risks that would feel relevant to consumers. It’s likely nothing bad will happen with most of the products in this guide. However, it’s also good to think through what could happen if something goes wrong. This question looks a potential worst-case scenarios for each product, in some cases for fun and in some cases based on things that have already happened with the product.