Executive Summary

From the beginning, the Fellowships and Awards program has worked with partners to invest millions of dollars in individuals and ideas that improve transparency in and the accountability of technology and pave the way for increased internet health. Along the way, F&A’s funding strategy has evolved in many directions, but there has been one throughline that’s embedded in Mozilla’s DNA: openness. As an investment strategy, that manifests by putting money into programs that improve openness in technology — and pushing other sectors and the institutions that undergird them to adopt openness as a foundational value.

This evaluation is an analysis of the Mozilla Foundation’s grantmaking and fellowship work led by the F&A team from the beginning of 2016 through mid-2020. It seeks to understand: 1) the impacts of the programs on individual funding recipients and organizational partners, on the issues the programs endeavor to address, and on the people most affected by those issues, 2) the strengths and challenges of the programs, for Mozilla staff, program participants and other stakeholders 3) how F&A’s work supports the impact goals laid out in Mozilla’s AI Theory of Change and 4) the ways in which the F&A program contributes to and is perceived by the broader internet health ecosystem.

This evaluation was carried out in two parts. First, Simply Secure carried out an analytical review and initial impact assessment based on Mozilla-provided internal and external planning, strategy, management, and output documents. Kenrya Rankin and Ayana Byrd then provided a further synthesis of those findings and conducted additional independent interviews to complete the impact narrative that follows. The second part of this report is the program evaluation, for which Simply Secure collected primary survey and interview data to evaluate the experience and impacts of the F&A program: 84 responses to three surveys and 47 anonymous interviews with current and former fellows, awardees, Foundation and F&A staff members, funders, and ecosystem stakeholders.

The evaluation found that the program furthers the foundation’s internet health goals and overall theory of change by investing in research, writing, art, and code that explores the role of the internet in society. The program attracts and grows a network of technologists, coders, educators, scientists, journalists and activists who are committed to advancing Mozilla’s mission. It also found that impact has been limited by a lack of clarity and communication around how program design and theory of change inform one another. Those surveyed felt that making the connection explicit would help the ecosystem and participants have clarity around the intentions of Mozilla’s work, crystalize strategy internally and externally, and help the foundation develop more effective impact measurements.

Respondents indicated that the strengths and challenges of the F&A program were largely consistent across program tracks, types and models. This finding indicates that individual F&A programs have more in common than the team currently acknowledges, and that the different programs ultimately offer similar experiences to funding recipients.

A multi-tiered approach was used to measure the ways the F&A program—and by extension, Mozilla—has impacted the tech landscape:

• Individual impact measures if a person or group who received a fellowship or award was able to further their career or devote more energy to their area of expertise. The fellowship program has been successful in helping early- and mid-career technologists become leaders in their fields.
  
• Organizational impact measures how partner organizations’ strategies evolve after hosting an embedded fellow. In many cases, not only have organizations amended their internal structure as a result of F&A program knowledge sharing, but they have made those amendments in ways that align with Mozilla’s strategic goals around trustworthy AI.
  
• Movement impact directly examines the effects of a project that was completed by a fellow or grantee. In the most impactful scenarios, there was a major shift in how everyone from activists to the general public understood a topic, there was an increase in philanthropic investment around an issue, or there was some other positive change on the internet health or social justice movements.
  
  

We also analyzed how fellowships and awards disbursed funds from 2016 through 2020 to determine if the Fellowships and Awards program’s investments furthered the impact goals laid out in the AI Theory of Change. While the data make it clear that F&A is moving closer to those goals, it is also clear that there is an opportunity to better align the investment strategy and timeline with program end goals to scale programmatic impact. And while Mozilla’s focus is squarely on the consumer tech space, the waves created by this work have the potential to have influence in other key areas. As one funding partner put it, “Every field needs this.”

Mozilla is poised to use the data of its impact on the tech landscape and civil society uncovered by this evaluation, to more effectively achieve its impact goals and to power movements. The challenges and pain points identified through the interviews and surveys highlight opportunities for improvement around four main areas: strategy, participant experience, data collection and operations. Findings also emphasized the need for the organization and the tech philanthropy field to collaborate around impact measurements, as the primary benefits and beneficiaries of the F&A program and partner efforts emerge over time and in ways that may be difficult to quantify. There is an opportunity for common metrics to aid the field as a whole.

Strategy

• This evaluation spans three different strategic visions and plans for the organization. Internal and external respondents related a general lack of cohesive and consistent strategic vision for the program, which participants also echoed in their experience of Mozilla and its programs. Lack of internal clarity around program goals and intentions is one major sticking point. The broader field of funders and stakeholders, also, are aware of Mozilla’s many leadership, strategy and staffing changes over time, which has led to a perception that the F&A program lacks focus. Strategic consistency is also challenging due to tensions and misalignments between program design, Mozilla’s goals and the various funders’ goals.
  
• More closely aligning F&A’s investment strategy and timeline with Mozilla Foundation’s now-adopted Trustworthy AI Theory of Change has the potential to steer the movement to develop an artificial intelligence ecosystem that is worthy of consumers’ trust. Launching programs—and selecting fellows and awardees—with impact in mind will better support the Foundation’s mission.
  
• Diversity, Equity and Inclusion (DEI) are important priorities to Mozilla and the F&A team, but the program lacks consistent vision and implementation around DEI practices. Some funding recipients, especially those from the Global South, articulated that their experience with the F&A program was too US-centric, in both operations and cultural positioning.
  
  

Participant Experience

• Fellowship length and schedule make it difficult to achieve the fellows’ desired goals, and can be an interruption to their careers that, without support for planning their next step, can feel like a post-award “cliff.”
  
• Funding recipients reported varied experiences with the F&A programs’ support structures. Additional and more robust systems of support were desired by participants across programs to improve participant experience, and provide a spectrum of support for their varying, individual needs.
  
• Participants who were hosted by an organization emphasized that expectation setting, boundaries and communication are absolutely essential to these partnerships flourishing—lack of role definition has led to difficulty and conflict historically.
  
• Staff reported that F&A is missing an opportunity to extend and deepen its relationship with fellows and awardees. Establishing an alumni network would allow Mozilla to tap their expertise while also putting them in community with each other, expanding their networks and their ability to impact movements.
  
  

Data Collection

• The primary impacts of the F&A program—ecosystem change, narrative shifts and ripple effects—are hard to measure and require metrics that can be deployed over time. Fostering longer-term data collection relationships is critical to both understanding and maximizing the program’s impact.
  
• Inconsistent data collection and management has meant that Mozilla cannot easily use the huge pool of information, resources and connections generated through the F&A program to its maximum potential.
  
• F&A needs to create a qualitative and quantitative measurement and evaluation framework that both protects the data of the people who are directly impacted by the programs and standardizes the information that is collected so it can yield useful insights into the programs.
  
  

Operations

• The F&A staff is rich in expertise and experience, and has an excellent reputation in the field and amongst funding recipients. At the same time, there is an immense opportunity to empower the F&A staff through improved leadership, communication and collaboration. High-level changes, such as leadership, strategy shifts and staff turnover have been destabilizing and have left a lasting impact on the staff. Staff voiced concerns about job security, lack of clarity around their work in relation to current strategy and siloing of F&A work.
  
• The operational end of running the F&A programs is complex and challenging; past participants illuminated various pain points, particularly regarding the logistical complexities of running global programs.
  
• Some staff urged Mozilla Foundation to consider the ways the F&A teams and projects are united within the organization. This evaluation can be used to guide conversations about how fellowships and awards—and the ways they are deployed—can best be used to support the impact goals.
  
  

Mozilla has an exciting opportunity to use the reflections of program participants and current and former staff and the learnings of this evaluation to build a more strategically focused, supportive and communicative set of programs that furthers its essential work within the digital rights ecosystem. The final section of this report details key opportunities and suggested design interventions that respond to these findings and work to situate Mozilla and the F&A program more powerfully on the side of program participants and the future of a healthy internet. The Management Response attached to this report is a Q&A with J Bob Alotta, VP Global Programs, and Hanan Elmasu, Director of F&A, illuminating what’s next for F&A, changes already in play, and how Mozilla plans to grow from this evaluative process.