This section unpacks the various impacts of the F&A program and the ways in which that impact is accounted for, measured, and recognized. This data comes from interviewees across the board, as well as from a review of internal and external strategy documents and survey responses.
- 95.2% of alumni survey respondents felt that they achieved their self-defined goals of the program.
- 87.1% say because of their funding they have had an impact in the social/tech movements & communities that they identify with.
- 85.5% felt that the program had an impact on them personally and professionally – many referenced that the program built their confidence, gave their work legitimacy, and introduced them to influential networks of peers and collaborators.
In the F&A Program Survey, we asked alumni about their personal goals for their funding period. Respondents generally clustered into the following categories:
- Seeking funding for a specific project building tools or technology, or to fund work they already were doing on the side – strong representation from WINS challenge, and programs across the board.
- Career path exploration into public interest technology and/or more “activism” related work – strong representation from Open Web Fellows
- Networking and learning from peers/ Mozilla Leadership – strong representation from Senior Fellows/Fellows in Residence
- Bringing technology into civil society – strong representation from Open News Fellows
- Building public understanding (through community-engaged work, public-facing projects, etc) – strong representation from Gigabit Community Fund participants
- Building the field (through tools and research meant for the ecosystem) – strong representation from Open Science/Mozilla Science Fellows
- Policy – largely Policy Fellows
- Education/curriculum building – strong representation from Responsible CS and Gigabit Community Fund participants
- Art and creative projects – strong representation from Creative Media Awards participants
The impacts of the fellowship program have rippled throughout the internet health movement and ecosystem through narrative shifts and knowledge production.
- Many different interviewees (recipients, staff, funders, etc.) described that the impact of the Mozilla F&A program is seen intangibly through cultural shifts, ripple effects, and growing emergent movements around public interest technology broadly. The program also has brought a fresh new set of voices into the internet health movement, like artists.
“Developing thinking in a specific field”
“The fellowships are about creating space where tech and civil society meet, with the aim being to enhance each other’s movements”
"Technology is about imagining. So who better than the creative people to think about this.”
- Interviewees who participated in the Open Science, Tech Policy, and Open News programs in the U.S. said they saw the cultural conversation shift around big tech and other internet health issues, and cited that many Mozilla funding recipients have been quoted in major publications pushing that conversation forward. The discourse changed, the fellows have been seen as experts, and they've brought their and Mozilla’s perspectives into the public square.
Funding programs in adjacent fields can greatly expand Mozilla’s reach and potential network (e.g. RCS opening and exposing MoFo to more academics/universities).
Increasing program reach globally or allowing for regional continuity and depth can help fellows have the institutional infrastructure and the Foundation to have the regional political context necessary to bring these knowledge production and narrative change ripple effects to geographies outside the U.S.
The F&A program helped to build the internet health space by exposing technologists to civil society organizations, opportunities, and activism.
- Mozilla programs supported many in making important career shifts into internet health/public interest technology careers, giving recipients the “confidence” and “credibility” to shift careers (often from industry and big tech to non-profit or civic technology).
- Mozilla F&A program built “leaders, teachers, and creators” in “tech activism.”
- For some more established and senior recipients, association with Mozilla served as "washing them of their previous associations."
- The Mozilla Science program, for example, was able to expand participants’ understandings of what it meant to be a scientist or researcher. An Open Science fellow shared that the fellowship “helped me see the parallels between activism in the web space, what it means to have an open and free web, and how, in parallel, you can see that in research and academia.” Mozilla’s principles for the open and free web were brought into new communities, like the scientific or artistic community.
Since a primary goal of the F&A program is to build the internet health space, there is interest in measuring impact by keeping track of where fellows and awardees have gone, how many of them have remained in and continued to build the space, and trying to understand the longer-term investment in these individuals and projects. Currently the infrastructure to do this is not in place, as it would require increasing staff capacity, but could be instituted when onboarding new funding recipients.
- Affiliation with Mozilla was described as “advantageous” for recipients because of credibility around the “Mozilla brand” in the ethical technology space. Funding recipients said that participating in the program increased their own credibility in the space: it raised their profile in a way that allowed them to be more publicly opinionated and heard by a wider audience.
The fellowship kick started my career so much...the quantity and quality was so much higher suddenly.“
"I was kind of a nobody open source contributor – nobody followed me on Twitter… I got talks occasionally at conferences when I would apply and follow up a bunch, but I wasn't a big name. And when I got the fellowship, all of a sudden, all these random people all over the world that I didn't know were contacting me because they were Firefox contributors, or they had been involved with Mozilla's growth, and were interested in what I was working on. And I was getting more talks at conferences.”
“[I] had an incredible experience. It's really powerful to be attached to the Mozilla name. You get amazing credibility from that.”
- Fellows who were more junior when they entered the program reflected the most impact on a personal level (career-amplification, increased opportunities, access to networks). More experienced/established fellows felt they had less impact or the fellowship didn’t uplift their career or profile very much.
- Receiving Mozilla funding, even at small dollar amounts, e.g. travel funds or small awards of $5-10k, was cited as a pivotal career moment for many recipients. Many said these funding opportunities “opened a lot of doors” for them.
There is a desire from funders, participants, and staff, to see a visualization of the network effects of the F&A program – who has been touched and what connections have been forged. This type of information-sharing can also mean increased visibility of the different niches and spheres of the ecosystem. One stakeholder, for example, referenced an organization that uses a “ripple map” to track the influence attending a conference has on individuals. They check in with people who go to the conference, ask what they learned, and then check in again 3, 6, and 9 months after and ask if they have any new connections, ideas, work, etc because of their experience at the event. They then map the ways that experience impacted those who attended. Mozilla could implement such a map internally, but they could also help develop a larger-scale map of the rings of experience, connection, work, and ideas over time that can help give a real sense of impact on an ecosystem or sub-ecosystem level.
- Tracking long-term impact and stewarding long-term relationships has historically been lacking. There has been a strong need to keep recipients engaged and connected, which one staff member cited as a “gaping missed opportunity.” This has resulted in over-emphasis on the same few, specific stories that are memorable, hyped up, or fresh in people’s mind. These stories have often been about “rockstar” fellows, who sometimes had significant influence in the space already, with little to no focus uplifting the work of other – often junior – fellows, and awardees.
- That being said, some funding recipients, notably some more junior grantees from outside of the U.S., reported a more active, continued involvement in the Mozilla community – like through MozFest – and said they felt very welcomed into the Mozilla ecosystem and encouraged to participate.
Mozilla can extend the value of a funding recipient by amplifying the recipient’s work post-funding. Interviewees broadly highlighted that the power of Mozilla funding should lie in leveraging and utilizing the network and its combined assets. Some recipients feel that they don’t have a continuing sense of what’s happening within the Mozilla community, which is a missed opportunity. In that way, there’s an opportunity to strengthen the identity of being a Mozilla fellow or awardee.
The primary impacts of the F&A program – ecosystem change, narrative shifts and ripple effects – have been difficult to measure.
- Other funders and stakeholders in the space shared that they, too, face difficulty developing measurement models and understanding the extent of the impacts of their programs.
- Compared to awards, funders and staff reflected that the impact of fellowships was frequently more individualized, difficult to measure, and strongest often long after the fellowship, sometimes 1-2 positions/roles/projects post-fellowship. Funding recipients agreed that the programs have ripple effects that continue long after the funding period ends. As one funder stated: “the sum total of good in the world has increased; but we didn't have ... more concreteness beyond that.”
- It is challenging to “measure the sum total of impact across the board” because “grantees set their own milestones.” - Funder
A clear theory of change for the F&A programs themselves and goals/metrics are desired by staff and funders to create a framework to understand the impacts of all of the programs, and are desired by funding recipients to understand how to use their funding period to its maximum potential.
There is a desire for a baseline set of impact measurements to be applied to all programs, with the understanding that additional program-specific measurements will be added and considered as needed. Many staff expressed that having these measurements would greatly simplify their work, but emphasized that team members across the board must commit to sticking with them once they’re established.
Measuring impact is easier when deliverables are pre-defined, e.g. in awards (project-based, tech-based, how many clicks did the project get, collaborations, reports/outputs/findings etc) or when fellows have well-defined projects or outputs.
Measurements of success can also be understood qualitatively, and the organization has an opportunity to develop a consistent reporting structure for these kinds of successes. These can include unexpected outcomes or lessons learned from projects that did not “succeed” by traditional standards.
The F&A program has achieved aspects of the Foundation’s theories of change, but impact measurement can be improved by program design that intentionally and specifically operationalizes the theories of change.
- Focusing on internet health and movement building, the F&A programs clearly and successfully added to the Foundation’s theory of change (ToC) by introducing more research, writing, art, and code exploring the role of the internet in society to the ecosystem.
- At the same time, the Foundation hasn’t yet homed in on the right tools, impact measurement resources, or overall framework to understand how the program is directly influencing some of the medium and long-term goals of the Movement Building ToC. Without that framework, it has been difficult to measure or understand Mozilla’s impact on the “Movement” itself.
- Participants, staff, and stakeholders shared that they were unclear as to whether the introduction of the Trustworthy AI framework was intended to be a subsect of the Internet Health and Movement Building ToC, or whether it was a new operational and strategic direction intended to apply to all of the Foundation’s programs. Having spoken mostly to people engaged with Mozilla pre-transition to the AI focus, it’s unclear whether the same confusion around messaging affects newer members of the F&A community.
- The AI-specific outcomes outlined in 2019 Trustworthy AI ToC either 1) didn’t apply to many fellows and awardees because the scope of their work or program was defined prior to or outside of the Trustworthy AI framework, or 2) there has not been enough time or resources dedicated to developing metrics of understanding the AI impacts of the program specifically.
Mozilla can develop both program strategy and frameworks of understanding how these issues, ideas, and technologies are moving through and growing the ecosystem at the same time. In this way, Mozilla has an immense opportunity to be a leader in the space by trailblazing around ecosystem impact measurement and network stewardship.
Clarifying and communicating which aspects of each F&A program are intended to address which aspects of the Foundation’s theory of change will help give the ecosystem and participants clarity around the intentions of Mozilla’s work, and will help the Foundation develop more effective impact measurements.
As time passes and allows for visibility of the medium-term outcomes outlined in the AI ToC – shifting industry norms, building new tech and products, generating demand, and creating regulations and incentives – the Foundation has the opportunity to define its methods of understanding or measurement now.