Evaluation Process

Mozilla approached Simply Secure in November of 2020 to undertake a comprehensive evaluation of the last five years of their Fellowships and Awards program (2016-2020) with the goal of refining and maturing programming and examining the effectiveness of their offerings. The Simply Secure team then reviewed internal documentation and data,conducted three surveys, and held 47 interviews in order to qualitatively and quantitatively analyze the F&A program’s five-year impacts, strengths, and challenges. An extensive draft of the findings and recommendations was produced in March-June 2021 and forms the bulk of this evaluation.

With the first draft of the evaluation, two external consultants – Kenrya Rankin and Ayana Byrd – were brought on to build on the findings and carry out additional interviews to create an impact narrative to provide greater detail on the evolution of the Mozilla F&A program, which formed the beginning of this report.

This evaluation is reflective of the last five years of Mozilla programming, and as such, it looks back upon the strengths and challenges of the F&A program. Since mid-2020, the Mozilla Foundation Global Programs teams have undertaken a large-scale project to evaluate and improve upon their programming and strategy – F&A and beyond – and this report does not encompass all of those efforts. To delve into the points of intervention that are in the works from Mozilla related to this evaluation, a Management Response based on interviews carried out by Kenrya Rankin and Ayana Byrd is appended to this report. Below are the evaluation's guiding questions and methodology employed by Simply Secure, along with potential limitations.

Table 5: Mozilla Fellowships and Awards Programs, 2016-2020

The chart below provides an overview of programs the Foundation operated during the evaluation period (2016-2020).

Methodology

Simply Secure began the evaluation process by reviewing Mozilla-provided materials on the past five years of Foundation programming. These included strategy documents, spreadsheets, program analyses, applications, and narratives. Our guiding questions, listed below, were refined through a collaborative workshop with the F&A team and an external consultant.

Guiding Questions:

• What benefit did each of the different fellowships and awards streams bring?
• To what extent does investing in fellows and awards result in community level and/or systemic change? Are there ways in which it may be hindering change?
• What lasting impact does the fellowship/award have on the individual participants?
• How is the Fellowships and Awards program perceived in the field?
• How does the Fellowships and Awards program advance or hinder Mozilla Foundation’s overall theory of change?
• How does the impact of the program (individual and system level) vary across fellowship models (hosted, non-hosted) and/or tracks? Across types of awards?

We collected research input from a diverse group of stakeholders for this evaluation during January and February 2021, and held an internal F&A staff workshop in February to engage with initial themes. The research collection phase began in January, using the the following methods:

Anonymous interviews (47):

• 22 current and former fellows and awardees (some fit in both categories)
  • 5 grants/awardees
  • 18 fellows
• 13 current and former Mozilla Foundation and F&A staff
• 2 current and former Mozilla Corporation staff
• 15 funders and ecosystem stakeholders (“stakeholders” include individuals who run similar programs or work for mission-aligned organizations in the space; “funders” include individuals speaking on behalf of philanthropic organizations who are currently supporting or have previously supported Mozilla Foundation programming).

Geographic distribution of all interviewees (these distributions are not reflective of the F&A programs as a whole) (see fig.1):

• 32 - North America (USA, Canada, Mexico)
• 7 - Europe (including UK)
• 3 - Middle East
• 2 - South America
• 3 - Africa

Figure 1: F&A Evaluation Interviewees: Geographic Distribution

Anonymous surveys (3):

• Community Engagement Survey – intended to capture the perceptions of the broader Mozilla Foundation community, including those interested in applying to the F&A program in the future, those who have applied in the past, and those who follow the Foundation’s work.
  • 15 responses

• Host Organization Survey – captured the experiences of organizations that have hosted Mozilla Foundation fellows.
  • 7 responses
  • Representing years 2016-2020 (one organization participated for 5 years)

• F&A Program Survey – captured the experiences of F&A program participants and alumni.
  • 62 responses (see charts below for program breakdown and year represented)
    • 26 Awards
    • 43 Fellows

Figure 2: F&A Program Survey: Participant Start Years

Figure 3: Program Survey: Recipient Type

Evaluation Limitations

• Awards were underrepresented in our interviews, largely due to the self-selecting nature of the voluntary interview and survey process. Several targeted attempts were made via email to recruit more awardees to participate in the research, with a lower response rate than with fellows.
• Interviews and surveys do not represent every program.
• Limited timeline – we could have performed more interviews and done further synthesis with more time.
• Internal program data access and consistency was a challenge, including updated and full contact information for past participants.
• Although we did conduct additional outreach to participants from outside the U.S., Canada, and Europe, we were limited by who responded to the survey and who responded to the follow-up interview and direct survey invitations.
• 5 of 22 participant interviewees and 18 of 62 program survey respondents took part in Mozilla programs that have been retired and are no longer in operation.