Teaching Responsible Computing Playbook

Accreditation and Ethics

Authors: Matthew Hertz, Atri Rudra

For undergraduate degrees that are accredited by ABET, assessing students’ learning of ethical aspects of computing will be required. For example, for Student Outcome (4) of the Computing Accreditation Commission (CAC), a part of ABET accreditation standards for computer science programs, states that “the Graduates of the program will have an ability to: Recognize professional responsibilities and make informed judgments in computing practice based on legal and ethical principles.” Other institutions may also include ethics training as part of their program or institutional goals (e.g. if they want to follow the ACM/IEEE CS curriculum guidelines). These institutions would also need to assess learning of ethical topics by their regional accreditor, such as the Middle States Commission on Higher Education or the Higher Learning Commission. Because ABET is more general, ABET will be used as an example in this section.

However, assessing ethical outcomes is a documented challenge (see Shuman et al., 2013 for a discussion of these issues centered around ABET’s learning outcomes). This makes assessing student outcome (4) in the CAC criterion one of the harder outcomes to assess.

One bonus of incorporating responsible computing in computing courses in the curriculum is that they provide a very natural way to assess student outcome (4) of CAC. In particular, using well-developed student learning outcomes and using proper assessments of ethics in these courses could lead to a systematic way to assess learning outcome (4) of CAC.

Perhaps the biggest challenge in assessing learning outcome (4) in CAC is that it is vague when it states “make informed judgments in computing practice based on [...] ethical principles.” This is an intentional decision by ABET to provide schools the flexibility in how this learning outcome. This also means that schools and computing departments have the flexibility to include responsible computing in various courses and meet the ABET requirement.

Key Questions:

Is ethical training an existing learning outcome for the program? If not, perhaps the Learning Outcomes and Assessments section of the playbook are more relevant.
Is an existing course already being used to assess ethical learning for accreditation? If a course is already being used to assess ethical learning, then that course should be used to incorporate responsible computing. If no such course exists, then choose a course in the curriculum to incorporate responsible computing.
Is the chosen course a required senior (or at least junior) level course? The ABET learning outcome asks that the graduates of the program satisfy a learning outcome. Institutional accreditors (for example, WASC and SACS) also want to see end-point assessment. Hence, doing the assessment in a course towards the end of the curriculum makes sense. (Of course, assessment can be done in addition earlier.) Also accreditors prefer to see that all students in the program are assessed for a given learning outcome-- doing the assessment in a required course ensures that all students are assessed (in the same manner) for a given learning outcome,
Is there an assessment cycle? An ABET visit happens every six years, visits from regional accreditors may have even longer visit cycles, but they do not specify the frequency with which assessment should be done in that period. It is left to the department to establish its own assessment cycle, i.e. how frequently should the various learning outcomes be assessed (and in which courses). Of course, assessment can be done every time the relevant course is taught but just for the purposes of accreditation, assessing each learning outcome once every two (or even three years) is fine as long as the process is documented and followed.
Will the learning outcomes be broken up into more specific learning outcomes? ABET learning outcomes are fairly general and have many moving parts. One option would be e.g. to break the overall learning to have separate ones on legal and ethical aspects of computing.
Is the instructor of the course on-board doing the assessment? Having instructor buy-in is necessary to do an assessment (as well as including responsible computing content). If not, does the instructor need more support/material to do so?
Is there an assessment plan in place before the semester starts? Planning which specific assignments will be used to do assessment before the semester starts goes a long way in doing a consistent job with an assessment (also allows for getting feedback on the assessment plan if the instructor or the department so desires). Also this might not be needed if the assessment plan is repeated across subsequent offerings of the course. See the learning outcomes and assessment section for more on assessment.
Is there a rubric in place to do assessment/grading? The easiest way to do an assessment of the ethics learning outcome is to tie it to grading of an assignment/project (or question on an exam). However, if the grading is being done by TAs then having a clear rubric helps in doing consistent grading (and hence assessment). In addition, the rubric can be used to do additional assessment for the sole purpose of accreditation. Finally, the rubric could also be used to structure the assignment itself.

Checklist

☐ Identify whether the computing department is already required to assess ethical learning for its accreditation.

☐ Identify a required junior/senior level course for assessment of the ethical learning outcomes.

☐ Establish an assessment cycle and determine the timeline for assessment.

☐ Identify sub-learning outcomes, if needed.

☐ Acquire instructor support for and willingness to do an assessment.

☐ Identify which questions/assignments will be targeted for an assessment.

☐ Create a grading rubric for the assessment.

Examples

University at Buffalo

University at Buffalo (UB)

Two courses (both required for the BS Computer Science program) that were part of computing courses that were targeted as part of the responsible computer science grant were also used to assess student outcome (4) for ABET CAC: CSE 331 (Algorithms and Complexity) and CSE 442 (Software Engineering).

CSE 331 already had an existing project where students created a short video about societal impacts of algorithms (the students picked both a specific algorithm as well as a case study for its application in real life). Starting Fall 2019, the video project was made more focussed so that students explicitly talked about ethical impacts of their chosen algorithm in its application.

CSE 442 students review a made-up dossier of government data to identify if an individual should be added to the government’s No-Fly list. Students must then discuss biases in the data provided and, finally, how they would respond if their job assigned them to the team automating this analysis. A second assignment asks students to document the ethical implications of the capstone project they chose. CSE 442 is the capstone course for the BS Computer Science program.

Both courses used the same rubric to grade the corresponding assignments as well as doing the assessment of student learning outcome (4) as part of the ABET CAC accreditation process. The rubric was also helpful in creating the final form of the assignment to make sure that students were explicitly asked to do things that the rubric was going to assess them on.

Check-list walk-through. We now give a quick walk-through of the check-list items to show how they were resolved at UB. (Warning: The checklist above did not exist when the effort was started so the mapping might not be perfect. However, the hope is that this gives an example of how you might go about using the checklist.)

Identify whether the computing department is already required to assess ethical learning for its accreditation.
- The BS Computer Science program at UB was already ABET accredited.
Identify a required junior/senior level course for assessment of the ethical learning outcomes.
- CSE 442 was already being used to assess student learning outcome (4) for ABET. As part of the Mozilla responsible computing grant, CSE 331 included an ethics specific assignment and was also selected to assess student learning outcome (4) for ABET as well. Both courses are required by the BS Computer science program.
Establish an assessment cycle and determine the timeline for assessment.
- The CSE department at UB was doing assessment every year in each chosen course, which turned out to be more cumbersome. We moved to a two-year cycle so that we could complete two assessment cycles before the the 2020 ABET visit (the learning outcomes of ABET changed two years into the cycle for 2020). In the next cycle, we plan to do assessment in each chosen course every three year.
Identify sub-learning outcomes, if needed.
- We separated out the student learning outcome (4) into three performance indicators:
  - Recognizes professional responsibilities in computing practice.
  - Makes informed judgments in computing practice based on legal principles.
  - Makes informed judgments in computing practice based on ethical principles.
- CSE 331 only assesses the last performance indicator while CSE 442 assesses all three performance indicators.
Acquire instructor support for and willingness to do an assessment.
- Both the instructors for CSE 331 and CSE 442 were co-PIs on the Mozilla responsible computing grant. In addition, both instructors were already either already doing ethics or responsible computing related assignments in their courses so the switch (in the case of CSE 331) was smooth.
Identify which questions/assignments will be targeted for assessment.
- These already existed or were created as part of the Mozilla grant.
Create a grading rubric for the assessment.
- As part of the Mozilla grant, we created a rubric to assess ethics related assignments, which was then used in both CSE 331 and CSE 442. Somewhat interestingly, when we started assessing student work the rubric was not effective. The problem was not with the rubric or student work, but that the assignments discussed ethical issues but did not require students perform any analysis. We first used this rubric while rewriting existing assignments and creating ones to make certain they require students respond to the ethical issues they raised. Once this was complete, we then presented the rubric to teach students the different criteria for them to consider when evaluating the ethics of a situation and, finally, to evaluate their responses.

Resources

Webpages of relevant courses at University at Buffalo: CSE 331, CSE442
Grading rubric for ethics related assignment developed at University at Buffalo.
2013 ACM/IEEE CS Curriculum guidelines has the following as part of its recommendation of “Characteristics of Graduates”:
- Commitment to professional responsibility Graduates should recognize the social, legal, ethical, and cultural issues inherent in the discipline of computing. They must further recognize that social, legal, and ethical standards vary internationally. They should be knowledgeable about the interplay of ethical issues, technical problems, and aesthetic values that play an important part in the development of computing systems. Practitioners must understand their individual and collective responsibility and the possible consequences of failure. They must understand their own limitations as well as the limitations of their tools.