A regulatory focus on public law leaves out the bulk of algorithmic decision-making that we encounter in sectors such as finance, private healthcare, insurance, entertainment, and a host of other services. However, the features of administrative law which render it suitable for public sector use of AI could also be extended meaningfully to private sector uses. 

AI

__

Read Part I and Part II of this series.

In the previous essays [1], we saw administrative principles that drew from a rights-based framework extending to the exercise of public functions. While we also included privatisation of public functions in our scope of public law, it still leaves out the bulk of algorithmic decision-making that we encounter in sectors such as finance, private healthcare, insurance, entertainment, and a host of other services. The rationales used above for mandating transparency requirements in Section V above, do not hold true for private use-cases of algorithmic decision-making. However, for the past few years, several scholars have written about the rationale for regulation of artificial intelligence not only through public law, but also through private law.

Transparency provisions under AI Act as an example

There have been several piecemeal examples of regulations which regulate aspects of machine learning enabled algorithms. However, the first comprehensive attempt at regulating AI has emerged in the form of EU’s AI Act. The attention of the global technology policy community and industry was squarely on the AI Act. This was for two reasons. First, despite the proliferation of several ethical AI principles and policy documents over the last few years, there has been no other comprehensive regulatory proposal of note. Most policy documents, while acknowledging the need for ethical AI and regulation had shied away from venturing into this tricky terrain, thus, positioning the AI Act as uniquely influential legislative example for other jurisdictions which venture into regulation of AI. Second, the strict regulations in the EU have had a domino effect, particularly in the digital technology domain. Most global corporations use it as a benchmark to avoid having to comply with multiple jurisdictions. The global impact of EU’s regulation of digital technologies has perhaps been more profound than any other regime, as they influence emerging economies striving to offer adequate protections to its citizens as well as helping its local firms compete globally. The AI Act makes the case for algorithmic transparency in private sector use as well. Instead of using a rights-based framework, it adopts a risk based approach to regulation, classifying different uses of AI systems based on the degree of risk they pose, and having a graded set of regulation ranging from no mandatory obligations (with encouragement to create codes of conduct) for other low-risk AI systems at one end, and prohibition on use cases which involve unacceptable risks (manipulative subliminal techniques, exploitation of disability, social scoring purposes, remote biometric identification by law enforcement) at the other end.

A large set of use cases fall under high-risk AI systems (Biometric identification; Education and vocational training; Migration, asylum, and border control management; worker management; law enforcement; access to essential private and public services and entitlements) which would be subject to a wider range of regulations including registrations, sector-specific assessments, self-assessments, certifications, testing, risk management and transparency. In this paper, we are concerned only with transparency provisions, and more specifically, their application. Below, we will look at the provisions of the AI Act as a suitable example to evaluate how algorithmic transparency may be delivered.

Interpretability, and the Duty to give reasons

Perhaps the most critical transparency provision of the AI Act, Section 13 sets transparency requirements for high-risk systems. Aside from being the most important transparency requirement in the AI Act, and thus, having extreme significance for AI systems deployed in the EU, it also offers a useful and replicable template for future regulation in other parts of the world. 

The AI Act classifies high-risk systems in two categories. The first are AI systems which are  embedded in products subject to third-party assessment under sectoral legislation. Here, it refers to AI systems that can be used as a safety component of a product, covered by any of the nineteen EU regulations designed to harmonize standards for certain products across the market. This means that when an AI system becomes a part of a product which, independent of the AI Act, needs to undergo third party assessment, then it will be classified as a high-risk system. The regulatory assumption here is that the need of third party assessment signifies a high degree of risk The second category of high risk AI systems are stand-alone, but are deemed to be high-risk when utilised in certain areas, or deployed in any of the following high-risk verticals. These verticals areas include “critical infrastructures (e.g. transport), that could jeopardise the life and health of citizens; educational/vocational training, that may determine the access to education and professional course of individuals (e.g. scoring of exams); safety components of products (e.g. AI application in robot-assisted surgery); employment, workers’ management and access to self-employment (e.g. CV-sorting recruitment software); essential private and public services (e.g. loan scoring); law enforcement that may interfere with fundamental right (e.g. evaluation of the reliability of evidence); migration, asylum and border control (e.g. verification of authen-ticity of travel documents); and administration of justice and democratic processes (e.g.applying law to a concrete set of facts). Broadly speaking, this means that AI/ML systems which can harm a person’s health or safety or infringe on their fundamental rights are classified as high risk.

G

The operative part of the provisions sets a legal threshold— sufficiently transparent to enable users to interpret the system’s output and use it appropriately. If the proposal were to become law in its current form, its implementation would be complicated. Based on how we answer multiple questions about the nature of transparency required the implementation could vary. Article 13 ties transparency to ‘interpretability’, but there is little consensus on what interpretability means in XAI literature. One source defines it as the AI system’s ability to explain or to provide the meaning in ‘understandable’ terms to an individual. In XAI literature, understandability is often defined as the ability of a model to make a human understand its function without any need for explaining its internal structure or the algorithmic means by which the model processes data internally. [2] Another source defines interpretability in a contrasting manner, requiring traceability from input data to the output.

It is worth noting that the standard of ‘interpretability’ is comparable to the administrative law standard of duty to give reasons that we discussed in detail here. Before delving into the XAI literature for suitable examples of methods that can deliver interpretability, it would be worth our while to first establish the contours of interpretability by tying it to the duty to give reasons.

Depending on the nature of algorithms and the functions that it performs, there could be multiple ways in which the ‘duty to give reasons’ test can be applied here. Let us begin with reminding ourselves that algorithms exist within a socio-technical context, and any form of evaluation for suitability must take that into account. Much like socio-technical systems in safety-critical fields such as aviation, medical devices, weapons systems, large scale manufacturing, which must clearly respond to and conform to a design problem that must always meet well-defined outcomes, when algorithms are used in high-risk systems, they need to deliver ways in which they can be validated for correspondence to key goals and requirements. [3] The duty to give reasons in an administrative context does not merely explain why a decision was taken or not taken in a vacuum but also records the knowledge of the context within which the decision can be understood. Thus, algorithm audits which do not technically serve as explanatory tools or interpretable models, can aid the delivery of the duty to provide reasons by explaining the broader context of the outcomes a system leads in its socio-technical context.

If we have to look for a comparable XAI technique to the output interpretability requirement, it is perhaps best served by the idea of global model explanations. One prominent way to classify XAi methods is by drawing a distinction between explaining a model locally (i.e., a single prediction) or globally which is the whole model. The right to explanation is analogous to global explanations except it is perhaps even a lower standard in that information about the algorithmic logic need not include information included in several global methods such as TCAV. The transparency duty in Article 13 of the AI Act most decidedly goes beyond it to also include output interpretability. Thus, it entitles a user to receive information about an output or a single prediction aside from global explanations.  

Human Supervision and Administrative Discretion

Another provisions of the AI Act that will help us anchor our discussion is Article 14. It mandates that high-risk AI systems are designed with appropriate human-machine interface tools for persons overseeing them to be able to understand the capacities and limitations of the system fully; to remain aware of potential automation biases in (over)relying on the decision-making outputs of an AI system, and to decide or intervene in the operation thereof when safety and fundamental rights are at risk. 

I have discussed earlier how Article 14 notably takes a different route from Article 13. Instead of tying transparency to an XAI term, it merely states what this transparency must achieve. Therefore, the implementation under Article 14 is agnostic to any specific form of transparency so long as it achieves human supervision in the manner described. This approach is useful in the absence of academic or legal consensus. Article 14’s scope is limited only to human oversight in the implementation of the AI system and does not apply to any other stakeholders including both users and adjudicators. Article 14 primarily deals with human oversight to be exercised by domain experts. Sub-clause (4) does a reasonable job of spelling out the level of oversight and consequently the nature of transparency that is contemplated. It sets several metrics which can be used to test the transparency design of the oversight system including output interpretability, ability to monitor anomalies and dysfunction and kill-switch options to reverse or override the outputs. To satisfy these tests, the design of the high risk AI/ML system needs to rely on XAI techniques suitable to the context such as local and global model explanations, examination of gradients in case of neural networks and meta explanations. Importantly, the human oversight agent needs to also be empowered to apply their minds to an output and take decisions to override them. 

Yet again, we may find some guidance in the rich jurisprudence of public administration. A key aspect of public administration is the responsibility on actors to exercise administrative discretion. The structure of public administration presupposes that laws and regulations will provide guidance, yet for effective government, it is essential that discretionary power is exercised appropriately. In the previous section, we dealt with one aspect of improper exercise of discretionary power—when it is based on irrelevant consideration. Here, we are concerned with a requirement that precedes it—that public bodies exercise discretion where needed. It is expected that where an element of discretion is called for, the decision-making authority is required to apply its mind to the decision and not follow policy or any other diktat, in this case an algorithmic feed blindly.

While attempting to define the contours of human supervision as envisaged under Article 14, we may find it instructive to draw from the idea of administrative discretion and ensure that the degree of transparency afforded by the AI system to the persons overseeing them does not ‘fetter administrative discretion’ as Oswald puts it or lead to ‘judgmental atrophy’ as Hildebrandt puts it.

Several real-world situations, particularly in high-risk systems will require the evaluation of multiple factors. When this analysis is delegated to algorithms, what follows is a datafication of factors which would otherwise have been analysed by humans. The first real risk is the possibility of their being some form of ‘loss in translation’ during this process of datafication as is to be expected in any such process. The human-in-the-loop must be trained to exercise the same level of judgment as expected from administrative discretion, which is this situation would include applying their mind and recognising factors that may have been lost in translation, or other relevant factors on which the model was not trained or irrelevant factors that do not correspond to the purpose.

Even outside of the purview of public law, several decisions are made by private platforms where they play arbiters of free speech, privacy and data protection, financial and health services, and access to information, news, and entertainment. The tried and tested concepts of reasonableness, necessity, risk-assessment, and foreseeability which are well-entrenched in public law can suitably inform the nature of supervisions that humans can exercise in high-risk systems employing algorithms. The above concepts do not render themselves well to datafication or feature extraction. Therefore, creating suitable points for intervention of human discretion is critical. 

Regulatory Prescriptions

When high-risk systems employ algorithmic systems, their deployment must be preceded by the following conditions being met.

  • The algorithmic systems must be designed to be interpretable to users such that the high-risk system may discharge the duty to give reasons for its decisions, as understood in administrative law.
  • Where the technical nature of the algorithmic system poses fundamental interpretability challenges, it needs to be designed to flag sufficient information for independent human assessment to verify the machine’s inferences. Sufficient information may include the input data, the nature of model in use and the likely factors which informed the decision.
  • The algorithmic system must be designed for supervision by a human agent within the high-risk system.
  • The nature of transparency afforded to the human agent in (c) must permit them to independently verify or validate decision made by the algorithmic system.

_____

[1] For detailed analysis of administrative to public sector decisions, please see here.

[2] See detailed exposition at Grégoire Montavon, Wojciech Samek and Klaus-Robert Müller. 2018. Methods for interpreting and understanding deep neural networks. (February 2018) Digital Signal Processing 73. 1-15. DOI: https://doi.org/10.1016/j.dsp.2017.10.011.

[3] Joshua A Kroll beautifully explains intensional and extensional aspects of scrutability of machine learning systems. Joshua A. Kroll. 2018. The fallacy of inscrutability. Phil. Trans. R. Soc. A. 376, 2133. (November 2018). DOI: https://doi.org/10.1098/rsta.2018.0084.