Second of the Six Key Challenges
AI systems may pose new challenges to human safety, a key concern for regulators. Human safety must be a primary consideration in the design of any AI system.12
We distinguish a few key areas in assessing the safety of AI technologies:
The general topic of reliability engineering, which ensures that products operate as intended with defined performance characteristics, without failures, and under diverse though expected hostile environmental conditions, is not within scope here.
Controllability is a specific risk associated with AI systems, which is designed to perform actions without human intervention, and this risk is heightened in systems with the ability to self-repair, self-improve, or self-replicate.13 In this paper, controllability is understood in terms of system failure prevention (reliability) and ensuring outcomes are as intended (predictability) and not in the general sense of ability to control and contain a generic AI system.
Security is crucial to ensure that the AI system is safe from malicious actors. For more information about security, see section 4.1.
4.2.1 Predictability
The following areas strongly affect the predictability of an AI system:
Constraining the Outputs. An AI system is almost always implemented as part of a larger system or application, with other components or indeed other AI systems relying on its output. To achieve some measure of predictability, the outputs of an AI system must be bounded and designed to serve as input in a larger system.
Reproducibility. Predictability is also associated with the issue of reproducibility specific to an AI system. Should the system react in the same way when inputs are equivalent? The answer to the question has implications on how the entire system responds and is a key measurement for predictability of the entire system.
Access and Availability of Internal Tooling and Infrastructure. Lack of access to resources used by AI systems is cited as one of the main blocking points for predictability.14 For predictability, the relevant resources used by the system include the data and the code base of the frameworks used, as well as the hardware version and the associated software releases.
4.2.2 Reliability
We deal specifically with the risk of unreliability in safety-critical applications. To prevent loss of reliability, we need to understand potential sources of failure. The following causes of failure were highlighted in recent research.15
Bad or inadequate data. Errors introduced through bad or inadequate data at development or deployment stage can lead to differential performance, to the extent that the data is not fit for purpose for certain cases.
Shifts in environment. Differences or shifts in environment between development and deployment can, again, lead to worse performance in unanticipated environments. This is where reproducibility and predictability mitigations are important considerations.
Faulty model assumptions and/or fragile models. Errors can be introduced both by faulty model assumptions and/or fragile models. For more information on recommendations for protecting models, see section 5.2.4.
Arm's Erik Jacobson describes Arm's efforts to implement better security measures at the chip level.
Seeking Assurance
Providing Detailed Information
