News  |    |  November 18, 2020

When AI Systems Fail: Introducing the AI Incident Database

News article by Sean McGregor. Published by the Partnership on AI.


Governments, corporations, and individuals are increasingly deploying intelligent systems to safety-critical problem areas, such as transportation, energy, health care, and law enforcement, as well as challenging social system domains such as recruiting.

Failures of these systems pose serious risks to life and wellbeing, but even well-intentioned intelligent system developers fail to imagine what can go wrong when their systems are deployed in the real world. These failures can lead to dire consequences, some of which we’ve already witnessed, from a trading algorithm causing a market “flash crash” in 2010 to an autonomous car killing a pedestrian in 2018 and a facial recognition system causing the wrongful arrest of an innocent person in 2019.

Worse, the artificial intelligence community has no formal systems or processes whereby practitioners can discover and learn from the mistakes of the past, especially since there is not a widely used centralized place to collect information about what has gone wrong previously.

Therefore, today we introduce a systematized collection of incidents where intelligent systems have caused safety, fairness, or other real-world problems: The AI Incident Database (AIID).

The AIID is inspired by, and combines the strengths of, similar databases in aviation and computer security.

In aviation, an “accident” is a case where substantial damage or loss of life occurs. “Incidents” on the other hand are cases where the risk of an accident substantially increases. For example, when a small fire is quickly extinguished in a cockpit it is an “incident,” but if the fire burns crew members in the course of being extinguished it becomes an “accident.” Decades of iterative incident-motivated improvements have decreased fatalities eightyfold since 1970.

The second incident database inspiring the AIID is the Common Vulnerabilities and Exposures (CVE) system, which contains more than 141,000 publicly disclosed cybersecurity vulnerabilities and exposures. The CVE site serves as critical security infrastructure across all industries by enabling vulnerabilities to be circulated and referenced with a consistent identifier. [ . . . ]