Within the last 24 hours (as The Attributer writes this) the CEO of Boeing, Dennis Muilenburg, has publicly acknowledged that bad data feeding into an automated flight system on the company’s popular 737 Max jets played a role in two recent crashes .
“With the release of the preliminary report of the Ethiopian Airlines flight 302 accident investigation it’s apparent that in both flights the Maneuvering Characteristics Augmentation System, known as MCAS, activated in response to erroneous angle of attack information.”
We are beginning to see emergent properties of autonomous (and semi-autonomous) systems, including their human interfaces, that are causing major concerns over the resolution of conflicting requirements. In a much earlier incident, Air France flight 447 (an Airbus 330) crashed on 1st June 2009 over the Atlantic Ocean. In that case ice crystals built up on speed sensors and caused the auto-pilot to disconnect. That convinced the pilots to take erroneous action that led to a catastrophic aerodynamic stall of the aircraft.
We might expect that this type of incident is one that will occur again as we roll out more and more autonomous systems for transportation and other industrial applications. It concerns the conflict that can occur between the humans and the technology, and how we resolve these cases in real time to save human lives. If the humans are given false (or fake) data, how should they handle that? How should they ‘know’ for certain that the data is false? To what extent should an autonomous system insist that it is a better decision maker than the humans? And to what extent should the human operator be considered as an effective backstop for a malfunctioning autonomous system. These are complex issues.
Resolving design conflict is by no means new. We have had to deal with the need for heavily controlled access and ingress into buildings whilst at the same time making it easy for those inside to escape if the building is on fire. Special fire escape doors have been used, and yet in certain venues, often nightclubs, the management has chained up the escape doors to prevent people getting in through the back without paying. There are many recorded tragedies of hundreds dying in nightclub fires for this very reason. Human behaviour is the key issue; technology attempts to mitigate the worst threats to human life but the designers cannot foresee or forestall every type of future behaviour…or can they?
In an earlier Attributer blog article (Safe, April 2016) we discussed the synergy that exists in the approach taken in SABSA for security architecture and that taken in the safety engineering frameworks: STAMP (System-Theoretic Accident Model and Processes), CAST (Causal Analysis using Systems Theory), and STPA (System Theoretic Process Analysis). These methods are from the team at the MIT Department of Aeronautics and Astronautics led by Dr Nancy Leveson.
The key differentiator between the MIT methods and traditional safety engineering approaches is the emphasis on hierarchical governance that modern thinking embraces. No longer is there a belief that an accident is caused by a linear chain of events, or even by the ‘swiss cheese model’ where all the ‘holes’ align. Yet already Mr Dennis Muilenburg has publicly asserted in his video statement that these accidents were caused by a ‘chain of events’. No doubt that part of his statement will occupy safety analysts and lawyers for several years to come. Many industry experts are already pointing to poor governance, driven by competitive issues with respect to Airbus and the need to maintain profitability.
Perhaps now is the time for The SABSA Institute to reach out again for some joint thinking and working with the MIT team. We are facing the prospect of safety engineering becoming more and more dependent on the quality of digital data feeds and therefore on the security of digital sub-systems. Autonomous piloting systems in all forms of transportation are becoming ubiquitous. Safety authorities in different sectors and jurisdictions are moving to a position that ‘a system cannot be safe, if it is not secure’. Both schools of thought (STPA and SABSA) have major contributions to make towards a holistic architectural approach to the design of safety-critical digital systems of all types. What is not yet clear is exactly how to achieve the goals. Further research and development of ideas is needed to move forward.
Here’s another ‘call to arms’ from The SABSA Institute. We shall be pleased to hear from anyone interested in working on a joint project to look at these issues in autonomous systems. The Attributer suspects that we might find that designers are too focused upon the behaviour of the technology under stress and not sufficiently attentive to the vast array of possible human behaviours that might accompany technical malfunction. As a starting plan The Attributer suggests the creation of a taxonomy of scenarios including the detailed analysis of possible human interventions. That would give us some insight into the scale of the problem as it emerges in this fast-changing world of digital automation. Contact firstname.lastname@example.org. Lion Air flight 610 on 29th October 2018 and Ethiopian Airlines flight 302 on 10th March 2019.  Video plus transcript statement at: http://www.boeing.com/commercial/737max/737-max-update.page#/message Accessed 5th April 2019.  The wording is important here. In systems theory there is a difference between a simple system (which can be fully known), a complicated system (which has many parts, is not simple but is still fully knowable), a complex system (which is not fully knowable because of emergent properties, but which has some level of predictability) and a chaotic system (which is unknowable and unpredictable).
http://noop.nl/2008/08/simple-vs-complicated-vs-complex-vs-chaotic.html . Accessed 5th April 2019.  See http://psas.scripts.mit.edu/home/wp-content/uploads/2016/01/Systems-Theoretic-Process-Analysis-STPA-John-Thomas.pdf for an overview of these safety engineering methods. Accessed 5th April 2019.