Trustworthy AI: An operator’s view (but a systems-engineering approach)
The art of being a professional mariner is about making decisions, whether related to shipping situations; routing and weather; engine configurations, through to threat evaluation and weapon allocation in a military context.
The ‘OODA loop’[1] (Observe, Orient, Decision, Action) decision-making cycle has been in use by the military for decades. In a human context, assurance of the accuracy of each phase can be achieved through techniques such as observing the process and challenging assumptions. However, if part of or all of the system-of-systems that delivers the action is automated, assurance of the process is difficult, particularly if the OODA loop operates at machine speed.
This article takes an operator’s view of the challenges, with a brief introduction of the technical: It is proposed that assurance of the Observe, Orient, and Decide functions in a system-of-systems involving AI is vital, and can be achieved by balancing an objective measure of system trustworthiness with a subjective assessment of how much users trust the system.
Decision Making – a human context
The ‘OODA loop’ (Observe, Orient, Decision, Action) is a widely-known decision-making model that originated in the US Air Force but is now used in a variety of civilian and military fields. It translates well from human to machine, with automation bringing benefits of assimilation of vast quantities of data, rapid assessment, and machine-speed repeatability. However, operating at machine speed, governed by AI with impenetrable algorithms and overlaid with machine learning, all stages of the decision-making cycle become opaque, with only the ‘action’ being observable.
To place this into context, the OODA loop output in a human based, ‘Rules of the Road’[2] scenario is neatly demonstrated in the well-practiced phrase of a Royal Navy (RN) Officer of the Watch (OOW) in a ‘shipping report’:
‘Captain Sir/Ma’am – at red 40, range 1 mile, I have a vessel engaged in fishing [Observation]
we are 30 degrees on their port bow and its bearing is drawing left
with a Closest Point of Approach of 2 cables to port. [Orientation]
It is a Rule 18 situation, and we are the give way vessel. [Decision]
My intention is to alter course 10 degrees to starboard to open the CPA to 5c, it is safe to do so. [Action]
Assurance of that decision-making cycle starts at basic training, and is formalised through qualifications, such as STCW (International Convention for Standards of Training, Certification and Watchkeeping for seafarers). It is then built upon by supervised experience, with defined parameters that the OOW can act within. Assurance that they have oriented themselves based on correct interpretation of all the information would come from questioning the decision and verifying key pieces of information – for example, an oft repeated phrase on a bridge of a RN ship is ‘don’t trust the radar – look out of the window’!
There are many softer skills and human factors that also contribute to decision-making assurance, such as, how much sleep has the OOW had; what is their arousal levels; are they under any other stress; how confident are they – does their tone of voice give away anything; have they made mistakes before. Most mariners with experience of Command have a story about being jolted awake in the middle of the night by something that ‘didn’t feel quite right’, often called ‘intuition’, but more accurately should be called ‘experience’ – with a rapid re-location to the bridge preventing a more serious incident developing.
It could be assumed that getting to the right ‘Action’ is the point of the OODA loop, which would imply that the journey to get there is immaterial. That would be the case if each incident is taken in isolation, but, for a OOW to improve and gain experiential learning (‘machine learning’ in an automated system) throughout their career, the component parts of the OODA loop are just as important. An action which is apparently correct, but based on an incorrect interpretation of the situation is more insidious than an obviously incorrect decision, which can be observed and corrected.
Decision making at machine speed
Substitute AI or ‘autonomy’ into a maritime system-of-systems, particularly in autonomous navigation and collision avoidance, and it is virtually impossible to assure each part of the OODA loop: The decision/action which is the output of the OODA loop can be assured through a variety of means such as repeated testing in live or synthetic environments. Alternatively, if required assurance levels cannot be met, risk can be managed through imposing control such as limiting operating areas to account for its capabilities or requiring a human to be embarked for supervision.
But how can we assure the ‘Observe’ and ‘Orient’ functions? How do we know why a decision has been made, and therefore assure its repeatability? This assurance is important as the level of automation in a system increases; particularly important if AI is applied; and vital if machine learning is overlaid.
A vignette from early autonomy trials of a RN vessel can illustrate the necessity of understanding the entirety of the OODA loop: Whilst transiting an empty Solent on a clear day at speed, the autonomous vessel (jet propelled) came to an instant stop (causing consternation to the human inhabitants!) just before the Coastguard helicopter flew past the bow at 100kts / 100 feet. The action of giving way to an unknown, bow-crossing AIS fitted vessel doing 100kts was probably correct, but it relied on interpretation of an AIS fitted vehicle doing 100kts being on the surface, vice the air.