The “three knows” for improving human-AI communication
Does the AI know that I know that it knows?
Applications of AI are blossoming across industries, especially for repetitive individual tasks: looking for lesions in a mammogram, translating an article to French, or predicting when factory machinery will break down. But as AI systems are deployed into environments where they need to interact with us, a need for back and forth communication, understanding of intentions, and sharing of knowledge will be needed. Without some of the psychological constructs we learn as small children, even sophisticated AI systems will struggle in real world settings.
Here is an example. I was cycling to work last week, down a hill, and saw a pedestrian crossing the road. If we had both carried on, we would certainly have collided.
I checked behind me and there were no cars nearby, so I had three options: screech to a halt (somewhat risky) or curve around her either to the left or the right.
The pedestrian changed her angle slightly, so she would end up crossing the road behind me as I passed. So I curved to my left to give her a bit more room, and we passed one another uneventfully.
Had I decided to curve to the right or screech to a halt in this scenario, we could have still collided. How did I realise her intent when she changed direction? For instance, how did I know she wasn’t just meandering across the street in a distracted zig zag fashion? Simple. I had seen her as we approached, and she had seen me. We made brief eye contact. Moreover, I knew that she had seen me, and she knew that I had seen her. I also knew that she knew that I had seen her. So when she changed her angle slightly, she could be confident that I would correctly interpret that as an attempt to pass behind me, and as a signal that I should curve to the left.
Had I not made eye contact with her, I would have either tried to stop in a hurry, or perhaps pass on the other side giving a very wide berth, neither of which would be as safe. Had she not made eye contact with me, she may have paused while I passed, thinking perhaps that I hadn’t noticed her.
Similar decisions and interactions happen every few minutes cycling through a city and we give them no thought. They are an example of what psychologists call “theory of mind”: the ability we have as humans to infer one another’s mental states, and understand that they differ from our own. In this example, we actually need a “second-order” cognitive theory of mind for the interaction to succeed (I knew that she knew that I had seen her). This is an advanced ability: our children only develop the ability to understand second-order concepts between the ages of 6 to 10. Until this point in development, contextual learning is generally somewhat supervised, by parents or siblings who have already developed second-order ability. It has been tricky to be sure whether this ability exists in other species, although there is some evidence chimpanzees demonstrate theory of mind in their interactions. If you’re interested to test your theory of mind skills, try the puzzle of the island of the blue- and brown-eyed people.
How would an autonomous vehicle have coped in this example, assuming for a moment that someone is creating an autonomous bicycle? Many of the architectures being developed today do not explicitly attempt to model theory of mind, even if they model the various actors in a perceived scene. So even if the autonomous vehicle is smart enough to spot the pedestrian and predict their trajectory, it would not necessarily attempt to infer whether the pedestrian had seen it, let alone whether the pedestrian knew that it had seen her.
Current machine learning techniques such as deep learning revel in solving problems where they have access to a copious quantity of training examples, using very few preconceptions about the nature of the problem. Computer vision could detect direction of gaze, whether someone has their phone out or are listening to music. It is certainly possible that a vehicle could learn how to take an appropriate action the majority of the time, just based on the probabilities of previous encounters and the sensory information it gathers. But a more likely near-term scenario is that the vehicle would err on the side of caution, assume pedestrians are unpredictable, and slow down and stop given the scenario above.
Jaguar Land Rover’s experiments with virtual eyes on autonomous vehicles have found a way to address this issue. The car visibly “looks at” a pedestrian to indicate it has seen them. In this case the pedestrian knows the car has seen it, but the car doesn’t know if the pedestrian has noticed the “look”, so it is not at the second order level yet. More literal ways of signalling a vehicle’s intent to pedestrians have been tested, such as displaying messages like “cross now” on visible screens, projecting graphics onto the road ahead, or even communicating with special street lighting. Ford are hoping to standardise new signalling conventions. Stanford University’s “JackRabbot 2” will use hand gestures to show intentions, like “you go ahead”.
In today’s early versions of autonomous vehicles (what’s called “level 3” autonomy) there is still a human driver who can take back control. As well as communicating with pedestrians and other road users, the cars must also keep an eye on their drivers. Companies like Affectiva, whose work on emotion recognition was originally commercialised to measure responses to advertising, are now producing systems to detect the mental state of both drivers and occupants in autonomous vehicles, checking for anger, fatigue, distraction or confusion. Letos are working on controlling the internal conditions and ambiance in cars based on emotional reactions. This ability of AI systems to look at us and interpret our emotions has its roots in research designed to help autistic children understand emotions better, children who also struggle with theory of mind concepts. These kinds of tools, inspired by insights from cognitive psychology, seem essential to creating more human communication with our AI systems.
Deep learning techniques have recently seen much success, and propelled AI into the limelight, but it is worth remembering the history and breadth of AI research. This includes work on “intelligent agents”, explicitly modelling concepts like knowledge, beliefs, desires, intentions, reasoning about knowledge, negotiating between agents, and attempts to simulate theory of mind (in social robots for example). Combining techniques such as formal AI logics with machine learning systems could lead to further advances in intelligent interaction.
Meanwhile, commercial pressures for real world AI deployments are pushing in the same direction, and not just for autonomous vehicles. Language learning applications try to model the knowledge and memory of their pupils in order to personalise and optimise the teaching process. In a sense, the application knows what a pupil knows, since it is constantly testing them. The recent work by DeepMind and Moorfields on interpreting eye scans includes two systems rather than a single “black box”: one to segment and classify the eye scans, and one to then suggest diagnoses and recommendations. This allows doctors to inspect the process in order to better know what the AI system knows. Indeed DeepMind are also researching ways for neural networks to learn theory of mind concepts. The next step would be for the AI system to be more certain that the doctor has understood its reasoning, reacting to either doubt or lack of attention, or to participate in conversations with patients about clinical consent.
In this article, we’ve discussed the advantages for AI systems of modelling second-order theory of mind, in their interactions with people around them. Getting to the “three knows”, so the AI knows that I know that it knows, enables more flexible and human interactions, and could lead to more useful systems in real world settings.
One of the hardest aspects of theory of mind for children to learn is called “false belief”: understanding that another person’s understanding is not right, as demonstrated by the Sally-Anne test. In the original cycling example, this would be an autonomous vehicle realising that the pedestrian hadn’t correctly inferred its intention. More realistic scenarios will introduce layers of complexity and doubt. Perhaps the pedestrian had inferred correctly, but the AI system hadn’t realised. Good crime dramas have us constantly updating our theories of the characters’ minds. We will end on a note of caution: if we succeed in modelling these more complex beliefs in future AI systems, we will need to be careful! Count how many “knows” in the classic interchange between the computer system Hal and the astronaut Dave Bowman that concludes 2001: A Space Odyssey:
Dave: Open the pod bay doors, Hal.
Hal: I’m sorry, Dave. I’m afraid I can’t do that.
Dave: What’s the problem?
Hal: I think you know what the problem is just as well as I do.
Dave: What are you talking about, Hal?
Hal: This mission is too important for me to allow you to jeopardise it.
Dave: I don’t know what you’re talking about Hal.
Hal: I know that you and Frank were planning to disconnect me. And I’m afraid that’s something I cannot allow to happen.Thanks to Peter Bloomfield for many edits, contributions, comments and suggestions.
