When robots observe their operators

We humans give more and more jobs to machines, but still we do not fully trust them! Take, for example, self-driving cars. We travel with doubt: Will the machine always make the right decision? Or will it make a - possibly fatal - mistake? We feel safer when a human sits up front with the robot, a person who can hit the brakes or an emergency stop button.

But does this work the other way around? Why should machines trust humans more than humans trust machines? After all, isn't "human error" often the cause of accidents, even in automated environments such as commercial planes?

Recognizing when the operator isn't doing well

Researchers from Wachtberg near Bonn, Germany, have looked into this. They have developed a system in which robots watch humans. As a potential application, they set up a simulator for the training of air traffic controllers.

Computer scientist Andreas Werger plays the role of the air traffic controller. He sits in front of the simulator at the Fraunhofer Institute for Communication Information Processing and Ergonomics (FKIE) and undertakes various tasks, similar to those of real air traffic controllers.

Psychologist Jessica Schwarz looks over Werger's shoulder.

"The planes in the virtual airspace are flying randomly. They are experiencing air contact," says Schwarz. "The controller has to make sure the planes don't collide and that they stay within a designated airspace, which is marked with colors."

The computer knows how you're doing

Another screen shows an analysis of Werger's performance and condition. "Here we can see, when his performance suffers," Schwarz explains.

Suddenly a collision alarm starts to beep - the air traffic controller frantically types instructions for the virtual pilot into his keyboard.

While Schwarz watches Werger, a computer watches Werger too. An eye tracker does the job. The device can see exactly where the air traffic controller is looking at any given moment.

"It is a small box underneath the monitor," says project director, Sven Fuchs. "It includes a camera with an infrared filter and infrared lamps that shine at the face. The camera does not register the entire face, just reflections from the retina. This can then be calculated into coordinates on the screen."

Often, eye trackers are used by paraplegics instead of computer mice. Here, however, the data which the device generates is used to monitor the behavior of the air traffic controller. Does he maintain oversight of the screen at all times? Is he distracted? Has he spotted a text message on his phone or is a colleague chatting to him over coffee?

Microsleep is not unannounced

An eye tracker can detect when an operator gets tired after long hours of monotonous work. "The frequency and time of eyelid closure are good indicators to detect tiredness," says the psychologist, Schwarz.

And the danger is microsleep.

You can tell when someone is heading into microsleep, because their eyelids close for longer periods than usual when they blink. This is something the eye tracker can detect with certainty - the computer can raise an alarm.

Aside from eye trackers, scientists have other detectors at their disposal. For example, there is a special belt, originally developed for soldiers and firemen. It constantly provides information about vital signs of the body.

"You wear the belt around your chest. It includes a sensor unit, which can measure physiological indicators such as heart beat, breathing frequency, body temperature and even their posture," Schwarz explains. So if an operator starts to lean forward or backwards - because they are tired - the sensor will detect the precise angle of the body.

Changing the body position under stress

By combining the data from the eye tracker with those from the sensor belt, the computer can detect all kinds of things. Is the air traffic controller under stress because her pulse is high, or she's breathing erratically, keeps an eye focused on one spot on the screen and leans more and more forward?

"When the work demand increases, many people tend to lean forwards more and more. They almost crawl into their computer screens," Sven Fuchs says. "This is something you can detect very well with the sensor belt."

An alternative would be a seat cushion equipped with sensors, which can also detect how an operator sits. If they are relaxed, they may lean back, resulting in a lower pulse rate and calm breathing. But it could also mean they are about to fall asleep. The computer has to be alert as well.

Many practical applications

The system is not just for air traffic controllers. It can be used anywhere where humans make mistakes that can lead to severe damage, injury or death. This could include car or truck drivers, or operators in the control rooms of power plants and chemical factories.

The aim of the research project is to see whether computers can make the right decisions when we fail. Project director Fuchs says it is like computers becoming one of the team.

"People can adjust to situations. I understand that if I see a colleague [has to concentrate], I'd better leave him alone. Or they see someone has too much time on their hands and is bored. That colleague may be able to help out with a little task. I believe we are also moving in that direction with interactive intelligent systems."