Despite the expanding capabilities and increasing sophistication of automation, human operators still play vital roles in most military systems today. As the complexity of these systems continues to grow, so does the importance of understanding how humans interact with these technologies.
 

Scientists at the Georgia Tech Research Institute (GTRI) are using wearable instrumentation to study human performance indicators such as electrical activity and oxygenation of the brain, heart rate, respiration, eye movements, and facial expressions to understand how operators respond to stressful conditions while interacting with military systems.
 

The researchers are adapting compact, portable and unobtrusive wireless equipment to take their cognitive studies beyond traditional laboratories to field locations where frontline aircrews, vehicle drivers, and specialized system operators work. What they are learning in these real-world conditions may help engineers improve the human-machine collaboration needed by today’s advanced systems, including new AI-supported systems.
 

“Different patterns of brain activity are statistically correlated with certain cognitive states and can be measured using wearable devices,” said Emily Maw, a research scientist in GTRI’s Human Performance Branch. “Using electroencephalography (EEG), we can tell if a person is experiencing higher workload, or lower workload, or if they are feeling drowsy and are not paying attention. That allows us to track that person’s performance of the task over time and pair it with physiological data, like EEG, to provide a comprehensive timeline for what they were experiencing while completing that task.”
 

This cognitive performance data can help designers develop systems better able to collaborate with aircrews or system operators, taking into account the workload and how the human brain handles it. These human-system interactions can be studied with simulated equipment in a GTRI laboratory, or in real-world conditions at military facilities without interrupting warfighter activities.
 

Maw is an experimental psychologist and human factors engineer. She specializes in a field called neuroergonomics that combines neuroscience and ergonomics – which studies how products or systems should be designed to work with the people who will use them. 
 

“We take information that we glean from fields such as neuroscience and cognitive psychology and apply that to hands-on, fielded systems or scenarios in which we can observe an operator as they are performing a task, either in a field environment or a simulated environment,” she said. “Over the years, this area of research has expanded the constellation of physiological tools that we can use to track internal biomarkers which give us a glimpse of the operator’s internal states.”
 

Neuroergonomic Assessment of Warfighter Cognitive State
 

Fluctuations in cognitive state can indicate the potential for lapses in operator performance that could have critical impacts on military missions. Traditional ways of studying operator workload and engagement can be subjective and can potentially skew the data collected by creating their own disruptions if not carefully implemented. They may also depend on operators accurately recalling what they were doing before and during those lapses.
 

Thanks to new stick-on electrodes that record brain activity, physiological measures of operator activity can now be more easily collected, providing a real-time objective snapshot of the human performance during a task being studied. 
 

Maw and her team used this “brain at work” technique to study operator vigilance. In their research, experimental subjects drawn from GTRI research staff watched a gauge to detect small (3 degrees of change) and larger (15 degrees of change) variations in a readout – an experimental activity adapted from the traditional Mackworth Clock Task, which was originally designed to simulate monitoring activities completed by radar operators during World War II. In two 15-minute vigils, low and high demand conditions were differentiated.
 

The experiment’s results showed an increase in reaction times across periods of watch for both conditions and higher reaction times in high-demand conditions. These differences were reflected in physiological changes recorded via EEG.
 

Supported by GTRI’s Independent Research and Development (IRAD) program, the researchers have expanded on conventional measures used to understand what’s happening inside the human body. For example, remotely measuring oxygen levels in the human brain’s prefrontal cortex can give insights into executive functions that control decision-making.
 

Cerebral Hemodynamics During Simulated Turbulence
 

Because they fly at low altitudes, helicopters experience considerable turbulence that can negatively affect their human operators. Leveraging the Human Centered Engineering’s Simulation Lab, Maw used a helicopter seat installed atop a three-axis motion platform to create simulated turbulence to study brain blood flow in the prefrontal cortex areas of 10 experimental subjects.
 

Using functional near-infrared spectroscopy (fNIRS), an emerging technology that uses optical sensors to track the relative concentration of oxygenated and deoxygenated hemoglobin within the brain to indicate cognitive load, Maw and her team evaluated the impacts of the simulated turbulence on the quality of recorded hemodynamic data. The goal of the study was to see whether fNIRS could address the shortcomings of other neuroergonomic measurement techniques in a dynamic field environment.
 

With an auditory working memory task to induce cognitive load on the test subjects, the study observed an increase in brain oxygenation for both moving and stationary conditions, and as expected, the difference was not significant. The successful measurement of induced cognitive load regardless of motion suggests that fNIRS could be a useful neuroergonomic technique for field measurements of critical executive functions, the researchers concluded.
 

Mobile Equipment Supports Testing in the Field
 

Maw and her collaborators have a well-equipped laboratory, but most of their equipment can also be packed up for use in the field. Mobile sensing technology has been available for years, but recent improvements have taken the capabilities to a new level.
 

“What we’re seeing now is an incredible upswing in the abilities of the technology we can use,” Maw said. “Developers are getting better at building smaller devices that are less expensive, and a lot of the improvements are really in the data-processing backends of the systems that allow us to do more with the data we have.”
 

This new wearable equipment is expanding what the researchers can do in the field, a capability that makes the studies more realistic. “The sensors are wireless, so once they are in place, they are pretty comfortable and have minimal impacts on the person doing the task we are observing,” Maw said.
 

Facilitating AI Collaboration with Humans
 

The growing importance of artificial intelligence (AI) in military systems is another area where human performance research may play an important role by helping humans understand what the machines are doing – and helping the AI understand how to collaborate with humans.
 

“How do we tell the human about the state of the system, and how do we tell the system about the state of the human it is interacting with?” she asked. “The human needs to understand what the computer is saying about its state, and the computer needs to know how the human is feeling and what the human’s workload is.”
 

Machine learning, another component of AI, is becoming increasingly important to studies of human performance by accelerating how data produced can be analyzed and studied. “Frequently, the data sets we generate can be huge,” Maw said. “Machine learning speeds up the timeline. That means we can return our results a lot faster.”
 

Evaluating Changes in System Interfaces
 

Upgrades to electronic warfare and other military systems often require changes in how humans interact with the equipment that helps protect them. Because they are used during stressful situations, these changes must be designed to avoid interrupting the split-second interactions that experienced aircrews expect.
 

“In the military world, the environments are incredibly complex and safety-critical,” Maw noted. “A single mistake can have severe consequences, so being able to analyze the workflows and performance interactions at a granular level is important to understanding how someone’s workload is changing. This is also important to the next step in research, which would be predictive analysis, to catch a mistake before it happens.”
 

The terms “mistake” and “lapse” are often used interchangeably, but to human factors engineers, they are different. Humans have a limit to the cognitive workload they can accommodate, and lapses are a specific type of error that may occur when well-trained operators are overloaded in situations that may pop up unexpectedly. In these situations, it can be easy, even for seasoned operators, to skip an important step or misremember a critical procedure. For aircrews, these lapses are more likely to occur during landings and takeoffs in high-stress scenarios. These types of errors are one reason why detailed checklists are a key component to many military procedures.
 

While stressful environments may be associated with an increase in errors, the other end of the stress spectrum, low cognitive workloads, can also create problems by inducing boredom that may lead to drowsiness. This can be especially challenging for long-duration missions, or tasks that require long-term monitoring at night when an operator must battle normal circadian rhythm.
 

Team Brings Military Experience to Research
 

Maw and her team have documented recent success testing these technologies outside of a laboratory setting, and are talking with various organizations about applications for this capability in the future.
 

“We have a unique capability specifically designed for military applications,” she said. “We are able to be successful in these areas because we have a deep internal knowledge about the context of where warfighters are working. We have a very deep tool bench and deep expertise in neuroergonomics and neurocognitive studies that help us offer custom solutions to our government sponsors.”
 

Writer: John Toon (john.toon@gtri.gatech.edu)
GTRI Communications
Georgia Tech Research Institute
Atlanta, Georgia USA

About the Georgia Tech Research Institute (GTRI)
The Georgia Tech Research Institute (GTRI) is the nonprofit, applied research division of the Georgia Institute of Technology (Georgia Tech). Founded in 1934 as the Engineering Experiment Station, GTRI has grown to more than 3,000 employees, supporting eight laboratories in over 20 locations around the country and performing more than $919 million of problem-solving research annually for government and industry. GTRI's renowned researchers combine science, engineering, economics, policy, and technical expertise to solve complex problems for the U.S. federal government, state, and industry.