While the crew is away from a space station orbiting near Mars, a critical battery panel module begins to show signs of failure. With no humans available to retrieve and install a spare module from a storage area, the issue could begin to affect systems in the outpost – and require immediate attention when the crew returns. 
 

A knowledge-driven robotics (KDR) system now under development at the Georgia Tech Research Institute (GTRI) could be integrated with the future outpost’s diagnostic and maintenance systems to provide a solution for such equipment-related scenarios. Using its unique capability to autonomously execute complex tasks – even those it had never done before – the KDR system would retrieve a spare module and replace the failing battery. 
 

Unlike conventional autonomous robotic systems that must be programmed to carry out specific tasks planned ahead of time, the knowledge-driven robot – like a jack-of-all-trades custodian – will figure out a solution to the battery problem and other issues that may arise. The system will provide a novel approach to automating multi-agent planning, drawing from a database of actions that it knows how to perform and using its comprehensive knowledge of equipment onboard the station. 
 

The KDR software architecture will provide novel capabilities for autonomously executing complex coordinated robotic behaviors in a variety of locations, such as space or planet Earth. Its autonomous planning and execution capabilities will benefit from the ability to continuously respond to new requests, detect and learn from errors, and incrementally revise its plans as priorities and knowledge change. 
 

And if it should end up stumped for what to do or hung up in an error, the robotic system could contact a human “call center” for help – and even send detailed images of the problem.
 

Helping Robots Understand What They’re Doing
 

“We’re trying to create a robotic system that understands what it’s doing,” said Stephen Balakirsky, a GTRI Regents’ Researcher who leads the project. “In Knowledge Driven Robotics, we’re trying to create a robot that is much more flexible in how we can change its tasking and what it can do. Our hope is that when the space station crew is away, the robots can keep the lights on and everything running until the humans return.”
 

Balakirsky compares the knowledge-driven strategy to teaching a robot how to use saws, hammers, drills, and other construction tools to build structural components such as a home’s walls and floors. Such a construction robot could then build a house based on conventional plans. That would be a lot simpler than having to give a house-building robot detailed instructions on every single step of the construction process and make it easier to build different homes or structures.
 

“That will be treating the robot like a skilled craftsman with a limited vocabulary of basic skills,” he said. “Once we train the robot with low-level skills, those skills generalize so that it understands what it is doing. We can tell the robot what we want it to do in higher-level terms, and let the robot worry about the lower-level details of how to get those things done.”
 

Combining a Series of Small “Atomic” Tasks 
 

In most existing robotic systems, complex jobs such as replacing a battery would have to be pre-programmed into the robots. For a space structure located far from human controllers on Earth, the list of possible tasks would be immense and couldn’t include every possible scenario. And because of the time required for signals to travel from Earth to the space station, teleoperation of the robot would be impractical.
 

The KDR software being developed at GTRI will combine a large number of low-level actions, or atomic tasks – such as grasping a plug – and combine them to handle larger tasks, such as disconnecting a battery module, moving to a storage area, opening an access panel, and connecting a new battery. The atomic tasks are contained in what robotics engineers call a “behavior tree” – a hierarchical structure where atomic tasks are organized in a tree-like structure with control flow mechanisms that allow for modular and adaptable decision-making. Combining these simple atomic tasks into larger mid-sized composite tasks would ultimately allow the system to handle a complex job like replacing a battery module.
 

Knowledge the system gains from each new task it completes could be generalized to help with future tasks.
 

“The major thing that’s unique to our system will be the amount of knowledge we can bring to the system to make it more flexible and generalizable,” said Balakirsky. “We are coupling a database of activities in our behavior trees. We can have much more context and knowledge that will be available for the robot tasking to take place to address unexpected situations.”
 

Easy for Humans, Harder for Robots
 

Another example of a task suited to the knowledge-based robot would be connecting a cable, which involves determining the correct configuration of the cable’s connector, using the correct angle to insert the cable, and applying the right level of force to insert the connector. Humans can do that without much thought, but the robot will have to know the many different kinds of connectors that it may have to work with, and how insertion for each kind differs.
 

“We can teach the robot generic skills and have a database of those skills that allows us to generalize the task to an HDMI cable, a USB cable, or a coaxial cable,” Balakirsky said. “It’s the same programming in the robot, but with differences to account for the differences in connectors. So not only can we do a wide variety of tasks, but also a wide variety of similar categories of tasks.”
 

Being in a space structure far from Earth may seem like a disadvantage for a maintenance robot, but there are also advantages. “Because we’re in a space station, we’re limited in what could be there,” Balakirsky noted. “The robot knows what is on a space station and can have an inventory, which makes it much easier to identify what it is dealing with.”
 

But even a very knowledgeable robot can’t know everything or avoid all errors. For that reason, human experts will be available in a “call center” to help the robot figure out what to do if something goes wrong. An advanced imaging system capable of sending stereo images of the problem will help humans in the call center figure out what’s wrong and help the robot determine how to fix the trouble. The humans will be able to suggest new tasks for the robot to complete, and will not directly teleoperate the robots. The system will have built-in error checking to detect and correct anything that goes wrong, allowing the robot to learn from its mistakes.
 

Potential Applications Back on Earth
 

GTRI researchers have been working on the KDR concept for more than five years, collaborating with researchers at Carnegie-Mellon University and the University of California Davis. In a recent demonstration for NASA – which is one sponsor of the work – Balakirsky and collaborators Research Engineer William Freidank and Senior Research Engineer Konrad Ahlin demonstrated how a pair of robotic arms operating on a six-meter shared track can work together. Their next step will be to show how the robotic system can actually remove and install a battery module.
 

Beyond the space applications, knowledge-driven robots could also be useful on Earth, where they might be tasked with keeping machinery going after hours in manufacturing operations. “If an alert happens somewhere in the facility, the mobile system could go to the problem area and provide an overview to a remote operator, who would be able to task the robot to fix whatever the issue was,” Balakirsky explained. The robots could also be used in national security applications such a bomb disposal to handle some of the steps – such as opening a cover – that are now done with tele-operation. 
 

Future plans call for coupling the robotic system to augmented reality to help remote operators see and better understand error situations. In space, the robot might also be connected to a 3D printer that would be able to manufacture a replacement part, if needed.
 

Helping Define Standards for Robot Tasking
 

Though the NASA funding for the knowledge-driven robotic software has now concluded, GTRI is working with the National Institute of Standards and Technology (NIST) and the IEEE Robotics and Automation Society on a robotic tasking standard (1872.1-2024) and an application guide (P1872.1.1) to help apply the standards.
 

“We’re using the system as a model for how to send in a standard task to a robot and know how the robot will execute the task,” Balakirsky said. “We’ll be trying to standardize the way that humans interact with the system to provide as much flexibility and understandability as possible.”
 

Balakirsky hopes the project will change the way humans interact with robots, moving from an emphasis on programming to assigning high-level tasks that the robots are knowledgeable enough to carry out. Improved communications could also be part of that interaction.
 

“In the not-too-distant future, we’re hoping to be able to interact with robots through natural language,” he said. “That will be a much more intuitive and direct way of interacting with robots than what we currently have.”

Writer: John Toon (john.toon@gtri.gatech.edu)
GTRI Communications
Georgia Tech Research Institute
Atlanta, Georgia USA

The Georgia Tech Research Institute (GTRI) is the nonprofit, applied research division of the Georgia Institute of Technology (Georgia Tech). Founded in 1934 as the Engineering Experiment Station, GTRI has grown to more than 2,900 employees, supporting eight laboratories in over 20 locations around the country and performing more than $869 million of problem-solving research annually for government and industry. GTRI's renowned researchers combine science, engineering, economics, policy, and technical expertise to solve complex problems for the U.S. federal government, state, and industry.