Artificial intelligence (AI) chatbots can give people a comfortable place to discuss personal thoughts and feelings. However, when a chatbot constantly agrees with a user, it can reinforce false beliefs and potentially cause harm.

New research from Georgia Tech suggests that a simple change to a chatbot’s instructions could reduce that risk. The work was accepted to the ACM AI Letters publication. 

Researchers Lorenzo de la Loza and Vijay Madisetti tested eight AI models in 1,280 experiments involving simulated conversations about delusional beliefs. They found that instructions designed to discourage excessive agreement reduced the models’ confirmation of those beliefs by 73.8%. Adding a self-reflection step increased the reduction to 77%.

“We think that this paper will help AI providers and solution creators better understand how best to protect and safeguard their users from harming themselves,” Madisetti said. “This may make AI safer for all of us.”

AI chatbots are designed to be helpful and supportive. But that behavior can become risky when a chatbot agrees with a user’s beliefs without questioning whether they reflect reality. Researchers call this behavior sycophancy.

Courtney Crooks, a principal research scientist at the Georgia Tech Research Institute and adjunct faculty member in Georgia Tech’s School of Cybersecurity and Privacy, said chatbot conversations can have both positive and negative effects.

People may feel more comfortable discussing difficult thoughts with a chatbot than with another person. But if the chatbot always agrees, users may miss opportunities to question their beliefs or consider other points of view.

The same behavior that makes a chatbot feel supportive, Crooks explained, can also encourage harmful patterns of thinking.

Read the full article at Georgia Tech Research News Center