Now for the first post since the update, about chatbots in fact-checking, from China.

The November article by Zehang Xie from Nanyang Technological University, Hui Hui from Shanghai Jiao Tong University, and Yunxiang Xie from Quanzhou Vocational and Technical University investigated the effectiveness of chatbot interventions in fake news detection.

It is hard to distinguish between credible from false content as fake news spreads rapidly on digital social media. This is a struggle for traditional fact-checking such as professional journalism and third-party verification. AI chatbots offer real-time assistance for identifying and verifying content.

There is a gap in the previous studies, in that they typically assess detection accuracy but not the effectiveness of specific effectiveness, and they typically analyze static one-off exposures. This is addressed in this study with a 6 (fake news type) × 3 (chatbot strategy) mixed design in HCCs. Fact-checking, contextual explanations, and authority endorsements and tested pre-versus and post-intervention accuracy.

The experiment was conducted with 540 participants from December 2024 to January 2025. They had a mean age of 21,95 years, with 246 identifying as female. The authors obtained widely discussed topics from China’s official misinformation debunking platform. The fake news were generated with ChatGPT, with two associate professors with PhD’s in journalism refining them. Each type of false news had six examples.

There were three chatbot strategies. The chatbot provided participants with factual corrections of each fake news article. It could also provide background information and broader context. The chatbot also could leverage official government sources and expert/institutional opinions, these included the Chinese government and WHO. After the chatbots had provided assistance to the participants, they classified the articles as fake or real and discussed them in their assigned groups, and repeated the classification task.

There was a significant effect caused by time – improvement from pre-test to post-test was seen, but it was qualified by interactions with fake news type and chatbot strategy and thus varied. The interaction between time and fake news type was not significant. On the other hand, the three-way interaction between time, fake news type, and chatbot strategy was significant.

For fabrication and photo manipulation, factual corrections were the most effective. In the case of parody and satire, contextual explanations were more effective than the alternatives. For advertising, context was better than authority, roughly equal to fact-checking. Authority endorsements worked best against propaganda.

In conclusion, no single strategy proved the most effective, and dealt with chatbot interventions in human-computer communities (HCCs). HCC’s support adaptable correction styles from quick cues to concise reasons. The theoretical implication of the study is that there is no one size that fits all in interventions. Second, it shifts focus from static interventions to socially embedded interventions. Third, it integrates Heuristic-Systematic Model (HSM) into chatbot research.

The article “Fake or fact? Evaluating chatbots’ performance to help users detect fake news in human-computer communities” by Zehang Xie, Hui Hui, and Yunxiang Xieb is in Journalism. (Free abstract).

Picture: A group of artificial intelligence robots answering the question by Mohamed Nohassi.

License Unsplash.