Do you ever see comments on social media that seem way off topic, but still manage to wrench the discussion around to divisive political debate?
A discussion about the cost of living suddenly becomes an argument about immigration. A conversation about the war in Ukraine turns into claims about government corruption. It can feel jarring – and sometimes this is deliberate.
As generative AI becomes more powerful, malicious groups are increasingly using it to produce and spread disinformation online. Automated accounts can flood social media with convincing comments designed to sow division, inflame political debate and undermine trust in reliable information.
But latest research offers a way to spot these attempts. Rather than trying to identify whether a post was written by AI, researchers focus on something different: whether it’s trying to derail the conversation.
Until recently, identifying malicious accounts was often quite straightforward. Many campaigns relied on people writing in a second language. So, posts sometimes contained grammatical mistakes or unusual word choices. Detection systems could look for these patterns in the language used.
But generative AI has changed that. AI systems can now produce fluent, natural-sounding text that is much harder to distinguish from human writing. For example, patterns like use of em-dashes and the word “delve” used to be telltale signs of a text being generated by AI. But AIs are adapting, and these older systems are increasingly ineffective.
Trying to detect AI purely from the words people use is becoming a losing battle. Researchers believe the better approach is to look at what a message is trying to achieve.
Looking for signs
Attempts to spread disinformation often work by steering conversations away from their original topic, towards more polarising issues. So, instead of analysing individual words, researchers set out to build a system that could recognise this phenomenon in online discussions.
They analysed comments posted beneath BBC News videos on YouTube, a platform that has previously been targeted by organised disinformation campaigns.
For example, imagine a comment about Ukraine’s president, Volodymyr Zelensky, interacting with senior UK political figures: “Zelensky must be wondering how many foreign secretaries the UK goes through.” Now, imagine another person responding: “Mind you, Zelensky has barely been president for four years. Maybe that’s why the little tyrant bans his opposition.”
Whether that second point is true or false is not the issue. Instead of responding to the original comment, it redirects the conversation towards a different, more divisive topic.
This is known as a red herring: introducing an unrelated issue that distracts from the original discussion. These kinds of shift are difficult for conventional disinformation detection systems to identify, because they are not tied to particular words or phrases.
Also read: How AI speaks: The science behind lifelike computer voices
Researchers manually analysed more than 1,600 comments under BBC News videos, labelling them according to 25 different features of online discussion – and discovered some clear patterns.
Researchers found that 36 per cent of derailing messages had red herrings, 65 per cent had leaps in logic known as “non sequiturs”, and 20 per cent contained personal attacks. They were also much less likely to acknowledge previous comments or express empathy.
Spotting manipulation
The next step was to see whether an AI system could recognise these patterns automatically. Researchers used an AI to catch an AI.
For every genuine online comment, researchers asked an AI large language model to generate several reasonable, relevant responses. Returning to the example of UK foreign secretaries, the AI suggested replies such as: “The current situation in this country must come as quite a shock” or “One too many?”. Both responded directly to the original point.
The system then compares the real response with AI-generated replies. If the actual comment differs substantially, it may indicate that someone is attempting to steer the conversation in a different direction. So, rather than searching for suspicious words, the system looks for unexpected changes in the flow of the discussion.
How the system works:
Researchers tested this approach using manually labelled dataset. In the second study, the system correctly identified derailing comments around 77 per cent of the time.
That’s far from perfect, but no detection system is – particularly when analysing something as complex as human conversation. However, the approach performed around twice as well as existing systems based on word-level sentiment analysis. It also achieved results comparable with the level of agreement between human researchers.
The approach is effective because the AI learns what a typical response to a conversation looks like. When a reply unexpectedly changes the discussion, the system can identify that change and analyse patterns that earlier methods couldn’t detect.
Of course, going off topic isn’t necessarily a sign of malicious intent or disinformation. People naturally take conversations in unexpected directions, and there are many legitimate reasons why discussions evolve.
For that reason, this technology may act as an early-warning system rather than a replacement for human judgment. It could help moderators identify conversations that deserve closer attention – but any final decisions should remain with trained experts.
There are also important ethical questions to address. AI systems can reflect biases in the data they are trained on, and they still do not understand conversations in quite the same way that people do. Improving how AI represents and interprets human discussion remains a challenge.
As AI-generated content becomes increasingly difficult to distinguish from human writing, detecting disinformation requires more than simply searching for telltale words. It requires understanding how conversations work, how they are manipulated, and when someone is trying to quietly steer them off course.
Via The Conversation
Follow us
