As AI lies spread on social media, often outstripping the ability of professional fact-checkers, fake detectors risk adding another layer of deception to an already fragmented information ecosystem.
While even reliable AI detectors can produce false results, researchers say a series of fraudulent tools have emerged online that can be easily weaponized to discredit authentic content and tarnish reputations.
AFP fact-checkers have identified three text detectors that claim to estimate what percentage is AI-generated. The tool – piloted in four languages – not only misidentified authentic text as AI-generated, but also attempted to monetize those errors.
One detector, Justdon AI, processed a human-written report about the US-Iran war and incorrectly concluded that it contained “88% AI content”. He then offered to clean up any traces of the AI for a fee.
“Your AI is humanizing text,” the site claimed, leading to a page where “100% unique text” was locked behind a paywall charging up to $9.99.
Two other tools – TextGuard and Refinely – generated similar false positives and tried to monetize them.
‘Scam’
The AFP presented its findings to all three detectors.
“Our system works using modern AI models, and the results it provides are known to be accurate within our technology,” TextGuard’s support team told AFP.
“At the same time, we cannot guarantee or compare results with other systems.”
Justdon also reiterated that “no AI detector can guarantee 100 percent accuracy.”
It acknowledged that the free version of its AI detector “may provide less accurate results due to the use of a lightweight model designed for high demand and quick access”.
Echoing AFP’s findings, a user on a review forum complained that “even with 100% human-written content, Justdon still flags it as AI.”
AFP provided the equipment with several human-written samples in Dutch, Greek, Hungarian and English. All were incorrectly marked as having high AI content, including excerpts from the acclaimed 1916 Hungarian classic.
The tools return AI flags regardless of the input – even for meaningless text.
Justdon and Refinely appeared to work even without an Internet connection, suggesting that their results may be scripted rather than genuine technical analysis.
“These are not AI detectors, but scams to sell a ‘humanizing’ tool that often return what we call ‘atrocious phrases'” – disjointed jargon or nonsensical substitutes – Germany-based academic Deborah Weber-Wolf, who has researched the detection tool, told AFP.
‘Liar’s Dividend’
Illustrating how such tools can be used to discredit individuals, pro-government influencers in Hungary claimed earlier this year that a document outlining the opposition’s election campaign was created entirely by AI.
To support the baseless allegation, he circulated screenshots showing Justdon’s results on social media.
The tools tested by AFP tried to lure students and academics as customers, with two of them claiming their users came from top institutions such as Cornell University.
Cornell University told AFP it had “no established relationships with AI detector companies.”
“Generative AI provides an increased risk that students could use it to submit work that is not their own,” the university said.
“Unfortunately, it is unlikely that detection technologies will provide a practical solution to this problem. Accurately detecting AI-generated content may be very difficult.”
Fact-checkers, including those at AFP, often rely on AI visual detection tools developed by experts, which typically look for hidden watermarks and other digital clues.
However, they can also sometimes introduce errors, making it necessary to supplement their findings with additional evidence such as open-source data.
The risk is huge as inaccurate readings from unreliable detectors risk ruining Trust AI Verification More broadly – and to promote a disinformation strategy researchers have dubbed the “Liar Dividend”: dismissing authentic content as AI creations.
“We frequently report on misinformation purveyors and other fraudsters who use AI to create false photos and videos,” said Waqar Rizvi of misinformation tracker Newsguard.
“Now, (we’re) monitoring the opposite, but no less deadly phenomenon: claims that a scene was created by AI when, in fact, it is authentic.”