China’s anti-fraud app uses AI to combat fake content

A new feature on a national anti-fraud app in China is using artificial intelligence to detect AI-generated traces in images, videos, audio, and text, creating significant buzz. Reported by China Central Television, this development aims to identify fake content and prevent scams, offering relief to online store owners struggling with fraudulent refund requests.

Recently, online retailers have faced challenges from buyers using AI-generated images to claim refunds without returning their purchases. The report highlights a common practice where buyers photograph defective goods to request refunds or returns. However, digitally manipulated images allow unscrupulous buyers to keep their products, leaving merchants frustrated and sparking public debate.

One case involves a fig tree farmer, surnamed Zhang, from Xinji, Hebei province. Zhang opened a store on a livestreaming platform to sell saplings but soon faced a "refund request only". A buyer claimed the saplings arrived dried out, providing a photo of a yellowed, lifeless plant as evidence. Zhang, however, was confident in the quality of his shipment, noting that he had sent a particularly healthy sapling with moist soil, making it improbable for the plant to dry out in just two days.

Drawing on his extensive cultivation experience, Zhang spotted inconsistencies in the photo. He noted that truly dehydrated plants would have wrinkled, drooping leaves, unlike the altered image presented. Despite Zhang's defense, the platform sided with the buyer, approving the refund based on the manipulated photo. Disheartened, Zhang decided to close his store, abandoning the platform.

Zhang's ordeal is not isolated. Many sellers have shared similar experiences, encountering AI-generated fake images in after-sales disputes, according to the report. Some have complained about identical breakages in multiple hangers, clearly indicating digital manipulation.

In today's world, where AI-generated images are increasingly common, distinguishing between authentic and fake content is a growing challenge. To address this, the national anti-fraud app, promoted by the Ministry of Public Security and several internet operators, is enhancing research and deploying AI detection tools.

Lan Jun, a security content visual algorithm specialist at an online platform, explained to CCTV that their detection method involves extensively learning the patterns of AI-generated content. This process uses large models to scrutinize content for physical flaws, such as garbled text or logical inconsistencies.

"We aim to minimize false identifications," Lan said, advising users and businesses to employ external AI authentication tools to support their judgments.

The key strategy to combat AI forgery is leveraging AI itself. According to the report, using AI to create fraudulent images for refunds constitutes civil fraud. If unchecked, it could disrupt the e-commerce ecosystem.

AI authentication tools can significantly reduce the cost of identifying fraud for businesses, acting as both a deterrent to wrongdoers and a robust evidence base for negotiating with platforms. Xie Yongjiang, head of the Internet Governance and Law Research Center at Beijing University of Posts and Telecommunications, emphasized the responsibility of platforms to adopt these technologies to maintain transaction integrity and protect the rights of businesses and consumers alike.