New guardrails to make anthropomorphic AI safer

Editor's note: China has issued new rules for regulating providers of artificial intelligence systems that simulate human personality traits, thinking patterns and communication styles in emotional interactions with users. Science and Technology Daily spoke to Huang Wenhong, a researcher at the China Center for Information Industry Development, and Sang Jitao, a professor at the School of Computer Science and Technology at Beijing Jiaotong University, about the importance of the new rules that will come into effect on July 15. Below are excerpts of the interviews. The views don't necessarily represent those of China Daily.

The new rules focus on AI services that simulate human personality traits and thinking patterns, establishing clear boundaries for anthropomorphic AI interaction and placing greater responsibility on platform operators, which are prohibited from excessively catering to users, fostering emotional dependence or addiction, or manipulating users emotionally in ways that could influence them to make unreasonable decisions.

Platforms are also barred from offering minors virtual intimate relationship services, including virtual family members or romantic partners. In addition, they are required to identify high-risk situations, such as users exhibiting extreme emotional distress or suffering significant financial losses, and provide appropriate reassurance and, where necessary, timely intervention.

AI agents are applications designed to accomplish specific objectives. An AI agent can understand a user's objective, break complex tasks into manageable steps, use external tools, access relevant knowledge bases and data, monitor task progress, and complete an entire workflow within the limits of its permissions. Many AI agents are also equipped with long-term memory, enabling them to retain conversation history, user preferences, role settings and the status of ongoing tasks, while coordinating with other AI models to execute complex assignments.

The rules do not apply the brakes to the development of custom AI agents. Instead, they establish guardrails for anthropomorphic interaction. They are likely to accelerate the transition from an early stage characterized by low barriers to entry, rapid expansion and unrestricted publication toward a more mature ecosystem featuring categorized supervision, risk-based governance, prerelease review and built-in safety mechanisms. As the rules take effect, platforms are expected to strengthen both the review process for creating and publishing AI agents and associated safety testing.

The rules accept that custom AI agents built on general-purpose large models remain an important direction for the future development of AI applications. But, going forward, a clearer distinction will need to be made between task-oriented AI agents and anthropomorphic agents designed for emotional companionship.

Task-oriented agents, which include intelligent customer service, knowledge bases, work assistants, and educational and scientific research applications do not involve continuous emotional interaction and therefore fall outside the primary scope of the new rules. By contrast, anthropomorphic, companion-oriented and role-playing AI agents will be subject to a more stringent oversight framework.

Although the new rules are likely to increase compliance costs in the short term and reduce the space for low-quality, borderline-legal and traffic-driven AI agents, they are expected to foster healthier long-term development of compliant AI applications serving fields such as education, work, scientific research, cultural communication, eldercare companionship and support for people with special needs.

Before the new rules were released, some AI platforms had already allowed users to create and publicly share a large number of customized AI agents, whose ability to maintain long-term memory and adopt character settings makes it easy for some of these applications to cross the regulatory boundaries governing anthropomorphic interaction.