Domestic AI models tackle unique demands of Cantonese

"Is there a wife in a wife cake?" Heyson He Lixi, an influencer promoting Cantonese, asked.

"There is no actual wife in a wife cake, as it is just a traditional Hong Kong pastry," answered Sensechat, the world's first Cantonese large language model, a ChatGPT-like product, in fluent Cantonese.

He raised a more challenging question, "What is sorrowful rice?"

"Sorrowful rice, a well-known delicacy in Hong Kong, derives its name from the martial arts masterpiece The Return of the Condor Heroes by Hong Kong writer Jin Yong. In the novel, the character Yang Guo experiences deep sorrow after losing his beloved, Xiaolongnyu," Sensechat responded.

He was impressed — it was the first artificial intelligence tool he had encountered that exhibited a profound understanding of Cantonese culture.

Born in 1998, He is fond of Cantonese and has been creating videos promoting the language since college. Over the years, he has amassed a fan base that surpasses 700,000 across various online platforms, including YouTube and popular Chinese mainland social media platforms like Bilibili, Xiaohongshu, and Douyin. Due to the recent AI craze, He started producing videos evaluating the tools' ability to understand Cantonese.

The outcomes are underwhelming. Many AI large language models (LLMs) and AI software developed on the mainland fail to recognize Cantonese phonetically. Some Western-developed AI software can listen to Cantonese, but cannot speak it accurately. ChatGPT, for instance, often blends Cantonese with Mandarin. Suno, an AI large language model tool that specializes in generating songs, can pronounce Cantonese to a degree, but its primary focus remains music creation.

In July, the Sensetime Group, an AI developer based in Hong Kong, introduced Sensechat, a Cantonese version of its proprietary LLM, and announced that it would be available for free to Hong Kong users indefinitely.

Upon a friend's recommendation, He downloaded Sensechat.

"I felt 85 percent satisfied with Sensechat," he said. "The application still requires to be further refined, but it is one of the few that can truly understand Cantonese."

The application emphasizes one of the unique traits of Cantonese — its colloquial nature.

Pronunciation of Cantonese involves extensive use of modal particles, which are often used at the end of sentences to indicate mood. These particles usually go unnoticed by most AI tools, but Sensechat captures them effectively.

In terms of written text, Sensechat can understand and reflect the nuances between the two forms of written Cantonese. It has a standardized form used in formal situations, similar to Mandarin, and a phonetic style for everyday use. This characteristic, He said, is often overlooked by other large language models.

He recorded his interactions with Sensechat, and shared it online, garnering over 150,000 views. "Cantonese speakers truly need such a tool," He said.

Data size matters

Training an LLM typically involves three stages, said Cao Jiannong, the chair professor in the Department of Computing at Hong Kong Polytechnic University.

The first stage requires pre-training using extensive data, followed by fine-tuning with high-quality data. In the third stage, humans are needed to align the output of the LLM with local culture, ethics, morals, laws, and other rules to restrict the risk of generating inaccurate, biased, or unlawful content.

Developing a Cantonese LLM faces difficulties in all three stages, Cao said.

While Hong Kong's internet infrastructure is relatively well-developed, there is a scarcity of Cantonese content available online. A major factor contributing to this scarcity is that while Cantonese is widely spoken in daily life, the written form of Cantonese is Chinese.

Moreover, English has long served as the official language in Hong Kong. Consequently, a significant portion of the city's online information, including official archived documents in areas such as law, finance, politics, and medicine, is predominantly available in English, Cao said.

LLMs rely heavily on abundant data for their training, said Francis Fong Po-kiu, honorary president of the Hong Kong Information Technology Federation, a local IT-related business association. Without data, there is simply no way to develop a language model, he said.

Literature scarcity

Cantonese web resources suffer not only from a shortage in quantity, but also a lack of quality, said Cao.

When it comes to written material, Hong Kong has not prioritized literature, resulting in a scarcity of quality Cantonese literary works, said Keith Li King-wah, chairman of Hong Kong Wireless Technology Industry Association.

Most available Cantonese texts come from online forums and social media, and often contain low-quality and even offensive language, potentially leading AI models to produce crude content, Li said.

Collecting speech data presents another problem.

Despite access to Cantonese videos online, such as movies and TV dramas, they cannot be used due to background noise, said Albert Lam Yun-sang, the chief technology officer and chief scientist at Fano Labs, a Hong Kong-based startup focusing on speech and language technologies.

Besides insufficient data, Cantonese's intricate linguistic characteristics are another obstacle in training an AI model.

The Economist magazine analyzed language learning time, and found that mastering Cantonese requires 88 weeks of study, placing it alongside Mandarin, Arabic, Japanese, and Korean in the top five most difficult languages to learn.

Lu Lewei, director of the Sensetime Research Institute, said that Cantonese is highly colloquial with numerous inflections. It has nine tones and even a slight variation in pronunciation can alter a word's meaning.

The language also features a blend of Chinese and English and a mix of old and modern terms.

In language modeling, the simplicity of a language offers advantages. The more complex the language is, the harder for the AI model to learn about it, Lam said.

Furthermore, underlying Cantonese is the local culture, which can be challenging for those tasked with aligning the output of large language models, Cao said.

Urgent need

Despite the difficulties involved in creating Cantonese AI models, demand for them is undeniable, said Fong from the Hong Kong Information Technology Federation.

The global Cantonese-speaking population is nearly 120 million, and 85.2 million of those are native Cantonese speakers.

In Hong Kong, 6.3 million residents, or 88.2 percent of the city's population, use Cantonese as their spoken language. In other cities within the Guangdong-Hong Kong-Macao Greater Bay Area, Cantonese is the predominant dialect, with 67 million residents in Guangdong province conversing in it.

In the future, AI will be akin to today's computers and fundamentally a tool for the general public. Without Cantonese AI tools, Cantonese-only speakers may encounter significant inconvenience and marginalization in both the offline and online world, Cao said.

For a city, lack of AI expertise could result in decreased productivity in sectors such as education, healthcare, finance, and law. These limitations could impede the whole city's development, Cao added.

Fong said AI models from other countries or regions may struggle to grasp Cantonese culture accurately. This could lead to cultural or political misinterpretations, resulting in the spreading of incorrect messages.

Dependence on outside AI models could make privacy and security vulnerable, Fong said.

Government officials, for instance, might face national security risks and local companies might leak data if they inadvertently disclose sensitive information to the models developed in foreign jurisdictions, he added.

Fong urged the Hong Kong Special Administrative Region government and local organizations to develop Cantonese LLMs.

In July, Sun Dong, Hong Kong's Secretary for Innovation, Technology, and Industry, announced that the SAR government is cooperating with local universities to develop a Hong Kong-based large language model.

A document co-pilot application for civil servants is now being used on a trial basis.

The model has already been implemented in Sun's department and the system will eventually become available to all Hong Kong residents, the secretary said.

The bureau said plans are underway to expand the pilot application to three other government bureaus, but it gave no indication when Hong Kong residents would gain access to it.

Fong said if it could be launched successfully, the government LLM would have many benefits.

It would be a positive step in resolving the issue of some Western AI models limiting their usage in Hong Kong. Also, implementing a localized AI model could safeguard privacy and provide more convenience to residents, Fong said.

Cao said it's unclear what specific features the government's AI model could offer and how it would distinguish itself from other similar products.

"I don't think the government has done enough research on what they want to do," Cao said.

Local startups

Local technology companies, meanwhile, are actively meeting the needs of the Cantonese-speaking market.

One startup, Votee AI, developed an opensource Cantonese LLM this year.

After years of operating in the local market, Votee AI has gathered substantial amounts of open-source Cantonese data along with primary data.

Taking a community-centered approach, they have also collaborated with local Cantonese linguists and AI researchers, including the team behind the online Cantonese dictionary "words.hk", to capture the nuances of Hong Kong speech.

Sensetime has also accumulated a vast reservoir of internal open-source data.

The company has synthesized data by leveraging advanced technologies and bought supplementary information from external channels to collect data.

To combat the shortage of high-quality Cantonese data, Sensetime also collected audio Cantonese data from hundreds of its local employees.

Sensechat's clients include customer service providers, financial institutions, legal firms, healthcare companies, and others.

For Hong Kong residents, the company promises to provide the service for free indefinitely for free on both the web version and mobile application.

A local tech industry insider, who chose to stay anonymous, said Sensechat should opensource its technology to allow more residents and organizations to access it freely, to benefit the city.

After trying the Sensechat platform, he said its understanding of some Hong Kong slang could be more precise. Nonetheless, "it should be recognized that Sensechat filled a void in the local market," he said.

Cultural roots

In addition to developing local AI models, existing mainstream language models should be encouraged to improve their Cantonese functions, said Li from the Hong Kong Wireless Technology Industry Association.

However, mainstream AI language models are primarily developed by commercial entities in the West. Without market demand, they may not be willing to enhance their products' Cantonese capabilities.

Li believes the Hong Kong SAR government and local organizations should take the lead in collecting Cantonese data, digitize cultural content, and share these resources openly to enrich the Cantonese body of information.

Cantonese speakers can also actively use the language to engage with mainstream AI language models.

These actions can demonstrate to AI model developers that there is a market demand for Cantonese, while interaction with these models can also enhance their understanding of Cantonese culture.

The key to encouraging more people to use Cantonese lies in making Cantonese culture appealing, Li said.

Language is not just a communication tool; it encapsulates the cultural essence and identity of its speakers, he said.

The marginalized status of Cantonese in the digital sphere is a reflection of the decline of the cultural significance of the region.

In the 1970s and 1980s, Hong Kong, although just a city, was so culturally influential that Cantonese was a popular language around the world, Li said.

"At that time, the whole world watched Hong Kong movies and TVB(television shows), knew Jackie Chan and Bruce Lee, and sang Cantonese songs. However, in the present day, even many students in Hong Kong cannot speak Cantonese," he said.

"The focus of government policies should not only be on technology, but also on culture."

He, the influencer, said he learned Cantonese from his grandparents when he was a child, which later made him more proficient in the language than other school students. The confidence this gave him motivated him to become a Cantonese blogger.

However, as He aged, Cantonese became so marginalized that even voice-operated devices and software in his home failed to understand Cantonese commands.

While He could communicate with these devices in Mandarin and English, his grandparents, who only speak Cantonese, struggled to keep pace.

He hopes that Cantonese LLMs will one day help his elderly grandparents manage their daily lives through voice-controlled apps capable of understanding Cantonese.