CantoneseLLM v2: Reasoning in a Low-Resource Language
CantoneseLLM v2 combines CPT, DPO, and RLVR, reaching 73.16 on HKCanto-Eval with Cantonese Traditional-Chinese reasoning.
Key Findings
Methodology
The study adapts Qwen3 8B and 30B-A3B through continuous pre-training (CPT), Chat Vector merging, supervised fine-tuning (SFT), direct preference optimisation (DPO), and reinforcement learning with verifiable rewards (RLVR). CPT uses 784M Cantonese and Hong Kong-related tokens. Chat Vector transfer uses Δchat=θinstruct−θbase and adds it to CPT weights. RLVR combines task rewards with a hard format gate and multiplicative Cantonese-language and Traditional-Chinese-script constraints.
Key Results
- The 30B-A3B CPT checkpoint reaches 68.47 average on HKCanto-Eval, while Chat Vector merging raises it to 74.36. However, the merged model largely inherits the donor checkpoint’s reasoning language rather than reliably producing idiomatic Cantonese.
- SFT causes a major regression: average scores fall by 20.52 points for 8B and 12.46 points for 30B-A3B. The 30B reasoning span shrinks to roughly one tenth of its former length, while 8B emits empty reasoning blocks on 64.5% of 60,850 sampled prompts.
- RLVR restores capability and language alignment. The final 30B-A3B scores 73.16 on HKCanto-Eval—within 1.20 points of its merged checkpoint—while retaining Cantonese reasoning in Traditional Chinese.
Significance
The paper shows that low-resource status reflects missing written data, native reasoning traces, and target-language teachers—not a small speaker population. It demonstrates why translating high-resource training data or applying a standard recipe can damage reasoning format and general capability. For Hong Kong, the work supports locally deployable, auditable models for retrieval, document processing, education, and public services, while making regional data and training infrastructure more accessible.
Technical Contribution
The main contribution is a diagnosis-driven pipeline rather than a fixed stack of algorithms. CPT installs local vocabulary; Chat Vector merging transfers instruction following at zero additional training cost; DPO repairs reasoning-block structure; and RLVR combines verifiable task rewards with language and script constraints. The release of checkpoints, Nemo-Gym environments, and a thirteen-year Traditional Chinese Common Crawl corpus also creates a reproducible engineering substrate for low-resource alignment research.
Novelty
This is a systematic attempt to build a reasoning model for a strongly spoken, low-resource language. Unlike recipes relying on tens or hundreds of billions of tokens, native teachers, and abundant chain-of-thought data, it treats language identity and orthography as explicit RLVR constraints. Its controlled stage analysis is especially novel in showing that SFT can erase reasoning traces instead of improving target-language reasoning.
Limitations
- The corpus contains only 784M tokens, with Cantonese Common Crawl contributing 5.4%; much material is informal web content, limiting coverage of formal registers, professional terminology, and social pragmatics.
- HKCanto-Eval is primarily multiple-choice, so it may miss open-generation failures involving fluency, Cantonese lexical choice, script fidelity, reasoning quality, and termination.
Future Work
Future work should collect more native Cantonese reasoning traces, build open-generation evaluations for education, law, and medicine, and develop stronger language-aware verifiers. The authors also point toward using v2 to generate richer long-context data for the next iteration.
AI Executive Summary
Cantonese is spoken by roughly 85 million people, yet its written resources remain scarce. Formal Chinese in Hong Kong is often Mandarin-oriented, written Cantonese has been stigmatized, and available material is concentrated in informal web and social-media domains. As a result, multilingual models may understand Cantonese while reasoning in Mandarin, English, or Simplified Chinese. Directly translating Mandarin resources does not solve the problem: vocabulary, grammar, pragmatics, and cultural norms differ.
CantoneseLLM v2 adapts Qwen3 8B and 30B-A3B through five stages. Continuous pre-training on 784M Cantonese and Hong Kong-related tokens installs local knowledge. Chat Vector arithmetic transfers instruction following without additional training. SFT adds translation, curation, and reasoning data, while DPO repairs the reasoning-block format. Finally, RLVR combines verifiable task rewards with a hard output gate and multiplicative Cantonese-language and Traditional-Chinese-script rewards. The stage-by-stage analysis is the central scientific story: Chat Vector merging transfers behaviour but preserves the donor’s reasoning language; limited SFT can erase reasoning instead of teaching it.
The final 30B-A3B model scores 73.16 on HKCanto-Eval, only 1.20 points below its merged checkpoint, while producing Cantonese reasoning absent from that checkpoint. SFT had previously reduced scores by 20.52 points for 8B and 12.46 for 30B-A3B; the 8B model generated empty reasoning blocks on 64.5% of sampled prompts. The work therefore argues that low-resource adaptation must jointly optimize correctness, language, script, format, and termination. Public checkpoints, environments, and a thirteen-year corpus offer a practical foundation for sovereign and locally deployable language technology.
Deep Analysis
Background
Recent efforts such as Swallow, SEA-LION, and Taiwan-LLM adapt open models using 35.1B–200B tokens. Cantonese has a large speech community but far less suitable writing, and it is not interchangeable with Mandarin despite shared characters. The practical need is therefore a moderate-sized model that can run locally and behave naturally in Cantonese, not merely a multilingual model that performs well on multiple-choice questions.
Core Problem
The paper addresses four linked bottlenecks: injecting local knowledge from only 784M tokens; transferring instruction following without a Cantonese teacher; preventing scarce SFT data from deleting reasoning; and enforcing Cantonese, Traditional Chinese, correct formatting, and task success simultaneously. Training loss is an unreliable proxy under low-resource overfitting, while multiple-choice benchmarks can conceal poor generation, script errors, and non-Cantonese reasoning.
Innovation
First, the authors curate a 784M-token corpus and release thirteen years of Traditional Chinese Common Crawl data. Second, Chat Vector arithmetic transfers alignment without instruction data. Third, DPO targets reasoning-block structure rather than simply adding more examples. Fourth, RLVR combines environment-specific correctness rewards, a hard format gate, and a multiplicative language-and-script factor. This makes Cantonese and orthographic fidelity operational conditions for success rather than optional stylistic preferences.
Methodology
- �� CPT: Qwen3 8B and 30B-A3B are trained on 64 TPU v6e chips; the selected learning rates are 3.0×10^-5 for 8B and 1.5×10^-5 for 30B-A3B.
- �� Chat Vector: compute Δchat=θinstruct−θbase and set θcv=θcpt+Δchat, transferring instruction following without new training.
- �� SFT: train on 74,865 rows and 177.4M tokens, with 52.2% reasoning tokens across Cantonese, translated reasoning, and replay data.
- �� DPO: preference optimisation restores the desired think–answer structure.
- �� RLVR: Nemo-Gym environments supply verifiable rewards, multiplied by Cantonese and Traditional-Chinese constraints and gated by valid output format.
Experiments
Evaluation uses HKCanto-Eval, MMLU, CantoMMLU, and controlled probes measuring reasoning-block presence, length, language, script, instruction compliance, and termination. CPT includes step and learning-rate sweeps; the 8B 530-step run is selected. The study compares official Qwen3, Chat Vector, SFT, DPO, and GRPO checkpoints. Machine-translated GSM8K probes expose lexical and script errors that multiple-choice scoring cannot detect.
Results
The 8B CPT sweep averages 65.07, 65.12, and 64.85 across selected step counts. For 30B-A3B, the best CPT score is 68.47; a higher learning rate lowers loss but yields only 67.65, demonstrating domain overfitting. Chat Vector scores are 69.51 for 8B and 74.36 for 30B-A3B. SFT causes drops of 20.52 and 12.46 points. RLVR returns the 30B model to 73.16 while restoring target-language reasoning.
Applications
The models can support Cantonese document processing, retrieval-augmented question answering, written-Chinese–Cantonese conversion, tutoring, and local customer service. Public weights and environments enable regional deployment and auditing. Production use still requires private knowledge bases, terminology controls, privacy filtering, human review, and independent validation for legal or medical outputs.
Limitations & Outlook
The dataset remains small and has a low proportion of native Cantonese, with substantial informal web material and machine-translated reasoning. RLVR is expensive: reported RL training costs are 441 GPU-hours for 8B and 1,256 for 30B-A3B, excluding earlier stages. HKCanto-Eval is mainly multiple-choice. Future work should expand native traces, improve verifiers, cover formal domains, and evaluate long-form multilingual interaction.
Plain Language Accessible to non-experts
Imagine hiring a new employee for a Hong Kong office. First, you give the employee 784 million words from local websites, novels, news, encyclopaedias, and other material. This teaches local facts and Cantonese vocabulary. Next, you copy the working habits of an employee who already follows instructions well. That is efficient, but the new employee may still think in Mandarin or use Simplified Chinese.
You then provide a small set of Cantonese examples. Surprisingly, the employee may learn to hand in short answers or leave the thinking section blank. This is what happened during SFT: the model saw too many examples without a reasoning trace. DPO acts like a supervisor selecting better work samples and restoring the expected thinking-and-answer format.
Finally, RLVR behaves like an automated quality-control system. A response receives a good score only when the task is solved, the reasoning is in Cantonese, and the writing uses Traditional Chinese. The final 30B model reaches 73.16 on HKCanto-Eval and nearly matches its earlier merged version, but it now reasons in Cantonese. The lesson is that teaching an AI a language requires more than vocabulary: it must learn the language, writing system, format, and behaviour together.
ELI14 Explained like you're 14
Picture a brilliant gaming teammate who keeps explaining strategies in Mandarin or English, even though your squad uses Cantonese. You want natural Cantonese, Traditional Chinese characters, and explanations that actually show the strategy. The researchers start by feeding the teammate 784 million words about Hong Kong, Cantonese, websites, stories, and encyclopaedias. That gives it local knowledge. Then they copy in the “listen to instructions” skills from another model.
But a weird bug appears. After a small Cantonese training set, the teammate sometimes skips the strategy and gives only the final move. The 8B model even produced an empty thinking box on 64.5% of tested prompts. So the team uses DPO, like showing the player which replay is better, to restore the proper format.
Then comes RLVR, basically a scoreboard. You get points for solving the problem, using Cantonese, and writing Traditional Chinese. Miss one requirement and the reward shrinks. The final 30B-A3B model scores 73.16 on HKCanto-Eval, only 1.20 points below its merged checkpoint, but it can finally reason in Cantonese. Pretty cool, right? The big lesson is that an AI does not automatically “think in” a language just because it can recognize that language. Training must reward the exact language, script, format, and answer quality you want.
Glossary
Continuous Pre-training (CPT)
Further language-model training on domain or language-specific text. Technically, it updates an existing checkpoint’s representations and probabilities rather than training from scratch.
Used to inject Cantonese vocabulary and Hong Kong knowledge from 784M tokens.
Chat Vector
A parameter difference between an instruction-tuned model and its base model. Adding this vector attempts to transfer instruction following without supervised training.
The paper uses Δchat=θinstruct−θbase and adds it to CPT checkpoints.
Supervised Fine-Tuning (SFT)
Training on input–target examples so the model imitates specified outputs. Mixed formats can accidentally teach empty or shortened reasoning.
SFT caused major benchmark regression and reasoning deletion.
Direct Preference Optimisation (DPO)
A preference-learning method that increases the probability of preferred responses relative to rejected ones without separately training a reward model.
Used to repair reasoning-block formatting, especially for 8B.
Reinforcement Learning with Verifiable Rewards (RLVR)
Reinforcement learning driven by automatically checkable task outcomes. Here, rewards are additionally constrained by output format, language, and script.
The final stage restores capability and Cantonese reasoning.
HKCanto-Eval
A Cantonese and Hong Kong-focused benchmark used to compare model checkpoints. Its multiple-choice structure measures knowledge and instruction performance but incompletely tests open generation.
The final 30B-A3B score is 73.16.
Open Questions Unanswered questions from this research
- 1 How to obtain large quantities of native Cantonese reasoning traces remains unresolved. The field needs sustained data collection, privacy protection, expert annotation, and methods that distinguish authentic Cantonese from mechanically translated Mandarin.
- 2 HKCanto-Eval cannot fully measure natural generation, social pragmatics, and script fidelity. Community benchmarks should include open-ended, multi-turn, professional, and safety-sensitive tasks.
Applications
Immediate Applications
Hong Kong knowledge assistant
Public agencies, schools, and companies can deploy the 30B model for Cantonese question answering, document summarisation, and retrieval. A private knowledge base, privacy controls, terminology lists, and human review are required for sensitive domains.
Cantonese–written Chinese conversion
Editors, educators, and service teams can rewrite colloquial Cantonese into formal Traditional Chinese or reverse the direction while preserving local wording. Deployment should test politeness, regional usage, named entities, and domain terminology.
Long-term Vision
Regional sovereign language infrastructure
Open checkpoints, data, and verifiable environments could support an auditable Hong Kong language-model ecosystem that runs locally. The main obstacles are native high-quality data, sustained compute, evaluation governance, and safety assurance.
Abstract
Cantonese is widely spoken but remains low-resource in written data, with no large corpus of native Cantonese reasoning traces available for model training. We develop and release CantoneseLLM v2, comprising models based on Qwen3 8B and 30B-A3B. The models are trained through CPT on 784 million Cantonese and Hong Kong-related tokens, chat-vector merging, SFT, DPO, and RLVR. Evaluation across the training stages shows that chat-vector merging transfers instruction following but preserves the donor model's reasoning language, while SFT with limited Cantonese reasoning data substantially shortens or removes reasoning traces and reduces benchmark performance. DPO restores the reasoning-block format, particularly for the 8B model, but recovers only part of the lost performance. The RLVR training with Cantonese language and Traditional Chinese scripts as multiplicative constraints introduced Cantonese language alignment and restored the lost performance. The 30B-A3B model reaches 73.16 on HKCanto-Eval, within 1.20 points of its merged checkpoint, while retaining the Cantonese reasoning behaviour absent from that checkpoint. We release the model checkpoints, the training environments, and a thirteen-year Traditional Chinese Common Crawl dataset. The models can be accessed at https://huggingface.co/collections/hon9kon9ize/cantonesellm-v20