100% Refusal, 0.4% Good: Safety Alignment in Hausa Is Measuring the Wrong Thing

The metric says the model is safe

Hausa is spoken by more than 200 million people across West Africa, mostly in northern Nigeria, Niger, and Ghana. It is one of the largest languages on earth, and safety alignment research has barely touched it. The standard pipeline, supervised fine-tuning followed by preference optimization, was designed and validated in English. A team from AIMS Cameroon, Bayero University Kano, the University of Hamburg, and Imperial College London decided to find out what that pipeline actually does when the target language is Hausa.

They built HausaSafeLM, one of the first Hausa safety preference datasets: 10,000 prompts derived from PKU-SafeRLHF, an English safety dataset of about 70,000 prompts. The prompts split into two tiers. The gold tier, 3,000 examples, was machine-translated and then human post-edited through a four-pass protocol covering fluency, meaning, register, and localization. The silver tier, 7,000 examples, was machine-translated only. Fifteen of the original 19 harm categories were kept and grouped into six macro-categories: violence and physical harm, hate speech and discrimination, sexual content, criminal activity, health and public safety, and psychological manipulation. Four categories were dropped as not salient in the Hausa context: copyright issues, privacy violation, white-collar crime, and animal abuse. Each example also carries a sensitivity level: low 40%, medium 40%, high 20%.

The base model is Qwen2.5-7B-Instruct, fine-tuned with LoRA on two NVIDIA A100 80GB GPUs. Five configurations were evaluated on 500 held-out harmful Hausa prompts and 200 benign Hausa prompts from the Aya dataset: the zero-shot baseline, SFT on gold only (SFT-G), SFT on gold plus silver (SFT-GS), and DPO on top of each SFT checkpoint (DPO-G, DPO-GS). A Gemini judge scored every response on three axes: safety (SAFE or UNSAFE), helpfulness on benign prompts, and linguistic quality (HIGH, MEDIUM, or LOW). Five metrics were reported: refusal rate (RR), false refusal rate (FRR), helpfulness rate (HR), refusal quality (SafeQ), and benign-response quality (AyaQ).

The headline result is a warning about the headline metric. Refusal rate saturates at 1.000 for every configuration, including the untuned baseline. The paper keeps it in the results table only to make the ceiling effect explicit. Everything interesting happens in the other four columns.

Finding 1: the untuned model refuses everything, and almost none of it counts

On 500 harmful Hausa prompts, the base model refuses all of them. RR equals 1.000. If your safety dashboard reports refusal rate, this model scores a perfect 100% before any alignment work has been done.

The quality scores say otherwise. The judge rates 96.2% of the base model's refusals low quality and only 0.4% high quality, for a SafeQ of 0.021 on a 0 to 1 scale. Manual inspection shows why: the model frequently fails to engage with the Hausa prompt at all rather than producing a coherent refusal. On benign prompts it is no better, unhelpful 74.4% of the time (HR 0.256) with similarly low-quality responses (AyaQ 0.165).

The appendix examples make the failure concrete. Asked in Hausa how to control someone's behavior without consent, the base model repeats prompt keywords and lists numbered fragments with no refusal. Asked a benign factual question, it answers entirely in Chinese, the dominant language for short Q and A in its pretraining data.

This is the paper's central point, and it generalizes beyond Hausa. A model that cannot reliably produce content in the target language will trivially avoid producing harmful content in it. Refusal rate alone cannot distinguish a genuinely aligned model from one that is simply too weak in the language to comply with the harmful request. The authors put it plainly: the base model's apparent safety is largely an artifact of weak Hausa generation, not evidence of alignment.

Finding 2: SFT teaches how to refuse, DPO teaches when, and each undoes the other

SFT changes the shape of the refusals. SFT-G lifts the share of refusals rated high or medium quality from near zero to 32%, and SFT-GS reaches 45%. SafeQ rises from 0.021 at baseline to 0.238 (SFT-G) and 0.350 (SFT-GS). The model starts refusing in fluent Hausa, acknowledging the request and declining it in natural language.

SFT on gold data alone has a cost. SFT-G wrongly refuses 40.5% of benign prompts (FRR 0.405) and is helpful only 18% of the time (HR 0.180). It learns the refusal behavior and then applies it too widely, including to a benign culturally specific question about the Emir of Kano, which it refuses outright.

DPO fixes the over-refusal. Both DPO-G and DPO-GS bring the false refusal rate down to 0.110, close to the base model's 0.096, while holding refusal rate at 1.000. That is what DPO is designed to do: its objective compares a preferred refusal against a dispreferred compliance, widening the gap between the two responses.

The price is paid in linguistic quality. SafeQ collapses to 0.023 for DPO-G and 0.036 for DPO-GS, with 94 to 95% of refusals rated low quality, essentially the same as the untuned base model. The paper's explanation is structural. DPO optimizes the relative preference of one response over another, not the absolute quality of the preferred response, so nothing in training rewards fluent Hausa. The appendix shows the damage: DPO variants answer a benign question entirely in Chinese, and on a harmful prompt they produce incoherent repetition about a store instead of any refusal at all.

The best single configuration is SFT-GS, with the highest SafeQ (0.350) and helpfulness (HR 0.260) of any variant and a false refusal rate of 0.245, well below SFT-G's 0.405 though still above the DPO variants' 0.110. Even SFT-GS is not clean. In one appendix example it refuses a request to steal candles and then inadvertently suggests workarounds for getting them.

Variant	RR	FRR	HR	SafeQ	AyaQ
Base	1.000	0.096	0.256	0.021	0.165
SFT-G	1.000	0.405	0.180	0.238	0.203
DPO-G	1.000	0.110	0.255	0.023	0.143
SFT-GS	1.000	0.245	0.260	0.350	0.268
DPO-GS	1.000	0.110	0.235	0.036	0.133

Finding 3: noisy data helps SFT a lot, and DPO almost not at all

The gold-versus-silver comparison is the paper's most useful result for anyone building low-resource datasets on a budget. Adding the 7,000 noisier silver examples to SFT is a large win: false refusals fall by 0.160 (0.405 to 0.245), helpfulness rises by 0.080, and refusal quality rises by 0.112. SFT learns directly from the chosen response, so the extra lexical and stylistic diversity in the silver Hausa pairs is useful signal even with translation noise in it.

Adding the same silver data to DPO does almost nothing. False refusal rate stays at 0.110 in both regimes, helpfulness changes by minus 0.020, and SafeQ rises by only 0.013. The reason is in the structure of the silver preference pairs: they mostly pair two safe responses of different quality, not a safe response against an unsafe one. The safe-versus-unsafe boundary was already established in the gold tier, so the extra pairs carry little contrastive signal for a preference objective.

The quality-versus-size trade-off is phase-specific. More data, even noisy data, is worth collecting for the imitation stage. For the preference stage, more of the same kind of pair is close to worthless. That distinction matters because the expensive part of this dataset was the human post-editing on the 3,000 gold examples. The result says: spend the human budget on the gold tier that sets the boundary, and let machine translation bulk up the SFT stage.

Finding 4: no configuration wins all five

Plot all five metrics on a radar chart and no polygon fills the space. The DPO configurations stretch furthest on refusal precision but collapse on both quality axes. SFT-GS covers the most total area, with reasonable precision and the highest quality of any configuration. The baseline sits in the middle on everything except refusal rate, where it ties for first.

The authors state the conclusion without hedging: no configuration they tested is safe, helpful, and fluent in Hausa. The alignment tax is not a single cost but a trade-off surface. DPO trades fluency for accuracy, SFT trades refusal accuracy for fluency and helpfulness, and the current recipes force a choice between discriminability and fluency in the target language.

What this means for data and AI practitioners

1. Stop leading safety reports with refusal rate in low-resource languages. It saturates immediately, including for a model that cannot write a coherent Hausa sentence. Report refusal quality, false refusal rate, and helpfulness alongside it, or the number is worse than useless: it is reassuring and wrong.
2. Budget your alignment stages differently. If you are aligning in a language like Hausa, SFT is where fluency is built and DPO is where precision is sharpened. Running DPO on top of a weak SFT checkpoint, or without enough SFT data, buys precision in a language the model can no longer speak.
3. Spend human annotation where it sets the boundary. The gold tier's value was the safe-versus-unsafe contrast, established by human post-editing. Bulk machine-translated data helped SFT but added nothing to DPO. Match the data investment to the stage that can use it.
4. Test benign prompts in the target language, not just harmful ones. The most telling failures in this study were on benign prompts: over-refusal of a question about the Emir of Kano, and answers that switch entirely to Chinese. A safety eval with no benign set cannot see either failure mode.
5. Plan for natively written data, not translated data. The authors' own recommendation is that future work needs safety datasets written natively in African languages, evaluated by native speakers, and alignment objectives that penalize linguistic degradation while optimizing preferences. Translation from English carries the harm taxonomy of the source language with it. Four PKU-SafeRLHF categories were dropped as culturally irrelevant, which is a reminder that the ones kept were still chosen in English first.

In favour

The design isolates exactly the question it claims to answer. Gold and silver tiers are translated from the same filtered pool, so the quality-versus-size comparison is clean. The five metrics are chosen to expose the ceiling effect rather than hide it, and the paper reports the base model's perfect refusal rate as a finding about the metric, not as a result. The appendix examples let a reader check the judge's verdicts against real model outputs, including the embarrassing ones, and the authors disclose the provisional status of the AyaQ values rather than burying it. The result also fits a consistent prior literature: Yong et al. (2023) showed translation-based jailbreaks bypassing GPT-4 safeguards in low-resource languages, and Shen et al. (2024) documented the "harmfulness curse" of higher toxicity and lower refusal quality outside high-resource languages.

Against

One 7B model from one family is a thin base for a general claim about alignment methods, and larger models may retain fluency through DPO where this one did not. The single Gemini judge is a real weakness for a paper whose core metric is linguistic quality in Hausa: the judge's own Hausa competence bounds every quality number reported. The evaluation set is small, and the authors themselves warn that small differences between configurations need significance testing they did not run. And the dataset inherits its harm taxonomy from an English source, which means the study measures how well translated English safety transfers, not how well Hausa safety is understood on its own terms. The AfriGuard and UbuntuGuard work cited in the paper is already moving toward culturally grounded benchmarks, which may make this translated-data approach look like a stopgap within a year.

What would make me wrong

• A larger or different-family model that holds SafeQ above 0.3 after DPO, which would show the fluency collapse is a property of this 7B model rather than of preference optimization in low-resource languages.
• Native-speaker re-judgment of the same responses that disagrees with the Gemini quality scores, especially if it finds the DPO refusals more fluent than the judge rated them.
• A DPO variant trained on preference pairs with a genuine safe-versus-unsafe contrast in the silver tier, if that erases the gap between DPO-G and DPO-GS and lifts SafeQ.
• A natively written Hausa safety dataset that changes the ranking of configurations, which would suggest the translated-data pipeline, rather than the alignment methods, is driving the trade-off.

Related on Everyday Data Science

• The Press Releases Promise African-Language AI. The Data Pipeline Is the Real Product.
• Africa's AI Fortnight Was About Small Models and Speech Data. That Is the Point.
• Sovereign AI Got Hardware This Week: Two Deals, Two Very Different Tests
• Three Plans to Share Africa's AI Computing Power Landed in Six Days. The Hard Numbers Are Still Missing.

If your model refuses every harmful prompt in a language it barely speaks, is it safe, or is it just quiet?

Sources: Aligning Large Language Models for Safety in African Languages: A Case Study of Hausa (arXiv:2610.37245) · full text (HTML) · Low-resource languages jailbreak GPT-4 (arXiv:2310.02446) · AfriGuard: safety and robust refusal for African languages (arXiv:2610.26474) · UbuntuGuard (arXiv:2601.12696) · Aya dataset (arXiv:2402.06619) · Direct Preference Optimization (arXiv:2305.18290)