The Broken Ruler: Why "LLMs Now Translate African Languages" Doesn't Mean What You Think

Every few weeks a new headline promises that large language models have "cracked" another African language. A benchmark score ticks up, a press release goes out, and the story writes itself: the gap is closing, the machines are learning our languages, progress is inevitable.

I want to show you why that story is, at best, half-true — and why the other half is a measurement problem hiding in plain sight.

A new paper, Evaluating Large Language Models for Hausa and Fongbe Machine Translation: Benchmarks, Failures, and Metric Reliability (arXiv

.22269), tested four frontier LLMs on translating English into two West African languages. The headline result is easy to summarize. The uncomfortable result — the one nobody is putting on a slide — is this: for one of these languages, the automatic scores we use to declare progress disagreed with the humans who actually read the translations. The ruler we measure with is bent, and we've been reading numbers off it as if it were straight.

What the study actually found

The researchers evaluated four LLMs on English→Hausa and English→Fongbe translation, and they did something most benchmark papers skip: they brought in human evaluators and then asked whether the automatic metrics agreed with them.

Two things fell out.

First, the quality gap between the two languages is enormous. For Hausa, the models produced translations humans rated between roughly 3.37 and 4.46 out of 5 — usable, sometimes good. For Fongbe, the best system managed about 2.20 out of 5. That is not a rounding error. That is the difference between a tool you can deploy and a tool that will quietly mislead the people relying on it.

The paper is careful to say this gap reflects how much data each language has in training, not that Fongbe is somehow "harder" as a language. That distinction matters, and I'll come back to it.

Second — and this is the real story — the metrics themselves broke down. The automatic scores (the BLEU/chrF/COMET/BERTScore family that the whole field leans on) lined up reasonably well with human judgment for Fongbe, but for Hausa they misfired. Human evaluators preferred a system that none of the automatic metrics ranked first. Read that again: the humans and the machines disagreed about which translation was best, and in the world of benchmark leaderboards, the machines' answer is the one that gets published.

It gets worse when you look under the hood:

• BERTScore collapsed. For both languages, the embeddings the metric relies on were so similar to each other (within-language similarity above 0.99) that the metric could no longer tell a good translation from a bad one. It was returning confident numbers that meant essentially nothing.
• COMET behaved erratically for Fongbe, at one point ranking the worst system first — the exact opposite of its job.

The authors' recommendation is refreshingly blunt: use chrF++ as your primary automatic metric for these morphologically rich languages, and treat human evaluation as mandatory, not optional, before you make any quality claim.

The angle everyone else will miss

Here is what the celebratory coverage gets wrong. When a leaderboard shows an African language's score climbing, we instinctively read that as "the translations are getting better." But a score is only as trustworthy as the metric producing it — and this paper shows that for a language as widely spoken as Hausa, the standard metrics can rank the wrong system at the top.

So a rising number can mean three very different things:

1. The translations genuinely improved.
2. The model got better at the surface patterns the metric happens to reward, without getting better at meaning.
3. Nothing changed at all, and you're looking at metric noise.

From the outside, all three look identical. That is the trap. And it is a distinctly African trap, because these metrics were built, tuned, and validated overwhelmingly on high-resource languages. When we import them wholesale to Hausa or Fongbe, we're using a thermometer calibrated for one climate to declare the temperature in another.

This is why I keep telling practitioners: a benchmark score in a low-resource language is a hypothesis, not a result. Treat it that way.

Why Fongbe struggles — and why it isn't the language's fault

Fongbe (also called Fon) is spoken by roughly 3.5 million people, mostly in southern Benin, and it's an official language there. It's a tonal Gbe language with three tones and a diacritic-heavy writing system — and crucially, tone is often not fully marked in everyday text. A model trained on scraped web text is therefore learning from writing that is systematically missing some of the information a correct translation depends on.

None of that makes Fongbe intrinsically "hard." It makes Fongbe under-resourced and under-represented — a completely different problem with a completely different fix. The first framing invites shrugging ("some languages are just difficult"). The second points at the actual lever: build and clean the data. That reframing is the whole game, and it's the kind of thing the press-release version of this story erases.

The tax you're already paying

There's a second cost that compounds all of this, and it's worth naming because it hits African builders directly in the wallet. A related line of work — The African Language Tax (arXiv

.24460) — measured how many tokens frontier models spend to represent African-language text. Because tokenizers were optimized for English, the same sentence costs more tokens in an African language, which means higher inference cost, higher latency, and a smaller effective context window before you've done anything at all. In that study Hausa carried roughly a 1.35× penalty on one common tokenizer, and some African scripts ran far higher.

Put the two papers together and the picture is stark: for many African languages you pay more to run the model and you can't fully trust the score that tells you whether it's working. That is the real state of play — not the one in the launch tweet.

What to actually do about it (the practitioner's checklist)

If you're building or evaluating translation for an African language, here's how to stay honest:

1. Never ship on a single automatic metric. For morphologically complex languages, make chrF++ your default and treat BLEU as a weak secondary signal. If someone hands you a BLEU-only leaderboard, ask what the humans said.
2. Budget for human evaluation from day one. Even a small, well-designed human eval with native speakers will catch failures that every automatic metric misses. This paper is the receipt.
3. Be suspicious of neural metrics on low-resource languages. BERTScore's embedding collapse here is a warning: a metric can look sophisticated and still be measuring nothing. Sanity-check it against human ratings before you rely on it.
4. Pick models per language, not per benchmark. The study found no single winner — different systems led for Hausa versus Fongbe. Your "best model" is a function of which language you're serving, not a global ranking.
5. Fix the data, not the blame. If a language scores poorly, the productive question is "what representative, tone-marked, cleanly transcribed data is missing?" — not "is this language too hard?"

The bottom line

The good news in this paper is real: for Hausa, today's models produce translations that native speakers find usable. That's genuine progress and worth celebrating.

But the deeper contribution is the warning. We are entering a phase where African languages appear on more and more leaderboards, and the temptation to read those numbers as gospel is enormous. This work shows that the instruments themselves need auditing before the outputs can be trusted. For those of us building AI for the continent rather than about it, that's not a footnote. It's the assignment.

The headline says the machines are learning our languages. The paper says: check the ruler first.

---

Primary source: Evaluating Large Language Models for Hausa and Fongbe Machine Translation: Benchmarks, Failures, and Metric Reliability (arXiv

.22269). Supporting context: The African Language Tax (arXiv

.24460); Fon language — Wikipedia.