I Ran 500 Translation Pairs Through Kimi K3, DeepL, and GPT-5.6 Sol — Reviewers Scored Them Blind

The Test

I've been a translator-adjacent person my whole career — localization PM at two startups — and I've watched every model launch claim "near-human translation." The claim is always vaguely true and specifically misleading, so I built a test I could defend. 500 source segments, four registers, 125 pairs each: legal clauses from a real software license, marketing copy from a DTC brand, code comments and error strings from a codebase I maintain, and a grab bag of idioms and cultural references.

Three engines: Kimi K3, DeepL Pro (the professional tier), and GPT-5.6 Sol. Both directions: Chinese to English and English to Chinese. Then I stripped all identifying marks, shuffled the outputs, and handed them to two bilingual reviewers — one native Chinese speaker working as an editor in London, one native English speaker who ran localization at a Shenzhen hardware company. They scored each output 1-5 on accuracy (meaning preserved), fluency (reads like native writing), and tone (register preserved). Neither knew which engine produced which line. Inter-rater disagreement: 9% of segments needed a tie-break.

The Headline Numbers

Composite scores across all registers and both directions (out of 5):

The averages hide the story: those wins are not evenly distributed. In legal and code comments all three engines were within half a point — genuinely hard to separate. The separation lives in marketing copy and idioms, where K3 and DeepL traded blows by direction, and Sol quietly fell out of contention. If you only translate technical documentation, the cheapest engine is already good enough. The war is over tone.

Where K3 Wins

Three patterns kept showing up in K3's winning segments. Chinese to English marketing: a launch line that literally means "small and beautiful, bright and brilliant" came out as "small but brilliant" from K3 — DeepL gave "small and exquisite, bright and dazzling," which is accurate and unusable on a billboard. The reviewers called K3's instinct for dropping redundant four-character parallelism "the single hardest skill in Zh-En advertising copy."

Implied subjects. Chinese drops pronouns; English sentences cannot. In 30-ish segments where the subject was ambiguous in the source, K3 inferred the right referent from context 27 times. DeepL guessed wrong or produced passive constructions that changed the actor 8 times. My London reviewer, who has edited both machine and human translations for a decade, said K3's choices read "like a translator who understood what the meeting was actually about."

Register matching in code comments. Weirdly specific but real: terse Chinese dev comments ("这里先这么写，以后再说") came out as terse English ("TODO: revisit this") rather than DeepL's oddly formal "This shall be implemented in this manner for the time being." Tone, not accuracy — but tone is why humans get paid.

Where DeepL Still Wins

Fluency scores tell the truth: DeepL's English output is smoother, especially in long sentences. Where K3 sometimes translates "almost right" — a collocation that's grammatical but 5% off, like "adopt a policy" where a native would say "set a policy" — DeepL lands native phrases more consistently. In legal text, DeepL's clause structures also mirrored the source's formality more predictably; for contracts, mirroring is a feature, not a bug. And in the idioms round, DeepL won the "known idiom" segments (成语 it has clearly seen a million times) while K3 won the "recent slang" segments DeepL simply misread.

The Idiom Problem Nobody Solves

The nightmare category for all three: wordplay with no equivalent. A slogan built on 谐音 (homophone) humor produced three different flavors of failure — K3 explained the joke in a footnote (reviewers: "correct, but a footnote is not a translation"), DeepL produced a fluent sentence with no joke in it, and Sol invented an English pun that was "grammatically brave and semantically unrelated," per the Shenzhen reviewer. The honest conclusion after 500 segments: for creative copy, the correct pipeline in 2026 is machine first-pass, human final-line, and the human is not optional. But the machine first-pass that saves the most human minutes is, on this evidence, Kimi K3 for Chinese and DeepL for European pairs.

Full cost of the experiment: under $9 across both directions and all engines — K3's share was $2.40. If you want the broader picture of where K3 earns its keep, my multimodal test and the academic writing test cover the neighboring registers.