We built a measurement harness for a problem dozens of hospitals are currently trying to solve: automatically suggesting ICD-10-GM codes from unstructured physician notes, fully on the hospital's own infrastructure, with no external API calls.
We didn't start with a model. We started with a question that is usually asked last: what is the upper bound of accuracy this architecture can deliver at all — and got three results that changed the order of work.
- The system's ceiling is set by search over the catalogue, not by the language model. We measured it: 0.854. No model can exceed it — that follows from the shape of the pipeline, not the weights.
- The most natural implementation — "turn the note into a vector and find the nearest code" — has a ceiling of 3.9%. It doesn't fail because the model is weak. It fails because the task is set up wrong.
- The standard way of parsing the official BfArM catalogue silently loses 27% of the code space. No error, no warning — some codes simply never enter the index.
All three were obtained on CPU, before we rented a single GPU hour. Below is exactly how, and what follows for the hospital.
Why this is worth money in the first place
A German inpatient hospital does not bill line by line. It bills per case, mapped to a G-DRG group. A coder picks the primary diagnosis, secondary diagnoses, and OPS procedures; a certified grouper turns that set together with age, sex, length of stay, and ventilation hours into a payment group. The group's weight is multiplied by the regional base rate — that's the amount.
From this, two economic facts drive the system's requirements:
One additional secondary diagnosis changes the bill by hundreds or thousands of euros. Coding completeness is not tidiness — it's revenue.
Overcoding is penalised. On behalf of the sickness fund, the Medizinischer Dienst reviews cases in samples: an expert requests the documentation and checks whether the billed code is supported by what's written. If it isn't, the case is downcoded, money is returned, and a penalty is added on top.
The asymmetry here is the point, and it dictates the architecture. Missing specificity loses a little. Adding a code that isn't supported returns money and adds a fine. A system optimised for "hitting the right code" and a system optimised for "hitting the right code that survives an audit" are two different products.
And a third factor: the data does not leave the hospital. Not to save money — because of regulation. Physician notes are health data in the sense of Art. 9 GDPR. In German healthcare, no cloud-provider contractual guarantee usually closes that conversation. So inference lives inside the hospital's perimeter, on fixed hardware — and this changes the engineering more than people assume: capacity cannot be added, it is planned up front.
One architectural decision from which everything else follows
The model does not generate a code. The model picks a code from a list.
ICD-10-GM 2026 is a closed catalogue of 16,905 codes, re-published every year by BfArM. Free generation gives you two failure modes that no amount of prompt phrasing eliminates: codes that don't exist, and codes that exist but are inadmissible in the role they were placed in. Both are invisible in average accuracy and expensive at audit time.
So the task is reframed so that neither is representable:
| Stage | Executed by | Why this way |
|---|---|---|
| 1. Extraction | LLM | only a language model turns clinical prose into a normalised fact with a pointer to the source |
| 2. Candidate search | no model | search over a closed set — this is what "search" is; and this is where the system's ceiling sits |
| 3. Selection | LLM | the code field is a JSON-enum over exactly the found candidates, forced at decoding time |
| 4. Validation | no model | the catalogue asserts these as facts; a field comparison is more reliable and a thousand times cheaper than asking a model |
| 5. Review | human | Art. 22 GDPR, and money depends on the code |
The difference between "we asked the model not to make things up" and "there are no symbols in its output alphabet with which to make things up" is the difference between hope and a guarantee. The first gets discussed on a call; the second gets shown at an audit.
A direct consequence for the order of work: if stage 3 can only choose from what stage 2 found, then search recall is a hard ceiling on the accuracy of the whole system. It has to be measured first. It is measured without a GPU.
What turned out to be in the BfArM data
BfArM publishes all the reference material for free and machine-readably. Terms of use require the classifier to remain unmodified and the source to be attributed — for a product, that means correct attribution with the version, and that gets solved explicitly, not by default.
ClaML loses 4,571 codes
The ClaML XML format is the first thing you reach for: structural, official, made for this. It contains 12,606 classes. The release contains 16,905 codes.
The difference is that five-digit codes are defined via Modifier / ModifierClass templates, not as separate elements. The parser doesn't see them and doesn't report it: no exception, no warning. The index simply doesn't contain part of the code space — mostly trauma codes, i.e. exactly what surgical protocols are full of.
The correct skeleton of the catalogue is the kodes.txt file from the metadata package, where BfArM has expanded these codes themselves, officially — not as our interpretation of templates. ClaML remains the source for what isn't in the CSV: inclusions, exclusions, coding notes.
The catalogue already contains an official billability flag
Fields §295 and §301 SGB V state whether a code is admissible as a primary diagnosis in outpatient and inpatient settings, respectively. Of 16,905 codes, 14,370 are terminal, and 13,041 are eligible to carry an inpatient primary diagnosis.
That's the target set for search. And this is an argument of a different class: not "we decided this code is terminal," but "BfArM asserts that this code is used to code under §301." The first gets discussed at audit; the second doesn't.
The rules layer comes for free
Sex constraints (983 codes female-only, 161 male-only), age boundaries, mandatory-notification flag under IfSG, "rare in Central Europe" flag. 2,316 billable codes have a sex or age restriction.
Asking a 27-billion-parameter model whether a spontaneous-childbirth code is plausible for a 34-year-old man is slower, more expensive, and less reliable than a single field comparison against the catalogue. Plus, every objection traces back to a specific field of the official reference — that's what you present to the expert.
89,304 labelled pairs, curated by the regulator
Alpha-ID-SE maps living wording to a code: "blutendes Magengeschwür → K25.0". 89,304 valid entries, curated by BfArM.
Normally this kind of labelling is commissioned or done by hand over months. Here it is official, free, and in a ready-to-use format — simultaneously today's index for search and tomorrow's fine-tuning dataset.
Coverage: 77.6% of billable codes have at least one synonym. The structure of the gap matters more than the number — the 2,914 codes without synonyms are almost entirely residual categories of the Sonstige and nicht näher bezeichnet variety. Trash bins don't have spoken names. But that's exactly what coders reach for when the documentation is incomplete — meaning that in the most common hard case, search has only the bureaucratic title to lean on. That's a structural limitation of any retrieval-first design, and it's resolved with rules and explicit refusal, not by improving search.
Small traps that would have cost integration time
Separators are not unified across files in the same release — and parsing with the wrong separator gives an empty synonym table, not an error, and quietly worse search. Encodings are historically mixed. Alpha-ID is cumulative: it contains entries retired in previous rounds, with a validity flag; indexing them means coding with terms the regulator has withdrawn.
What the retrieval measurements showed
The evaluation set is built by leave-one-synonym-out: for codes with multiple terms, one term is held out as the query and is simultaneously removed from the indexed text. Without that removal, the measurement measures string retrieval — our first run without it gave recall@20 = 0.994, which was a sign of a broken measurement, not a solved task. After stratifying across the 21 chapters and hard leakage checks:
| Retriever | recall@1 | recall@20 | cat3@20 |
|---|---|---|---|
| BM25 | 0.412 | 0.710 | 0.798 |
| Dense (multilingual-e5-base) | 0.518 | 0.838 | 0.904 |
| Hybrid (RRF) | 0.478 | 0.844 | 0.916 |
| Hybrid (multilingual-e5-large) | 0.494 | 0.854 | 0.900 |
Four conclusions, three of which contradict the received wisdom.
Vector search beat lexical by 13 points. We expected the opposite: on domain terminology with German compounds, intuition speaks for BM25. The cause is in the shape of the measurement — with one term held out, the query becomes a paraphrase, and paraphrase is exactly what word overlap doesn't catch.
Hybrid is worse than pure dense at the first position and better at the twentieth. RRF works with ranks, and a strong dense-top-1 gets diluted by a weak BM25-top-1. The pattern reproduced on two independent embedders, so it's a property of the method. Practical takeaway: the metric that matters depends on what you hand the model — one code or a list. "Hybrid is always better" is a slogan, not a result.
Quadrupling embedder size gave a gain indistinguishable from zero. 0.844 → 0.854 with a standard error of about ±1.6 points on 500 queries. Model capacity is not the binding constraint here. The lever is domain adaptation on those 89 thousand pairs that already exist.
A ceiling of 0.854 means that in roughly one case in seven, the correct code is not among the candidates. Widening the list from 20 to 50 buys about 4 points and costs prompt tokens on every query. That's a precision-vs-cost knob, and it needs to be set deliberately, not left at the default.
The main result: task formulation matters more than the model
We ran retrieval over a corpus of physician notes without any language model at all, feeding in three different queries and taking top-1 as the prediction.
| What gets fed into search | top-1 | ceiling |
|---|---|---|
| The full note, 312 words on average | 0.010 | 0.039 |
| The exact evidence quote from the document | 0.049 | 0.196 |
| The official catalogue title of the code | 0.843 | 0.951 |
The first row is the verdict on the naive architecture. A 312-word note is the primary diagnosis plus three or four secondaries, plus five rejected differentials, plus lab values, plus medication, plus travel history. The embedding of such a text is the centroid of the whole hospitalisation; the target diagnosis contributes a small share of the signal, and negated versions contribute alongside confirmed ones.
The second row is more surprising. Even a perfect quote from the document gives 19.6%. A physician writes clinically; the catalogue is written bureaucratically. The conclusion worth stating on its own: evidence spans exist for audit, not for search.
The third row proves the index is fine. The problem is not search but the text you address it with.
From this comes the redefinition of the first stage. The job of the language model is not "extract diagnoses" but "emit a short canonical term in the register of the catalogue that search will hit." And the diagnosis must be inferred: the text doesn't say "acute viral hepatitis"; it says elevated transaminases, negative serology, and a trip to South East Asia.
In code, this is one config line — the query is assembled from normalised fields, not from the quote. Defending that line without three measured numbers is impossible. With them, it's obvious.
A side observation from the third row, important for the product: top-1 = 0.843 when queried with the exact official code title means that around 15% of codes are not uniquely identifiable even by their own title — residual categories collide with neighbours. This isn't fixed by model quality. It's fixed by refusal and showing a group of similar codes to a human.
Throughput: this is a nightly process, not an interactive one
Benchmark on a single L40S card, two-stage pipeline:
| Concurrency | Cases/s | Multiplier | p50 | p95 |
|---|---|---|---|---|
| 1 | 0.022 | 1× | 46.3 s | 55.0 s |
| 8 | 0.144 | 6.5× | 59.6 s | 71.8 s |
| 32 | 0.336 | 15.3× | 85.2 s | 113.9 s |
46 seconds per case at concurrency 1 means this pipeline cannot be interactive. But batching is almost linear: 15× the throughput at 1.84× the latency.
For the hospital this is an architectural consequence, not an optimisation. Coding in a German inpatient setting happens after discharge, in controlling, in batches — a nightly run coincides with the process, it isn't a compromise. In the morning the coder opens a ready worklist.
And this is also what determines whether the existing hardware suffices for the hospital's document flow. On fixed capacity, throughput planning is an early and expensive-to-revise decision. Design therefore starts from the load profile — documents per day, tolerable delay, tolerable queue depth — and only then picks the model size. Not the other way round.
What this gives the hospital
What follows is a calculation model, not a measured result. Parameters get substituted with yours; we show the shape of the calculation because it matters more than the specific numbers.
The impact is not in replacing coders. It sits in three places.
Completeness. Secondary diagnoses supported by the documentation but not billed are unclaimed revenue for care that was actually delivered. The system reads every document in full, every time, without end-of-shift fatigue. Effect estimate = (fraction of cases with an additional supported diagnosis found) × (average DRG weight delta) × (base rate) × (cases per year).
Audit resilience. Every suggestion comes with a verbatim quote from the document and the catalogue version in the audit log. When MD queries a case, the response is assembled in minutes, not hours of chart work. Plus the deterministic layer strips a portion of overcoding before it hits the bill — and it's overcoding that carries the fine.
Coder time. The work shifts from searching a 17-thousand-item reference to confirming a ready suggestion with the evidence highlighted. Estimate = (average time per case) × (expected share of cases passing without edits) × (cases per year).
The key metric here is not per-code accuracy. Per-code accuracy is a presentation metric. For the hospital, the numbers to compute are:
- DRG-level match: did the case group into the same payment group
- CMI delta
- Fraction of cases passing review without edits
- Precision and recall at case level, separately, because an extra code and a missed code cost different amounts
And the most valuable input is the retrospective data you already have. Years of coded, MD-reviewed cases are a ready evaluation set on the order of tens of thousands of examples with verified answers. It doesn't need to be generated or labelled. On it the real ceiling is measured, and on it retrieval is fine-tuned.
There is a non-obvious risk here worth knowing up front. Training on historical codes trains the system on the coding habits of a specific hospital — including those the MD contested. So the training sample is filtered on cases that passed the review, not on all billed cases, and the quality metric is agreement with codes that survived audit, not with codes historically billed. The difference between these two framings is the difference between a useful system and a machine for systematically reproducing past mistakes at scale.
What we did not measure, and why we say so
The corpus is synthetic. Real German clinical notes are practically not available publicly. We generated the corpus "backwards": first the code, then a plausible note under it, with rejection of texts into which the code itself or the verbatim catalogue title leaked. The method gives near-zero label noise and lets us share the corpus. It does not reproduce the real gaps in documentation, the errors of dictation transcription, or the templates of a specific hospital. So all the accuracy figures are a ceiling, not a forecast.
The model matrix isn't fully swept. Comparing models, quantisations, and FP8 degradation on rare German terminology is work done on the client's hardware, because the answer depends on which hardware is installed. On a 24 GB card and on a 48 GB card these are different answers, and taking someone else's answer is pointless.
We do not write the grouper. A certified grouper is licensed, not rebuilt. Primary-diagnosis rules (DKR) work at the case level, not on a single note — that is a separate layer.
We list this as the first item, not the last, because in a regulated domain, the list of what the system does not do is part of its description. A vendor without that list either hasn't measured or won't tell you.
How we propose to start
Weeks 1–2. Evaluation set from your data. Retrospective coded cases that passed MD. We measure the real retrieval ceiling on your documentation — before any development. If it turns out to be low, that changes the plan, and learning it early is cheaper.
Weeks 3–4. Load profile and benchmark on your hardware. Documents per day, tolerable delay, available VRAM. The model/quantisation matrix is run on the card you actually have. Output: a defensible model choice with numbers, not a recommendation.
Months 2–3. Pipeline and review worklist. Extraction, search, deterministic rules layer, confirmation UI with evidence highlighting. Audit log with catalogue version and weights hash on each decision.
Beyond that. Fine-tuning retrieval on your data. 89 thousand BfArM pairs plus the corrections from your coders that accumulate from mandatory review. The system improves off work that is happening anyway.
The order is exactly this because the measured ceiling sits in retrieval, not the generator, and because the first steps are an order of magnitude cheaper than the later ones.
The study was conducted on official BfArM publications: ICD-10-GM 2026 (Systematik, Metadaten, Alphabetisches Verzeichnis, Alpha-ID-SE) and OPS 2026. The classifiers are used without modification. No personal data or clinical documentation was used in the study; the evaluation corpus is synthetic.
Happy to walk through methodology and numbers in a technical meeting — including the repository with the benchmark harness.