Most teams that ship a speech-to-text feature test the audio pipeline thoroughly. Sample rates are checked, silence is trimmed, formats are converted, and the file that reaches the model is exactly what it should be. Then the model returns a transcript, and the test asserts that the string is not empty. What the model actually understood is rarely measured, so when a provider quietly rolls out a new version and "Q3" starts coming back as "Q three", nobody finds out until a customer does.

Word Error Rate is the standard way to measure transcription accuracy, and it is simple enough to compute in a few lines of Python. The catch is that a naive comparison will report failures for casing, punctuation, and hyphens, none of which your users care about, while a properly normalized one will surface the substitutions and dropped words that they do. This post walks through building a small WER gate with jiwer: pairing reference transcripts with model output, normalizing both sides so that only real errors count, and turning the result into a pass/fail check that fails your pipeline before a regression reaches production.

The Blind Spot in Audio Testing

Audio testing in most QA suites stops at the signal. There are checks that a recording exists, that it has the right sample rate and channel count, that it isn't silent, that it converts cleanly from one format to another. These are useful checks and they catch real bugs, but they all share the same limitation: they verify the input to the speech-to-text model, not the output.

The output side usually gets a much lighter touch. A typical integration test sends a known clip to the transcription service and asserts that a response came back, that it wasn't an error, and maybe that it contains a keyword or two. Sometimes there is an exact string match against a transcript someone typed out once, which passes until the model changes how it writes a number or where it puts a comma, and then it fails for reasons nobody cares about and gets loosened or deleted.

Neither approach tells you whether the transcript is accurate. A model can return a fluent, well-formed sentence that drops a negation, swaps a digit in an order number, or turns a product name into a similar-sounding word, and every one of those tests will stay green. The failure only becomes visible downstream, when a voice command does the wrong thing or a support agent reads back a number the customer never said.

This matters more now than it used to, because speech-to-text is rarely something a team builds and controls end to end. It is a hosted API or a model checkpoint that gets updated on the provider's schedule. Accuracy can shift between versions without any change in your own code, and without a metric that captures how well the model heard what was said, there is nothing in the pipeline that would notice.

What Word Error Rate Actually Measures

Word Error Rate is the metric the speech recognition field settled on decades ago, and it has stuck because it is easy to reason about. You take a reference transcript, which is what was actually said, and a hypothesis, which is what the model produced. You then line the two up word by word and count the minimum number of edits needed to turn the hypothesis into the reference. Three kinds of edit are possible:

• Substitution: the model output a different word in place of the correct one. "Q three" where the reference says "Q3" is a substitution of one word for another.
• Deletion: a word in the reference has no counterpart in the hypothesis. The model simply didn't produce it. A dropped "not" is a deletion, and one of the more dangerous kinds.
• Insertion: the hypothesis contains a word that isn't in the reference at all. Fillers, hallucinated words, and a single reference token split into two all show up here.

WER is the sum of those three counts divided by the number of words in the reference. A score of 0.00 means the transcript is perfect. A score of 0.10 means roughly one word in ten is wrong in some way. Because insertions are counted against the reference length, the score can exceed 1.0 if the model produces a lot of extra words, which surprises people the first time they see it but is a legitimate result.

The alignment step is what makes this non-trivial. Given a deletion early in a sentence, a naive position-by-position comparison would mark every subsequent word as wrong. The Levenshtein alignment that WER uses finds the cheapest explanation instead, which in the example above is a single missing word rather than three mismatches. Libraries like jiwer do this for you, and they also give you the per-category counts, which turn out to be more useful than the headline number when something breaks. A model update that produces mostly deletions is telling you something different from one that produces mostly substitutions.

Character Error Rate is the same calculation applied at the character level instead of the word level. It is the better fit for languages without whitespace word boundaries, and it is often worth tracking alongside WER for short utterances, where a single wrong word in a four-word command swings WER by 25 points while CER stays proportionate to how wrong the word actually was. For the English sentences in this post, WER will be the gate and CER will be reported next to it for context.

One more thing worth being clear about before writing any code: WER is a formatting-sensitive metric. "Twenty-five" and "twenty five" are different words to it. So are "Let's" and "let's". If the reference transcripts were written by a person following one style and the model follows another, a large part of the score will be measuring the gap between those styles rather than the gap between what was said and what was heard. That is the problem the next two sections deal with.

Setting Up the Test Cases

The whole thing needs one dependency. jiwer handles the alignment, the WER and CER arithmetic, and the text normalization, and it has no heavy transitive dependencies of its own, so it installs in seconds and is safe to add to a CI image. Pin the version, because the transform API has changed shape between major releases.

                
pip install jiwer==4.0.0
                

Each test case is a pair: a reference transcript and a hypothesis. The reference is what a person would write down after listening to the clip, and it is the ground truth the model is measured against. The hypothesis is what the speech-to-text model returned for that same clip. In a real pipeline the hypothesis comes from calling your provider's API on a fixed set of recordings; here the responses are hardcoded so the example runs anywhere without credentials. Keeping the cases in a JSON file rather than in the script means the transcripts can be reviewed and extended by someone who never touches the Python, and it makes it easy to swap in a different file for a different language or domain.

The four cases below are chosen to exercise different failure modes. The first has a genuine formatting mismatch on a number. The second has a wrong digit in an order number and a contraction written out in full. The third and fourth differ from their references only in casing, punctuation, and a hyphen, which a good normalization pipeline should reduce to zero.

                
[
  {
    "id": "meeting-intro",
    "reference": "Good morning, everyone. Let's start with the Q3 numbers.",
    "hypothesis": "Good morning everyone, let's start with the Q three numbers."
  },
  {
    "id": "support-call",
    "reference": "My order number is 48213 and it hasn't arrived yet.",
    "hypothesis": "my order number is 4813 and it has not arrived yet"
  },
  {
    "id": "voice-command",
    "reference": "Set a timer for twenty-five minutes.",
    "hypothesis": "set a timer for twenty five minutes"
  },
  {
    "id": "address-lookup",
    "reference": "The office is at 221B Baker Street, London.",
    "hypothesis": "The office is at 221B Baker Street London."
  }
]
                

A note on the references themselves, because this is where most WER setups go wrong before any code is written. The reference transcripts should be produced by a person, not by a model, and they should follow a written style guide: how numbers are spelled, whether contractions are expanded, how proper nouns are capitalized. It does not matter much which conventions you pick. What matters is that every reference follows the same ones, because inconsistency in the references shows up as noise in the score and there is no normalization step that can remove it.

Loading the file is the least interesting part of the script, but it is worth making the path an optional argument from the start. The same gate can then run against a smoke set of a dozen clips on every commit and a larger set nightly, without any change to the code.

                
import json
import sys
from pathlib import Path

import jiwer


def load_cases(path):
    with open(path, encoding="utf-8") as f:
        return json.load(f)


cases_file = Path(sys.argv[1]) if len(sys.argv) > 1 else Path("test_cases.json")
cases = load_cases(cases_file)
                

The Normalization Trap

The most direct way to get a score is to hand the two strings to jiwer.wer and read the number back. It takes a reference and a hypothesis, splits both on whitespace, aligns them, and returns the rate. This is the call most people write first, and it is worth writing it first here too, because the numbers it produces are the reason the rest of this section exists.

                
for case in cases:
    raw_wer = jiwer.wer(case["reference"], case["hypothesis"])
    print(f"{case['id']}: {raw_wer:.2f}")
                

Taken at face value, these say the model got more than half the words wrong in three of the four clips. Look at the voice command case again: the reference is "Set a timer for twenty-five minutes." and the model returned "set a timer for twenty five minutes". A person reading those two would say the transcript is perfect. WER says 0.67, because "Set" and "set" are different tokens, "minutes." with a trailing period is not "minutes", and "twenty-five" is one word in the reference but two in the hypothesis, which counts as a substitution plus an insertion. Four edits over six reference words. Every one of them is a formatting difference, and none of them is a transcription error.

This is the trap. Raw WER measures the distance between two writing styles as much as it measures the distance between what was said and what was heard, and the style component is usually the larger of the two. A gate built on raw scores will either be set so loose that it catches nothing, or it will fail every time the provider changes its punctuation model. Either way it stops being trusted.

The fix is to normalize both sides through the same pipeline before comparing them. jiwer ships a set of composable transforms for exactly this. Each one is a small, single-purpose text operation, and jiwer.Compose chains them into a single callable that is applied to the reference and the hypothesis identically. The final transform in the chain has to be one that turns the string into a list of words, because that is the shape the scoring functions expect.

                
normalize = jiwer.Compose([
    jiwer.ToLowerCase(),
    jiwer.ExpandCommonEnglishContractions(),  # "hasn't" -> "has not"
    jiwer.SubstituteRegexes({r"-": " "}),     # "twenty-five" -> "twenty five"
    jiwer.RemovePunctuation(),
    jiwer.RemoveMultipleSpaces(),
    jiwer.Strip(),
    jiwer.ReduceToListOfListOfWords(),
])
                

Read top to bottom, this lowercases everything, expands common English contractions so that "hasn't" and "has not" become the same three tokens, replaces hyphens with spaces, strips the remaining punctuation, collapses any double spaces that the previous steps left behind, trims the ends, and finally splits into words. The result is that the only differences left between the two sides are differences in which words were recognized.

The order of the transforms is not cosmetic, and two of the lines above are there specifically because the obvious ordering produces wrong results. ExpandCommonEnglishContractions matches on the apostrophe, so it has to run before RemovePunctuation deletes it. Run them the other way round and "hasn't" becomes "hasnt", which then fails to match "has not" and counts as a substitution plus a deletion. Similarly, RemovePunctuation removes hyphens by deleting them, so "twenty-five" becomes "twentyfive" rather than "twenty five". The SubstituteRegexes line ahead of it turns the hyphen into a space first, and the model's "twenty five" then matches. Both of these were found by running the pipeline and reading the alignment output, not by reading the documentation, which is a good argument for always looking at the per-word alignment when a score is higher than expected.

What the pipeline deliberately does not do is normalize numbers. "Q3" and "Q three" remain different tokens, and the wrong order number in the support call remains wrong. That is a judgment call. If your reference style guide says numbers are always written as digits and your provider can be configured to do the same, then a mismatch there is a real defect and should count. If the provider's number formatting is out of your control, adding a SubstituteWords step that maps spelled-out numbers to digits is reasonable. The point is that the decision should be explicit and written into the pipeline, rather than left to whatever the raw comparison happens to do.

Scoring and Gating

With the pipeline in place, the scoring function can be written. It computes three things for each case. The raw WER is kept purely so the output can show how much of the original score was formatting noise, which is useful when someone asks why the numbers in the report look nothing like the numbers the provider's dashboard shows. The normalized result is the one that matters, and for that the call is jiwer.process_words rather than jiwer.wer. The two accept the same arguments, but process_words returns an object carrying the substitution, deletion, and insertion counts alongside the rate, plus the alignment data needed to print a readable diff later. The transforms are passed in explicitly for both sides, which keeps the normalization visible at the call site rather than hidden in a global default. CER is computed on the raw strings, since character-level scoring is less sensitive to the formatting issues that make raw WER misleading.

                
WER_THRESHOLD = 0.15


def evaluate(reference, hypothesis):
    """Return raw WER, normalized word-level output, and CER."""
    raw_wer = jiwer.wer(reference, hypothesis)
    words = jiwer.process_words(
        reference,
        hypothesis,
        reference_transform=normalize,
        hypothesis_transform=normalize,
    )
    cer = jiwer.cer(reference, hypothesis)
    return raw_wer, words, cer
                

The threshold is a per-case limit, not an average across the set. That is a deliberate choice. An average lets one badly transcribed clip hide behind nine good ones, which is the same problem as gating a load test on mean response time. A per-case limit means every clip has to clear the bar on its own, and a single regression on a single clip is enough to turn the run red. The value of 0.15 is a starting point for clean English audio and should be revisited once there is a baseline, which the last section comes back to.

The main loop applies the scoring function to every case and prints a compact summary line for each: the pass or fail status, the three scores, and the error breakdown. The breakdown earns its place in the output because it is the fastest way to characterize a regression. A model update that produces mostly deletions is dropping words, which usually points at audio quality or endpointing. One that produces mostly substitutions is mishearing them, which is more often a vocabulary or accent problem. When a case fails, the loop also prints the word-level alignment from jiwer.visualize_alignment, so the person reading the CI log can see exactly which words differ without re-running anything locally.

                
failures = 0
for case in cases:
    raw_wer, words, cer = evaluate(case["reference"], case["hypothesis"])
    passed = words.wer <= WER_THRESHOLD
    failures += not passed

    print(f"[{'PASS' if passed else 'FAIL'}] {case['id']}")
    print(f"  WER raw={raw_wer:.2f}  normalized={words.wer:.2f}  CER={cer:.2f}")
    print(f"  subs={words.substitutions}  dels={words.deletions}  ins={words.insertions}")
    if not passed:
        print(jiwer.visualize_alignment(words, show_measures=False))
                

The last piece is what makes this a gate rather than a report. After the loop, the script prints a one-line summary and exits with a non-zero status if any case failed. Every CI system treats a non-zero exit as a failed step, so this single line is the entire integration: no plugin, no custom reporter, no parsing of the output by a downstream job. If the transcripts are within tolerance the step is green and the pipeline continues; if they are not, the step is red and the alignment output above it says why.

                
print(f"\n{len(cases) - failures}/{len(cases)} cases within WER <= {WER_THRESHOLD}")
sys.exit(1 if failures else 0)
                

Reading the Results

Running the script against the four cases produces the output below. Three cases pass, one fails, and the process exits with status 1. It is worth going through each line, because the interesting part of this output is not the pass/fail column but the gap between the raw and normalized scores, and what the failing case's alignment shows.

Start with the two clean cases. The voice command scored 0.67 raw and 0.00 normalized, and the address lookup went from 0.12 to 0.00. In both, every single edit that the raw comparison counted was casing, punctuation, or a hyphen. The normalization pipeline removed all of it and left nothing behind, which is exactly the outcome it should produce for a transcript that a person would call correct. If either of these had a non-zero normalized score, that would point at a gap in the pipeline rather than a problem with the model, and the fix would be another transform, not a bug report.

The support call also passes, at 0.09. The raw score was 0.50, and the drop is mostly the contraction: "hasn't" expanded to "has not" on the reference side, which matched the model's output word for word. The one remaining substitution is the order number, 48213 against 4813. A single wrong token in an eleven-word reference is under the threshold, so the case is green. That is the correct result according to the metric and a worrying one according to common sense, and the next section deals with it.

The meeting intro is the case that fails, and the alignment output is what makes the failure actionable. The marker row underneath the two transcripts flags an insertion where the model produced "q" against nothing in the reference, and a substitution where it produced "three" against the reference's "q3". In other words, one reference token became two hypothesis tokens. Two edits over ten words is 0.20, above the 0.15 limit. Nothing was misheard. The model simply formats quarter names differently from the person who wrote the reference, and the pipeline, which deliberately leaves numbers alone, reported that as an error.

Whether that failure is a true positive depends on the decision made in the normalization section. If the product displays transcripts to users and the style guide says "Q3", then a model that writes "Q three" is producing output that has to be post-processed, and the gate catching it is doing its job. If nobody downstream cares how quarters are spelled, the right response is to add a number normalization step and rerun, and the case will pass. Either way, the CI log shows exactly which two tokens caused the failure, which is the level of detail needed to make that call in under a minute rather than by reproducing the run locally.

The CER column is doing quiet but useful work throughout. For the voice command it reads 0.08 against a raw WER of 0.67, which on its own is a strong hint that the word-level disagreement is formatting rather than content. For the support call it reads 0.10, closer to the normalized WER, because a wrong digit is a genuine character-level difference. When the two metrics diverge sharply, the divergence is usually normalization noise; when they move together, the error is usually real.

What WER Won't Catch

Go back to the support call. The model returned "my order number is 4813" for a customer who said 48213, and the gate passed it at 0.09. From the metric's point of view this is correct: one token out of eleven is wrong, and one in eleven is under the limit. From the point of view of whoever is looking up that order, the transcript is useless. The same thing would happen with a dropped "not", a wrong dosage, or a product name that came back as a different product name. Each is one word, and WER counts every word the same.

This is not a flaw in the metric so much as a description of what it is. WER is an aggregate. It answers the question "how much of this transcript is wrong?" and does so well. It does not answer "is the part of this transcript that matters correct?", and no threshold will make it do so, because lowering the limit far enough to catch a single wrong digit in a long sentence also makes the gate fail on every minor disfluency. The two questions need two checks.

The second check is a targeted assertion on tokens that must survive transcription intact. What counts as critical depends on the domain, but numbers and negations are a reasonable default for almost any product. The check below reuses the same normalization pipeline, since the whole point is to compare like with like. It extracts the critical tokens from the reference, then verifies that each one appears somewhere in the hypothesis. Note that it uses normalize(text)[0] because the pipeline ends with ReduceToListOfListOfWords, which returns a list of sentences, each a list of words.

                
NEGATIONS = {"not", "no", "never"}


def critical_tokens(words):
    """Tokens a transcript must not get wrong: numbers and negations."""
    return {w for w in words if w.isdigit() or w in NEGATIONS}


def missing_critical_tokens(reference, hypothesis):
    ref_words = normalize(reference)[0]
    hyp_words = normalize(hypothesis)[0]
    return critical_tokens(ref_words) - set(hyp_words)
                

Run against the four cases, this returns an empty set for three of them and {'48213'} for the support call. The "not" in that case passes, because the contraction expansion in the pipeline turned "hasn't" into "has not" on both sides. Wiring the check into the main loop is a matter of treating a non-empty result the same as a threshold breach: mark the case failed, print which tokens went missing, and let the existing exit code handle the rest. With that in place the support call fails for the right reason, and the gate answers both questions instead of one.

A membership check is deliberately loose. It confirms the number is present somewhere in the hypothesis, not that it is in the right place or attached to the right noun, and it will not notice a number that was transcribed correctly but repeated. For most products that is enough, because the failure mode being guarded against is the model mishearing a digit, and a mishearing produces a different token rather than a misplaced one. If position matters, the alignment data from process_words already has it, and the check can be tightened to require the critical token to sit in a matched column rather than anywhere in the sentence.

There is a second category of thing WER does not see at all, which is everything the transcript does not contain. Timestamps, speaker labels, confidence scores, and punctuation placement are all outputs of a modern speech-to-text service, and a regression in any of them is invisible to a metric that only looks at the words. If the product depends on them, they need their own checks. WER is the right first gate for transcription accuracy, and it should be understood as that: the first gate, not the only one.

Conclusion

The 0.15 threshold used here is a placeholder. Replace it with a baseline: run the gate against the current model with a loose limit, record the normalized WER per case, and set the limit a little above the worst case you are willing to accept today. The gate is then green on the version you already ship and turns red the moment a provider update makes any clip measurably worse. Wiring it into CI is a two-line step, install jiwer and run the script, since the exit code does the rest. Against hardcoded hypotheses it is cheap enough for every commit; against live model output, a nightly run on a fixed recording set is usually the right cadence.

A speech-to-text integration tested only at the signal level has no idea whether the model understood anything, and a model behind someone else's API can get worse without a line of your code changing. Word Error Rate closes that gap. Normalized properly and paired with a targeted check on the tokens that matter, it turns transcription accuracy from something you learn about in a support ticket into something your pipeline tells you before the deploy goes out.

The complete example is available on our GitHub page. Swap the hardcoded hypotheses for calls to your provider and run it against a handful of your own recordings. The raw scores will almost certainly look alarming, and the normalized ones will tell you what is actually going on.