Some test failures are just a model choosing a different word. A snapshot that stored "Standard shipping takes 3 to 5 business days" fails because today's response says "Orders usually arrive within 3 to 5 business days", and the build goes red over a difference no customer would ever notice. Teams usually deal with this by updating the snapshot, loosening the check to a keyword match, or deleting the test. This post shows a fourth option: keep the snapshot, but compare meaning instead of characters. We will build a small matcher that scores how close two responses are in meaning, pick its threshold from data instead of guessing, and prove it still catches the changes that matter.

The Snapshot Failed. The Answer Was Fine.

Take a support assistant that answers shipping questions. The first time the test runs, it stores this response as the snapshot:

Standard shipping takes 3 to 5 business days. Orders placed before 2 PM ship the same day.

A week later someone tweaks the system prompt, or the provider quietly updates the model behind the same name. The test runs again and the assistant now says:

Orders usually arrive within 3 to 5 business days, and anything placed before 2 PM goes out the same day.

The facts are identical, but almost none of the words are. The snapshot assertion fails and prints a diff that is mostly red and green noise. A reviewer reads both versions, concludes nothing is wrong, and runs the update command. This happens a few times, and the team learns that a red snapshot usually means "regenerate it".

That habit is where the real damage starts. One day the response changes to "Standard shipping takes 5 to 7 business days". It is a genuine regression, but it fails the same way the harmless ones did, and it gets approved along with them. A snapshot that fails on harmless rephrasing and one that gets approved without reading are equally useless, because neither tells you whether the answer is still right.

The usual fixes do not solve it. Updating snapshots on every run turns the test into a recorder. Replacing the snapshot with a keyword or substring check catches "5 to 7" only if you thought to list it in advance. Deleting the test removes the problem along with the coverage. The cause is that the snapshot stores wording, while what you actually care about is meaning, and string equality can only measure the first one.

Why Exact-Match Snapshots Break on LLM Output

A traditional snapshot works because the code under test is deterministic: same input, same output, so any difference is a change someone made. A language model gives up that guarantee in several ways, and most of them are outside your control:

The first row is the one teams try to fix. Setting the temperature to 0 and fixing a seed reduces the variation, but most providers describe the result as mostly deterministic, not guaranteed. Small differences in how requests are batched and processed on the server can still flip a word. The rest of the table cannot be pinned at all. A model update or a prompt edit is a legitimate change that you want to make, and it will rewrite the wording of nearly every response.

So the problem is not that the output varies. It is that the variation comes in two kinds that a string comparison cannot tell apart. Noise is a different way of saying the same thing. Signal is a different thing being said. Both produce a failing diff, and only the second one deserves a red build. Fixing the model's randomness will never make the first kind go away, which means the fix has to live in the comparison.

The Idea: Snapshot the Meaning, Not the Words

An embedding model turns a piece of text into a list of numbers, called a vector, so that texts with similar meaning end up with similar vectors. "Standard shipping takes 3 to 5 business days" and "Orders usually arrive within 3 to 5 business days" share very few words, but they land close together. "We do not ship to your country" lands far from both. You do not need to understand what the individual numbers mean. What you need is a way to measure how close two vectors are.

The usual measure is cosine similarity, which compares the direction two vectors point in and returns a score between -1 and 1. In practice, the scores for text land between 0 and 1, where values near 1 mean "says the same thing" and lower values mean "says something different". The semantic snapshot test then becomes three steps:

• Store the approved response, along with its embedding, as the snapshot.
• On each run, embed the new response with the same model.
• Pass if the cosine similarity is at or above a threshold, fail if it is below.

The structure is the same as a normal snapshot test, and so is the workflow. The first run records the snapshot, later runs compare against it, and a deliberate change is approved by updating it. The only part that changes is the definition of "equal".

Two cautions before we build it. First, the threshold is a real decision. Set it too high and harmless rephrasing still fails the build, set it too low and genuine regressions slip through. We will measure it instead of guessing in a later section. Second, embeddings blur details. In the experiments later in this post, a response that says "5 to 7 days" instead of "3 to 5 days" scores higher against the original than a harmless rewording does, because the topic and tone are identical and only a few digits differ. Semantic comparison is great at ignoring wording and poor at noticing small factual changes, so we will measure that limit and then close it.

Building a Semantic Snapshot Matcher

The matcher is a few dozen lines of Python: a function that embeds text, a small class that stores and reads snapshots, and a pytest fixture that wires it into a test. The example uses sentence-transformers with the all-MiniLM-L6-v2 model. It runs on the test machine, so there is no API key and no cost per run, and the same text always produces the same vector, which is what you want from a measuring instrument. The model is also small enough to cache in CI.

The first piece embeds text and compares two embeddings. The vectors are normalized to unit length, which turns cosine similarity into a plain dot product. The model is loaded on first use, because loading takes a few seconds and a test run should only pay for it once.

                
import numpy as np
from sentence_transformers import SentenceTransformer

MODEL_NAME = "all-MiniLM-L6-v2"

_model = None


def _get_model():
    global _model
    if _model is None:
        _model = SentenceTransformer(MODEL_NAME)
    return _model


def embed(text):
    """Return a unit-length embedding, so cosine similarity is a plain dot product."""
    return _get_model().encode(text, normalize_embeddings=True)


def cosine_similarity(a, b):
    return float(np.dot(a, b))
                

Next comes the snapshot store. A snapshot is a JSON file with three fields, and each one is there for a reason:

• response: the approved text, so a reviewer can read what was approved in a pull request instead of staring at 384 numbers.
• embedding: the vector that future responses are compared against.
• model: the name of the model that produced the vector, because scores from two different embedding models cannot be compared.

                
class SemanticSnapshot:
    def __init__(self, directory, update=False, threshold=DEFAULT_THRESHOLD):
        self.directory = Path(directory)
        self.update = update
        self.threshold = threshold

    def _path(self, name):
        return self.directory / f"{name}.json"

    def _write(self, name, response):
        self.directory.mkdir(parents=True, exist_ok=True)
        payload = {
            "model": MODEL_NAME,
            "response": response,
            "embedding": [round(float(x), 6) for x in embed(response)],
        }
        self._path(name).write_text(json.dumps(payload, indent=2) + "\n", encoding="utf-8")
                

The comparison is a method on the same class. This is reduced to its core: load the stored snapshot, refuse to compare if the model has changed, embed the new response, and fail if the similarity is below the threshold. The threshold is a parameter, and DEFAULT_THRESHOLD is a constant we will set from data in the next section instead of picking a number that feels right.

                
def assert_matches(self, name, response, threshold=None):
    stored = json.loads(self._path(name).read_text(encoding="utf-8"))
    if stored["model"] != MODEL_NAME:
        raise AssertionError("Snapshot was embedded with a different model. Re-record it.")

    similarity = cosine_similarity(stored["embedding"], embed(response))
    limit = threshold or self.threshold
    if similarity < limit:
        raise AssertionError(f"similarity {similarity:.3f} is below the threshold of {limit:.2f}")
                

The model check matters more than it looks. Upgrade the embedding model in a requirements file and every stored vector is suddenly measured with the wrong ruler. A loud failure that says "re-record this" is better than a pass or a fail that nobody can explain. The version in the repository goes through a compare function that also runs a second check. We will add it once the numbers show why it is needed.

The workflow around the matcher copies what normal snapshot tools do:

• First run: a missing snapshot is recorded and the test passes.
• CI: when the CI environment variable is set, a missing snapshot fails instead, so a pipeline never records one silently.
• Deliberate change: pytest --update-snapshots re-records every snapshot with the response the test received.

The last piece is the pytest wiring. The fixture points the matcher at a snapshots folder next to the test file and reads the --update-snapshots option, which is registered in the same conftest.py. The test itself is two lines. ask_assistant stands in for whatever call your application makes to the model. In the example it returns a fixed string so the code runs offline.

                
# conftest.py
@pytest.fixture
def semantic_snapshot(request):
    directory = Path(request.fspath).parent / "snapshots"
    return SemanticSnapshot(directory, update=request.config.getoption("--update-snapshots"))


# test_shipping_assistant.py
def test_shipping_answer_still_matches_snapshot(semantic_snapshot):
    answer = ask_assistant("How long does shipping take?")
    semantic_snapshot.assert_matches("shipping_policy", answer)
                

Run it against the snapshot of the approved wording, with the stand-in returning the reworded answer that broke the exact-match test in the first section:

The test that used to fail on wording now passes. That is the whole matcher, but two questions remain. The pass depends on a threshold we have not justified yet, and we have not checked that a wrong answer still fails. The next section deals with the first one.

Picking the Threshold: Sweep It, Don't Guess It

Semantic comparisons tend to come with a threshold that somebody picked once: 0.8, because it feels like a high score. There is nothing wrong with 0.8 except that nobody knows whether it is right for this model and these responses. The way to find out is to measure it, using pairs where you already know the answer.

Start with a small labeled set. Each entry is a snapshot, a new response, and a label for what kind of change the new response represents:

                
{
  "kind": "number_change",
  "snapshot": "You can return any item within 30 days of delivery for a full refund.",
  "response": "You can return any item within 14 days of delivery for a full refund."
}
                

The example uses four labels, and each one has a clear expectation:

• paraphrase (10 pairs): same facts, different words. These must pass.
• number_change (7 pairs): a count, a time, or a price is different, such as 30 days becoming 14. These must fail.
• word_change (6 pairs): the numbers are the same but the meaning is not, such as a negation or "full refund" becoming "store credit only". These must fail.
• unrelated (6 pairs): a refusal, an off-topic answer, or a different policy altogether. These must fail.

The 29 pairs in the example repository are about a support assistant: shipping, returns, password resets, and similar topics. Yours should come from your own application. Scoring uses the same embed and cosine_similarity as the matcher, applied to every pair and grouped by label:

                
KINDS = ["paraphrase", "number_change", "word_change", "unrelated"]


def score_pairs():
    scored = []
    for pair in json.loads(DATA.read_text(encoding="utf-8")):
        scored.append(
            {
                "kind": pair["kind"],
                "similarity": cosine_similarity(embed(pair["snapshot"]), embed(pair["response"])),
            }
        )
    return scored


def print_ranges(scored):
    for kind in KINDS:
        scores = [p["similarity"] for p in scored if p["kind"] == kind]
        print(f"{kind:<15}{len(scores):>4}{min(scores):>9.3f}{max(scores):>9.3f}")
                

Each row shows the kind, the number of pairs, the lowest score and the highest score:

Read it from the top. Harmless rewordings score between 0.788 and 0.950, a wide band for responses that say the same thing. Unrelated answers top out at 0.315, nowhere near it, so any threshold in between catches all of them. The number changes are the surprise. They score between 0.935 and 0.998, which puts almost every one of them above the harmless rewordings. The word changes straddle the paraphrases, from 0.717 to 0.976.

Now sweep the threshold. For each value, count how many pairs of each kind score below it. For paraphrases that count is false failures: harmless changes that would break the build. For the other kinds it is regressions caught. A second function turns the lowest paraphrase score into a recommended threshold.

                
def sweep(scored):
    for step in range(50, 100, 5):
        threshold = step / 100
        counts = []
        for kind in KINDS:
            scores = [p["similarity"] for p in scored if p["kind"] == kind]
            counts.append(f"{sum(s < threshold for s in scores)}/{len(scores)}")
        print(f"{threshold:.2f}", *counts)


def recommend(scored, margin=0.05):
    lowest = min(p["similarity"] for p in scored if p["kind"] == "paraphrase")
    return round(lowest - margin, 2)
                

Here is the sweep at six of its ten steps:

The popular 0.80 already fails one harmless rewording out of ten, and 0.85 fails four. A team using it would see red builds for answers that are fine and would learn to regenerate snapshots, which is the habit this post started out trying to break. Raising the threshold to catch more regressions is a losing trade: at 0.95 it catches 3 of 7 number changes, but fails 9 of 10 harmless paraphrases.

The rule that falls out is to set the threshold from the paraphrase side. Its job is to never fail a harmless rewording, so it goes just under the lowest paraphrase score. Here that is 0.788 minus a margin of 0.05, which recommend rounds to 0.74. At 0.74 the matcher passes all 10 paraphrases and catches all 6 unrelated answers. It catches none of the 7 number changes and only 1 of the 6 word changes.

Two caveats apply. The margin is a judgment call, and ten paraphrases is a thin sample. The lowest score will move as you add pairs, so rerun the script whenever you add pairs or change the embedding model. The number belongs to this model and this data, not to semantic snapshots in general.

That leaves the part that matters most. A threshold only separates answers that are about different things. A response with a wrong number is about the same thing, and similarity cannot see the difference. The next section closes that gap and proves the result with regression tests.

Proving the Matcher Catches Real Regressions

The gap is easy to describe: the meaning check ignores wording, which is why it cannot see a changed number. The fix is a second check with the opposite blind spot. It ignores wording too, but it looks at the digits. Every number, time, price and percentage in the snapshot has to appear in the new response the same number of times, and the new response cannot add any. A response passes only when both checks pass.

                
import re
from collections import Counter

# Numbers, times, prices and percentages: "3", "9:30", "$50", "15%".
FACT_PATTERN = re.compile(r"[$€£]?\d+(?:[.,:]\d+)*%?")


def extract_facts(text):
    """Count every number-like token. Counter equality ignores order and wording."""
    return Counter(FACT_PATTERN.findall(text))
                

A Counter is the right container because order and wording do not matter, but repetition does. "Within 3 to 5 business days" and "3 to 5 business days" produce the same facts, while "5 to 7" produces a different set. The two checks come together in a compare function that returns a result object. It records the similarity, the facts that went missing, and the facts that appeared out of nowhere.

                
@dataclass
class MatchResult:
    similarity: float
    threshold: float
    missing_facts: list = field(default_factory=list)
    unexpected_facts: list = field(default_factory=list)

    @property
    def meaning_ok(self):
        return self.similarity >= self.threshold

    @property
    def facts_ok(self):
        return not self.missing_facts and not self.unexpected_facts

    @property
    def passed(self):
        return self.meaning_ok and self.facts_ok


def compare(snapshot_text, snapshot_embedding, response, threshold=DEFAULT_THRESHOLD):
    """Score a new response against a stored one. Pure function, no file access."""
    expected = extract_facts(snapshot_text)
    actual = extract_facts(response)
    return MatchResult(
        similarity=cosine_similarity(np.asarray(snapshot_embedding), embed(response)),
        threshold=threshold,
        missing_facts=sorted((expected - actual).elements()),
        unexpected_facts=sorted((actual - expected).elements()),
    )
                

In the repository, assert_matches now calls compare and raises with a readable message when the result does not pass. That replaces the reduced version from the earlier section. Now for the proof, which starts with a test that pins down why the second check exists. It compares the harmless rewording and the wrong delivery window against the same snapshot and asserts that the wrong answer scores higher:

                
def test_similarity_alone_cannot_tell_rewording_from_a_wrong_number():
    """The reason the matcher checks facts as well as meaning."""
    snapshot = embed(APPROVED)
    reworded = compare(APPROVED, snapshot, TODAY)
    wrong_number = compare(APPROVED, snapshot, WRONG_WINDOW)

    assert wrong_number.similarity > reworded.similarity
    assert wrong_number.meaning_ok and not wrong_number.facts_ok
                

With this model, the reworded answer scores 0.808 and the wrong delivery window scores 0.998. The meaning check passes both, and only the fact check fails the wrong one. This test is not there to test the matcher. It documents a property of embeddings, so that the next person who wonders why the fact check exists, or tries to delete it, gets a failing test with an explanation.

Next come the regression tests. Each one breaks the approved answer in a different way and expects the matcher to fail it. The wrong answers are constants at the top of the test file: the delivery window changed to 5 to 7 days, the cutoff moved from 2 PM to 5 PM, and a refusal in place of the answer. These tests use their own checker fixture because --update-snapshots would otherwise overwrite the snapshot with the wrong answer, and the tests would pass for the wrong reason.

                
@pytest.fixture
def checker():
    """A matcher that never re-records, so the regression tests below stay deliberate."""
    return SemanticSnapshot(SNAPSHOTS)


def test_changed_delivery_window_fails(checker):
    with pytest.raises(AssertionError, match="facts:"):
        checker.assert_matches("shipping_policy", WRONG_WINDOW)


def test_changed_cutoff_time_fails(checker):
    with pytest.raises(AssertionError, match="facts:"):
        checker.assert_matches("shipping_policy", WRONG_CUTOFF)


def test_unrelated_answer_fails(checker):
    with pytest.raises(AssertionError, match="meaning:"):
        checker.assert_matches("shipping_policy", UNRELATED)
                

The match argument checks which check failed, not only that something did. It also shows what a reviewer sees when one of these fires for real. This is the message for the 5 to 7 days answer:

Compare that with the wall of red and green from the exact-match snapshot. The failure names the exact fact that changed, the "3" that went missing and the "7" that appeared, and prints both versions underneath. It takes a few seconds to judge instead of a few minutes. Running the whole file gives:

The same labeled pairs from the previous section show the effect across the whole set. At the 0.74 threshold, adding the fact check changes the number of regressions caught like this:

Number changes go from none caught to all seven, and the harmless paraphrases still pass, because a reworded answer keeps the same numbers. The word changes did not move. One test in the output above is marked XFAIL on purpose, and it is the clearest example of what the matcher still cannot see.

Where Semantic Snapshots Fall Short

Every limit below was reproduced by running the matcher, and each one is a reason to treat a semantic snapshot as a cheap first line of defense, not a verdict on correctness.

Negations and swapped words. The XFAIL test feeds the matcher "Standard shipping does not take 3 to 5 business days" and expects a failure:

                
@pytest.mark.xfail(
    strict=True,
    reason="Known gap: a negation keeps the same topic and the same numbers, so both checks pass.",
)
def test_negated_answer_fails(checker):
    with pytest.raises(AssertionError):
        checker.assert_matches("shipping_policy", NEGATED)
                

It scores 0.957 and keeps every number, so both checks pass and the test fails as expected. With strict=True, the day the matcher improves and the test starts passing, the suite turns red and tells you to remove the marker. The labeled data shows the same gap. "Store credit only" scores 0.850, "Amex" in place of "PayPal" scores 0.877, and "Saturday to Sunday" in place of "Monday to Friday" scores 0.976. All three pass. Only one of the six word changes is caught, and only because its score happens to fall under the threshold.

Number formatting. The fact check cuts both ways. "Three to five days" has no digits, so against a snapshot that says "3 to 5" it reports both numbers missing and fails a harmless answer. The same happens with "2:00 PM" against "2 PM", "14:00" against "2 PM", and "1000" against "1,000". These are false failures, the exact problem this post set out to remove. If the snapshot itself spells numbers out, the opposite happens: "five to seven days" scores 0.968 against "three to five days" and passes. Either normalize numbers before extracting them or instruct the model to always write digits. The example does neither.

Long responses. The embedding model reads at most 256 tokens and ignores the rest. In a test with a long answer whose last sentence changed from "Refunds take 5 days" to "Refunds are never issued", the similarity was 1.000. The meaning check saw nothing, and only the stray digit made the fact check fail. Keep the snapshotted text short and factual, or split long answers into chunks.

Scope. This approach suits short answers that should stay stable: policies, confirmations, summaries with figures. For open-ended or creative text, "same meaning" has no stable definition and a threshold will not give you one. For contradictions, the usual next step is a model-graded check that asks a second model whether the new response disagrees with the snapshot. We covered that in grading at scale with an LLM judge, and it needs its own checks, as detecting a hallucinating judge explains.

Takeaways

A snapshot test should fail when the answer changes, not when the wording does. Comparing meaning gets most of the way there: store the approved response with its embedding, compare new responses by cosine similarity, and set the threshold from labeled pairs instead of a number that feels right. Then add the check that embeddings cannot do on their own, an exact comparison of the numbers, because a wrong figure looks almost identical to the right one in vector space. What is left are negations and swapped words, and those need a model-graded check. Until you add one, treat the labeled set as a regression suite for the matcher itself, and grow it every time a real failure gets through.

The full code example from this post is available on our GitHub repository.