@kendrastoate381
Profile
Registered: 2 hours, 37 minutes ago
Semantic caching: why your similarity threshold needs a near-miss set
" (video: //www.youtube.com/embed/Lgg-jir_AhQ)
A semantic cache stores a large-language-model response and returns it when a new query’s embedding is similar enough to a previously answered query. In deployed products, that decision rests on a single similarity threshold, typically shipped as a default between 0.80 and 0.95. The logic feels clean: embed both queries, compute cosine similarity, and if the score clears the bar, reuse the old answer. But the threshold is not a property of language. It is a property of one embedding model on one traffic distribution. And for many real workloads, a single number quietly fails.
Consider a customer who writes "I want to cancel my subscription" and later writes "I want to pause my subscription for a month." These share high lexical overlap — nearly every content word matches. An embedding model will place them close together, and a threshold around 0.85 will happily return the cancellation answer to the pause request. That is a false cache hit: the response is confident, fluent, and wrong. The opposite failure is just as common. "How do I reset my password?" and "What do I do if I forgot my login?" mean the same thing but share almost no lexical overlap. Their embeddings may sit far apart, so the cache misses and the model regenerates a response that was already computed minutes ago.
A 2026 preprint by Guerin Green of Novel Cognition built a small labelled set to measure this problem directly. The set contains 243 English customer-support question pairs across eight domains: 101 heavily reworded paraphrases, 102 near-miss pairs that differ by a word or by word order but need different answers, and 40 lightly reworded paraphrases reported separately. Scored by cosine similarity with the open embedding model Qwen3-Embedding-0.6B, the near-miss pairs scored higher (median 0.801) than the genuine paraphrases (median 0.691). The area under the ROC curve was 0.144, with a 95% bootstrap interval from 0.097 to 0.195. That means the distributions were not merely overlapping; they were inverted. The model thought near-misses were more similar than true paraphrases.
At a threshold of 0.85, which is the default of one commercial service, that model matched none of the 101 paraphrases and 29 of the 102 near-misses — 28.4% of the wrong answers would have been served from cache. False matches reached zero only above a threshold of 0.985, and even then, 2 of the 40 near-verbatim repeats were still missed. So raising the threshold does not fix the problem; it just trades false cache hits for cache misses, and in this case the trade is terrible. The preprint also found that similarity correlated with word-set Jaccard overlap (Pearson r = 0.695), including within each label, which suggests the embedding model leans on surface form rather than meaning. Adding a query-instruction prefix halved false matches at 0.85 (to 12.7%) without changing the ordering of the pairs — a cheap mitigation, but still not a solution.
The practical lesson for engineers is to stop trusting a default. A threshold is a measurement, not a constant. You need your own labelled set of paraphrases and near-misses, drawn from your own traffic. Collect a few hundred pairs: some that should share an answer, some that must not, even though they look similar. Score them with your chosen embedding model, plot the two distributions, and find the threshold that separates them — if one exists. In the data above, no single threshold works for that model. Other embedding models need their own measurement; the numbers here do not transfer.
The preprint argues that this protocol is essential for semantic caching to be safe. The pairs, https://zenodo.org/records/23667256 the scores, and an analysis script that recomputes every statistic are released under CC BY 4.0, so you can inspect the logic without trusting the prose. The core message is calm and blunt: lexical overlap is a strong predictor of embedding similarity, but it is not the same as semantic equivalence. A customer asking to cancel versus to pause a subscription will always look close on the surface. Your cache must know the difference before it answers.
Website: https://zenodo.org/records/23667256
Forums
Topics Started: 0
Replies Created: 0
Forum Role: Participant
Location: France, Bondy