UpcomingGovernment AI Adoption Management Summit 2026Thursday, Dec 3 at 11 AM CSTThe Blindspots of AI AdoptionGov AI Adoption SummitDec 3, 11 AM CSTRegister Now
Book a demo Start free
Pricing
Start free
Book a demo

AgentNuro Control · Semantic Caching

Semantic caching: reuse AI responses instead of paying for them twice

AgentNuro checks every request against answers you have already paid for. Exact repeats, and differently worded versions of the same question, come back from cache with no provider call. You set how close a match must be, and see the dollars the cache saves.

Try Control Lite, freeSee when a cache hit is safe
app.cloudnuro.ai/cacheSemantic match

New request

“Give me a summary of the Q3 board report”

0.97 similaritythreshold 0.94

Matched stored request, answered 3 hours ago

“Summarize the Q3 board report”

Served from cache

40 ms

No provider call
If sent to gpt-4.11.2 s  $0.0021

Saved $0.0021 on this request

What is semantic caching?

Semantic caching stores AI responses and serves them again when a new request means the same thing as one already answered, even if it is worded differently. Exact-match caching catches only word-for-word repeats. Both skip the provider call, so a cache hit has no provider charge and returns faster.

Without a cache

  • Paid in full, every time.The same question asked by fifty people is generated and billed fifty times.
  • Waiting on routine work.Every request waits for a fresh model call, however common the question.
  • Repeat work stays invisible.Provider bills show total tokens, not how much of it was the same question again.

Two ways a request can match

=Exact match

“Summarize the Q3 board report”

“Summarize the Q3 board report”

Word for word. Fast, and only catches true repeats.

~Semantic match

“Summarize the Q3 board report”

“Give me a summary of the Q3 board report”

Same meaning, different words. Matched by similarity, within the threshold you set.

How does semantic caching cut AI cost?

Every request is checked against the cache before it reaches a provider. A hit skips the call entirely. A miss goes to the provider as normal, and its answer is stored for next time.

  1. 1

    Check the cache

    Before any provider call, the request is compared with stored answers, exact first, then by meaning.

    Every request, before the provider
  2. 2

    Hit: serve it

    A close-enough match returns the stored answer. The provider is never called, so there is no provider charge.

    No provider charge
  3. 3

    Miss: call as normal

    No match, and the request goes to the provider exactly as it would without a cache.

    Billed as usual
  4. 4

    Store the new answer

    The fresh answer is saved, so the next identical or similar request can be a hit until it expires.

    Ready for the next repeat

Even a miss can cost less

The repeated part of the prompt, billed at the cached rate

On a cache miss, AgentNuro arranges the prompt so the parts that rarely change, such as system instructions and tool definitions, come first. That lets the provider’s own prompt caching bill them at its lower cached-token rate.

One request, 2,005 input tokens

1,150 cached855 fresh

Stable prefix (system prompt, tool definitions) billed at the provider’s cached rate; only the new part billed at the full input rate.

When is a cache hit safe?

A wrong answer from cache is worse than no cache at all. “Summarize the Q2 report” sounds close to “Summarize the Q3 report,” but it needs a different answer. The similarity threshold decides where that line sits. Try it.

Stored answer for

“Summarize the Q3 board report”

Similarity threshold

Incoming requestSimilarityOutcome

Similarity scores depend on the embedding model and the workload.

  • Similarity threshold

    Set per workload. A request has to be close enough to a stored one to count, not merely similar sounding.

  • Expiry

    Every cached answer has a time to live. When it expires, the next request goes to the provider and refreshes it.

  • Kept out of the cache

    Workloads you exclude, and requests that depend on personal or live data, are never served from cache.

How much does caching save?

It depends on how repetitive the work is, so AgentNuro shows it per workload instead of promising a number. Help desks and policy Q&A repeat a lot. Summaries of unique calls barely repeat. Some workloads should not be cached at all.

app.cloudnuro.ai/cache/workloadsSample data

Cache savings by workload, month to date

WorkloadRequestsHit rateSaved
Internal policy Q&AHighly repetitive questions96,20058%$2,410
Support ticket triageCommon ticket types repeat182,40041%$1,860
Sales call summariesEach call is unique44,9006%$140
Code review assistantExcluded: code differs every time71,300OffNot cached
Contract reviewExcluded: confidential documents12,800OffNot cached
Response cache total407,600$4,410
Provider prompt caching, on misses$1,180

Every hit is credited with what the request would have cost at the provider, and cache savings are reported separately from routing savings.

How is AgentNuro caching different?

Provider prompt caching discounts part of a request but still generates and bills the answer. Many gateway caches match only exact repeats. AgentNuro skips the call for exact and similar requests, makes misses cheaper too, and shows the dollars either way.

Provider prompt cachingExact-match gateway cachesSemantic cache plugins
AgentNuro Control
What it skips
Provider prompt cachingRe-reading a repeated prompt prefix. The answer is still generated and billed.Exact-match gateway cachesThe provider call, for word-for-word repeats only.Semantic cache pluginsThe provider call, for exact and similar requests.
AgentNuro ControlThe provider call for exact and similar requests. On a miss, the repeated prefix is billed at the cached rate.
Match control
Provider prompt cachingNot applicable.Exact-match gateway cachesExact match only.Semantic cache pluginsA similarity setting, often configured per plugin.
AgentNuro ControlThreshold, expiry, and exclusions set per workload.
Savings shown
Provider prompt cachingA cached-token line on the invoice.Exact-match gateway cachesHit counts. Dollar savings are left to you.Semantic cache pluginsUsually hit rate.
AgentNuro ControlDollars saved per workload, with cache and routing reported separately.
Works with
Provider prompt cachingOne provider at a time.Exact-match gateway cachesSeparate from routing and budgets.Semantic cache pluginsOften a separate add-on.
AgentNuro ControlThe same gateway as model routing, budgets, and policy.

The first three columns describe common approaches in the market, not a specific vendor.

Frequently asked questions

An exact cache matches requests word for word. A semantic cache matches by meaning, using similarity rather than string comparison, so differently worded requests can return the same stored answer. AgentNuro checks for an exact match first, then a semantic one.

It can if the threshold is too loose or answers live too long, which is why both are set per workload. A hit is served only above the similarity threshold, cached answers expire after their time to live, and excluded workloads are never cached.

Provider prompt caching bills the repeated start of a prompt at a lower rate, but the request still goes to the provider and the answer is still generated and billed. AgentNuro’s cache can skip the provider call entirely. On a miss, AgentNuro also arranges the prompt so provider prompt caching applies more often. The two add up.

You do, per workload. Start strict, watch the hit rate and the answers served, and loosen it only where repeats are clearly the same question.

Yes. You choose which workloads use the cache. Anything that should not be cached, such as confidential documents or answers that depend on live data, stays out.

Each cache hit is credited with what the request would have cost at the provider, on the model it would have used. Savings appear per workload and month to date, reported separately from routing savings.

One line. Caching runs in the AgentNuro gateway, so your calls go through it with a one-line endpoint change, or through the SDK.

Control fails open. If the gateway cannot be reached, requests go to the provider directly, so an outage pauses caching, not your application.

Stop paying full price for answers you already have.

Set the threshold, expiry, and exclusions per workload, and see the dollars saved on every hit.

Try Control Lite, free