Why should you care?

Reps ask the same discount policy question daily — caching saves thousands of identical GPT calls.

See it live — copy this example

Paste into an ASP.NET Core 8+ / AgentNest project, then run with dotnet run (set your API keys in user-secrets).

// AgentNest.Caching/SemanticAnswerCache.cs
public sealed class SemanticAnswerCache(IDistributedCache cache, IEmbeddingGenerator embedder)
{
    public async Task<string?> TryGetAsync(string tenantId, string question, CancellationToken ct)
    {
        var hash = Convert.ToHexString(SHA256.HashData(Encoding.UTF8.GetBytes(tenantId + question)));
        return await cache.GetStringAsync($"ans:{hash}", ct);
    }
    public Task SetAsync(string tenantId, string question, string answer, CancellationToken ct)
    {
        var hash = Convert.ToHexString(SHA256.HashData(Encoding.UTF8.GetBytes(tenantId + question)));
        return cache.SetStringAsync($"ans:{hash}", answer,
            new DistributedCacheEntryOptions { SlidingExpiration = TimeSpan.FromHours(6) }, ct);
    }
}

What happened?

• SemanticAnswerCache keys by tenant plus question hash.
• RagPipeline checks cache before embedding search and LLM call.

Practice next

1. Insert cache lookup at start of RagPipeline.AskAsync.
2. TTL cache by content type — policies 6h, live CRM data 60s.
3. Invalidate on KB chunk update events.
4. Add semantic near-duplicate match via embedding similarity before exact hash.
5. Use HybridCache for L1 in-memory plus Redis L2.

Remember

Cache RAG answers and embeddings with tenant-scoped keys. Invalidate on knowledge base updates. Track hit rate to validate savings.

Policy FAQ savings

500 reps query enterprise discount policy daily.

Outcome: Answer cache hits 92%; OpenAI spend on that FAQ drops to near zero.