Why should you care?

Token throughput limits and OpenAI TPM caps require backpressure — not unlimited synchronous HTTP threads.

See it live — copy this example

Paste into an ASP.NET Core 8+ / AgentNest project, then run with dotnet run (set your API keys in user-secrets).

// AgentNest.Api/Scaling/TokenBucketLimiter.cs
public sealed class TokenBucketLimiter(IDistributedCache cache)
{
    public async Task<bool> TryAcquireAsync(string tenantId, int estimatedTokens, CancellationToken ct)
    {
        var key = $"tpm:{tenantId}:{DateTime.UtcNow:yyyyMMddHHmm}";
        var used = int.TryParse(await cache.GetStringAsync(key, ct), out var u) ? u : 0;
        if (used + estimatedTokens > 100_000) return false;
        await cache.SetStringAsync(key, (used + estimatedTokens).ToString(),
            new DistributedCacheEntryOptions { AbsoluteExpirationRelativeToNow = TimeSpan.FromMinutes(1) }, ct);
        return true;
    }
}

What happened?

• TokenBucketLimiter tracks estimated tokens per tenant per minute in Redis, rejecting overload before calling Azure OpenAI.
• Follow the steps below — typing the code yourself is the fastest way to learn.

Practice next

1. Estimate tokens from prompt length before IChatClient call.
2. Return 429 with Retry-After when bucket full.
3. Scale API replicas based on concurrent stream gauge.
4. Read actual usage from response.Usage instead of estimates.
5. Add priority queue for hospital tier tenants.

Remember

Scale horizontally for connections; queue for bulk AI work. Enforce per-tenant token buckets in Redis. Coordinate K8s HPA with provider TPM limits.

Tenant traffic spike

CRM customer runs company-wide copilot day — 10x normal TPM.

Outcome: Token bucket smooths calls; KEDA scales workers; no regional OpenAI ban.