Integrating OpenAI Into .NET Applications: Streaming, Retries, and Production Patterns

A production OpenAI integration in .NET differs from a working demo in three specific ways: it streams responses instead of waiting for the full completion, it retries transient failures with backoff instead of surfacing errors to the user immediately, and it enforces token budgets before requests go out rather than discovering cost problems after the invoice arrives. Here's how each of those actually gets built.

Streaming: Why It's Not Optional for Production

A non-streamed OpenAI request means your user stares at a loading spinner for however long the full response takes to generate — often several seconds for anything non-trivial. Streaming the response token-by-token as it's generated, and rendering it incrementally on the frontend, is the difference between an AI feature that feels responsive and one that feels broken, even though the total generation time is identical either way.

In .NET, this means using the streaming variant of the chat completion call and processing the response as an async stream, forwarding chunks to the client as they arrive — typically over Server-Sent Events or a WebSocket connection — rather than awaiting the entire completion before sending anything back.

Retries: Handling Rate Limits and Transient Failures Gracefully

OpenAI's API returns rate limit errors (HTTP 429) and occasional transient server errors under normal production load — this isn't a sign something's wrong, it's expected behavior at scale that needs to be handled, not treated as a fatal error. A production integration needs exponential backoff with jitter on retryable errors, distinguishing them from genuinely fatal errors (invalid API key, malformed request) that shouldn't be retried at all since retrying them just wastes time before failing anyway.

Getting this distinction wrong in either direction causes real problems: retrying non-retryable errors wastes time and delays the actual error surfacing to the user; failing to retry legitimate rate-limit errors surfaces spurious failures to users during periods that would have succeeded seconds later with a retry.

Token Budgeting: Controlling Cost Before It's a Problem

The demo-to-production gap that costs real money: a demo happily sends an entire conversation history or an entire document as context on every single request. In production, this needs active management — truncating or summarizing conversation history once it exceeds a defined token budget, and setting explicit max_tokens limits on responses rather than letting the model generate as long as it wants.

For document-processing or RAG-style use cases specifically, this means chunking documents deliberately and retrieving only the relevant chunks for a given query, rather than stuffing an entire document into context on every request — both for cost control and because model performance on very long contexts is often worse than on focused, relevant context.

Prompt Patterns Worth Standardizing

Rather than hand-writing prompts inline at each call site, production systems benefit from a small internal library of reusable prompt templates with clearly defined inputs — this makes prompt behavior consistent across features, makes it possible to update a prompt's behavior in one place rather than hunting through the codebase, and makes it much easier to add logging and evaluation around specific prompt templates over time.

Error Handling That Doesn't Leak to Users

When something does fail after retries are exhausted, users should see a clear, actionable message — not a raw API error message or a silent hang. Logging the actual error details server-side for debugging, while returning a clean fallback message client-side, is a small detail that meaningfully affects how an AI feature feels when things occasionally go wrong (which, at scale, they will).

FAQ

If you're building an AI feature and want it to behave like a production system rather than a demo that breaks under real usage, get in touch.