Skip to content
Back to Blog
February 10, 202610 min read

Patterns for integrating Claude API in production

We've been shipping Anthropic's Claude API in production across multiple products for the better part of a year. The model is excellent. The wrapper around the model is what determines whether you have a usable feature or a flaky one. This post is the set of patterns we keep coming back to: what works, what bites you, and how to keep the bill predictable.

The default that doesn't scale

The starter pattern looks like this: user clicks a button, a route handler calls the Claude API with a prompt and the user's input, the response comes back, you render it. It works in development and on the first 100 users. Then it starts breaking in three predictable ways.

  • Latency is variable.Median is fine (say 1.2 seconds), but the p99 is 8 seconds. If your handler is synchronous, your serverless function timeout will hit before some responses arrive, especially on Vercel's default 10-second limit.
  • Rate limits are real.A traffic spike pushes you over your tokens-per-minute allowance and you start getting 429s. If your code doesn't handle them gracefully, users see errors.
  • Costs scale with input. Token usage is dominated by the input (the prompt + any context you stuff in there), much more than by the output. A naive integration that re-sends the same context with every request burns money fast.

Pattern 1: Stream the response

For anything user-facing, stream. The Anthropic SDK supports SSE streaming and Next.js' route handlers + ReadableStream make this almost trivial:

// app/api/chat/route.ts
import Anthropic from "@anthropic-ai/sdk";

export const runtime = "edge";

export async function POST(req: Request) {
  const { messages } = await req.json();
  const client = new Anthropic();

  const stream = await client.messages.stream({
    model: "claude-sonnet-4-6",
    max_tokens: 1024,
    messages,
  });

  return new Response(stream.toReadableStream(), {
    headers: { "Content-Type": "text/event-stream" },
  });
}

Streaming solves the latency-perception problem entirely. The user sees the first token in 300-500ms even on a long generation, and nobody cares that the full response took 4 seconds because they were already reading.

Pattern 2: Use Haiku for the cheap stuff

Not every AI call needs the most capable model. We default to Sonnet for anything user-facing where quality matters. We use Haiku for:

  • Classification ("is this transaction a rent payment, an expense, or something else?")
  • Short summaries ("summarize this email in one sentence")
  • Quick rewrites ("make this paragraph more concise")
  • Anything that runs in a background job at scale

Haiku is roughly 10× cheaper than Sonnet and roughly 3× faster. For the right tasks, you can't tell the difference in output quality, and the cost savings let you say yes to features you'd otherwise have to gate.

Pattern 3: Cache prompts with prompt caching

Anthropic's prompt caching is the single biggest cost lever we've found. If your prompt has a long system message, a document the user uploaded, or a long context block that's reused across multiple turns, marking it as cached drops the price of those tokens by 90% on subsequent requests.

For example, in PropertyTools.ai, when a user is iterating on a tenant email (try this draft, no try that draft, what about more formal), we cache the lease summary, the tenant's payment history, and the system prompt. Each subsequent draft only pays for the new instruction and the generated output. A 5-turn iteration session costs about 1.5× a single uncached call instead of 5×.

Pattern 4: Validate structured outputs

When you ask Claude for JSON, ask it to use the JSON output mode and validate the result with Zod (or your validator of choice) before using it. Even with the JSON output mode, you can occasionally get edge cases: extra prose around the JSON, missing fields, the wrong type for a field. Validate, and on failure, retry once with the validation error appended to the prompt.

const Schema = z.object({
  category: z.enum(["rent", "expense", "other"]),
  confidence: z.number().min(0).max(1),
});

let parsed = Schema.safeParse(JSON.parse(response));
if (!parsed.success) {
  // Retry once with the error fed back
  const retry = await client.messages.create({
    ...,
    messages: [
      ...originalMessages,
      { role: "assistant", content: response },
      { role: "user", content: `Validation failed: ${parsed.error.message}. Return only valid JSON matching the schema.` },
    ],
  });
  parsed = Schema.safeParse(JSON.parse(extractJson(retry)));
}

The retry loop catches almost all the failures we've seen in production. We log every retry as a warning so we can spot prompts that need to be fixed at the source.

Pattern 5: Background jobs for batch work

If a user uploads a 50-page PDF and you want Claude to summarize it, don't do that in the request handler. Drop a job into a queue (we use Inngest for most projects) and let it run async. Notify the user when it's done: email, in-app notification, whatever fits.

This also lets you batch. Group 10 transaction-categorization requests into a single Claude call instead of 10 sequential calls. You amortize the system-prompt overhead across all 10 inputs.

Pattern 6: Respect rate limits with exponential backoff

On 429 (rate limit) responses, back off and retry. The Anthropic SDK does some of this for you, but for production we wrap calls in our own retry layer with jitter:

async function callWithRetry(fn, maxRetries = 3) {
  for (let i = 0; i <= maxRetries; i++) {
    try { return await fn(); }
    catch (err) {
      if (err.status !== 429 && err.status !== 529) throw err;
      if (i === maxRetries) throw err;
      const delay = Math.min(1000 * 2 ** i, 8000) + Math.random() * 500;
      await new Promise((r) => setTimeout(r, delay));
    }
  }
}

529 (overloaded) is the other one to handle. It means Anthropic's side is briefly saturated; retry shortly. Anything non-2xx that's not 429 or 529 should propagate to your error tracking.

Pattern 7: Observability is non-optional

Log every Claude call with: prompt size in tokens, response size in tokens, latency, the model used, the user / org ID. Push these to whatever you use for structured logs (we use Axiom on most projects). When something goes wrong, you want to be able to answer "was this slow because of us, or because of the model?" and "is one customer eating all our token budget?" in seconds, not days.

For cost monitoring specifically, we ship a per-org token-usage dashboard so we can see who the heavy users are and price the product accordingly.

What we don't do

A few things we've consciously steered away from:

  • We don't fine-tune.Prompt engineering plus examples in the system prompt has been enough for every use case we've hit. Fine-tuning is expensive, freezes you to a model version, and the rate of base-model improvement makes the ROI questionable.
  • We don't chain models. If you find yourself building a 5-step LLM pipeline, you usually have a prompt-design problem, not a chaining problem. Claude is good enough that one well-designed call beats five mediocre ones.
  • We don't expose raw model output. Every Claude response goes through at least a render layer that sanitizes, validates, or formats. The user never sees raw markdown-with-think-tags.

The summary

Stream user-facing responses. Use the cheapest model that works. Cache aggressive. Validate structured output and retry on failure. Background-job the batch work. Back off on rate limits. Log everything. Resist the urge to over-architect. The model is capable; the wrapper should be boring.

If you're integrating Claude (or any LLM) into a product and the API edges are getting in the way of shipping, we've done this enough times to help you skip the expensive lessons.