Adding an LLM Feature to a Node.js API — A Practical Integration Pattern

Don't Hardcode One Provider

The first mistake is calling openai.chat.completions.create() directly from route handlers scattered across the codebase. Providers change pricing, rate limits, and availability — wrap the call behind a small interface from day one:

// lib/llm/client.ts
interface LLMProvider {
  complete(prompt: string, opts?: { json?: boolean }): Promise<string>
}
 
class OpenAIProvider implements LLMProvider {
  async complete(prompt: string, opts?: { json?: boolean }) {
    const res = await openai.chat.completions.create({
      model: 'gpt-4o-mini',
      messages: [{ role: 'user', content: prompt }],
      response_format: opts?.json ? { type: 'json_object' } : undefined,
    })
    return res.choices[0].message.content ?? ''
  }
}
 
class ClaudeProvider implements LLMProvider {
  async complete(prompt: string, opts?: { json?: boolean }) {
    const res = await anthropic.messages.create({
      model: 'claude-sonnet-4-5',
      max_tokens: 1024,
      messages: [{ role: 'user', content: prompt }],
    })
    return res.content[0].type === 'text' ? res.content[0].text : ''
  }
}
 
export const llm: LLMProvider = new OpenAIProvider() // swap without touching call sites

This paid off directly on an AI CV-parsing feature I built — the provider changed mid-project for cost reasons, and it was a one-line swap instead of a rewrite.

Structured Output, Not Regex-Parsed Prose

If you need the model to return data your code will consume (not just display), don't ask for prose and regex it apart. Use JSON mode / structured output and validate what comes back:

import { z } from 'zod'
 
const ExtractedProfile = z.object({
  name: z.string(),
  yearsExperience: z.number(),
  skills: z.array(z.string()),
})
 
async function extractProfile(resumeText: string) {
  const raw = await llm.complete(
    `Extract name, years of experience, and skills as JSON from this resume:\n\n${resumeText}`,
    { json: true }
  )
  const parsed = ExtractedProfile.safeParse(JSON.parse(raw))
  if (!parsed.success) {
    throw new Error(`LLM returned unexpected shape: ${parsed.error.message}`)
  }
  return parsed.data
}

The safeParse step matters — models occasionally drop a field or return the wrong type even in JSON mode. Validate before your code trusts the shape.

Streaming for Anything User-Facing

If a response takes more than ~1-2 seconds and a user is waiting on it, stream it. Node.js APIs can proxy the provider's stream straight through as Server-Sent Events:

// routes/chat.ts
app.get('/api/chat/stream', async (req, res) => {
  res.setHeader('Content-Type', 'text/event-stream')
  res.setHeader('Cache-Control', 'no-cache')
  res.setHeader('Connection', 'keep-alive')
 
  const stream = await openai.chat.completions.create({
    model: 'gpt-4o-mini',
    messages: [{ role: 'user', content: req.query.prompt as string }],
    stream: true,
  })
 
  for await (const chunk of stream) {
    const token = chunk.choices[0]?.delta?.content ?? ''
    if (token) res.write(`data: ${JSON.stringify({ token })}\n\n`)
  }
  res.write('data: [DONE]\n\n')
  res.end()
})

Nginx needs proxy_buffering off on this route, or it'll buffer the whole response before forwarding — silently turning your stream back into a blocking call.

Rate Limits and Retries Are Not Optional

Providers throttle you, and they will do it in production at the worst time. A bare fetch call with no retry logic means one 429 becomes a user-facing error:

async function withRetry<T>(fn: () => Promise<T>, attempts = 3): Promise<T> {
  for (let i = 0; i < attempts; i++) {
    try {
      return await fn()
    } catch (err: any) {
      if (err.status === 429 && i < attempts - 1) {
        await new Promise((r) => setTimeout(r, 2 ** i * 1000))
        continue
      }
      throw err
    }
  }
  throw new Error('unreachable')
}

Exponential backoff on 429s, and a hard cap so a genuinely down provider fails fast instead of hanging a request queue.

Cost Control: Cache and Cap

LLM calls are the most expensive thing in most API request paths. Two guards that pay for themselves quickly:

• Cache identical prompts where the input is likely to repeat (same document reprocessed, same classification question) — a Redis cache keyed on a hash of the prompt avoids paying twice for the same answer.
• Cap max_tokens deliberately per use case instead of leaving it unset — an open-ended generation task with no cap is how a single bad request turns into a large bill.

const cacheKey = `llm:${crypto.createHash('sha256').update(prompt).digest('hex')}`
const cached = await redis.get(cacheKey)
if (cached) return cached
 
const result = await llm.complete(prompt)
await redis.set(cacheKey, result, 'EX', 60 * 60 * 24) // 24h

What Actually Breaks This In Practice

• No timeout on the provider call. If the SDK doesn't set one, a hung connection can tie up a request handler indefinitely. Set an explicit timeout and treat it as a failure path, not an edge case.
• Treating the model's output as always well-formed. Even with JSON mode, validate before trusting the shape — see the Zod example above.
• No fallback when the primary provider has an outage. Providers do go down. The interface pattern above makes a fallback provider a config change, not a rewrite, if you've already abstracted the call.
• Logging full prompts/responses containing user PII without thinking about it. Decide what's safe to log before you ship, not after a data review flags it.

Checklist

• LLM calls go through a provider interface, not scattered openai.*/anthropic.* calls
• Structured output is validated (Zod or similar) before your code trusts it
• User-facing responses stream; proxy_buffering off is set where needed
• Retries with backoff on rate limits; a hard timeout on every call
• Repeatable prompts are cached; max_tokens is capped per use case
• A fallback provider path exists, even if it's not wired up by default

The interesting engineering problem in "adding AI" to an API isn't the API call — it's making that call behave like the rest of your reliable, observable, cost-aware backend.