Why Token Limits Silently Break AI Structured Output (And How to Fix It)
A hardcoded 1024-token limit truncates large JSON responses mid-object. JSON.parse throws. Your API returns 500. Users see nothing. This is a real engineering failure mode in production AI pipelines — and it is invisible until it hits.
Structured JSON output from language models is foundational to production AI applications. When it silently breaks, the failure mode is brutal: a truncated JSON string that looks like a response, fails JSON.parse, gets caught by a generic error handler, and returns a 500 to the user with no indication of what went wrong. The root cause is almost always a token limit set too low for the output the caller expects. This is not a hypothetical — it is a pattern we encountered in production with the SiteNexis Intelligence Report.
The Failure Mode in Detail
The Intelligence Report generates a structured JSON object with six sections, each containing a subscale score and 150–250 words of prose. The total token budget for the response is approximately 2000–3000 tokens. Our AI client function callAI had a hardcoded GROQ_MAX_TOKENS constant set to 1024 — appropriate for focused scoring tasks that return small JSON objects, but wrong for the full executive summary. Groq would return a response that ended mid-sentence in the middle of the third section. JSON.parse would throw a SyntaxError. The route would catch it and return 503. Users would see a blank Intelligence Report panel.
▲Groq does not return an error when it truncates a response at the token limit. It returns a normal 200 response with finish_reason: "length" — indicating the model stopped because it hit the limit, not because it completed the output. If your caller ignores finish_reason, you will get silently broken output.
The Fix: Caller-Specific maxTokens
The correct fix is to make maxTokens a parameter of the callAI function, not a module-level constant. Scoring functions that return small JSON objects can use 1024 tokens. The executive summary route passes 3000. The signature change is straightforward: callAI<T>(userPrompt, systemPrompt, maxTokens = DEFAULT_MAX_TOKENS). Each call site passes the token budget appropriate for what it expects back. The default constant still applies for all existing callers — no breaking changes.
The Double-Parse Antipattern
A secondary failure mode appeared in the same route: the executive summary was called as callAI<string> — requesting a raw string output — then passed through parseAIResponse<ExecutiveSummaryOutput> to decode the JSON. This created a double-parse chain: first the AI client tried to parse the response as a string (which it was), then the caller tried to parse that string as structured JSON. With a truncated response, neither parse succeeded cleanly. The fix: call callAI<ExecutiveSummaryOutput> directly, letting the built-in parsing handle the JSON decoding in one step. The type parameter on callAI controls the expected output shape — use it.
How to Detect This Before It Hits Production
- 1Log finish_reason from every model completion call — a "length" reason is a truncation event that should alert
- 2Write integration tests that generate realistic-size outputs and assert the full JSON schema is present
- 3Inspect the raw response string length before JSON.parse — a suspiciously short response for a large output schema is a red flag
- 4Use explicit type parameters on callAI to enforce structured output expectations at compile time
- 5For OpenRouter fallback paths: confirm maxTokens is passed through correctly — some proxy layers have their own defaults that override caller values
See how the SiteNexis Intelligence Report generates a complete structured executive summary — no truncation, no silent failures.
Try the Intelligence Report