Claude Code with Groq: Fast Inference Done Right
Why Groq without CLAUDE.md ships inference that breaks on the second model change
Groq runs open-weight language models on custom inference hardware that returns tokens at speeds well past what general-purpose GPUs deliver. The product is GroqCloud: an OpenAI-compatible API in front of models like the Llama family, Mixtral, and Qwen, billed per token, fast enough that the latency stops being the bottleneck in a generation pipeline. The appeal is obvious for any feature where time-to-first-token matters: chat, autocomplete, agentic loops, anything a user waits on. The catch is that "OpenAI-compatible" tempts an assistant into treating Groq like a drop-in clone, which papers over the places the two differ and the operational realities a production integration has to handle.
A Claude Code session that wires up Groq without project rules produces a client that sends a prompt and prints the response, which looks complete in a terminal. What it leaves out is everything that keeps the integration alive in production. It hardcodes a model ID that Groq deprecates on a known schedule, so the feature returns 404 the day the model retires. It ignores the rate-limit headers Groq returns on every response, so the integration finds out it is throttled by failing rather than by backing off. It puts the API key in code that ships to the browser. It treats streaming and non-streaming as the same call shape, so the streaming path drops the final usage record. Each of these is fine in a demo and a liability in a service, and none of them surfaces until the second model retirement or the first traffic spike.
This guide covers the CLAUDE.md configuration that locks Claude Code into Groq patterns that survive production: exact model IDs pinned against deprecation, rate-limit handling driven by the headers Groq actually returns, a clean split between streaming and batch paths, the API key kept server-side, and a fallback chain for when a model retires or the service is busy. For the local-inference alternative when you want models running on your own hardware, Claude Code with Ollama covers the patterns for self-hosted models. For the multi-provider abstraction that routes across Groq and others, Claude Code with the Vercel AI SDK covers the provider-agnostic layer.
The Groq CLAUDE.md template
The CLAUDE.md at your project root is read at the start of every Claude Code session. For a Groq integration it needs to declare: the client library, the pinned model IDs, the rate-limit policy, the streaming conventions, the fallback chain, the key handling rule, and the hard rules that block the failure modes Claude introduces most often.
# Groq rules
## Client
- groq-sdk (official) OR openai SDK pointed at api.groq.com/openai/v1
- Base URL: https://api.groq.com/openai/v1 when using the openai SDK
- API key from GROQ_API_KEY env, server-side ONLY
## Models (MANDATORY)
- Pin exact model IDs in a config module, NEVER inline string literals
- Primary, fallback, and cheap models declared as named constants
- Check the Groq models endpoint in CI to catch deprecations early
- NEVER assume an OpenAI model name maps to a Groq model
## Rate limits (MANDATORY)
- Read x-ratelimit-remaining-* headers on every response
- On 429, respect retry-after, exponential backoff with jitter after that
- Token budget tracked per request, requests sized to the model context
- NEVER retry a 429 immediately without backoff
## Streaming
- Streaming and non-streaming are SEPARATE call paths, not one with a flag
- Streaming path accumulates the final usage chunk, never drops it
- Aborted streams clean up the connection (AbortController)
- SSE parsing handles the [DONE] sentinel explicitly
## Fallback chain
- Primary model -> fallback model on 404/503, cheap model on budget pressure
- Fallback is explicit and logged, never silent
- A total failure returns a typed error, never a fabricated response
## Hard rules
- NEVER hardcode a model ID outside the model config module
- NEVER ship GROQ_API_KEY to a browser bundle
- NEVER ignore the rate-limit headers on a response
- NEVER retry a 429 without honoring retry-after first
- NEVER treat streaming and batch as the same code path
- NEVER fabricate output when every model in the chain fails
- ALWAYS set a request timeout, never rely on the default
- ALWAYS log which model actually served a request
Four rules here prevent the majority of the production failures Claude generates without them.
The model pinning rule is the one that breaks integrations months after they ship. Groq retires older model versions on a published schedule, and a hardcoded model string keeps working right up to the retirement date, then returns a 404 with no warning. The rule moves every model ID into a single config module with named constants for the primary, fallback, and cheap models, so a deprecation is a one-line change and a CI check against the models endpoint catches it before users do.
The rate-limit header rule turns throttling from a surprise into a managed condition. Groq returns x-ratelimit-remaining-requests and x-ratelimit-remaining-tokens on every response and a retry-after on a 429. An integration that reads these headers can slow down before it hits the wall; one that ignores them only learns it is throttled when a request fails. The rule requires reading the headers and honouring retry-after before any backoff.
The streaming separation rule prevents the subtle bug where the final usage record is lost. Streaming responses arrive as server-sent events ending in a [DONE] sentinel, with the token usage in the last data chunk before it. Code that treats streaming as "the same call with stream: true" tends to stop reading at the first content chunk and never captures usage, so billing and budget tracking silently break. The rule keeps the two paths separate so each handles its own response shape correctly.
Install and the model config module
Install the official Groq SDK or the OpenAI SDK pointed at the Groq base URL. Both work; the official SDK tracks Groq-specific features more closely.
# Official Groq SDK
npm install groq-sdk
# OpenAI SDK pointed at Groq
npm install openai
The first thing to build, before any call site, is the model config module the CLAUDE.md mandates. This is the single place every model ID lives.
// src/groq/models.ts
export const MODELS = {
// Pin exact IDs. Update here when Groq publishes a deprecation.
primary: "llama-3.3-70b-versatile",
fallback: "llama-3.1-8b-instant",
cheap: "llama-3.1-8b-instant",
} as const;
export type ModelKey = keyof typeof MODELS;
Every call site references MODELS.primary rather than a string literal. When Groq retires a model, the fix is one line in this file and the CI check confirms the new ID resolves. Without this module, model IDs scatter across the codebase and a deprecation becomes a search-and-replace through every file that calls the API.
// src/groq/client.ts
import Groq from "groq-sdk";
export const groq = new Groq({
apiKey: process.env.GROQ_API_KEY, // server-side only
timeout: 30_000, // explicit, never the default
maxRetries: 0, // we handle retries ourselves
});
The client sets an explicit 30-second timeout and disables the SDK's built-in retries because the integration handles retry logic itself, where it can read the rate-limit headers and apply the fallback chain. Letting the SDK retry blindly would mask the 429 the rate-limit handler needs to see.
Calls that read the rate-limit headers
A Groq response carries the rate-limit state in its headers. The integration reads them on every call so it can back off before it gets throttled rather than after.
// src/groq/complete.ts
import { groq } from "./client";
import { MODELS, ModelKey } from "./models";
export async function complete(prompt: string, model: ModelKey = "primary") {
const res = await groq.chat.completions
.create({
model: MODELS[model],
messages: [{ role: "user", content: prompt }],
temperature: 0.2,
})
.withResponse();
const remaining = Number(res.response.headers.get("x-ratelimit-remaining-requests") ?? "0");
if (remaining < 5) {
logger.warn("groq rate budget low", { remaining, model: MODELS[model] });
}
return {
text: res.data.choices[0]?.message?.content ?? "",
model: MODELS[model], // log which model served it
usage: res.data.usage,
};
}
The .withResponse() gives access to the raw headers alongside the parsed body. The handler reads the remaining-requests count and logs a warning when the budget runs low, which gives operations a signal before the integration starts failing. It returns the model that actually served the request, satisfying the rule that every call logs its model. This is the visibility that turns a rate-limit incident from a mystery into a known condition.
Handling 429 with backoff and fallback
When a request is throttled or a model is unavailable, the integration honours retry-after, applies exponential backoff with jitter, and falls back through the model chain. The CLAUDE.md rule is that the fallback is explicit and logged, never silent.
// src/groq/resilient.ts
import { complete } from "./complete";
import { ModelKey } from "./models";
const CHAIN: ModelKey[] = ["primary", "fallback", "cheap"];
export async function resilientComplete(prompt: string) {
let lastErr: unknown;
for (const model of CHAIN) {
for (let attempt = 0; attempt < 3; attempt++) {
try {
return await complete(prompt, model);
} catch (err: unknown) {
lastErr = err;
const e = err as { status?: number; headers?: Headers };
if (e.status === 429) {
const retryAfter = Number(e.headers?.get("retry-after") ?? "1") * 1000;
const backoff = retryAfter + Math.random() * 250 * 2 ** attempt;
logger.warn("groq throttled, backing off", { model, backoff });
await sleep(backoff);
continue;
}
if (e.status === 404 || e.status === 503) {
logger.warn("groq model unavailable, falling back", { model });
break; // move to next model in chain
}
throw err; // non-retryable
}
}
}
throw new GroqUnavailableError("all models exhausted", { cause: lastErr });
}
The handler distinguishes three cases. A 429 means throttled, so it waits the server's retry-after plus jittered backoff and retries the same model. A 404 or 503 means the model is gone or busy, so it breaks out and moves to the next model in the chain. Any other error is non-retryable and rethrows immediately. When every model in the chain is exhausted, it throws a typed GroqUnavailableError rather than returning empty or fabricated text, which is the rule that keeps a downstream caller from treating a failure as a valid answer.
Streaming as its own path
Streaming is a different response shape and gets its own function. The CLAUDE.md rule is that the streaming path accumulates the final usage chunk and cleans up aborted connections.
// src/groq/stream.ts
import { groq } from "./client";
import { MODELS } from "./models";
export async function* streamComplete(prompt: string, signal?: AbortSignal) {
const stream = await groq.chat.completions.create(
{
model: MODELS.primary,
messages: [{ role: "user", content: prompt }],
stream: true,
stream_options: { include_usage: true },
},
{ signal }
);
let usage;
for await (const chunk of stream) {
const delta = chunk.choices[0]?.delta?.content;
if (delta) yield { type: "token" as const, value: delta };
if (chunk.usage) usage = chunk.usage; // final chunk carries usage
}
yield { type: "done" as const, usage };
}
The stream_options: { include_usage: true } tells Groq to send a final chunk with the token usage, which the loop captures into usage and emits in the terminating done event. The AbortSignal lets a caller cancel the stream, closing the connection cleanly instead of leaking it. A naive streaming implementation yields tokens and returns, dropping the usage record and leaving the budget tracker blind, which is exactly the bug the separation rule prevents.
The caller consumes the generator and reacts to each event type.
for await (const event of streamComplete(prompt, controller.signal)) {
if (event.type === "token") process.stdout.write(event.value);
if (event.type === "done") logger.info("stream complete", { usage: event.usage });
}
The token events drive the user-facing output and the done event records usage, so the streaming path tracks consumption as accurately as the batch path.
Keeping the key server-side
The API key handling rule exists because an OpenAI-compatible client is easy to instantiate in the wrong place. A key in a browser bundle is a key anyone can extract and bill against your account.
// app/api/chat/route.ts -- server-side route, key never leaves here
import { resilientComplete } from "@/groq/resilient";
export async function POST(req: Request) {
const { prompt } = await req.json();
const result = await resilientComplete(prompt);
return Response.json({ text: result.text, model: result.model });
}
The Groq client lives in a server route that the browser calls over HTTP. The key stays in the server environment, the browser receives only the generated text and the model name, and there is no path that exposes the credential to a client. For a Next.js app this is an API route or a server action; for any other framework it is the equivalent server-only boundary. Claude reads the rule from CLAUDE.md and keeps the client out of any module the frontend bundles.
Permission hooks for Groq operations
Groq operations are mostly read-only API calls, but a careless script can run up real cost by hammering the API or pulling large completions in a loop. Permission hooks in .claude/settings.local.json keep an automated session inside safe bounds.
{
"permissions": {
"allow": [
"Bash(curl https://api.groq.com/openai/v1/models*)",
"Bash(npm run groq:check-models*)"
],
"deny": [
"Bash(*GROQ_API_KEY*echo*)",
"Bash(*for*curl*api.groq.com*)",
"Bash(*while*curl*api.groq.com*)"
]
}
}
The allow list permits the read-only model listing Claude needs to verify model IDs and the CI check script. The deny list blocks any attempt to echo the API key and any shell loop that would batch-call the API, which is the pattern that turns into an unexpected bill. Combined with the Claude Code permissions model for the full policy surface, the result is a setup where Claude can inspect and validate the integration without being able to run up cost.
Common Claude Code mistakes with Groq
Six patterns Claude generates incorrectly without CLAUDE.md constraints.
1. Hardcoded model ID
Claude generates: model: "llama-3.3-70b-versatile" inline at the call site.
Correct pattern: reference MODELS.primary from the model config module so a deprecation is one line.
2. Ignored rate-limit headers
Claude generates: a call that reads only the body and never the headers.
Correct pattern: .withResponse() and read x-ratelimit-remaining-* to back off before failing.
3. Immediate 429 retry
Claude generates: a retry loop that retries instantly on 429.
Correct pattern: honour retry-after, then exponential backoff with jitter.
4. Streaming drops usage
Claude generates: a stream loop that yields tokens and returns, losing the usage chunk.
Correct pattern: include_usage: true and capture the final chunk before the loop ends.
5. API key in the browser bundle
Claude generates: a Groq client instantiated in client-side code.
Correct pattern: client in a server route, browser receives only the result.
6. Silent fabrication on total failure
Claude generates: returning empty text when every model fails.
Correct pattern: throw a typed error so the caller never treats a failure as an answer.
Add each pattern to CLAUDE.md with the corrected form. Combined with CLAUDE.md examples for the general structure, the result is Groq integration code Claude generates correctly the first time.
Testing the Groq integration
The tests that matter mock the HTTP layer and assert on behaviour under failure: that a 429 backs off, that a 404 falls back, that streaming captures usage. Testing against the live API is slow, costs money, and cannot reproduce a rate limit on demand.
import { describe, it, expect, vi } from "vitest";
import { resilientComplete } from "../src/groq/resilient";
describe("resilient completion", () => {
it("falls back to the next model on 404", async () => {
const calls: string[] = [];
mockGroq((model) => {
calls.push(model);
if (model === "llama-3.3-70b-versatile") throw { status: 404 };
return { choices: [{ message: { content: "ok" } }] };
});
const res = await resilientComplete("hi");
expect(res.text).toBe("ok");
expect(calls).toContain("llama-3.1-8b-instant");
});
it("throws a typed error when all models fail", async () => {
mockGroq(() => { throw { status: 503 }; });
await expect(resilientComplete("hi")).rejects.toThrow("all models exhausted");
});
});
The first test confirms the fallback chain advances to the next model when the primary returns 404. The second confirms that total failure throws the typed error instead of fabricating output. These are the behaviours the CLAUDE.md rules promise, verified in CI. For the wider testing setup, Claude Code with Vitest covers the configuration.
Building Groq inference that stays up
The Groq CLAUDE.md in this guide produces an integration where every model ID lives in one config module, every response is read for rate-limit headers, a 429 backs off on the server's schedule, a retired model falls back through an explicit chain, streaming captures its usage record, and a total failure throws rather than fabricates. The result is a fast-inference layer that keeps serving when a model retires or traffic spikes, instead of one that worked in the demo and broke in production.
The principle is the same as any external-service integration with Claude Code. Groq without a CLAUDE.md produces a client that works against today's models under today's load and conceals the operational realities that show up on the next model retirement or the first throttle. The CLAUDE.md is what makes the resilient patterns the default for Claude.
For local inference on your own hardware, Claude Code with Ollama covers self-hosted models. For the multi-provider abstraction that routes across Groq and others, Claude Code with the Vercel AI SDK covers the provider-agnostic layer. For the hosted-inference alternative with a broader model catalog, Claude Code with Replicate covers running open models behind an API.
Get Claudify. Ship a Groq integration that degrades gracefully when a model retires or the rate limit hits.
More like this
Ready to upgrade your Claude Code setup?
Get Claudify
