Claude Code with Mistral: A Pragmatic Model Router
Why a Mistral app without CLAUDE.md burns money on the wrong model
Mistral ships a family of models that span a wide cost and capability range: small models that handle classification and extraction for a fraction of a cent, mid-tier models that handle most production reasoning, and large models for the hard problems. The whole point of the family is that you route each task to the cheapest model that can do the job. The failure mode is that nothing in the API forces you to. Every call looks the same regardless of which model string you pass, so the difference between a bill that scales linearly with traffic and one that grows ten times faster is a single field most code sets once and forgets.
A Claude Code session that writes Mistral integration code without project rules reaches for the most capable model by default, because that is the safe answer when the task is unknown. It calls mistral-large-latest for a sentiment classifier that a small model handles for a hundredth of the cost. It parses the model's text output with a regex instead of asking for structured JSON. It fires requests with no timeout, so a slow response blocks a request handler indefinitely. It calls the embeddings endpoint inside a loop with no batching, turning one API call into a thousand. None of these are wrong in the sense of producing incorrect output. They are wrong in the sense that the same behaviour costs ten times more and fails ten times harder under load.
This guide covers the CLAUDE.md configuration that locks Claude Code into Mistral patterns that hold up in production: a model selection table that routes each task type to the right tier, structured output as the default for anything machine-parsed, a shared client with retry and timeout policy, batched embeddings, and permission hooks that keep Claude from running scripts that hammer the paid endpoints during development. For the agent framework that orchestrates multi-step Mistral calls, Claude Code with LangChain covers the chains and tools layer. For the application runtime, Claude Code with FastAPI covers the Python service that wraps these calls.
The Mistral CLAUDE.md template
The CLAUDE.md at your project root is read at the start of every Claude Code session. For a Mistral-backed application it needs to declare the SDK version, the model selection rules, the structured output policy, the retry and timeout baseline, the embeddings batching rule, and the hard rules that block the cost mistakes Claude makes most often.
# Mistral rules
## Stack
- mistralai Python SDK 1.x (or @mistralai/mistralai for TypeScript)
- All calls go through src/llm/client.py, NEVER instantiate Mistral() inline
- API key from MISTRAL_API_KEY env var, NEVER hardcoded
- Default region: codestral.mistral.ai for code, api.mistral.ai for chat
## Model selection (MANDATORY)
- Classification, extraction, routing, short rewrites: mistral-small-latest
- General reasoning, summarization, RAG answers: mistral-medium-latest
- Hard reasoning, long-context synthesis, agentic planning: mistral-large-latest
- Code generation and completion: codestral-latest
- Embeddings: mistral-embed
- NEVER default to mistral-large for a task a smaller model handles
- Document the chosen model with a comment explaining why
## Structured output (MANDATORY)
- Anything parsed by code MUST use response_format json_object
- Define a Pydantic / Zod schema for every structured response
- NEVER parse model text output with regex or string splitting
- Validate the parsed JSON against the schema before use
## Reliability (MANDATORY)
- Every call sets timeout_ms explicitly (default 30000)
- Retry on 429 and 5xx with exponential backoff, max 3 attempts
- Respect Retry-After header on 429 responses
- Circuit-break after repeated failures, fall back to cached or degraded mode
## Embeddings (MANDATORY)
- Batch inputs into a single call, max 512 per request
- NEVER call mistral-embed inside a per-item loop
- Cache embeddings keyed by content hash, NEVER re-embed unchanged text
## Cost controls
- max_tokens set on every completion call, NEVER unbounded
- Token usage logged per call for cost attribution
- Streaming used for user-facing responses to reduce perceived latency
## Hard rules
- NEVER instantiate the Mistral client outside src/llm/client.py
- NEVER use mistral-large for classification or extraction
- NEVER parse a structured response without json_object mode
- NEVER call an LLM endpoint without a timeout
- NEVER embed inside a loop, always batch
- NEVER log the full prompt if it contains user PII
- ALWAYS set max_tokens to bound the response cost
- ALWAYS validate structured output against a schema
Six rules here prevent the majority of incidents Claude generates without them.
The model selection rule is the one that controls the bill. Without it, Claude treats every task as if it needs the largest model, because that is the conservative default when the requirements are unstated. The table gives Claude a deterministic answer for each task type: a sentiment classifier gets mistral-small-latest, a RAG answer gets mistral-medium-latest, an agentic planning step gets mistral-large-latest. The comment requirement forces the reasoning to be visible, so a reviewer can challenge a choice that does not fit.
The structured output rule kills the regex parser. When a downstream system consumes the model's output, the output has to be machine-readable. Without the rule, Claude generates code that prompts for "JSON only" and then parses the text with brittle string operations that break the first time the model wraps the JSON in a markdown fence. With the rule, every machine-parsed response uses json_object mode and validates against a schema.
The embeddings batching rule prevents the silent thousand-call loop. The natural shape for embedding a list of documents is a for loop with one API call per document, which Claude generates because it reads cleanly. The right shape is a single batched call. The difference is a thousand round trips versus one, which shows up as both latency and rate-limit pressure.
The shared client
Every call goes through one module. This is where the retry policy, timeout, and logging live, so no individual call site has to remember them.
# src/llm/client.py
import hashlib
import logging
import os
import time
from functools import lru_cache
from mistralai import Mistral
from mistralai.models import SDKError
logger = logging.getLogger("llm")
_DEFAULT_TIMEOUT_MS = 30_000
_MAX_RETRIES = 3
@lru_cache(maxsize=1)
def _client() -> Mistral:
key = os.environ["MISTRAL_API_KEY"]
return Mistral(api_key=key, timeout_ms=_DEFAULT_TIMEOUT_MS)
def complete(
model: str,
messages: list[dict],
max_tokens: int,
response_format: dict | None = None,
temperature: float = 0.2,
) -> str:
client = _client()
last_error: Exception | None = None
for attempt in range(_MAX_RETRIES):
try:
resp = client.chat.complete(
model=model,
messages=messages,
max_tokens=max_tokens,
temperature=temperature,
response_format=response_format,
)
usage = resp.usage
logger.info(
"mistral call model=%s prompt_tokens=%s completion_tokens=%s",
model,
usage.prompt_tokens,
usage.completion_tokens,
)
return resp.choices[0].message.content
except SDKError as exc:
last_error = exc
if exc.status_code == 429 or exc.status_code >= 500:
backoff = (2 ** attempt) + (0.1 * attempt)
logger.warning(
"mistral retry attempt=%s status=%s backoff=%.1fs",
attempt,
exc.status_code,
backoff,
)
time.sleep(backoff)
continue
raise
raise RuntimeError(f"mistral call failed after {_MAX_RETRIES} attempts") from last_error
The client is created once and cached. The complete function takes the model as a required argument, which means the call site has to make a deliberate choice rather than inherit a default. Every call logs its token usage so cost can be attributed per feature. The retry loop only retries on 429 and 5xx, never on a 4xx that indicates a bad request, because retrying a malformed prompt just wastes three calls before failing.
The max_tokens argument is required with no default. This forces every call site to think about how long the response should be, which is the single most effective cost control. A summarization task that should produce two sentences gets max_tokens=120, not the implicit ceiling of the context window.
Model selection in practice
The CLAUDE.md table tells Claude which model to pick. In code, make the routing explicit with named functions so the choice is visible at the call site.
# src/llm/tasks.py
from src.llm.client import complete
_CLASSIFY = "mistral-small-latest"
_REASON = "mistral-medium-latest"
_PLAN = "mistral-large-latest"
def classify_intent(text: str) -> str:
"""Small model: cheap, fast, sufficient for routing a support ticket."""
messages = [
{"role": "system", "content": "Classify the support ticket intent."},
{"role": "user", "content": text},
]
return complete(
model=_CLASSIFY,
messages=messages,
max_tokens=20,
response_format={"type": "json_object"},
)
def answer_question(question: str, context: str) -> str:
"""Medium model: the workhorse for RAG answers."""
messages = [
{"role": "system", "content": f"Answer using only this context:\n{context}"},
{"role": "user", "content": question},
]
return complete(model=_REASON, messages=messages, max_tokens=600)
def plan_actions(goal: str, tools: str) -> str:
"""Large model: reserved for multi-step planning where it earns its cost."""
messages = [
{"role": "system", "content": f"Plan the steps. Available tools:\n{tools}"},
{"role": "user", "content": goal},
]
return complete(
model=_PLAN,
messages=messages,
max_tokens=1500,
response_format={"type": "json_object"},
)
The model constants are named by task, not by size, so a reader sees _CLASSIFY and understands the intent. The classifier uses max_tokens=20 because the answer is a single label. The RAG answer uses max_tokens=600 because it produces a paragraph. The planner uses max_tokens=1500 and the large model because multi-step planning is where the larger model's reasoning pays for itself. This is the routing discipline the CLAUDE.md enforces, expressed as code a reviewer can audit.
Structured output with a schema
Anything a downstream system parses needs structured output. The pattern is a Pydantic model that defines the shape, json_object mode on the call, and validation on the response.
# src/llm/extract.py
import json
from pydantic import BaseModel, ValidationError
from src.llm.client import complete
class TicketFields(BaseModel):
category: str
priority: str
customer_sentiment: str
requires_human: bool
def extract_ticket_fields(ticket_text: str) -> TicketFields:
schema = TicketFields.model_json_schema()
messages = [
{
"role": "system",
"content": (
"Extract structured fields from the support ticket. "
f"Return JSON matching this schema: {json.dumps(schema)}"
),
},
{"role": "user", "content": ticket_text},
]
raw = complete(
model="mistral-small-latest",
messages=messages,
max_tokens=200,
response_format={"type": "json_object"},
)
try:
return TicketFields.model_validate_json(raw)
except ValidationError as exc:
raise ValueError(f"model returned invalid ticket fields: {exc}") from raw
The Pydantic model is the contract. Its JSON schema is passed into the prompt so the model knows the exact shape to produce, and response_format forces JSON mode so the response is never wrapped in prose or a markdown fence. The model_validate_json call is the gate: if the model returns JSON that does not match the schema, the code raises rather than passing malformed data downstream. This is the difference between a typed boundary and a hope that the model behaved.
Function calling and tools
Mistral supports function calling, which lets the model request a tool invocation and then incorporate the result. The pattern is a tool registry, a loop that executes requested tools, and a hard cap on iterations.
# src/llm/agent.py
import json
from src.llm.client import _client
_MAX_TOOL_ITERATIONS = 5
def run_with_tools(messages: list[dict], tools: list[dict], tool_fns: dict) -> str:
client = _client()
for _ in range(_MAX_TOOL_ITERATIONS):
resp = client.chat.complete(
model="mistral-large-latest",
messages=messages,
tools=tools,
tool_choice="auto",
max_tokens=1500,
)
msg = resp.choices[0].message
messages.append(msg)
if not msg.tool_calls:
return msg.content
for call in msg.tool_calls:
fn = tool_fns[call.function.name]
args = json.loads(call.function.arguments)
result = fn(**args)
messages.append(
{
"role": "tool",
"name": call.function.name,
"content": json.dumps(result),
"tool_call_id": call.id,
}
)
raise RuntimeError("tool loop exceeded max iterations without resolving")
The loop has a hard ceiling of five iterations. Without it, a model that keeps requesting tools in a loop runs up an unbounded bill and never returns. The tool_choice="auto" lets the model decide whether to call a tool or answer directly. Each tool result is appended as a tool role message keyed by the call ID so the model can correlate the result with its request. When the model stops requesting tools and returns content, the loop exits. This is the controlled version of an agent loop, with the iteration cap that the CLAUDE.md rule mandates.
Batched embeddings
Embeddings are where the loop-versus-batch difference is starkest. The CLAUDE.md rule forbids per-item calls. The correct pattern batches inputs and caches by content hash.
# src/llm/embed.py
import hashlib
from src.llm.client import _client
_BATCH_SIZE = 512
_cache: dict[str, list[float]] = {}
def _hash(text: str) -> str:
return hashlib.sha256(text.encode()).hexdigest()
def embed(texts: list[str]) -> list[list[float]]:
client = _client()
results: dict[str, list[float]] = {}
to_embed: list[str] = []
for text in texts:
h = _hash(text)
if h in _cache:
results[h] = _cache[h]
else:
to_embed.append(text)
for i in range(0, len(to_embed), _BATCH_SIZE):
batch = to_embed[i : i + _BATCH_SIZE]
resp = client.embeddings.create(model="mistral-embed", inputs=batch)
for text, item in zip(batch, resp.data):
h = _hash(text)
_cache[h] = item.embedding
results[h] = item.embedding
return [results[_hash(t)] for t in texts]
The function first checks the cache by content hash, so unchanged text is never re-embedded. The remaining inputs are batched into groups of 512, the maximum the endpoint accepts, so a thousand documents become two API calls instead of a thousand. The results are reassembled in the original order. This single function replaces the naive loop that would otherwise call the endpoint once per document, and it is the pattern the CLAUDE.md rule exists to enforce. For a persistent cache backed by a real store rather than an in-process dict, Claude Code with Redis covers the caching layer.
Streaming for user-facing responses
For responses a user reads as they arrive, streaming reduces perceived latency. The pattern yields chunks as they come and accumulates the full response for logging.
# src/llm/stream.py
from src.llm.client import _client
def stream_answer(messages: list[dict], max_tokens: int = 800):
client = _client()
accumulated = []
stream = client.chat.stream(
model="mistral-medium-latest",
messages=messages,
max_tokens=max_tokens,
)
for event in stream:
delta = event.data.choices[0].delta.content
if delta:
accumulated.append(delta)
yield delta
full = "".join(accumulated)
# log full response for audit after the stream completes
The generator yields each delta as it arrives so the calling layer can forward it to the client, and accumulates the full text so the complete response can be logged once the stream ends. Streaming does not reduce the token cost, but it makes a multi-second response feel responsive because the first words appear immediately. Use it for chat and long-form answers, not for the structured extraction calls where the response is consumed whole.
Permission hooks for Mistral development
Mistral calls cost money on every invocation. During development, a Claude Code session that runs a test script in a loop or replays a production dataset against the API can run up a real bill. Permission hooks in .claude/settings.local.json keep the expensive operations behind an explicit gate.
{
"permissions": {
"allow": [
"Bash(pytest tests/unit*)",
"Bash(python -m mypy src*)",
"Bash(ruff check*)"
],
"deny": [
"Bash(python scripts/backfill_embeddings*)",
"Bash(python scripts/replay_*)",
"Bash(*mistral*--all*)"
]
}
}
The allow list permits the unit tests, type checks, and linting Claude needs for normal work, all of which run against mocked clients rather than the live API. The deny list catches the scripts that hit the paid endpoints at scale: an embeddings backfill, a dataset replay, any command that processes everything. These should run deliberately, with a human watching the cost, not as a casual step in an agent loop. Combined with the Claude Code hooks system for project-wide policy, the result is a setup where Claude can develop and test the integration without the risk of an accidental five-figure API bill.
Testing Mistral integration without hitting the API
Unit tests should never call the live API. The pattern is to mock the client at the boundary and assert on the request shape and the handling of the response.
# tests/test_extract.py
from unittest.mock import patch
from src.llm.extract import extract_ticket_fields
def test_extract_returns_validated_fields():
fake_json = (
'{"category": "billing", "priority": "high", '
'"customer_sentiment": "frustrated", "requires_human": true}'
)
with patch("src.llm.extract.complete", return_value=fake_json):
fields = extract_ticket_fields("I was charged twice and nobody replied")
assert fields.category == "billing"
assert fields.requires_human is True
def test_extract_raises_on_invalid_json():
with patch("src.llm.extract.complete", return_value='{"category": "billing"}'):
try:
extract_ticket_fields("incomplete")
assert False, "should have raised on missing fields"
except ValueError:
pass
The first test patches the complete function to return a known JSON string and asserts the Pydantic model parses it correctly. The second test feeds incomplete JSON and asserts the validation gate raises rather than returning a half-populated object. Neither test touches the network, so the suite runs in milliseconds and costs nothing. The model selection and prompt construction are tested separately by asserting on the arguments passed to the mocked complete. For the broader testing setup, Claude Code with pytest covers the fixtures and mocking patterns.
Building a Mistral app that scales with traffic, not cost
The Mistral CLAUDE.md in this guide produces an application where every call routes to the right model for its task, every machine-parsed response uses structured output with schema validation, every call has a timeout and a bounded retry policy, embeddings are batched and cached, and the development workflow keeps the paid endpoints behind a permission gate. The result is a Mistral integration where the bill scales with the value of the work, not with the size of the default model.
The underlying principle is the same as any application built with Claude Code. Mistral without a CLAUDE.md produces working code that quietly reaches for the largest model, skips structured output, and embeds in loops, because those are the readable defaults when the requirements are unstated. The CLAUDE.md is what makes the cost-aware, production-safe patterns the default for Claude.
For the agent orchestration layer that chains Mistral calls into multi-step workflows, Claude Code with LangChain covers the chains and tools. For the service that wraps these models behind an API, Claude Code with FastAPI covers the Python runtime. For the embeddings cache and rate-limit backing store, Claude Code with Redis covers the patterns that keep the model layer fast.
Get Claudify. Build a Mistral integration that routes every call to the right model and never embeds in a loop.
More like this
Ready to upgrade your Claude Code setup?
Get Claudify
