Claude Code with LangSmith: LLM Observability That Holds Up
Why LangSmith without CLAUDE.md produces traces nobody trusts
LangSmith is the observability and eval platform for LLM applications, built by the LangChain team but useful well beyond LangChain itself. The core idea is that every LLM call your service makes is an observable event with inputs, outputs, latency, token cost, and tool call structure, and that you should be able to replay any of those events, score them with an evaluator, and version your prompts across the timeline. The product is genuinely good at this. The catch is that it only works if your traces are well-structured, your datasets are versioned, and your evaluators are deterministic enough to score consistently across runs.
Claude Code, without explicit instructions, generates LangSmith integrations that ship traces successfully and produce the kind of waterfall view that looks great in a demo. But the traces lack the metadata that makes LangSmith powerful. Claude wires up the LangChain @traceable decorator, sets a project name, and stops. The result is a stream of traces with no tenant tagging, no session linking across multi-turn conversations, no prompt version captured, and no relationship to a dataset that lets you re-run the same inputs against a new model version. The traces tell you a call happened. They cannot tell you whether the call was correct, whether it regressed, or whether a specific user is hitting a failure mode others are not.
This guide covers the CLAUDE.md configuration that locks Claude Code into LangSmith's correct model: the client initialised once and shared across the codebase, every traced function carrying tenant and session metadata, datasets versioned with named splits, evaluators producing structured scores not free-form strings, and prompts pulled from the LangSmith registry rather than hardcoded in source. If you are building a backend that calls multiple LLM providers, Claude Code with Next.js covers the request lifecycle. For Claude-specific patterns, Claude Code with the Anthropic SDK covers the message and tool-use shapes that LangSmith captures.
The LangSmith CLAUDE.md template
The CLAUDE.md at your project root is read at the start of every Claude Code session. For a LangSmith integration it needs to declare: the SDK version, the API key environment variable, the project naming convention, the mandatory metadata on every trace, the dataset versioning rules, the evaluator scoring pattern, and the hard rules that prevent the silent metadata loss Claude generates most often.
# LangSmith observability rules
## Stack
- langsmith ^0.2.x (Python) or langsmith ^0.2.x (Node)
- Python 3.11+ or Node.js 20.x with TypeScript 5.x strict
- LANGSMITH_API_KEY in .env (never hardcode)
- LANGSMITH_PROJECT in .env (e.g. "claudify-api-prod")
- LANGSMITH_TRACING=true in production environments
## Project structure
- src/lib/langsmith.ts , LangSmith client singleton
- src/lib/traced.ts , Wrappers that add metadata to every traceable call
- src/lib/evaluators/ , Evaluator functions (one file per evaluator)
- src/lib/prompts.ts , Pulls prompt versions from LangSmith registry
- evals/datasets/ , Local JSON for dataset bootstrap (not for runtime)
## Client initialisation
- ALWAYS use a singleton: import { langsmith } from '@/lib/langsmith'
- Project naming: "{service}-{environment}" (e.g. "claudify-api-prod", "claudify-api-staging")
- NEVER use the default "default" project in production
## Trace metadata (MANDATORY on every traced function)
Every traced function MUST include in metadata:
- tenant_id , the customer/organisation identifier
- user_id , the authenticated user
- session_id , groups multi-turn conversations
- request_id , correlates with backend logs and Honeycomb spans
- prompt_version , the LangSmith prompt commit hash if a prompt was pulled
- model , the underlying model identifier
Optional but commonly useful:
- feature , a stable string identifying the feature path
- intent , the user-facing action that triggered the call
## Datasets
- Dataset names: "{task}-{version}" (e.g. "summarisation-v3", "classification-v2")
- Splits: "train", "dev", "test" (NEVER "validation", standardise on "dev")
- Versioning: dataset versions are immutable, NEVER mutate an existing version
- Bootstrap from evals/datasets/{name}.json on first sync, after that LangSmith is the source of truth
## Evaluators
- Evaluator returns: { key: string, score: number (0-1), comment?: string }
- key: stable identifier for the evaluator (e.g. "answer_relevance", "tool_call_correctness")
- score: float in [0, 1], NEVER negative, NEVER above 1
- NEVER return free-form text as a score, always quantise to [0, 1]
- Evaluators must be deterministic for the same (input, output) pair
## Prompts
- Production prompts live in LangSmith prompt registry, NOT in source
- Pull at startup or per-request via langsmith.pull_prompt("name")
- Cache pulled prompts for the lifetime of the request, NEVER for the lifetime of the process
- ALWAYS capture the commit hash of the pulled prompt in trace metadata
## Hard rules
- NEVER ship to production with the "default" project name
- NEVER trace a function without the metadata block above
- NEVER include raw user PII in the trace inputs (hash or redact email, phone, address)
- NEVER mutate a dataset version, create a new version
- NEVER write an evaluator that calls another LLM without setting check_jsonschema=True on the response
- ALWAYS link multi-turn conversations with a stable session_id
Three rules here prevent the majority of production failures Claude generates without them.
The mandatory metadata rule is the most impactful for actual usefulness. LangSmith without tenant, user, and session metadata is a stream of disconnected runs. You cannot filter to a tenant, you cannot replay a session, and you cannot correlate a LangSmith run to backend logs because there is no shared request ID. Claude omits this because it is not in the SDK quickstart. Declaring it explicitly in CLAUDE.md, alongside a traced wrapper that injects it, makes Claude generate the right pattern by default.
The dataset immutability rule prevents a class of subtle bugs in eval workflows. When you mutate a dataset, every run scored against the previous version becomes incomparable to runs against the new version. The chart shows a regression that is not real. Versioning datasets as "summarisation-v3" with a hard rule that v3 is immutable forces Claude to create "summarisation-v4" when the eval needs to change.
The deterministic evaluator rule counters Claude's instinct to write LLM-as-judge evaluators that call a model with a free-form prompt. Those work, but they are noisy, slow, and expensive. The CLAUDE.md rule pushes Claude toward structured evaluators that score on measurable properties (exact match, BLEU, ROUGE, schema validity, tool call correctness) and use LLM-as-judge only when no structured signal exists.
Install and client setup
Install the LangSmith SDK:
# Python
pip install langsmith
# Node
npm i langsmith
Set up the environment variables:
# .env
LANGSMITH_TRACING=true
LANGSMITH_API_KEY=ls__your_api_key_here
LANGSMITH_PROJECT=claudify-api-prod
LANGSMITH_ENDPOINT=https://api.smith.langchain.com
Create the singleton client:
// src/lib/langsmith.ts
import { Client } from 'langsmith';
if (!process.env.LANGSMITH_API_KEY) {
throw new Error('LANGSMITH_API_KEY is not defined');
}
if (!process.env.LANGSMITH_PROJECT) {
throw new Error('LANGSMITH_PROJECT is not defined');
}
export const langsmith = new Client({
apiKey: process.env.LANGSMITH_API_KEY,
apiUrl: process.env.LANGSMITH_ENDPOINT ?? 'https://api.smith.langchain.com',
});
export const PROJECT_NAME = process.env.LANGSMITH_PROJECT;
Add the singleton pattern to CLAUDE.md so Claude never instantiates inline:
## Client singleton (ENFORCE)
- The only LangSmith client lives at src/lib/langsmith.ts
- Every file that traces does: import { langsmith, PROJECT_NAME } from '@/lib/langsmith'
- Claude MUST NOT write: new Client({ apiKey: ... }) inside route handlers or services
The traced wrapper
The traceable decorator from the LangSmith SDK lets you mark any function as traced. By default it captures inputs, outputs, latency, and errors. To attach the metadata block from CLAUDE.md, wrap the decorator in a project-specific helper.
// src/lib/traced.ts
import { traceable } from 'langsmith/traceable';
import { PROJECT_NAME } from '@/lib/langsmith';
export interface TraceContext {
tenantId: string;
userId: string;
sessionId: string;
requestId: string;
promptVersion?: string;
model?: string;
feature?: string;
intent?: string;
}
export function traced<TArgs extends unknown[], TReturn>(
name: string,
fn: (...args: TArgs) => Promise<TReturn>,
ctx: TraceContext,
) {
return traceable(fn, {
name,
project_name: PROJECT_NAME,
metadata: {
tenant_id: ctx.tenantId,
user_id: ctx.userId,
session_id: ctx.sessionId,
request_id: ctx.requestId,
prompt_version: ctx.promptVersion ?? null,
model: ctx.model ?? null,
feature: ctx.feature ?? null,
intent: ctx.intent ?? null,
},
tags: [
`tenant:${ctx.tenantId}`,
`feature:${ctx.feature ?? 'unknown'}`,
...(ctx.model ? [`model:${ctx.model}`] : []),
],
});
}
Using it in a chat endpoint:
// src/app/api/chat/route.ts
import { NextRequest, NextResponse } from 'next/server';
import { traced } from '@/lib/traced';
import { extractTraceContext } from '@/lib/auth';
import { generateResponse } from '@/lib/chat';
export async function POST(req: NextRequest) {
const ctx = await extractTraceContext(req);
const body = await req.json();
const generate = traced('chat.generate_response', generateResponse, {
...ctx,
feature: 'chat',
intent: 'assistant_reply',
model: 'claude-opus-4-7',
});
const response = await generate(body.messages, body.systemPrompt);
return NextResponse.json({ response });
}
Every run captured by LangSmith now carries tenant_id, user_id, session_id, request_id, and model in metadata, plus filterable tags for tenant, feature, and model. You can pivot the LangSmith UI by any of those dimensions in a single click.
Run trees and multi-step traces
A single user request often triggers multiple LLM calls: a router that decides intent, a retrieval step, a generation step, a self-critique step. LangSmith captures these as a run tree where the root run is the request and the children are the steps.
The traceable decorator automatically nests when one traced function calls another:
// src/lib/chat.ts
import { traceable } from 'langsmith/traceable';
const routeIntent = traceable(
async (message: string) => {
// call a small model to classify the intent
return await classifyIntent(message);
},
{ name: 'chat.route_intent' },
);
const retrieve = traceable(
async (query: string) => {
return await searchKnowledgeBase(query);
},
{ name: 'chat.retrieve' },
);
const generate = traceable(
async (messages: Message[], context: string) => {
return await callLLM(messages, context);
},
{ name: 'chat.generate' },
);
export const generateResponse = traceable(
async (messages: Message[], systemPrompt: string) => {
const intent = await routeIntent(messages[messages.length - 1].content);
let context = '';
if (intent === 'knowledge_query') {
const docs = await retrieve(messages[messages.length - 1].content);
context = docs.map(d => d.content).join('\n\n');
}
return await generate(messages, context);
},
{ name: 'chat.generate_response' },
);
The LangSmith UI now shows a tree: chat.generate_response at the root, with chat.route_intent, chat.retrieve, and chat.generate as children. Each step has its own inputs, outputs, latency, and token cost.
Add a run tree section to CLAUDE.md:
## Run trees and multi-step chains
- ALWAYS wrap each LLM call in its own traceable function
- Function names follow the pattern: "{feature}.{step_name}" (e.g. "chat.route_intent")
- NEVER wrap an entire request handler in a single traceable, break it into the steps you want to see
- Child traces inherit parent metadata, but you can add step-specific tags
- For long chains (5+ steps), prefer parallel where the dependency graph allows
For Claude-specific tool use traces where each tool call should be its own step, Claude Code with the Anthropic SDK covers the message shape that maps cleanly to LangSmith run children.
Datasets and evaluations
LangSmith datasets are versioned collections of (input, output) pairs that you score evaluator functions against. The workflow is: create a dataset, sync it from local JSON, run an experiment that calls your application against the dataset inputs, score the outputs with evaluators, compare experiments across model versions.
Bootstrap a dataset from a local JSON file:
// scripts/sync-dataset.ts
import { langsmith } from '@/lib/langsmith';
import datasetData from '../evals/datasets/summarisation-v3.json' assert { type: 'json' };
interface DatasetExample {
inputs: { document: string };
outputs: { summary: string };
split: 'train' | 'dev' | 'test';
}
async function syncDataset(name: string, version: string, examples: DatasetExample[]) {
const datasetName = `${name}-${version}`;
let dataset;
try {
dataset = await langsmith.readDataset({ datasetName });
console.log(`Dataset ${datasetName} exists, skipping create.`);
} catch {
dataset = await langsmith.createDataset(datasetName, {
description: `Auto-synced from evals/datasets/${name}-${version}.json`,
});
console.log(`Created dataset ${datasetName}.`);
}
for (const example of examples) {
await langsmith.createExample(
example.inputs,
example.outputs,
{
datasetId: dataset.id,
metadata: { split: example.split },
},
);
}
console.log(`Synced ${examples.length} examples.`);
}
await syncDataset('summarisation', 'v3', datasetData as DatasetExample[]);
Add a dataset section to CLAUDE.md:
## Datasets
- Local source of truth lives at evals/datasets/{name}-{version}.json
- Sync to LangSmith via scripts/sync-dataset.ts (one-way push)
- Naming: "{task}-{version}" (e.g. "summarisation-v3")
- Splits: "train", "dev", "test" recorded in example metadata
- Versions are immutable after first push, increment to v4 for changes
## Hard rules for datasets
- NEVER push examples without a split label
- NEVER push a dataset over an existing version (immutability)
- NEVER include PII in examples, hash or synthesise inputs
Evaluators
An evaluator takes a (run, example) pair and returns a score. The score must be a float in [0, 1] with a stable evaluator key. LangSmith aggregates these across experiments and charts them over time.
A structured evaluator that checks tool call correctness:
// src/lib/evaluators/tool-call-correctness.ts
import type { Run, Example } from 'langsmith';
export async function toolCallCorrectness(
run: Run,
example: Example,
): Promise<{ key: string; score: number; comment?: string }> {
const expected = example.outputs?.tool_calls as Array<{ name: string; args: object }> | undefined;
const actual = (run.outputs?.tool_calls as Array<{ name: string; args: object }>) ?? [];
if (!expected || expected.length === 0) {
return {
key: 'tool_call_correctness',
score: actual.length === 0 ? 1 : 0,
comment: actual.length === 0
? 'No tool calls expected and none made.'
: 'No tool calls expected but model made tool calls.',
};
}
if (actual.length !== expected.length) {
return {
key: 'tool_call_correctness',
score: 0,
comment: `Expected ${expected.length} tool calls, got ${actual.length}.`,
};
}
let matched = 0;
for (let i = 0; i < expected.length; i++) {
if (
expected[i].name === actual[i].name &&
JSON.stringify(expected[i].args) === JSON.stringify(actual[i].args)
) {
matched++;
}
}
return {
key: 'tool_call_correctness',
score: matched / expected.length,
comment: `${matched}/${expected.length} tool calls matched.`,
};
}
Running an experiment against the dataset with this evaluator:
// scripts/run-experiment.ts
import { langsmith } from '@/lib/langsmith';
import { runOnDataset } from 'langsmith/evaluation';
import { toolCallCorrectness } from '@/lib/evaluators/tool-call-correctness';
import { generateResponse } from '@/lib/chat';
await runOnDataset({
client: langsmith,
datasetName: 'tool-use-v2',
llmOrChainFactory: () => generateResponse,
evaluators: [toolCallCorrectness],
experimentPrefix: 'opus-4-7-baseline',
});
Add an evaluators section to CLAUDE.md:
## Evaluators
- One file per evaluator at src/lib/evaluators/{key}.ts
- Function signature: (run, example) => { key, score, comment? }
- key: snake_case, stable across versions of the evaluator
- score: float in [0, 1], document the meaning in a comment
- Composite evaluators: write the structured pieces first, add LLM-as-judge ONLY for what cannot be measured structurally
## When to use LLM-as-judge
- ONLY for subjective qualities (helpfulness, tone, safety)
- ALWAYS use a smaller, faster model than the production model
- ALWAYS set response_format to JSON schema and validate the schema
- Cache LLM-as-judge results by (input_hash, model, prompt_version) tuple
Prompt versioning
LangSmith's prompt registry stores prompts as versioned objects, similar to a git repository for text. The production application pulls a named version at runtime, the prompt commit hash is captured in trace metadata, and changes to the prompt are reviewed through the LangSmith UI.
Pulling a prompt at startup:
// src/lib/prompts.ts
import { langsmith } from '@/lib/langsmith';
interface CachedPrompt {
template: string;
commitHash: string;
fetchedAt: number;
}
const cache = new Map<string, CachedPrompt>();
const CACHE_TTL_MS = 60_000; // 1 minute
export async function getPrompt(name: string): Promise<{ template: string; commitHash: string }> {
const cached = cache.get(name);
if (cached && Date.now() - cached.fetchedAt < CACHE_TTL_MS) {
return { template: cached.template, commitHash: cached.commitHash };
}
const prompt = await langsmith.pullPrompt(name);
const entry: CachedPrompt = {
template: prompt.template,
commitHash: prompt.commit_hash,
fetchedAt: Date.now(),
};
cache.set(name, entry);
return { template: entry.template, commitHash: entry.commitHash };
}
Using it in a generation function:
import { getPrompt } from '@/lib/prompts';
import { traced } from '@/lib/traced';
export async function generateSummary(document: string, ctx: TraceContext) {
const { template, commitHash } = await getPrompt('summariser');
const generate = traced('summary.generate', async (doc: string) => {
return await callLLM(template.replace('{document}', doc));
}, { ...ctx, promptVersion: commitHash });
return await generate(document);
}
Every trace now carries the exact prompt commit hash. If you change the prompt and quality regresses, the LangSmith UI shows which traces used the new version and which used the old.
Add a prompt versioning section to CLAUDE.md:
## Prompts
- Production prompts live in LangSmith prompt registry under the names declared here:
- "summariser"
- "router"
- "tool_use_system"
- "self_critique"
- Pull via getPrompt(name) which caches for 60s
- Capture commit_hash in trace metadata via the traced wrapper
## Hard rules for prompts
- NEVER hardcode a production prompt string in source
- NEVER bypass getPrompt() and call langsmith.pullPrompt() directly outside src/lib/prompts.ts
- ALWAYS test prompt changes against the dev split of the relevant dataset before promoting
For broader prompt engineering workflows that feed the LangSmith registry, the LangSmith prompt UI handles the review process, similar to how a code review tool works for pull requests.
Sampling and PII
At low traffic, capture every trace. At higher traffic, sample by importance: capture all error traces, capture all slow traces, sample healthy traces at a fixed rate. PII redaction is non-negotiable for traces that may include user content.
// src/lib/redact.ts
const EMAIL = /[\w.+-]+@[\w-]+\.[\w.-]+/g;
const PHONE = /\+?\d{1,3}[ -]?\(?\d{3}\)?[ -]?\d{3}[ -]?\d{4}/g;
export function redactPII(text: string): string {
return text
.replace(EMAIL, '[email]')
.replace(PHONE, '[phone]');
}
Apply redaction in the traced wrapper:
// in src/lib/traced.ts, modify the wrapper to redact inputs/outputs
import { redactPII } from '@/lib/redact';
return traceable(fn, {
name,
project_name: PROJECT_NAME,
metadata: { /* ... */ },
process_inputs: (inputs) => {
if (typeof inputs === 'string') return redactPII(inputs);
return JSON.parse(redactPII(JSON.stringify(inputs)));
},
process_outputs: (outputs) => {
if (typeof outputs === 'string') return redactPII(outputs);
return JSON.parse(redactPII(JSON.stringify(outputs)));
},
});
Add a PII section to CLAUDE.md:
## PII handling
- ALL traces pass through redactPII() in process_inputs and process_outputs
- Patterns redacted: email, phone (extend as needed)
- NEVER add a new PII pattern without also adding it to the redaction function
- For high-sensitivity flows (payments, auth tokens), set tags: ["pii:high"] and consider not tracing at all
Common Claude Code mistakes with LangSmith
Six patterns Claude generates incorrectly without CLAUDE.md constraints, with the correct replacement for each.
1. Inline client instantiation
Claude generates: const client = new Client({ apiKey: process.env.LANGSMITH_API_KEY }); at the top of every file.
Correct pattern: one singleton at src/lib/langsmith.ts, imported everywhere.
2. Missing metadata block
Claude generates: traceable(fn, { name: 'do_thing' }) with no metadata.
Correct pattern: every traced function goes through the traced() wrapper that injects tenant, user, session, request, and prompt version metadata.
3. One large traceable instead of a run tree
Claude generates: a single traceable around the entire request handler.
Correct pattern: separate traceable for each LLM call (route, retrieve, generate, critique), nested as a run tree.
4. Hardcoded production prompts
Claude generates: const SYSTEM_PROMPT = "You are a helpful assistant..." in source.
Correct pattern: getPrompt('summariser') pulled from the LangSmith registry, commit hash captured in trace metadata.
5. LLM-as-judge as the default evaluator
Claude generates: an evaluator that calls a model with a free-form "rate this from 1-10" prompt.
Correct pattern: structured evaluators (exact match, schema validation, tool call correctness) first, LLM-as-judge only where no structured signal exists, with JSON schema validation on the response.
6. PII in trace inputs
Claude generates: trace inputs that contain raw user emails, phone numbers, or document text without redaction.
Correct pattern: process_inputs and process_outputs run through redactPII() before the trace is uploaded.
Add a common mistakes section to CLAUDE.md with these six pairs. Claude benefits from explicit before/after comparisons because it can match the pattern it would otherwise generate to the corrected form.
Permission hooks for eval scripts
A LangSmith integration accumulates scripts: dataset sync scripts, experiment runners, prompt promotion utilities, project cleanup. Some are read-only. Some mutate the LangSmith state in ways that affect production. Permission hooks gate the destructive ones.
In .claude/settings.local.json:
{
"permissions": {
"allow": [
"Bash(node scripts/sync-dataset.js*)",
"Bash(node scripts/list-experiments.js*)",
"Bash(node scripts/run-experiment.js*)",
"Bash(node scripts/list-prompts.js*)"
],
"deny": [
"Bash(node scripts/promote-prompt.js*)",
"Bash(node scripts/delete-dataset.js*)",
"Bash(node scripts/purge-project.js*)"
]
}
}
Syncing a dataset (which is idempotent due to immutability) and running an experiment are safe operations. Promoting a prompt to the production tag or deleting a dataset would change production behaviour and destroy historical eval data. The deny list forces Claude to surface those operations as prompts.
Building LLM observability that survives the move to production
The LangSmith CLAUDE.md in this guide produces LLM observability where every traced function carries tenant, user, session, and prompt version metadata, datasets are versioned and immutable, evaluators return structured scores rather than free-form text, prompts live in the LangSmith registry with commit hashes captured in trace metadata, and PII is redacted before traces leave the application.
The underlying principle is the same as any observability integration with Claude Code. LangSmith without a CLAUDE.md produces traces that look great in a demo but cannot answer the questions a production team actually has: which tenant is regressing, which prompt version triggered the failure, whether the new model is better than the old one on the dev split. The CLAUDE.md template removes each failure mode by making the metadata-rich, version-aware pattern the only pattern Claude can generate.
For the next layer of LLM operational visibility, LangSmith pairs with Claude Code with the Anthropic SDK for Claude-specific trace shapes, and Claudify includes a LangSmith-specific CLAUDE.md template with the traced wrapper, dataset conventions, evaluator patterns, prompt registry usage, and all six common-mistake rules pre-configured.
Get Claudify. Ship LangSmith traces that answer eval questions on day one.
More like this
Ready to upgrade your Claude Code setup?
Get Claudify
