Claude Code with Honeycomb: Tracing That Tells You Why
Why Honeycomb without CLAUDE.md produces flat traces and useless dashboards
Honeycomb is the observability platform built around the idea that a production incident is a question, not a metric, and the answer is hiding somewhere in the dimensions of your event data. The product is genuinely different from Datadog, New Relic, or any other tool that grew up as a metrics store with traces bolted on. It is column-oriented, high-cardinality by default, and designed to let you ask "which users are seeing this slow path" and get an answer back in under a second. The catch is that Honeycomb is only as useful as the data you send it. A poorly instrumented service in Honeycomb looks the same as a poorly instrumented service in any other tool: flat traces, generic attributes, and no way to ask the questions that matter.
Claude Code, without explicit instructions, generates OpenTelemetry instrumentation that ships to Honeycomb successfully and produces traces that look fine in the UI. But the traces lack the attribute richness that makes Honeycomb powerful. Claude adds the OTel SDK, wraps a few HTTP handlers, sends spans with the default attributes (http.method, http.status_code, http.route), and stops there. The result is a tracing setup that tells you a request was slow, but cannot tell you that requests are slow only for users on the enterprise plan accessing the reports endpoint from European IP ranges between 2pm and 4pm UTC. The latter is the kind of question Honeycomb was built to answer, and the answer requires custom attributes on every span.
This guide covers the CLAUDE.md configuration that locks Claude Code into Honeycomb's correct model: OpenTelemetry SDK initialised once at process start, a wrapper that adds tenant and user attributes to every span, semantic conventions followed for HTTP and database calls, derived columns defined for common queries, and SLOs configured against meaningful service-level indicators. If you are building a Node.js service, Claude Code with Next.js covers the request lifecycle where these spans need to be created. For database-heavy tracing, Claude Code with Drizzle covers the query patterns that produce the most useful spans.
The Honeycomb CLAUDE.md template
The CLAUDE.md at your project root is read at the start of every Claude Code session. For a Honeycomb integration it needs to declare: the OpenTelemetry SDK and exporter, the service name and resource attributes, the environment variable for the API key, the mandatory attributes on every span, the high-cardinality fields Claude should prefer, and the hard rules that prevent the silent attribute loss Claude generates most often.
# Honeycomb tracing rules
## Stack
- @opentelemetry/sdk-node ^0.50.x
- @opentelemetry/exporter-trace-otlp-http ^0.50.x
- @opentelemetry/auto-instrumentations-node ^0.45.x
- @honeycombio/opentelemetry-node ^0.10.x (Honeycomb's bundled distro, recommended)
- Node.js 20.x, TypeScript 5.x strict
- HONEYCOMB_API_KEY in .env.local (never hardcode)
- HONEYCOMB_DATASET in .env.local (e.g. "api-service-prod")
## Project structure
- src/instrumentation.ts , OTel SDK setup, MUST be imported first
- src/lib/tracing.ts , Tracer singleton + helper functions
- src/lib/span-attributes.ts , Shared attribute builders (tenant, user, plan)
- src/middleware/tracing.ts , HTTP middleware that adds request-scope attributes
## SDK initialisation
- ALWAYS initialise in src/instrumentation.ts, imported as the FIRST line of the entry file
- Use @honeycombio/opentelemetry-node for sensible defaults
- ServiceName MUST match the deployed service name in your platform (Vercel, Render, Fly)
- Resource attributes MUST include: service.version, deployment.environment, service.namespace
## Span attributes (MANDATORY on every span)
Every span created in business logic MUST include:
- tenant.id , the customer/organisation identifier (high cardinality, this is the point)
- user.id , the authenticated user (high cardinality)
- plan.tier , free | pro | enterprise
- feature , a stable string identifying the feature path
- request.id , the request correlation ID
Optional but commonly useful:
- db.statement.hash , a hash of the SQL or NoSQL query (NOT the raw query)
- error.kind , a stable error class string, NOT the error message
- queue.depth , for any backpressure-aware operation
## High-cardinality is correct
- DO add user.id, tenant.id, order.id, request.id to spans
- DO NOT pre-aggregate or bucket these values in code
- Honeycomb is built for high cardinality, this is not a Prometheus dataset
## Hard rules
- NEVER initialise OTel SDK anywhere other than src/instrumentation.ts
- NEVER import instrumentation.ts other than as the FIRST import in the entry file
- NEVER log to console.log() instead of adding a span attribute
- NEVER use generic span names like "http_request" or "db_query", use the route or operation
- NEVER attach the raw SQL string as an attribute, hash it or use db.operation + db.collection
- NEVER catch an error without setting span.recordException(error) and span.setStatus({ code: 2 })
- ALWAYS end every span you start, in a finally block if needed
Three rules here prevent the majority of production failures Claude generates without them.
The mandatory attributes rule is the most impactful for actual observability. Honeycomb's value over a metrics tool is that you can pivot any query by tenant, user, plan tier, or feature. None of those attributes appear by default on OpenTelemetry auto-instrumented spans. Without a wrapper that adds them, your trace data is functionally indistinguishable from what you would get out of a generic tracing tool. Claude omits these because they are not in the SDK quickstart docs. Declaring them explicitly in CLAUDE.md, alongside a span helper that injects them, makes Claude generate the right pattern by default.
The single SDK initialisation rule prevents a class of subtle bugs. OpenTelemetry must be initialised before any instrumented library is imported. If Claude generates code that initialises OTel inside a route handler, or imports instrumentation.ts after another module that uses http or pg, the auto-instrumentation will not patch those modules. The result is spans that appear for some operations but silently miss others. The "FIRST import in entry file" rule is non-negotiable.
The high-cardinality directive counters Claude's training bias. Most observability tools punish high-cardinality attributes because they explode storage costs. Honeycomb is built for high cardinality. Claude has read thousands of Prometheus tutorials warning against user_id labels, so it generates low-cardinality bucketing code by default. The CLAUDE.md rule explicitly inverts that bias for Honeycomb's data model.
Install and SDK setup
Install the Honeycomb-bundled OTel distribution, which sets sensible defaults:
npm i @honeycombio/opentelemetry-node @opentelemetry/auto-instrumentations-node
Create the instrumentation file. This is the first file imported in your entry point:
// src/instrumentation.ts
import { HoneycombSDK } from '@honeycombio/opentelemetry-node';
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';
if (!process.env.HONEYCOMB_API_KEY) {
throw new Error('HONEYCOMB_API_KEY is not defined');
}
const sdk = new HoneycombSDK({
apiKey: process.env.HONEYCOMB_API_KEY,
serviceName: process.env.SERVICE_NAME ?? 'claudify-api',
instrumentations: [
getNodeAutoInstrumentations({
'@opentelemetry/instrumentation-fs': { enabled: false },
}),
],
});
sdk.start();
The fs instrumentation is disabled because it generates an enormous volume of low-value spans (every file read becomes a span). This is the most common bandwidth issue Claude leaves in by default.
Wire it into your entry point as the literal first import:
// src/index.ts
import './instrumentation.js'; // MUST be first
import { startServer } from './server.js';
startServer();
Add the startup pattern to CLAUDE.md:
## Startup order (ENFORCE)
- src/instrumentation.ts is the FIRST import in src/index.ts
- Nothing else may be imported before it
- Claude MUST NOT move the import order, even for "cleanliness"
- If the build tool reorders imports, configure it to keep instrumentation first
The span attribute wrapper
The single most important piece of code in a Honeycomb integration is the helper that adds business-context attributes to every span. Without it, your spans have http.method and http.status_code and nothing else useful. With it, every span carries the tenant, user, plan, and feature context that makes Honeycomb queries powerful.
// src/lib/tracing.ts
import { trace, context, Span, SpanStatusCode, Attributes } from '@opentelemetry/api';
const tracer = trace.getTracer('claudify-api', process.env.SERVICE_VERSION ?? 'unknown');
export interface SpanContext {
tenantId: string;
userId: string;
planTier: 'free' | 'pro' | 'enterprise';
requestId: string;
}
export async function withSpan<T>(
name: string,
ctx: SpanContext,
feature: string,
fn: (span: Span) => Promise<T>,
extraAttrs: Attributes = {},
): Promise<T> {
return tracer.startActiveSpan(name, async (span) => {
span.setAttributes({
'tenant.id': ctx.tenantId,
'user.id': ctx.userId,
'plan.tier': ctx.planTier,
'request.id': ctx.requestId,
feature,
...extraAttrs,
});
try {
const result = await fn(span);
span.setStatus({ code: SpanStatusCode.OK });
return result;
} catch (err) {
const error = err instanceof Error ? err : new Error(String(err));
span.recordException(error);
span.setAttribute('error.kind', error.name);
span.setStatus({ code: SpanStatusCode.ERROR, message: error.message });
throw error;
} finally {
span.end();
}
});
}
Using it in a route handler:
// src/app/api/reports/route.ts
import { NextRequest, NextResponse } from 'next/server';
import { withSpan } from '@/lib/tracing';
import { getCurrentSpanContext } from '@/lib/auth';
import { generateReport } from '@/lib/reports';
export async function POST(req: NextRequest) {
const ctx = await getCurrentSpanContext(req);
return withSpan('reports.generate', ctx, 'reports', async (span) => {
const body = await req.json();
span.setAttribute('report.type', body.type);
span.setAttribute('report.row_count', body.rowCount);
const report = await generateReport(body, ctx);
span.setAttribute('report.size_bytes', report.sizeBytes);
return NextResponse.json(report);
});
}
Every span this handler produces carries tenant.id, user.id, plan.tier, request.id, feature, report.type, report.row_count, and report.size_bytes. In Honeycomb you can now ask: "show me p95 latency for reports.generate, broken down by plan tier" and get a meaningful answer.
BubbleUp and query-friendly attributes
BubbleUp is Honeycomb's automatic anomaly detection feature. You select an interesting region in a chart (a latency spike, an error cluster) and BubbleUp tells you which attribute values are over-represented in that region versus the baseline. It works because Honeycomb indexes every attribute as a queryable dimension.
For BubbleUp to be useful, your attributes need to be high-cardinality and meaningfully distributed. Three patterns matter:
Use stable enums, not free-form strings. plan.tier: "free" | "pro" | "enterprise" is bubbleable. plan.description: "Our Pro plan with 50 seats" is not.
Hash or normalise where the raw value would be too unique. SQL queries with embedded parameters are unique per request, which makes them useless for BubbleUp. Hash the query template or extract the operation:
import { createHash } from 'crypto';
function hashQuery(sql: string): string {
// Replace literals with placeholders to group similar queries
const template = sql
.replace(/\d+/g, '?')
.replace(/'[^']*'/g, '?');
return createHash('sha256').update(template).digest('hex').substring(0, 12);
}
// Then in a span:
span.setAttribute('db.statement.hash', hashQuery(query));
span.setAttribute('db.operation', 'SELECT');
span.setAttribute('db.collection', 'orders');
Capture error kinds, not error messages. error.kind: "RateLimitExceeded" groups all rate limit errors. error.message: "Rate limit exceeded: 100 req/sec for tenant t_abc123" is unique per occurrence.
Add a BubbleUp section to CLAUDE.md:
## BubbleUp-friendly attributes
- Prefer stable enum values over free-form descriptions
- Hash unique identifiers when the raw form would defeat grouping:
- db.statement.hash, NOT db.statement.raw
- error.kind (the class name), NOT error.message (the formatted string)
- Always set BOTH error.kind AND span.recordException() for error rows
- For HTTP, set http.route (the templated path /users/:id), NOT http.url (the resolved path)
- For background jobs, set queue.name and job.kind
For a deeper Next.js context where these spans live, Claude Code with Next.js covers the middleware patterns where request-scope attributes are extracted.
Derived columns
Derived columns let you define computed dimensions in Honeycomb that combine, transform, or extract from raw span attributes without changing the instrumentation code. They are SQL-like expressions that Honeycomb evaluates at query time.
The four derived columns every Honeycomb dataset should have:
# Name: is_error
# Expression:
EQUALS($status_code, 2)
# Name: is_slow
# Expression:
GT($duration_ms, 1000)
# Name: trace_root
# Expression:
EQUALS($parent_id, "")
# Name: plan_tier_grouped
# Expression:
IF(
EQUALS($plan.tier, "enterprise"), "paid",
IF(EQUALS($plan.tier, "pro"), "paid", "free")
)
These derived columns make queries dramatically more expressive. Instead of writing the full expression in every query, you write WHERE is_slow = true GROUP BY plan_tier_grouped and the column does the work.
Add a derived columns section to CLAUDE.md so Claude documents the conventions:
## Derived columns (define in Honeycomb UI, document here)
Required:
- is_error: EQUALS($status_code, 2)
- is_slow: GT($duration_ms, 1000)
- trace_root: EQUALS($parent_id, "")
Project-specific:
- plan_tier_grouped: combines pro + enterprise into "paid"
- region_grouped: combines AWS regions into "us" / "eu" / "asia"
- user_segment: derives based on $user.signup_year and $plan.tier
## Rules
- NEVER change a derived column expression without updating the query library
- NEVER define a derived column with the same name as a raw span attribute
- Document every derived column in this CLAUDE.md so Claude can reference them in queries
SLOs
Honeycomb SLOs are defined as a Service Level Indicator (SLI) over a target percentage and time window. The SLI is a derived column that evaluates to true (success) or false (failure) for each event. The SLO tracks the percentage of true events over the window.
A canonical SLO definition for a Next.js API:
# SLO: API requests succeed in under 500ms (99% over 30 days)
#
# SLI derived column: api_request_fast_success
# Expression:
AND(
EQUALS($trace_root, true),
EQUALS($name, "http.request"),
EQUALS($http.status_code, 200),
LT($duration_ms, 500)
)
#
# SLO target: 99%
# Time window: 30 days
# Budget: 1% of 30 days = ~7.2 hours of failure budget
The SLI sits in a derived column. The SLO sits in Honeycomb's SLO configuration UI. The CLAUDE.md captures the intent:
## SLOs
### Customer-facing reads (read SLO)
- SLI: trace_root = true AND http.method = GET AND http.status_code = 200 AND duration_ms < 500
- Target: 99.5% over 30 days
- Alert: budget burn rate > 5x normal
### Customer-facing writes (write SLO)
- SLI: trace_root = true AND http.method IN (POST,PUT,DELETE) AND http.status_code IN (200,201,204) AND duration_ms < 1000
- Target: 99% over 30 days
- Alert: budget burn rate > 5x normal
### Background jobs (job SLO)
- SLI: name = "job.process" AND status_code = 0 AND duration_ms < 30000
- Target: 99.5% over 7 days
## Rules for editing SLOs
- NEVER tighten an SLO without consulting historical burn rate
- NEVER widen an SLO without documenting the customer-facing impact
- ALWAYS include a budget burn rate alert at 5x normal
Claude will not propose SLOs without prompting because they are a product decision, not a code change. Documenting them in CLAUDE.md gives Claude the context to reference the target latencies when generating performance-sensitive code, and to flag when a new feature might violate the SLI.
Triggers and heatmaps
Honeycomb triggers fire when a query result crosses a threshold. They are the alerting layer. Heatmaps are the visualisation primitive Honeycomb uses for latency distributions: each point is a bucketed count of events at a particular duration.
Common trigger patterns to document in CLAUDE.md:
## Triggers (document active ones here)
### High error rate
- Query: COUNT WHERE is_error = true GROUP BY service.name
- Threshold: > 50 events in 5 minutes
- Recipients: oncall pagerduty
### P99 latency spike
- Query: HEATMAP duration_ms WHERE http.route = "/api/checkout"
- Threshold: P99 > 2000ms for 5 minutes
- Recipients: backend slack channel
### SLO burn rate (per SLO)
- Query: SLO-specific, configured in SLO UI
- Threshold: 5x burn rate
- Recipients: oncall + slack
### Tenant-specific anomalies
- Query: COUNT WHERE is_error = true GROUP BY tenant.id
- Threshold: any tenant > 100 errors in 10 minutes
- Recipients: customer success slack channel
## Rules
- NEVER add a trigger that pages without a defined remediation playbook
- ALWAYS set a recipient channel, never leave it default
- Document trigger queries in this CLAUDE.md so Claude can suggest them
For payment-flow tracing where SLO violations are revenue-impacting, Claude Code with Stripe covers the webhook handlers that should be wrapped in spans and counted in SLOs.
Sampling
At low traffic, sample everything. At high traffic, sample by importance: keep all error spans, keep all slow spans, keep all spans from a tail-sampled fraction of healthy requests.
The Honeycomb-bundled OTel distro includes a tail-based sampler:
// src/instrumentation.ts
import { HoneycombSDK } from '@honeycombio/opentelemetry-node';
import { TraceIdRatioBasedSampler, AlwaysOnSampler } from '@opentelemetry/sdk-trace-base';
const sdk = new HoneycombSDK({
apiKey: process.env.HONEYCOMB_API_KEY,
serviceName: 'claudify-api',
// Sample 100% in dev, 10% in production unless error or slow
sampler: process.env.NODE_ENV === 'production'
? new TraceIdRatioBasedSampler(0.1)
: new AlwaysOnSampler(),
});
For tail-based sampling where the keep/drop decision happens after the trace completes, run the Honeycomb Refinery service in front of your exporters. Refinery is a separate component that buffers traces, applies sampling rules based on the completed trace, and forwards the kept ones to Honeycomb.
Add a sampling section to CLAUDE.md:
## Sampling
- Development: AlwaysOnSampler (100%)
- Staging: TraceIdRatioBasedSampler(0.5) (50%)
- Production: TraceIdRatioBasedSampler(0.1) (10%) head-based
- Production tail sampling (optional): Honeycomb Refinery in front of OTel exporter
## Rules
- NEVER sample below 100% in development
- NEVER drop error spans, configure tail sampler to keep all errors
- NEVER drop slow spans (duration_ms > 1000), keep them all
- ALWAYS include sample.rate as a resource attribute so Honeycomb can compute weighted counts
The sample.rate resource attribute matters. Without it, Honeycomb counts your sampled events as if they were the full population, undercounting actual throughput. With it, Honeycomb multiplies the sampled count by the inverse of the sample rate to estimate the true count.
Common Claude Code mistakes with Honeycomb
Six patterns Claude generates incorrectly without CLAUDE.md constraints, with the correct replacement for each.
1. SDK initialised in the wrong file
Claude generates: OpenTelemetrySDK setup inside server.ts or a route handler.
Correct pattern: setup in src/instrumentation.ts, imported as the first line of the entry file.
2. Missing business-context attributes
Claude generates: spans with only http.method, http.status_code, http.route.
Correct pattern: every span carries tenant.id, user.id, plan.tier, feature via the withSpan wrapper.
3. Raw SQL as an attribute
Claude generates: span.setAttribute('db.statement', 'SELECT * FROM orders WHERE id = 123') for every query.
Correct pattern: span.setAttribute('db.statement.hash', hashQuery(sql)) plus db.operation and db.collection for grouping.
4. Error message captured instead of error kind
Claude generates: span.setAttribute('error.message', err.message) only.
Correct pattern: span.recordException(err), span.setAttribute('error.kind', err.name), and span.setStatus({ code: SpanStatusCode.ERROR }).
5. Filesystem instrumentation left enabled
Claude generates: getNodeAutoInstrumentations() with no overrides.
Correct pattern: disable @opentelemetry/instrumentation-fs explicitly to avoid bandwidth waste.
6. Console logs instead of span attributes
Claude generates: console.log('User accessed report', userId, reportId) alongside an active span.
Correct pattern: span.setAttribute('report.id', reportId) and span.setAttribute('user.id', userId). The log becomes a queryable dimension.
Add a common mistakes section to CLAUDE.md with these six pairs. Claude benefits from explicit before/after comparisons because it can match the pattern it would otherwise generate to the corrected form.
Permission hooks for tracing scripts
A Honeycomb integration accumulates scripts: dataset reset scripts, trigger backup scripts, SLO export utilities, query library generators. Some are read-only against the Honeycomb API. Some create or delete resources. Permission hooks gate the destructive ones.
In .claude/settings.local.json:
{
"permissions": {
"allow": [
"Bash(node scripts/export-slos.js*)",
"Bash(node scripts/list-triggers.js*)",
"Bash(node scripts/query-dataset.js*)",
"Bash(node scripts/preview-derived-column.js*)"
],
"deny": [
"Bash(node scripts/delete-dataset.js*)",
"Bash(node scripts/purge-triggers.js*)",
"Bash(node scripts/reset-slos.js*)"
]
}
}
Exporting SLOs and listing triggers are safe operations with no external side effects. Deleting a dataset or resetting SLOs would destroy historical data and break dashboards. The deny list forces Claude to surface those operations as prompts rather than running them as part of an automated workflow.
Building traces that answer the right questions
The Honeycomb CLAUDE.md in this guide produces tracing setups where the OTel SDK is initialised exactly once in the correct order, every span carries the tenant, user, plan tier, and feature attributes that make Honeycomb queries pivot, derived columns capture common predicates so query authors do not repeat themselves, SLOs sit on top of meaningful SLIs with documented burn-rate alerts, and sampling preserves error and slow spans regardless of the head sample rate.
The underlying principle is the same as any observability integration with Claude Code. Honeycomb without a CLAUDE.md produces code that ships spans to the platform but does not include the attributes that distinguish Honeycomb from a generic metrics tool. The CLAUDE.md template removes each failure mode by making the high-cardinality, attribute-rich pattern the only pattern Claude can generate.
For the next layer of operational visibility, Honeycomb pairs with Claude Code with Vercel for deployment-tagged spans, and Claudify includes a Honeycomb-specific CLAUDE.md template with the span wrapper, derived column conventions, SLO definitions, sampling rules, and all six common-mistake patterns pre-configured.
Get Claudify. Ship Honeycomb instrumentation that answers production questions on the first query.
More like this
Ready to upgrade your Claude Code setup?
Get Claudify
