Claude Code with Consul: Service Mesh That Behaves in Production
Why Consul without CLAUDE.md ships a service mesh that leaks
Consul is HashiCorp's service mesh and service discovery system. It registers services, runs health checks, maintains a KV store, issues mTLS certificates, and routes traffic between services through sidecar proxies. The pitch is that any application becomes a discoverable, encrypted, authorisation-controlled service by registering with the local Consul agent. The reality is that the same flexibility makes the failure modes hard to spot. Consul will happily run with ACLs in default-allow mode, where any client can read any KV path and any service can call any other service. It will accept service registrations from anonymous tokens. It will issue mTLS certs to services that have no intention to talk to each other. It will hand out KV reads of paths that contain secrets to anyone with the right gossip key.
A Claude Code session that drafts Consul configuration without project rules generates a deployment that works in the demo and is hostile in production. The server config has acl.enabled: false because the quickstart shows it that way. The client config registers a service with no Connect.SidecarService block, so the sidecar runs in legacy mode without intentions. The KV write puts secrets at a top-level key with no token, so every other agent on the gossip network can read them. The intention * -> * allow rule is created by hand "to make things work" and never removed. The bootstrap token is logged to stdout because the script prints it during install and nobody removed the print statement. Each of these is a small mistake. Together they describe a service mesh that an attacker who reaches the gossip network can map and pivot through in minutes.
This guide covers the CLAUDE.md configuration that locks Claude Code into Consul patterns that survive production: a strict agent topology with three or five servers and one client per host, ACLs in default-deny mode, intentions defined per service pair, structured KV namespacing, mTLS via the built-in Connect CA, and permission hooks that gate the destructive operations. For the alternative service mesh that runs inside Kubernetes, Claude Code with Traefik covers the edge proxy patterns. For the application-level configuration management Consul KV often gets pressed into, Claude Code with Cloudflare Workers covers the simpler patterns where KV-as-config is overkill.
The Consul CLAUDE.md template
The CLAUDE.md at your project root is read at the start of every Claude Code session. For a Consul deployment it needs to declare: the Consul version, the agent topology, the ACL policy, the intention default, the gossip and RPC encryption, the KV namespace structure, the Connect CA, and the hard rules that block the misconfigurations Claude makes most often.
# Consul rules
## Stack
- consul 1.19.x or 1.20.x (LTS branch only)
- Server count: 3 (small deployments) or 5 (production at scale)
- One client agent per host, no client-on-server collocation
- Consul Connect with built-in CA for mTLS
- ACLs in default-deny mode (acl.default_policy: deny)
- Gossip encryption key in -encrypt flag, rotated annually
## Project structure
- consul/server.hcl , server config, used by every server
- consul/client.hcl , client config, used by every client
- consul/services/ , one HCL file per service registration
- consul/intentions/ , declarative intention definitions (consul intention CLI)
- consul/policies/ , ACL policies as HCL
- consul/kv-seed/ , initial KV bootstrap (idempotent)
- ops/consul/ , systemd units, terraform, monitoring
## Server topology (MANDATORY)
- Servers: 3 or 5, never 2 or 4 (no quorum with even numbers, single point of failure with one)
- Servers run in private subnets, no public exposure
- Server-to-server RPC on port 8300 restricted to server CIDR
- Bootstrap-expect matches the actual server count
- Raft snapshot interval explicit, autopilot enabled
## Client topology (MANDATORY)
- One client agent per host, running as a system service
- Client agent registered services via local config files (NOT HTTP API in production)
- Client agent never runs in -dev mode outside of explicit local testing
- Client agent token has node:write but no operator privileges
## ACL configuration (MANDATORY)
- acl.enabled: true
- acl.default_policy: deny
- acl.enable_token_persistence: true
- Bootstrap token stored in secrets manager, NEVER committed
- Anonymous token policy explicitly defined (default deny, override for specific reads only)
## Intentions (MANDATORY)
- default_intention_policy: deny
- One intention per service-pair (no wildcards in production)
- Intentions defined as HCL files, applied via consul intention create
- intentions/ directory is the source of truth, sync to Consul on deploy
## KV namespacing (MANDATORY)
- Top-level prefix: <environment>/<service>/<key>
- Secrets in kv-encrypted/ prefix, written with the connect-injector sidecar pattern
- KV writes go through the ACL token for the writing service, never with the bootstrap token
- NEVER store unencrypted secrets in KV without the encryption sidecar
## Gossip and RPC encryption
- -encrypt key rotated annually via consul keyring rotate
- TLS for HTTP API, RPC, and Connect CA (NOT optional)
- Auto-encrypt enabled for client-to-server TLS bootstrap
## Hard rules
- NEVER run consul agent -dev in production or staging
- NEVER set acl.default_policy to allow
- NEVER create an intention with src or dest of "*" in production
- NEVER write secrets to KV without the encryption sidecar
- NEVER expose the HTTP API on the public interface
- NEVER use the bootstrap token after the initial setup
- ALWAYS validate HCL with consul validate before reload
- ALWAYS register services via local config files, not HTTP API
Six rules here prevent the majority of incidents Claude generates without them.
The odd server count rule is operational hygiene. Consul uses Raft for consensus, which requires a majority of servers to be reachable for writes. Three servers can lose one and still operate; two servers can lose neither without losing quorum. Five servers can lose two. Four servers can only lose one before quorum is lost, which gives you no benefit over three and twice the operational cost. Claude's instinct is to write "two servers for redundancy", which is the worst possible topology.
The default-deny ACL rule kills the worst class of production incident. With acl.default_policy: allow, any token (or no token) can read any KV path, register any service, and modify any intention. With acl.default_policy: deny, every action requires an explicit ACL policy. The setup cost is one-time. The payoff is that an attacker who steals an unrelated token cannot pivot to your secrets.
The default-deny intentions rule is the same principle at the service-mesh layer. With default_intention_policy: deny, no service can call any other service until an intention is created. The intention says "service A can call service B." Claude's instinct without this rule is to create an intention with src: "*", dest: "*", action: allow "to make things work." That intention is the service mesh equivalent of disabling the firewall.
The KV namespacing rule prevents accidental key collisions and gives the ACL policies a natural shape. With keys structured as <environment>/<service>/<key>, the ACL policy for service A says key_prefix "production/service-a/" { policy = "write" } and nothing else. Service A cannot read service B's keys. Production cannot read staging. The hierarchy enforces itself.
Install and server bootstrap
Consul ships as a single Go binary. Install via the package manager or download the release.
# Debian/Ubuntu
sudo apt install consul
# macOS
brew install consul
Generate the gossip encryption key once and store it in your secrets manager:
consul keygen
# kVqQGQHNbTJDhM3PqEcLqGy5d/2dpEmZJqyMfb7XmHE=
Create the server config that every Consul server will use:
# consul/server.hcl
datacenter = "production"
data_dir = "/var/lib/consul"
log_level = "INFO"
node_name = "consul-server-{{ENV \"NODE_INDEX\"}}"
server = true
bootstrap_expect = 3
ui_config {
enabled = true
}
bind_addr = "{{ GetInterfaceIP \"eth0\" }}"
client_addr = "127.0.0.1"
advertise_addr = "{{ GetInterfaceIP \"eth0\" }}"
retry_join = [
"provider=aws region=eu-west-2 tag_key=Role tag_value=consul-server",
]
acl {
enabled = true
default_policy = "deny"
enable_token_persistence = true
tokens {
initial_management = ""
agent = ""
}
}
connect {
enabled = true
}
ports {
grpc = 8502
grpc_tls = 8503
https = 8501
http = -1
}
tls {
defaults {
ca_file = "/etc/consul.d/tls/ca.crt"
cert_file = "/etc/consul.d/tls/server.crt"
key_file = "/etc/consul.d/tls/server.key"
verify_incoming = true
verify_outgoing = true
}
internal_rpc {
verify_server_hostname = true
}
}
auto_encrypt {
allow_tls = true
}
performance {
raft_multiplier = 1
}
telemetry {
prometheus_retention_time = "60s"
disable_hostname = true
}
Every block here enforces a rule from the CLAUDE.md. The bootstrap_expect = 3 says we run a three-server cluster. The acl block enables ACLs in default-deny mode. The ports.http = -1 disables plaintext HTTP entirely; only HTTPS on 8501 is accepted. The tls block requires mutual TLS for both incoming and outgoing connections, and the internal RPC layer verifies server hostnames against the cert SAN. The auto_encrypt.allow_tls = true lets clients bootstrap their TLS certs from the server CA, so the operator does not have to distribute per-client certs by hand.
The retry_join uses Consul's cloud discovery to find other servers by tag, which removes the chicken-and-egg of "how does the third server find the first two when DNS is not ready yet." For non-AWS, the equivalents work for GCP, Azure, and DigitalOcean.
Bootstrapping ACLs
After the cluster forms, bootstrap the ACL system to generate the initial management token. This is a one-time operation per cluster.
consul acl bootstrap > /tmp/bootstrap.json
The output contains the management token, the accessor ID, and a secret ID. Copy the secret ID into your secrets manager immediately, then delete the file:
jq -r '.SecretID' /tmp/bootstrap.json | \
vault kv put secret/consul/bootstrap_token value=-
rm /tmp/bootstrap.json
The bootstrap token has every permission. Use it once to create policies and the per-service tokens you actually need, then never use it again. Operating Consul day-to-day with the bootstrap token is the equivalent of using a root account for every shell session.
Create the policies for the agents and services:
# consul/policies/client-agent.hcl
node_prefix "" {
policy = "write"
}
service_prefix "" {
policy = "read"
}
session_prefix "" {
policy = "read"
}
Apply the policy and generate a token for the client agents:
consul acl policy create -name client-agent -rules @consul/policies/client-agent.hcl
consul acl token create -description "client agent" -policy-name client-agent
The output token goes into the client agent config, never the bootstrap token. Repeat the pattern for each service: write a policy that only grants the keys and services that service actually needs, create a token bound to that policy, deploy the token with the service.
Client agent and service registration
The client agent runs on every host, registers services with the cluster, runs health checks, and proxies API calls to the servers. The client config is smaller than the server config but every field matters.
# consul/client.hcl
datacenter = "production"
data_dir = "/var/lib/consul"
log_level = "INFO"
node_name = "{{ env \"HOSTNAME\" }}"
server = false
bind_addr = "{{ GetInterfaceIP \"eth0\" }}"
client_addr = "127.0.0.1"
advertise_addr = "{{ GetInterfaceIP \"eth0\" }}"
retry_join = [
"provider=aws region=eu-west-2 tag_key=Role tag_value=consul-server",
]
acl {
enabled = true
default_policy = "deny"
tokens {
agent = "{{ env \"CONSUL_AGENT_TOKEN\" }}"
}
}
connect {
enabled = true
}
ports {
grpc = 8502
grpc_tls = 8503
https = 8501
http = -1
}
auto_encrypt {
tls = true
}
tls {
defaults {
ca_file = "/etc/consul.d/tls/ca.crt"
verify_incoming = false
verify_outgoing = true
}
}
The client's agent token is the one generated from the client-agent policy. The auto_encrypt.tls = true says the client should request a TLS cert from the server on join, which removes the need to distribute per-host certs.
Register services via dropped config files in /etc/consul.d/services/:
# consul/services/api.hcl
service {
name = "api"
port = 8080
tags = ["production", "v2"]
check {
id = "api-health"
name = "API health endpoint"
http = "http://localhost:8080/healthz"
interval = "10s"
timeout = "2s"
success_before_passing = 2
failures_before_critical = 3
}
connect {
sidecar_service {
proxy {
upstreams {
destination_name = "postgres"
local_bind_port = 5432
}
upstreams {
destination_name = "redis"
local_bind_port = 6379
}
}
}
}
token = "{{ env \"CONSUL_API_SERVICE_TOKEN\" }}"
}
The service registers itself with a name, port, tags, and a health check. The connect.sidecar_service.proxy.upstreams block declares which other services this service calls, with the local port the sidecar binds for each. The service code connects to localhost:5432 for Postgres and localhost:6379 for Redis, and the sidecar handles the mTLS to the actual upstream services.
The per-service token has write on its own service registration and read on the upstreams it declares. It cannot register other services, cannot read unrelated KV paths, cannot modify intentions. If the token is compromised, the blast radius is the one service.
Intentions for service-to-service authorisation
With default_intention_policy: deny, no service can call any other service until an intention permits it. Define intentions as declarative HCL files and apply them via the CLI.
# consul/intentions/api-to-postgres.hcl
Sources = [
{
Name = "api"
Action = "allow"
}
]
DestinationName = "postgres"
Apply with:
consul intention create -file consul/intentions/api-to-postgres.hcl
Repeat for every service pair: API to Postgres, API to Redis, worker to Postgres, scheduler to API. The directory becomes the source of truth for the service mesh topology, and a CI step on the directory applies any changes.
For services that need to deny specific peers within a broader allow, use deny actions:
Sources = [
{
Name = "audit-log-reader"
Action = "deny"
},
{
Name = "api"
Action = "allow"
}
]
DestinationName = "postgres"
Intentions evaluate in source-name order: the most specific match wins. The audit-log-reader source is denied even though a broader policy might allow it.
For Claude Code with custom subagents that operate inside a Consul-meshed environment, the same intention model applies: each agent gets a service registration, a token, and intentions to the upstreams it needs. The mesh enforces the boundary even if the agent code is buggy.
KV namespace and secrets
The KV store is convenient for application config, feature flags, and small operational state. The right pattern is a strict hierarchical namespace with ACL policies that grant write to the owning service and read to consumers.
# Application config for the api service in production
consul kv put production/api/feature_flags '{"new_checkout":true}'
consul kv put production/api/rate_limits '{"per_user":100}'
# Read by the api service token, denied to everyone else
The policy for the api service includes:
key_prefix "production/api/" {
policy = "write"
}
key_prefix "production/shared/" {
policy = "read"
}
The service can write to its own namespace and read from the shared namespace, where common config lives. The structure scales: adding a new service means adding a <env>/<service>/ prefix and a matching ACL policy. The bounded scope makes "what does this service have access to" a one-line answer.
For secrets, never write them to KV directly. Consul KV is not a secrets manager, and the gossip network has weaker guarantees than the secrets backends. The right pattern is to use Vault (or your cloud secrets manager) for secrets, with Consul reserved for non-sensitive config. If Consul KV must hold secrets, use the consul-template Vault integration that writes encrypted blobs from Vault into KV with the application token, and unwrap them in the application.
Permission hooks for Consul operations
Consul has CLI commands and HTTP API endpoints that can take down the cluster, leak the bootstrap token, or rewrite intentions out of band. Permission hooks in .claude/settings.local.json gate the destructive ones.
{
"permissions": {
"allow": [
"Bash(consul members*)",
"Bash(consul catalog*)",
"Bash(consul kv get*)",
"Bash(consul kv list*)",
"Bash(consul intention list*)",
"Bash(consul acl policy list*)",
"Bash(consul validate*)"
],
"deny": [
"Bash(consul leave*)",
"Bash(consul force-leave*)",
"Bash(consul kv delete*)",
"Bash(consul kv put*)",
"Bash(consul intention delete*)",
"Bash(consul acl bootstrap*)",
"Bash(consul acl token delete*)",
"Bash(consul agent -dev*)",
"Bash(consul snapshot restore*)"
]
}
}
The allow list permits read-only inspection. The deny list catches the operations that remove servers, delete keys, wipe tokens, restore from snapshots, or start a dev-mode agent. Combined with the Claude Code hooks system for project-wide policy, the result is a setup where Claude can investigate cluster state without being able to mutate it.
Common Claude Code mistakes with Consul
Six patterns Claude generates incorrectly without CLAUDE.md constraints.
1. acl.default_policy: allow
Claude generates: a server config with no ACL block or with default_policy = "allow" "for development."
Correct pattern: acl.enabled = true, acl.default_policy = "deny", with explicit policies for every agent and service.
2. Wildcard intention
Claude generates: consul intention create '*' '*' "to unblock service calls."
Correct pattern: one intention per source-destination pair, declarative HCL files in consul/intentions/.
3. Even number of servers
Claude generates: bootstrap_expect = 2 "for redundancy."
Correct pattern: 3 or 5 servers, never 2 or 4.
4. Bootstrap token in config
Claude generates: a script that pipes the bootstrap output to the agent config.
Correct pattern: bootstrap once, store the token in a secrets manager, never use it again, use per-service tokens for everything else.
5. KV writes with bootstrap token
Claude generates: consul kv put ... -token=$BOOTSTRAP_TOKEN because that token works.
Correct pattern: KV writes go through the ACL token of the writing service, scoped to that service's key prefix.
6. Service registration via HTTP API
Claude generates: a deploy script that POSTs to /v1/agent/service/register.
Correct pattern: drop a config file in /etc/consul.d/services/ and run consul reload, so the registration is in source control and idempotent.
Add each pattern to CLAUDE.md with the corrected form. Combined with CLAUDE.md examples for the general structure, the result is a Consul deployment Claude generates correctly the first time.
Observability and metrics
Consul exposes Prometheus metrics on the HTTP API endpoint. Configure scrape via the official Consul Prometheus exporter or directly from /v1/agent/metrics?format=prometheus.
- job_name: consul
metrics_path: /v1/agent/metrics
params:
format: [prometheus]
scheme: https
tls_config:
ca_file: /etc/consul.d/tls/ca.crt
insecure_skip_verify: false
static_configs:
- targets:
- consul-server-1:8501
- consul-server-2:8501
- consul-server-3:8501
The metrics that matter in production: consul_raft_leader_lastContact for Raft health, consul_raft_apply for write throughput, consul_kvs_apply_count for KV churn, consul_intention_apply for intention changes, consul_acl_resolveToken for token resolution rate, consul_session_apply_count for session lifecycle.
Alerts that pay back the setup: no leader for 30 seconds (cluster is wedged), Raft last contact above 200 ms (network partition or under-provisioned servers), ACL token resolution errors above zero (a service is using a missing token), intention apply rate above 10 per minute (someone is editing intentions manually). For dashboards, pair with Claude Code with Grafana and use the official Consul dashboard as a starting point.
For tracing the path of an individual request through the mesh, enable Connect tracing in the sidecar config and ship the traces to your observability stack. Claude Code with Datadog covers the patterns for sidecar tracing across the mesh.
Testing Consul changes
Every change to a service registration, intention, or ACL policy should be validated before apply. The consul validate command checks HCL syntax. For semantic validation, deploy to a staging cluster first and run smoke tests.
#!/bin/bash
set -euo pipefail
# Validate every HCL file
for file in consul/services/*.hcl consul/intentions/*.hcl consul/policies/*.hcl; do
consul validate "$file"
done
# Apply to staging
export CONSUL_HTTP_ADDR=https://consul-staging:8501
export CONSUL_HTTP_TOKEN=$(vault kv get -field=value secret/consul/staging-deploy)
for file in consul/intentions/*.hcl; do
consul intention create -replace -file "$file"
done
# Smoke test: a service that should be able to call its upstream
curl -sf https://api-staging.example.com/healthz
curl -sf https://api-staging.example.com/api/users
# Smoke test: a service that should be denied
output=$(curl -sf https://api-staging.example.com/admin/secrets 2>&1 || true)
echo "$output" | grep -q "denied" || { echo "expected denial"; exit 1; }
The smoke tests verify both the allow paths (services that should reach their upstreams) and the deny paths (paths that should be blocked by intentions). The second category is the one Claude misses: testing that the right calls fail.
Building Consul that does not become an attack surface
The Consul CLAUDE.md in this guide produces a deployment where the server count is odd, ACLs are default-deny, intentions are default-deny, KV uses a hierarchical namespace, secrets stay in a real secrets manager, the bootstrap token is used once and stored in vault, services register via local config files, and every change is validated before it reaches production. The result is a service mesh that operators can trust to enforce the boundaries the manifest declares.
The underlying principle is the same as any service mesh with Claude Code. Consul without a CLAUDE.md produces a working install that conceals an open mesh where any compromised service can call any other and read any KV path. The CLAUDE.md is what makes the production-safe defaults the default for Claude.
For the alternative cloud-native edge that runs in Kubernetes and uses its own CRDs for service routing, Claude Code with Traefik covers the equivalent patterns. For the single-host reverse proxy where a full service mesh is overkill, Claude Code with Caddy covers the simpler setup. For the workflow orchestration layer that often sits behind a Consul-meshed application stack, Claude Code with Temporal covers the patterns for durable execution.
Get Claudify. Ship a Consul mesh that enforces the boundaries you declared.
More like this
Ready to upgrade your Claude Code setup?
Get Claudify
