Claude Code with Databricks: Notebooks, Jobs, and Unity Catalog
Why Databricks without CLAUDE.md burns compute hours
Databricks is the most flexible data platform available to engineering teams in 2026. The platform combines Spark-based ETL, Unity Catalog governance, Delta Lake storage, MLflow experiment tracking, and Databricks SQL warehousing under one workspace. The flexibility is also the problem when Claude Code touches it. Without explicit constraints, Claude generates code that runs, but spins up an interactive cluster for a one-off job, uses two-part table names that break under Unity Catalog, mixes Spark and pandas in a way that pulls data to the driver node, and writes jobs in the legacy single-task format instead of the modern multi-task Asset Bundle structure.
The most common Claude defaults that hurt Databricks workflows: hardcoding cluster IDs in notebooks, using spark.read.csv() against a single file path when the upstream data lives in a Volume that needs the Unity Catalog three-part name, calling .toPandas() on a DataFrame the size of the source table (collecting gigabytes to the driver), writing notebooks as one giant cell instead of structured tasks, and missing the dbutils.widgets.get() pattern that lets a notebook accept parameters from a job definition. On the SDK side, Claude often initialises the Databricks client with personal access tokens hardcoded in the source file instead of reading from the unified authentication chain.
This guide covers the CLAUDE.md configuration that locks Claude Code into Databricks's correct model: Unity Catalog three-part names everywhere, the SDK loaded from environment-based authentication, jobs declared in Asset Bundles, and notebooks structured as parameterised tasks instead of monolithic scripts. If you are building a data warehouse layer that feeds Databricks from Postgres, Claude Code with PostgreSQL covers the source-side patterns. For Python notebook environments specifically, Claude Code with Python shows the package management approach that pairs with the Databricks Runtime.
The Databricks CLAUDE.md template
The CLAUDE.md at your project root is read at the start of every Claude Code session. For a Databricks integration it needs to declare: the SDK version, the Unity Catalog naming policy, the authentication chain, the cluster reuse policy, the Asset Bundle layout, and the hard rules that block the mistakes Claude makes most often.
# Databricks rules
## Stack
- databricks-sdk-py ^0.30.x, Python 3.11
- databricks-connect ^15.x for local Spark sessions
- Databricks Asset Bundles for job + pipeline declarations
- Databricks Runtime 15.x LTS on all clusters
- DATABRICKS_HOST and DATABRICKS_TOKEN in .env (never hardcode)
## Project structure
- src/etl/ : Python modules with @task-decorated functions
- src/notebooks/ : Notebook source files (.py with magic markers)
- src/jobs/ : Asset Bundle YAML for job + pipeline definitions
- databricks.yml : Root Asset Bundle config (bundle.target, workspace)
- src/lib/spark_session.py : Spark session factory for local + remote contexts
## Authentication (MANDATORY)
- Production: DATABRICKS_HOST + DATABRICKS_TOKEN via unified auth chain
- Local dev: databricks configure or .databrickscfg profile
- NEVER hardcode tokens in source files, notebooks, or Asset Bundle YAML
- NEVER commit .databrickscfg, .env, or any file with DATABRICKS_TOKEN
## Unity Catalog naming (HARD RULE)
Every table reference MUST use three-part names: catalog.schema.table
Correct:
spark.table("main.bronze.events")
CREATE TABLE main.silver.users (...)
Incorrect (Claude default):
spark.table("events")
CREATE TABLE users (...)
This applies to: spark.read.table, spark.sql, CREATE/DROP/ALTER statements,
%sql magic cells, and dbutils.fs paths to Volumes (/Volumes/catalog/schema/volume)
## Spark vs pandas (HARD RULE)
- NEVER call .toPandas() on a DataFrame without explicit row count gate
- NEVER use pandas.read_csv on a path that lives in DBFS or a Volume
- ALWAYS prefer Spark transformations over pandas operations on raw data
- pandas is acceptable AFTER .limit(N) where N <= 100,000 with comment justifying
## Cluster policy (cost rule)
- NEVER create a new interactive cluster for a notebook that runs once
- ALWAYS check for an existing job cluster definition before spinning up a new one
- Job tasks default to job_cluster, not existing_cluster_id
- existing_cluster_id ONLY for live debugging, never in committed code
## Delta Lake patterns
- Writes: df.write.format("delta").mode("append" | "overwrite").saveAsTable(...)
- Upserts: use DeltaTable.merge() pattern, never write-then-delete
- Reads: spark.table(name) preferred over spark.read.format("delta").load(path)
- Time travel: spark.read.option("versionAsOf", N) for replay scenarios
## Job definitions
- ALL jobs declared in src/jobs/*.yml as Asset Bundle resources
- Multi-task jobs use the tasks: list with depends_on
- Notebook tasks: notebook_task with notebook_path + base_parameters
- Python wheel tasks: python_wheel_task with package_name + entry_point
- NEVER use the legacy single-task format (task_type: notebook at root)
## Hard rules
- NEVER hardcode DATABRICKS_TOKEN, cluster IDs, or warehouse IDs
- NEVER use two-part names (schema.table) when Unity Catalog is enabled
- NEVER call .collect() or .toPandas() on a multi-million row DataFrame
- NEVER write a notebook as a single cell, structure cells around logical steps
- ALWAYS set explicit dbutils.widgets.text() for every parameter the notebook reads
- ALWAYS handle nulls in MERGE conditions explicitly
Four rules here prevent the majority of production failures Claude generates without them.
The Unity Catalog three-part name rule is the most impactful for governance and correctness. Two-part names (schema.table) resolve against the current catalog, which depends on cluster configuration and session state. Three-part names (catalog.schema.table) are unambiguous and survive any context. Claude omits the catalog by default because tutorial code still uses the older Hive-style references.
The .toPandas() gate rule prevents driver memory exhaustion. Spark DataFrames can hold billions of rows distributed across executors. Calling .toPandas() collects every row to the driver node, which has limited memory. Claude generates .toPandas() for convenience because pandas is more familiar from training data. The CLAUDE.md rule forces an explicit .limit(N) before any conversion, with N capped at 100,000.
The cluster reuse rule prevents cost blowouts. Each interactive cluster has startup time (typically 3 to 5 minutes) and minimum billing duration. Spinning up a new cluster for every notebook run multiplies cost without delivering more compute. Job clusters defined in Asset Bundles reuse pooled instances and shut down automatically when the job completes.
Install and authentication setup
Install the SDK, connector, and bundle CLI:
pip install databricks-sdk databricks-connect
Install the Databricks CLI for Asset Bundles:
brew tap databricks/tap
brew install databricks
databricks --version
Configure authentication with a profile:
databricks configure --token --profile DEFAULT
# Prompts for Databricks Host URL and Personal Access Token
This writes ~/.databrickscfg. The SDK and CLI both read from this file by default. For CI environments, use environment variables instead:
# .env (local) or CI secret store
DATABRICKS_HOST=https://your-workspace.cloud.databricks.com
DATABRICKS_TOKEN=dapi_your_token_here
Create the spark session factory that every notebook and ETL module imports:
# src/lib/spark_session.py
import os
from databricks.connect import DatabricksSession
from pyspark.sql import SparkSession
def get_spark() -> SparkSession:
"""Return a Spark session that works locally and on a Databricks cluster."""
if "DATABRICKS_RUNTIME_VERSION" in os.environ:
# Inside a Databricks cluster, use the existing session
return SparkSession.builder.getOrCreate()
# Local development via Databricks Connect
return DatabricksSession.builder.profile("DEFAULT").getOrCreate()
The DATABRICKS_RUNTIME_VERSION check is the canonical way to detect whether the code is running inside a Databricks notebook or remotely via Databricks Connect. Claude rarely generates this check without an explicit pattern in CLAUDE.md, because tutorial code assumes one execution context.
Unity Catalog and three-part names
Unity Catalog is the governance layer that sits above Delta Lake. Every table belongs to a catalog and a schema. A typical layout uses three catalogs for the medallion architecture: main for production data with bronze, silver, and gold schemas; dev for development; and sandbox for ad-hoc exploration.
Every table reference in code, notebooks, and SQL needs the three-part name. The Spark default of resolving against the current catalog leads to subtle bugs when the same code runs in dev and prod contexts:
# Correct: explicit three-part names
df = spark.table("main.bronze.raw_events")
silver_df = (
df
.filter("event_type = 'order_placed'")
.withColumn("processed_at", current_timestamp())
)
(silver_df
.write
.format("delta")
.mode("append")
.saveAsTable("main.silver.orders"))
For SQL inside notebooks or Spark SQL strings:
spark.sql("""
CREATE TABLE IF NOT EXISTS main.silver.orders (
order_id STRING,
user_id STRING,
amount DECIMAL(18, 2),
processed_at TIMESTAMP
)
USING DELTA
PARTITIONED BY (DATE(processed_at))
""")
For Volume paths (managed storage for non-tabular files):
csv_path = "/Volumes/main/landing/uploads/2026-05-31.csv"
df = spark.read.option("header", True).csv(csv_path)
Add a Unity Catalog reference section to CLAUDE.md:
## Unity Catalog layout
### Catalogs
- main: production data, three schemas (bronze, silver, gold)
- dev: development environment, mirrors main schema names
- sandbox: ad-hoc exploration, no SLAs, anyone can write
### Schema conventions
- bronze: raw ingestion, append-only, retains source schema
- silver: cleansed and conformed, source of truth for analytics
- gold: aggregated marts, dimensional models, denormalised
### Volumes
- main.landing.uploads: inbound files (S3 or ADLS mount)
- main.landing.exports: outbound files to downstream consumers
- main.ml.artifacts: ML model checkpoints and training data snapshots
Claude will not reproduce this naming convention without it in CLAUDE.md. The default pattern from training data is database.table or just table, which works in older Databricks workspaces but breaks in Unity Catalog enabled environments.
Asset Bundles for jobs and pipelines
Asset Bundles are the declarative layer for everything that runs on Databricks: jobs, pipelines, Lakehouse Monitoring, ML experiments, dashboards. The bundle YAML lives in version control. The databricks bundle deploy command syncs it to the workspace. This replaces the manual UI-based job configuration that older Databricks workflows used.
The bundle root:
# databricks.yml
bundle:
name: claudify-data-platform
include:
- src/jobs/*.yml
targets:
dev:
mode: development
default: true
workspace:
host: https://dev-workspace.cloud.databricks.com
prod:
mode: production
workspace:
host: https://prod-workspace.cloud.databricks.com
run_as:
service_principal_name: claudify-data-sp
A multi-task job that ingests data, transforms it, and writes the silver layer:
# src/jobs/daily_etl.yml
resources:
jobs:
daily_etl:
name: daily_etl
schedule:
quartz_cron_expression: "0 0 5 * * ?"
timezone_id: UTC
job_clusters:
- job_cluster_key: etl_cluster
new_cluster:
spark_version: 15.4.x-scala2.12
node_type_id: i3.xlarge
num_workers: 4
data_security_mode: SINGLE_USER
tasks:
- task_key: ingest_bronze
job_cluster_key: etl_cluster
notebook_task:
notebook_path: ../notebooks/ingest_bronze.py
base_parameters:
source_date: "{{job.start_time.iso_date}}"
- task_key: transform_silver
job_cluster_key: etl_cluster
depends_on:
- task_key: ingest_bronze
notebook_task:
notebook_path: ../notebooks/transform_silver.py
base_parameters:
source_date: "{{job.start_time.iso_date}}"
- task_key: build_gold_marts
job_cluster_key: etl_cluster
depends_on:
- task_key: transform_silver
python_wheel_task:
package_name: claudify_etl
entry_point: build_gold_marts
parameters: ["{{job.start_time.iso_date}}"]
Deploy and run:
databricks bundle validate --target dev
databricks bundle deploy --target dev
databricks bundle run daily_etl --target dev
Add an Asset Bundle section to CLAUDE.md:
## Asset Bundles
- Bundle root: databricks.yml (targets, include patterns)
- Job files: src/jobs/*.yml (one job per file, resources.jobs.<name>)
- ALWAYS define job_clusters at the job level, NEVER existing_cluster_id in tasks
- Multi-task: use tasks: list with explicit depends_on
- Notebook tasks: notebook_task.notebook_path RELATIVE to bundle root
- Python wheel tasks: python_wheel_task for production ETL, faster than notebooks
- Parameters: base_parameters for notebooks, parameters for wheels
- ALWAYS pin spark_version to a specific LTS release (15.4.x-scala2.12)
- ALWAYS use data_security_mode: SINGLE_USER for Unity Catalog jobs
The single-task vs multi-task distinction matters. Older Databricks tutorials show jobs as a single notebook with task_type: notebook at the root of the YAML. This format still works but is being deprecated. The multi-task format with tasks: [...] is the supported pattern going forward, and it lets you orchestrate dependencies, branching, and conditional execution within one job. Claude defaults to the single-task format because it appears in more training data.
Parameterised notebooks
A notebook that runs inside a job needs to accept parameters. The dbutils.widgets API is the canonical way to do this. Widgets work as both interactive controls (when you open the notebook manually) and as parameter receivers (when a job task passes base_parameters).
# src/notebooks/transform_silver.py
# Databricks notebook source
# COMMAND ----------
dbutils.widgets.text("source_date", "")
dbutils.widgets.text("catalog", "main")
dbutils.widgets.text("source_schema", "bronze")
dbutils.widgets.text("target_schema", "silver")
# COMMAND ----------
source_date = dbutils.widgets.get("source_date")
catalog = dbutils.widgets.get("catalog")
source_schema = dbutils.widgets.get("source_schema")
target_schema = dbutils.widgets.get("target_schema")
if not source_date:
raise ValueError("source_date is required")
print(f"Processing {catalog}.{source_schema} -> {catalog}.{target_schema} for {source_date}")
# COMMAND ----------
from pyspark.sql.functions import col, to_date
source_table = f"{catalog}.{source_schema}.raw_events"
target_table = f"{catalog}.{target_schema}.events"
df = (spark.table(source_table)
.filter(to_date(col("event_ts")) == source_date)
.withColumn("processed_at", current_timestamp())
.select("event_id", "user_id", "event_type", "event_ts", "processed_at"))
(df.write
.format("delta")
.mode("append")
.saveAsTable(target_table))
print(f"Wrote {df.count()} rows to {target_table}")
The # Databricks notebook source magic marker at the top tells Databricks to treat the file as a notebook when imported. The # COMMAND ---------- markers separate cells. This file format lets you commit notebooks to git as plain Python, which Claude can read and modify directly.
Add a notebook structure section to CLAUDE.md:
## Notebook structure
- File starts with: # Databricks notebook source (magic marker)
- Cells separated by: # COMMAND ----------
- First code cell: dbutils.widgets.text(...) for every parameter
- Second cell: read widgets into local variables, validate required ones
- Subsequent cells: one logical step per cell (read, transform, write, summarise)
- NEVER bundle all logic into a single cell
- NEVER read environment variables, ALWAYS use widgets for parameters
- ALWAYS print a summary line at the end (row count, output table name)
The widgets-only-for-parameters rule keeps notebooks portable. A notebook that reads environment variables works when run interactively but fails when scheduled as a job, because job tasks pass parameters via widgets, not the environment. Claude often mixes the two patterns without this rule.
Delta Lake MERGE for upserts
Most ETL workloads need upsert semantics: update rows that already exist, insert rows that do not. Delta Lake's MERGE operation handles this in one statement. The pattern looks like SQL MERGE INTO, but the Python API gives more control over the conditions.
from delta.tables import DeltaTable
from pyspark.sql.functions import col
target_table = "main.silver.users"
target = DeltaTable.forName(spark, target_table)
source_df = spark.table("main.bronze.user_updates").filter("event_date = '2026-05-31'")
(target.alias("t")
.merge(
source_df.alias("s"),
"t.user_id = s.user_id"
)
.whenMatchedUpdate(
condition="s.updated_at > t.updated_at",
set={
"email": "s.email",
"name": "s.name",
"updated_at": "s.updated_at",
}
)
.whenNotMatchedInsert(
values={
"user_id": "s.user_id",
"email": "s.email",
"name": "s.name",
"created_at": "s.created_at",
"updated_at": "s.updated_at",
}
)
.execute())
The whenMatchedUpdate(condition=...) clause prevents stale updates. Without it, every match overwrites the target row even if the source is older. This is the most common silent bug Claude generates with MERGE: copying the basic example from the docs without the timestamp condition that production workflows always need.
The equivalent SQL form:
spark.sql("""
MERGE INTO main.silver.users AS t
USING (
SELECT * FROM main.bronze.user_updates WHERE event_date = '2026-05-31'
) AS s
ON t.user_id = s.user_id
WHEN MATCHED AND s.updated_at > t.updated_at THEN
UPDATE SET
t.email = s.email,
t.name = s.name,
t.updated_at = s.updated_at
WHEN NOT MATCHED THEN
INSERT (user_id, email, name, created_at, updated_at)
VALUES (s.user_id, s.email, s.name, s.created_at, s.updated_at)
""")
Add a MERGE section to CLAUDE.md:
## Delta Lake MERGE
- Use DeltaTable.forName(spark, "catalog.schema.table"), NOT forPath
- whenMatchedUpdate MUST include a condition (typically updated_at comparison)
- whenNotMatchedInsert specifies the full target schema in values dict
- For deletes: whenMatchedDelete with a tombstone condition, NOT whenNotMatched
- After heavy MERGE: OPTIMIZE table_name (compaction)
- After ALL MERGEs: register the operation with a comment for audit trail
- NEVER use spark.write.format("delta").mode("overwrite") for incremental updates
Common Claude Code mistakes with Databricks
Six patterns Claude generates incorrectly without CLAUDE.md constraints, with the correct replacement for each.
1. Two-part table names
Claude generates: spark.table("silver.users") or CREATE TABLE users (...).
Correct pattern: spark.table("main.silver.users") and CREATE TABLE main.silver.users (...).
2. .toPandas() without a limit
Claude generates: df = spark.table("main.silver.events").toPandas() and then iterates with pandas.
Correct pattern: filter and aggregate in Spark, then .limit(100000).toPandas() only for downstream pandas-specific operations that genuinely require it.
3. Interactive cluster for a scheduled job
Claude generates a job YAML with existing_cluster_id: "0531-abc123" pinned to a developer cluster.
Correct pattern: job_clusters: [{ job_cluster_key: ..., new_cluster: { ... } }] at the job level, referenced from each task with job_cluster_key.
4. Hardcoded credentials
Claude generates: client = WorkspaceClient(host="...", token="dapi_xxx").
Correct pattern: client = WorkspaceClient() which reads from .databrickscfg or environment variables.
5. Single-cell notebooks
Claude generates a notebook with all imports, widgets, reads, transforms, and writes in one giant cell.
Correct pattern: cells separated by # COMMAND ----------, one logical step per cell, with print statements between steps for observability.
6. MERGE without timestamp condition
Claude generates: target.merge(source, "t.id = s.id").whenMatchedUpdateAll().whenNotMatchedInsertAll().execute().
Correct pattern: whenMatchedUpdate(condition="s.updated_at > t.updated_at", set={...}) to prevent stale updates from overwriting fresher rows.
Add a common mistakes section to CLAUDE.md with these six pairs. Claude benefits from explicit before/after comparisons because it can match the pattern it would otherwise generate to the corrected form.
Permission hooks for Databricks scripts
A Databricks project accumulates scripts: bundle deployments, local Spark tests, table maintenance jobs, ad-hoc data exports. Some are read-only. Some trigger expensive operations or modify production tables. Permission hooks gate the destructive ones.
In .claude/settings.local.json:
{
"permissions": {
"allow": [
"Bash(databricks bundle validate*)",
"Bash(databricks workspace ls*)",
"Bash(databricks clusters list*)",
"Bash(python -m pytest tests/*)"
],
"deny": [
"Bash(databricks bundle deploy --target prod*)",
"Bash(databricks bundle run *--target prod*)",
"Bash(databricks jobs run-now*)",
"Bash(python scripts/backfill*)"
]
}
}
Validation and read-only operations are safe to run automatically. Production deploys, production job runs, and backfill scripts that rewrite large tables require explicit confirmation. The deny list forces Claude to surface those operations as prompts rather than running them as part of an automated workflow. For more on permission hooks generally, Claude Code permissions covers the full deny-list pattern.
Building Databricks pipelines that scale
The Databricks CLAUDE.md in this guide produces pipelines where every table reference uses the Unity Catalog three-part name, Spark transformations stay in Spark instead of collecting to pandas, jobs run on declarative job clusters instead of pinned interactive ones, notebooks accept parameters through widgets, and MERGE operations include the timestamp condition that prevents stale overwrites.
The underlying principle is the same as any platform integration with Claude Code. Databricks without a CLAUDE.md produces code that runs, but accumulates cost through cluster spin-ups, breaks across environments because of two-part names, and corrupts data because MERGE operations overwrite newer rows with older ones. The CLAUDE.md template removes each failure mode by making the correct pattern the only pattern Claude can generate.
For the next layer up from Spark ETL, Databricks works alongside Claude Code with Drizzle for the application-layer Postgres reads that feed dashboards.
Get Claudify. A complete Claude Code operating system with platform integration templates pre-configured.
More like this
Ready to upgrade your Claude Code setup?
Get Claudify
