Skip to content

setup-databricks.md

Latest commit

 

History

History
127 lines (95 loc) · 6.21 KB

setup-databricks.md

File metadata and controls

127 lines (95 loc) · 6.21 KB
Raw

Databricks setup

What you need before running anything in this repo, and how to wire credentials.

1. Prerequisites in the workspace

  • Unity Catalog enabled, with a catalog you can write to (default in examples: workspace).
  • A SQL warehouse (Serverless or Pro). Both dlt and dbt connect through it via its HTTP path.
  • Permission to create schemas and tables in your catalog. dlt creates the raw schema (default raw); dbt creates the analytics schema (default analytics).

Find your connection details

SQL Warehouses → select your warehouse → Connection details:

  • Server hostname → DATABRICKS_HOST (without https://)
  • HTTP path → DATABRICKS_HTTP_PATH (e.g. /sql/1.0/warehouses/abcd1234efgh5678)

2. Authentication

Local dev: Databricks CLI OAuth (recommended)

One-time setup — no PAT, no secrets in files:

databricks auth login --host https://YOUR_HOST.cloud.databricks.com
# browser opens → login → done. Credentials saved to ~/.databrickscfg.
  • dbt: auth_type: oauth in profiles.yml reads ~/.databrickscfg automatically. Run uv run dbt debug — no env vars, no browser popup after first login.
  • dlt: still needs a PAT (see below). dlt doesn't yet read ~/.databrickscfg.

dlt: Personal Access Token (PAT)

User Settings → Developer → Access tokens → Generate new token. Put it in .env as DATABRICKS_TOKEN and DESTINATION__DATABRICKS__CREDENTIALS__ACCESS_TOKEN. dlt loads .env automatically via python-dotenv.

Production: OAuth machine-to-machine (M2M / service principal)

Replace the PAT/U2M token with a service principal for any non-local workload:

  • dlt: set client_id / client_secret in ingestion/.dlt/secrets.toml (env vars: DESTINATION__DATABRICKS__CREDENTIALS__CLIENT_ID / __CLIENT_SECRET).
  • dbt: add client_id: / client_secret: env vars to profiles.yml prod target (see the prod: block in profiles.yml.example).

3. Configure this repo

cp .env.example .env
# fill: DATABRICKS_HOST, DATABRICKS_HTTP_PATH, DATABRICKS_TOKEN, DATABRICKS_CATALOG, DATABRICKS_SCHEMA

cp transformation/dbt_databricks/profiles.yml.example transformation/dbt_databricks/profiles.yml
# profiles.yml reads the same env vars, so usually no edits needed

The Python scripts call load_dotenv(), so .env is picked up automatically under uv run. dlt also accepts the same values from ingestion/.dlt/secrets.toml (copy from the .example).

dlt 1.30 treats its entire credentials section as secret-bearing, so these values belong in environment variables or secrets.toml, not config.toml. The examples leave session_timezone unset because serverless SQL warehouses can reject the corresponding spark.sql.session.timeZone session configuration; timestamp-producing examples use explicit UTC-aware values instead.

Which schema does what

Layer Schema (default) Created by Set via
Raw / landed raw dlt DLT_DATASET_NAME
Models (staging→marts) analytics dbt DATABRICKS_SCHEMA

dbt's sources.yml points at the dlt raw schema — that's the handoff. Keep the two schema names consistent across .env and profiles.yml.

4. Verify connectivity

uv sync
uv run python -c "import dlt, dbt; print('imports ok')"
make doctor           # offline readiness check for env, dbt parse, CLI, and bundle config
make dbt-parse        # parses dbt project; needs profiles.yml but does not write tables
make dlt-rest         # first real write: loads a small REST API dataset into <catalog>.raw

If dlt-rest succeeds you'll see the new tables under your catalog's raw schema in the Databricks Catalog explorer.

Verify the Asset Bundle

Asset Bundles read the workspace host from Databricks CLI auth, DATABRICKS_HOST, or an explicit profile. Do not put ${var.workspace_host} in workspace.host; the CLI treats that field as authentication configuration and does not support variable interpolation there.

export DATABRICKS_HOST=https://YOUR_HOST.cloud.databricks.com
databricks bundle validate --var warehouse_id=YOUR_WAREHOUSE_ID

For the full deploy path, see deploy-databricks-bundle.md.

5. Staging for large/file loads (optional)

For file or bulk loads dlt stages data before COPY INTO. On Databricks the simplest staging is a Unity Catalog Volume. Create one (CREATE VOLUME <catalog>.<schema>.dlt_staging;) and reference it from ingestion/.dlt/config.toml. The REST/SQL examples here are small enough to load directly.

For push-based append workloads, ingestion/advanced/zerobus_append.py demonstrates Databricks Zerobus instead of object staging. It needs a supported region plus OAuth service-principal credentials, a Zerobus endpoint, and a managed Delta table outside Databricks default storage; see ../ingestion/README.md for the exact target DDL and least-privilege grants.

For a serverless staging investigation, compare SQL PUT and Files API uploads with the included probe before changing the loader.

Troubleshooting

  • Windows uv sync fails on ruff.exe with OS error 5: an editor's Ruff server can keep .venv/Scripts/ruff.exe open, preventing its replacement. This workspace uses ruff.importStrategy: useBundled so VS Code runs the extension's server instead. If the old server is still running, restart Ruff from the Command Palette, then retry uv sync. Command-line lint and formatting continue to use the project's locked Ruff version.
  • 403 / permission denied → your principal lacks USE CATALOG / CREATE SCHEMA / CREATE TABLE.
  • Zerobus “default storage” error → choose a catalog whose managed location is your own S3 storage. Zerobus cannot write to Databricks default storage.
  • Zerobus Arrow schema mismatch on _dlt_* → pre-create the target with _dlt_load_id and _dlt_id declared STRING NOT NULL, as shown in the ingestion guide.
  • Warehouse won't start / timeouts → confirm the warehouse is running and the HTTP path matches it.
  • dbt can't find profile → pass --profiles-dir transformation/dbt_databricks (the Makefile does).