Deployment

Deployment means one thing here: something launches your task on a schedule, in an environment, with credentials. The task folder itself never changes. This page is the map; each environment has its own page with the exact steps.

Where you can deploy today

Environment	Status	Page
Databricks	Proven end to end, runs in CI on every merge	Databricks
Kubernetes	Working manifests and images, run in CI on every pull request	Kubernetes
Docker	Working images, the base for Kubernetes and the clouds	Kubernetes
AWS EMR Serverless	Scripts and CI jobs written, waiting for an account	AWS
GCP Dataproc Serverless	Scripts and CI jobs written, waiting for an account	GCP
Anything with spark-submit	Supported through python -m ubunye	Anywhere

One rule applies everywhere and is worth learning once: the task never chooses its own cluster. Leave spark.master out of your config. Whoever launches the job owns that decision, and the engine refuses a config that tries to override a platform's choice, because a silent single machine run on paid compute is worse than an error.

---

The Databricks flow in detail

Ubunye pipelines run on Databricks via Databricks Asset Bundles (DABs) and are deployed through GitHub Actions.

DABs belong in the usecase repo

Bundle definitions (bundles/, databricks.yml) are usecase-specific and live in the usecase repository, not in the Ubunye Engine repo. The engine provides the runtime; the usecase repo defines the jobs.

---

Deployment flow

Push code → GitHub Actions validates → Merge to main → GitHub Actions deploys bundle → Databricks runs on schedule

Stage	Where	What happens
PR	GitHub Actions	ubunye validate --all + pytest
Merge to main	GitHub Actions	databricks bundle deploy --target nonprod
Production deploy	GitHub Actions (manual)	databricks bundle deploy --target prod
Execution	Databricks	Scheduled job runs the pipeline

---

Databricks Asset Bundles

Jobs are defined as code in the usecase repo.

databricks.yml (usecase repo root)

Defines bundle name and deployment targets:

bundle:
  name: "my-usecase-pipelines"

include:
  - "bundles/*.yaml"

targets:
  nonprod:
    mode: development
    default: true
    workspace:
      host: ${DATABRICKS_HOST}
    variables:
      mode: "nonprod"
      unity_catalog: "aws-db-nonprod-tl-catalog"

  prod:
    mode: production
    workspace:
      host: ${DATABRICKS_HOST}
    variables:
      mode: "prod"
      unity_catalog: "aws-db-prod-tl-catalog"

Job definitions (bundles/*.yaml)

Each YAML file in bundles/ defines a Databricks job:

resources:
  jobs:
    monthly_rewards:
      name: "ubunye-monthly-rewards"
      schedule:
        quartz_cron_expression: "0 0 6 1 * ?"
        timezone_id: "UTC"
      tasks:
        - task_key: "run_pipeline"
          python_wheel_task:
            package_name: "ubunye_engine"
            entry_point: "ubunye"
            parameters: ["run", "-d", "...", "-t", "..."]

---

GitHub Actions workflow

The deploy workflow (.github/workflows/deploy.yml in the usecase repo) handles CI and CD:

On pull request:

• Validates pipeline configs with ubunye validate
• Runs unit tests with pytest

On merge to main:

• Deploys the Databricks Asset Bundle to nonprod

Required secrets

Two auth flows are supported — pick one per workspace. See Databricks Authentication for the full setup.

Flow	Secrets
PAT (Free Edition / single-user)	DATABRICKS_HOST, DATABRICKS_TOKEN
Service principal + OAuth (recommended, paid workspaces)	DATABRICKS_HOST, DATABRICKS_CLIENT_ID, DATABRICKS_CLIENT_SECRET

The Databricks CLI auto-selects OAuth when DATABRICKS_CLIENT_ID and DATABRICKS_CLIENT_SECRET are both set; otherwise it falls back to PAT.

Example workflow

name: deploy
on:
  pull_request:
    branches: [main]
  push:
    branches: [main]

jobs:
  validate:
    if: github.event_name == 'pull_request'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: "3.11" }
      - run: pip install -e .[dev]
      - run: ubunye validate -d ./pipelines -u my_usecase -p my_package --all
      - run: pytest tests/unit/ -v --tb=short -m "not integration"

  deploy-nonprod:
    if: github.event_name == 'push' && github.ref == 'refs/heads/main'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install databricks-cli
      - run: databricks bundle deploy --target nonprod
        env:
          DATABRICKS_HOST: ${{ secrets.DATABRICKS_HOST }}
          DATABRICKS_TOKEN: ${{ secrets.DATABRICKS_TOKEN }}

---

Python API on Databricks

On Databricks, use the Python API instead of the CLI. The API reuses the active SparkSession:

import ubunye

# Single task
outputs = ubunye.run_task(
    task_dir="/Workspace/pipelines/fraud_detection/ingestion/claim_etl",
    mode="nonprod",
    dt="202510",
)

# Multiple tasks
results = ubunye.run_pipeline(
    usecase_dir="/Workspace/pipelines",
    usecase="fraud_detection",
    package="ingestion",
    tasks=["claim_etl", "feature_engineering"],
    mode="nonprod",
    dt="202510",
)

The Python API sets descriptive Spark app names (ubunye:<usecase>.<package>.<task>) for easy identification in the Spark UI and history server.

Why not the CLI on Databricks?

The CLI creates a new SparkSession, wasting the one Databricks already has running. The Python API detects the active session and reuses it.

---

Dev notebooks

ubunye init generates a dev notebook at notebooks/<task>_dev.ipynb for each task. The notebook mirrors the production pipeline interactively:

1. Parameters — dbutils.widgets for effective_year_month and mode
2. Setup — loads config, creates DatabricksBackend
3. Extract — reads all inputs, prints row counts
4. Inspect Sources — display() each input DataFrame
5. Transform — runs the Task class from transformations.py
6. Inspect Outputs — display() each output DataFrame
7. Load — writes outputs (commented out by default)
8. Sandbox — Spark session exposed for free exploration

The notebook reads from the same config.yaml so there is zero drift between dev and production.