Skip to content

Ubunye Engine

Ubunye runs your data and ML jobs from a small folder of YAML and Python, the same way on a laptop, a cloud or Databricks.

  • Write it once. A task is one folder: a config.yaml and a transformations.py.
  • Run it anywhere. pandas with no Java, or Spark on your laptop, a cluster, AWS, GCP, Azure or Databricks.
  • Prove it's the same. Every run can leave a run record with a hash of every row, so two runs can be compared exactly.

Get started GitHub

Try it in 60 seconds

On pandas, so all you need is Python (which version): no Java, no cloud account.

pip install "ubunye-engine[pandas]"
ubunye init -d pipelines -u demo -p starter -t filter_adults
ubunye run -d pipelines -u demo -p starter -t filter_adults --backend pandas --lineage
ubunye lineage list -d pipelines -u demo -p starter -t filter_adults

init makes a task folder with eight sample people in a CSV. run keeps the adults and writes them as Parquet. lineage list shows the run record the run left:

RUN_ID     STARTED              STATUS        DUR  ROWS
------------------------------------------------------------
a2da9fbd  2026-11-10T10:34:34  success      0.3s  in:8  out:5

Next: the Quickstart explains each step, then runs the same folder on Spark.

What you get

The same task, unchanged, has given the same rows on pandas, local Spark, Kubernetes, AWS Glue, GCP Dataproc, Azure Container Apps and Databricks, checked from the run records by the proving ground (latest results).

New to the words below? Concepts explains use case, pipeline, task, profile, write mode, run record and backend in plain words.

  • Config first. Inputs, outputs and Spark settings live in YAML, so no credentials or paths are hard coded. Config overview
  • One config for every environment. A profile (DEV, PROD) changes the Spark settings per environment, chosen with -m. Values can be filled in with Jinja2, a small template language: {{ dt }} becomes the run date. Profiles, Jinja
  • Connectors you declare, not code. Hive, JDBC, files on S3 or local disk, Delta, REST APIs, binary files and Unity Catalog (Databricks' catalog of tables). Connectors
  • Backends. Run on Spark, Databricks, or pandas with no Java (--backend). ubunye backends lists what each can do. Execution backends
  • Checks before a run. ubunye validate (the config), ubunye plan (a dry run: what it will read and write) and ubunye doctor (this machine). Which command catches what
  • Bad data stopped early. Expectations check inputs, put bad rows aside (quarantine) and reconcile input with output. Expectations
  • Run records. --lineage writes one run record per run to .ubunye/lineage/: what ran, on what data, with a hash of every row. ubunye gate and ubunye prove compare them. Run records
  • Safe reruns. A batch that was already appended is refused unless you pass --rerun; --resume finishes a pipeline that stopped half way. Safe reruns
  • Models. A library-independent UbunyeModel contract and a built-in registry to version, promote, roll back and gate models. Model contract, Registry
  • Observability. Prometheus, OpenTelemetry, MLflow and JSON event logs through hooks (ubunye.hooks), and export of run records to OpenLineage (an open standard for sharing what a job read and wrote). Hooks, OpenLineage
  • Orchestration export. One command writes an Airflow DAG, a Databricks job or a Spark Declarative Pipeline from the same config. Orchestration
  • Deployment. ubunye deploy to Databricks, AWS Glue, GCP Dataproc, EMR Serverless, Kubernetes and Azure Container Apps. Deployment
  • Plugins. Add your own readers, writers, transforms and backends through entry points (a standard Python way for an installed package to announce plugins): ubunye.readers, ubunye.writers, ubunye.transforms, ubunye.backends and more. ubunye plugins lists what is installed. Writing your own
  • Agents. ubunye mcp serves the engine over MCP (Model Context Protocol, the way AI agents call tools). It only reads unless you pass --allow-run. Agents over MCP
  • Tested three ways. A unit tier with no Java, a Spark tier, and a parity tier that checks Spark and pandas agree. Contributing

What Ubunye is not

  • Not an agent framework. Use LangChain or CrewAI for that.
  • Not an orchestrator. Use Airflow, Prefect or Dagster. (ubunye export writes an Airflow DAG or a Databricks job for you.)
  • Not a compute engine. It runs on Spark, or on pandas for laptop-sized data.

Ubunye is the layer between your data sources and your applications that makes the plumbing the same everywhere, so you can focus on the logic.

Two ways to run a task

ubunye run -d pipelines -u fraud -p etl -t claims -m PROD --lineage

import ubunye

outputs = ubunye.run_task(
    task_dir="pipelines/fraud/etl/claims",
    mode="PROD",
    dt="2024-06-01",
)

The Python API finds and reuses an active SparkSession on Databricks. When no session exists, it creates one, the same as the CLI. Pass backend="pandas" to run with no Spark at all.

A Spark example with profiles

This task reads a Hive table and writes Delta, so it needs the Spark backend (Spark on your machine, a cluster or Databricks). For a pandas example, see Try it in 60 seconds above.

# pipelines/fraud/etl/claims/config.yaml
MODEL: etl
VERSION: "1.0.0"

ENGINE:
  profiles:
    DEV:
      spark_conf:
        spark.sql.shuffle.partitions: "4"
    PROD:
      spark_conf:
        spark.sql.shuffle.partitions: "200"

CONFIG:
  inputs:
    raw_claims:
      format: hive
      db_name: raw
      tbl_name: claims
  transform: {}
  outputs:
    clean_claims:
      format: delta
      path: s3://data-lake/clean/claims
      mode: overwrite

ubunye plan -d pipelines -u fraud -p etl -t claims -m PROD   # what it will read and write
ubunye run  -d pipelines -u fraud -p etl -t claims -m PROD

-m PROD picks the PROD profile. Leave -m out and run uses DEV. Profile names must match exactly, case included. Profiles

Documentation map

Section What's in it
Installation Install, check it works, extras, Python and Java versions
Quickstart Your first task, end to end, in 5 minutes
Concepts The words the docs use, in plain language
Tutorials Same result on pandas and Spark; a nine-table pipeline with data checks
Writing tasks The full config.yaml reference
Connectors Every built-in reader and writer
Machine learning The UbunyeModel contract and the registry
Examples Working examples, grouped by what you want to do
Deployment Databricks, AWS, GCP, Azure, Kubernetes
How-to guides Notebooks, RAG, drift, REST ingestion, hooks, gates and more
CLI reference Every command and flag
Python API run_task and run_pipeline
Execution backends Spark, Databricks, pandas
Errors Every error, what it means and what to do
Proving ground One workload compared across environments
Developer guide Architecture, plugins, testing