Changelog¶

All notable changes to Ubunye Engine will be documented here.

The format is based on Keep a Changelog and this project adheres to Semantic Versioning.

[Unreleased]¶

[0.7.0] (2026-09-25)¶

One release where three were planned (0.6, 0.7 and 0.8). The same task folder runs on Spark or on pandas (no Java) and leaves the same run record hash for the same data; that record is now proof: code, environment, input hashes, timings and expectations, sent as OpenLineage and OpenTelemetry, checked in CI by ubunye gate, and deployable in one command to AWS Glue, GCP Dataproc Serverless, Kubernetes, Azure Container Apps and EMR Serverless, or exported to Airflow 2 and 3 and Spark Declarative Pipelines. The same data hashes were written on Glue, Dataproc, Kubernetes (kind), Azure Container Apps and Spark Declarative Pipelines; EMR Serverless is built but not yet run (the AWS free plan blocks EMR). A task can call a language model through one port that the engine sees: every call in the record, the bill capped before it is sent and priced before the run, the whole run replayable for nothing, and the cost exported as FOCUS 1.4 rows. Agents drive it all through ubunye mcp. Python 3.10 to 3.13 on Linux, Windows and macOS.

Not in this release, on purpose: a positional ubunye run ./task (four flags stay the interface), a DuckDB backend (it failed SQL parity with Spark), a tool port and divergence ledger for agents, Dagster and Prefect exporters, and a benchmark suite (a separate project, later).

Added¶

ubunye mcp: the engine as an MCP server for agents. Tools tasks, doctor, plan, runs, record, gate and focus only read; run exists only with --allow-run. Tasks are named, never paths, and must sit under -d. An agent's run replays its model calls unless the server starts with --allow-live-llm, and the run's limits apply. A task's prints go to stderr, so stdout stays the protocol's. A failed run is reported with its record, not raised. New mcp extra (the MCP Python SDK 2.x).
The gate covers model calls; a model step replays in CI. ubunye gate reports a run's model calls (count, replayed, tokens, list-price cost) and gains --require-replay (fail if a call went live) and --max-llm-cost-increase (fail if the list price of the tokens, or the tokens for an unpriced model, grew by more than a share; measured on replayed runs too). The gate Action's new llm-mode input defaults to replay. A new example, examples/production/llm_replay, labels reviews through ubunye.llm with answers recorded from a stub model, and a CI job replays it on Linux, Windows and macOS against a golden hash with nothing listening at the model's address.
ubunye lineage focus: the model bill as FOCUS 1.4 cost rows. A run's model calls become rows a FinOps tool loads next to the cloud bill: every mandatory FOCUS 1.4 column, one row per provider, model and token direction, cost equal to quantity times unit price, times in UTC, custom columns prefixed x_ (run, task, model, direction, price date, cost basis). Costs are tokens times the list price the run used; the provider's invoice is the authority and every row says so. Replayed calls make no rows; unpriced calls are left out and counted. CSV or JSON lines. Each logged call now carries its provider, service, unit prices and their date.
ubunye plan shows the bill before the run. For a task that calls a model, the plan prices its recorded calls (the replay file) at today's prices and sets them against UBUNYE_LLM_MAX_USD, with no data read and no model called. It fails on what would stop the run (replay with nothing recorded, a limit that is not a number, a dollar ceiling on an unpriced model) and warns on an estimate over the ceiling or live calls with no ceiling. --json adds an llm section per task.
The bill is capped before the run. UBUNYE_LLM_MAX_USD, UBUNYE_LLM_MAX_CALLS and UBUNYE_LLM_MAX_SECONDS set one budget per run, shared by every port (and by calls running at the same time); llm.port(max_usd=, max_calls=, max_seconds=) adds a port's own. Before each call the port reserves its worst case (prompt tokens counted high, plus the whole max_tokens); a call that could pass a limit raises LLMBudgetError and is never sent, and the task writes nothing. After the call the reservation becomes the real cost. Prices come from a dated table of Anthropic's current models taken from its pricing page, or from price= or a UBUNYE_LLM_PRICES file; an unpriced model has an unknown cost, not zero, and a dollar ceiling on it fails closed. Each call records cost_usd and estimated_usd; the run record's new llm_budget keeps the limits and the spend, and ubunye lineage trace prints them. No new config field: limits are environment variables and port arguments.
Record once, replay anywhere: model calls answered from a file, for nothing. UBUNYE_LLM_MODE=record calls the model and keeps each answer in the task folder's llm-replay.jsonl, next to config.yaml, to be committed with the task (or in UBUNYE_LLM_STORE), keyed by the request's hash; prompts are never stored. UBUNYE_LLM_MODE=replay answers from that file with no key, no network and no cost, and fails closed: a request with no recorded answer stops the run with its key, and never falls through to a live call. Every call in the run record now says its source (live, record or replay). A recorded run replays to the same data and row hashes on any machine.
ubunye.llm: one port for language model calls, seen by the engine. A task calls llm.port("anthropic" | "openai_compatible" | "databricks_serving", model=...) and then complete() or complete_many() (answers in prompt order, a few calls at a time). The three backends use the standard library only; openai_compatible covers OpenAI, Azure OpenAI, vLLM, Ollama and LiteLLM. Keys come from the provider's usual variable or api_key=, which can be a secret:// reference; a missing key fails before any call. Rate limits and server errors are retried, honouring retry-after; other refusals fail at once with the provider's reason and never the key. Every call made during a run is in the run record's new llm_calls (backend, model, tokens, seconds, attempts, status and a sha256: key of the request; never the prompt or the answer), and ubunye lineage trace sums them per model. More backends are plugins in the ubunye.llm_backends group.
ubunye doctor: what will fail, and why, before a run. One command checks the Python version, every backend (usable, or what it needs and the install command), whether a run without --backend would work, that Java suits the installed Spark, that delta-spark is built for the same Spark major (a mismatch installs cleanly and fails at run time), winutils.exe on Windows, and that every plugin loads. Given tasks (-d -u -p -t), it names the environment variables they use without a default that are not set, and checks their configs load. Warnings are environment problems that matter only for what you use; failures are what makes a run fail, and set exit code 1. --json prints one document. The config resolver gains required_env_references() for the variable check.
ubunye export spark-pipeline: a task as a Spark Declarative Pipeline. Writes spark-pipeline.yml, a definitions module and a copy of the task, for spark-pipelines run on Spark 4.1+: inputs become temporary views, the task's transform() runs unchanged, outputs become materialized views. What does not carry over (merge modes, output paths, expectations) is reported; secret:// references are refused. The run-anywhere example exported this way wrote the same row hashes on Spark 4.2 as on every other platform.
ubunye export airflow writes a DAG that runs, on Airflow 2.4+ and Airflow 3. The generated DAG used schedule_interval (removed in Airflow 3), imported BashOperator only from its Airflow 2 home, and passed env= without append_env=True, which replaces the whole environment, so ubunye was not on PATH and every run failed. It now uses schedule=, imports from the standard provider on Airflow 3 and falls back on Airflow 2, keeps the environment, passes Airflow's logical date as -dt {{ ds }}, quotes the command, and is compiled before it is written. New options: --usecase-dir (where the pipelines are on the Airflow workers), --backend, --lineage.
One command to Kubernetes, Azure Container Apps and EMR Serverless. ubunye deploy k8s runs a task as a Kubernetes Job (kubectl's context, no retries, cleaned up after a day); ubunye deploy container-apps as an Azure Container Apps job pulling with a managed identity, reading the run record back from Log Analytics; ubunye deploy emr-serverless starts a run in an EMR Serverless application (tested as a plan: the free plan blocks EMR). ubunye deploy dockerfile container writes a self-contained image (Java, Spark in local mode, Delta, engine, pipelines). The entry script also reads its arguments from UBUNYE_ENTRY_ARGS, for runtimes whose CLI cannot pass them.
One command to AWS Glue and GCP Dataproc Serverless. ubunye deploy glue and ubunye deploy dataproc run a task unchanged on either service, through the cloud's own CLI and login: the task goes up as a zip with a small entry script that runs it with a run record and prints the record back, so --record-out saves it and ubunye gate can compare runs across clouds. Glue pip-installs the engine (or an uploaded wheel) and supplies Delta; Dataproc runs in an image whose Dockerfile ubunye deploy dockerfile dataproc writes, following Dataproc's rules. --env, --var, --dry-run, --wait/--no-wait. See Glue and Dataproc.
ubunye gate and a GitHub Action: the receipt gates pull requests. The gate compares a run record with a baseline and fails when the run failed, a fail expectation broke, an output's data or schema changed without a VERSION bump, an output went missing, or a limit was passed (--max-slowdown, --max-seconds, --max-row-change). Each changed output says what else changed (config, code, environment, which inputs), and a change with none of them is reported as nondeterminism. Runs are named by record file, run id, previous or latest; --json and --summary (Markdown for the job summary). The composite action .github/actions/gate runs a task on a pull request's base and head and gates them. See Gate.
OpenTelemetry done properly. The OTel hook now follows OpenTelemetry's own configuration (OTEL_EXPORTER_OTLP_ENDPOINT, protocol, headers, OTEL_TRACES_EXPORTER / OTEL_METRICS_EXPORTER, OTEL_SERVICE_NAME), exports over OTLP (http/protobuf or grpc) with the new otel extra, reuses a host application's providers, and flushes when a task ends so short CLI runs export. It used to print every span to the console and ignore the endpoint. Spans now nest steps under the task, carry the run id, backend and config hash, and mark failures as errors with the exception recorded. New metrics: ubunye.task.runs, ubunye.task.duration, ubunye.step.duration, ubunye.rows.read, ubunye.rows.written (rows counted where free, on Spark only with UBUNYE_OTEL_COUNT_ROWS=1). UBUNYE_TELEMETRY is read when a run starts, not at import, so setting it in a notebook works. The Prometheus hook is unchanged; its row and byte counters are fed only by callers of observe_step(), never by the engine, as before. See OpenTelemetry.
OpenLineage events: the receipt lands in your catalogue. A recorded run sends START and COMPLETE or FAIL (OpenLineage 2-0-2) to any OpenLineage server (Marquez, DataHub, OpenMetadata, Google Dataplex) when OPENLINEAGE_URL is set, and/or to a JSON-lines file (UBUNYE_OPENLINEAGE_FILE). Datasets are named by the OpenLineage conventions with credentials removed; events carry the standard outputStatistics, dataQualityMetrics, dataQualityAssertions and errorMessage facets plus ubunye_evidence and ubunye_hash (schemas in docs/schemas). No new dependency; a server that is down never fails a run. ubunye lineage openlineage exports (and with --send, backfills) stored runs. Every event in the tests is validated against the vendored OpenLineage spec. See OpenLineage.
Run record v2: the receipt says why two runs differ. Each record now carries a hash of the task's code, the environment (Python, platform and the versions of the packages that can change a result, plus one hash of them), every input's row hash and count (like the outputs), per-step timings, and every expectation's result; timings and expectation results are kept when a run fails. ubunye lineage compare reports code, environment and inputs as changed or unchanged and names the packages whose versions moved; trace prints them. Input hashing costs one more scan per input and can be turned off (LineageRecorder(hash_inputs=False)). Monitors get the new evidence only if their task_end accepts it, so existing monitors are unaffected; v1 records still load. See the addendum to ADR 006.
The pandas content hash is 2.4 times faster (60,000 rows: 828 ms to 350 ms, the same hash). Each column's text is now built once, with the column name encoded once instead of on every row; a property test holds the fast path to the row-at-a-time reference byte for byte. It pays for hashing inputs in run record v2: a recorded run now hashes its input and its output in less time than it used to hash its output alone.
Secrets by reference: secret://<provider>/<reference>. A config names a secret (password: "secret://aws-sm/prod/db#password") and the engine fetches it only into the copy of the config a connector receives, at the moment it reads or writes. The config hash, plan, config, run records and logs keep the reference, so a secret cannot leak through them, and rotating it does not change the config hash (tested). Providers are plugins (ubunye.secrets): env, file, databricks, aws-sm, gcp-sm, azure-kv, with #field for JSON secrets and new aws, gcp, azure extras for the cloud SDKs. validate refuses an unknown provider with the closest name; doctor names the package a task's provider is missing; neither fetches anything. See Secrets.
CONFIG.expectations: what an output must look like, checked before anything is written. Declared rules per output (not_null, unique, between, one_of, matches, row_count), each with a severity: fail stops the run with nothing written, quarantine moves the breaking rows to a named output with a _ubunye_failed_rules column listing every rule each row broke, warn logs. max_quarantine_rate fails a run when too much is set aside. A missing value passes every rule but not_null, as in SQL. The rules run through Narwhals, so the same config gives the same verdicts on Spark and pandas (tested on both, and at the oldest supported versions, where pandas 2.2 and pandas 3 disagree about a missing value in a pattern match). Narwhals becomes a dependency (narwhals>=2.0, pure Python, no dependencies of its own). Every rule's result, passed or not, is on the error and in the run state for the run record. See Expectations.
A typo inside a connector block fails validation, with a suggestion. Input and output blocks accept keys the engine does not know, because each connector reads its own settings, so paht: data/in.csv validated and was ignored. Connectors can now declare CONFIG_KEYS; for one that does, any other key is an error that names the closest real key (inputs.src: 'paht' is not a setting of 's3'; did you mean 'path'?), reported before the "requires" error it usually causes. Every built-in reader and writer declares its keys; format and options are allowed everywhere, and mode, merge_keys and replace_where on every output. A connector that declares nothing (most third-party ones) accepts any key, as before. The s3 and unity writers each accept the other's key (table, path) so one output block can serve both, as the run-anywhere example does. All 53 task configs in the engine and examples repos and two downstream projects still validate.
A conformance suite every backend must pass, shipped for yours. ubunye.testing.backend_conformance is the set of tests that says a backend keeps the engine's promises: it is registered under its name and declares what it can do, gives transforms its own frames and the engine a port, reads a CSV file exactly as Spark does and so leaves the same run record hash as every other engine (checked against a reference built from plain Python values), reads back what it writes, and honours the write modes it claims. Subclass it in a backend's tests. The pandas backend passes it in the unit tier, and both Spark backends in the integration tier. ADR 006 now lists the type names the hash uses, which a port that is not Spark or pandas must report.
One transform for every engine, written with Narwhals (ADR 005). A transform written with the Spark API runs only on Spark, and one written with pandas only on pandas. Two ways to write it once were tested on the Titanic example's own logic against Spark: Narwhals gave the same data hash on both engines (with a sum cast to a 64 bit integer, which Spark does and pandas does not), and SQL did not (DuckDB and Spark type the same aggregate differently), so Narwhals ships and SQL waits for a DuckDB backend, not in this release. A transform may return a Narwhals frame; the engine unwraps it without importing Narwhals. There is no config field for portability: ubunye plan reads the imports in transformations.py and says what it is written for (pyspark, narwhals, pandas), in text and in --json, and warns when that is not what --backend gives it. The example in the docs is run on both engines by the test suite.
--json for scripts and agents. plan, validate, backends, every lineage command and models list/info/compare print exactly one JSON document on stdout with --json, errors included ({"ok": false, "error": ...}), and keep their exit codes. lineage compare --json reports each data hash as unchanged, changed, unknown or not comparable, the same verdict the text form prints.
The pandas backend understands Spark's mode option, and backends check their IO details before a run. Every Titanic example reads its CSV with mode: "FAILFAST", which the pandas backend refused, so none of them could run there. mode now works for CSV and JSON exactly as Spark 4.2 does it (checked against Spark): FAILFAST stops at a bad row, DROPMALFORMED skips it, and PERMISSIVE, Spark's default, cuts a row with too many fields and pads one with too few with null. A backend can also check an input's or output's details (options, schema) before anything runs (Backend.check_io); the pandas backend uses it, so plan, validate --backend and the run's own preflight report an option it cannot honour instead of failing on open.
--var key=value, documented for years, now works. It was in the README and three docs pages and was never implemented, so every example that used it failed. It now works on every command that renders a config (run, validate, plan, config, test run), and the Python API takes the same thing as variables= on run_task, run_pipeline and notebook. Names must be valid template names, env is reserved and mode has its own flag; --var dt=... works like -dt, and the same name with two values is refused instead of one silently winning. The variables a run used are kept in its run record. On the way, validate stopped setting dtf to the timestamp's value (it now has -dtf), and test run now renders with mode (its profile), as run does.
The run record's data hash now means "these exact rows" (ADR 006). It read a 1 percent sample, changed with row order on Spark, and on pandas quietly recorded the schema hash as the data hash. The new rows-v1 hash reads every row in the same pass as the count, ignores row and column order, changes when any cell changes, tells null from NaN, does not depend on the timezone, and is the same on Spark and pandas for the same data (Spark computes it in one aggregation on the cluster). When rows cannot be read the record says why instead of inventing a hash. Records also carry the Ubunye version, the backend, the run variables and each output's hash_method, and runs from run_task and run_pipeline are stored under the same folder and name as CLI runs, so ubunye lineage list finds them. lineage compare calls two missing hashes "unknown" (it said "unchanged") and a pre-0.7 hash "not comparable". sample_fraction is ignored and kept so old configs load.
A pandas transform gets a plain pandas DataFrame (ADR 004). It used to get an adapter and had to write sources["x"].native to reach the DataFrame. The Backend port gains to_native and to_port, and the engine converts at the edges: transforms get and may return native frames; writers, hooks and lineage get the port, where count() means rows (a raw pandas count() counts non-nulls per column). run_task, run_pipeline and the notebook's read() and transform() return native frames, so on pandas use len(frame) for rows. Both methods default to doing nothing, so Spark tasks and older backends are unchanged.
Backends are plugins, and say what they can do (ADR 001, 002). Spark, Databricks and pandas now register in a new ubunye.backends entry point group, exactly as a third party engine would, so adding an engine is a package with one entry point and no edit to Ubunye. Each backend declares its capabilities (features such as a SparkSession or path IO, file formats, write modes, distributed, needs Java) and each connector declares what it requires. Before anything starts, the engine checks every input and output against the backend and lists every problem at once, so a task that cannot run stops in the first second instead of halfway through. The core no longer imports SparkBackend (a test now reads every import in ubunye/core and fails on any engine). Backend.is_spark still works, read from the capabilities, and is deprecated. Backends and connectors written before 0.7.0 declare nothing and behave exactly as before.
Choose a backend by name everywhere, with one resolution order (ADR 003). --backend NAME on ubunye run, ubunye test run and ubunye validate; backend="pandas" (or an instance) on run_task, run_pipeline and notebook. With no choice: the platform's session if there is one (on Databricks, the notebook's), else Spark. The CLI now follows that order too: run inside a process that already has a SparkSession, it attaches to it instead of stopping it at the end. New ubunye backends (and --json) lists what is installed and what each backend can do; ubunye validate --backend pandas checks a task can run there without starting anything. An unknown or broken backend gives a clear error with the installed names or the pip install that fixes it. New page: Execution Backends; new section: Architecture Decisions.
A pandas backend, so ubunye run --backend pandas runs a task on a laptop with no Spark and no JVM (issue #38). Backend was a port with a single kind of adapter (Spark), and a port with one adapter has never really been tested as a port. It has a second one now. PandasBackend reads and writes the generic path formats (csv, parquet, json) with pandas; lakehouse formats and managed tables stay Spark's job, and their connectors say so, so an unsupported pairing fails with a clear message instead of a stray AttributeError. This works because the data plane gained a small read/write seam on Backend (read_frame / execute_write): a path connector like s3 asks the backend to do the IO instead of naming Spark, so the same task and the same config.yaml run on either backend. The proof is an integration test that runs one passthrough task through the engine on Spark and on pandas and asserts the output is identical. Install the extra with pip install 'ubunye-engine[pandas]'.
The pandas backend reads data exactly as Spark does. The first version used pandas' own defaults, so the same CSV gave different columns and types on the two backends: pandas assumed a header Spark does not, guessed types Spark leaves as text, read JSON as one array where Spark reads one object per line, and could not read a folder Spark had written. Reads now follow Spark: no header by default (_c0, _c1), text unless inferSchema, Spark's inferred types (int when every value fits, else bigint; an all empty column is text), JSON Lines with columns sorted by name, folders of part files, globs, explicit schema: strings, and timestamps read in spark.sql.session.timeZone (UTC when unset). Columns are Arrow backed, so a whole number column with nulls stays whole numbers. Options it cannot honour, nested schema types and remote paths are refused by name instead of ignored. Needs pandas 2.2 and pyarrow 14 or newer.
The pandas backend writes data exactly as Spark does, so each can read the other's output. It used to write one file where Spark writes a folder, so Spark could not read pandas output as a table, and append re-read and rewrote the whole file every run. It now writes Spark's layout (a folder of part-* files and _SUCCESS); append adds a part file; overwrite is staged and swapped in only when complete, so a failed write keeps the old data. The text formats match Spark byte for byte on the cases tested: CSV without a header by default, minimal quoting with a backslash escape, text trimmed, Java style numbers, Spark's timestamp text; JSON Lines without null fields. Parquet timestamps are written as UTC microseconds (Spark cannot read nanoseconds). A named index (what groupby leaves) is kept as columns. partition_by, unknown write options, nested values in CSV and non pandas frames are refused by name.
Connectors that need Spark say so on the pandas backend. hive, jdbc, delta, unity, binary and rest_api build on a SparkSession. On the pandas backend they used to fail with AttributeError: 'PandasBackend' object has no attribute 'spark', and rest_api only after fetching every page. They now stop first, before any network call, with a message naming the connector and the way out (--backend spark, or csv/parquet/json paths with format: s3).
Proof that the two backends agree, run against real Spark. The old parity test compared one passthrough task as strings. The new suite runs Spark 4.2 beside the pandas backend and compares Arrow types and Python values, never strings: eight reader cases (CSV with and without header, inferSchema, explicit schema, separators and null values; JSON Lines, multiLine and schema), a folder Spark wrote, each engine reading what the other wrote for parquet, CSV and JSON, CSV and JSON files byte for byte, and one task end to end on both backends. It runs in a timezone other than UTC so a timezone slip cannot hide. It found two gaps, now fixed: explicit TIMESTAMP schemas could not read text without an offset, and JSON was not written the way Spark writes it.

Changed¶

The Titanic examples run on Spark and on pandas, with the same receipt. Their three transforms (survival by class; clean, then aggregate) are written once with Narwhals, identically in the local and Databricks examples. CI runs each local example on Spark, then on the pandas backend into the same output, checks the golden output again, and requires the two run records to carry the same config hash and data hash (scripts/same_receipt.sh, which uses only ubunye lineage list and lineage compare). On the real 891-row file the pandas run takes 0.4 s against Spark's 4 s. The example tests run every transform on both engines. The Databricks notebooks install Narwhals. The docs now name the two known Spark and pandas differences under Narwhals: the type of a sum (cast first) and rounding exactly halfway.
Python 3.10 to 3.13, tested on Linux, Windows and macOS. Python 3.9 is no longer supported (it reached end of life in October 2025). CI now runs the unit tier on every supported Python on all three systems (Windows and macOS ran none of it before), the Spark tier on Spark 4 with Python 3.10 and Java 17 and with Python 3.13 and Java 21, and on Spark 3.5 with Java 11.
The oldest versions the package accepts are tested, and two were raised. A new CI job installs exactly the declared minimum of every dependency and runs the unit tier, and a test keeps that job and pyproject.toml in step. It found that typer>=0.12 could not work with today's click (the CLI could not start), so the minimum is now typer>=0.15.4, the first that works. The spark extra now asks for pyspark>=3.5 (it said 3.3, which was never tested); the whole Spark tier passes on Spark 3.5.
ubunye plan is a real dry run, and exits 1 when it finds a problem. It printed the config's names back and always exited 0. Built on the September work, it now checks each local input exists or is written by an earlier task in the same plan (the September version failed every multi task pipeline on that), loads the transform class, resolves every write mode (so merge without merge_keys is caught before a cluster runs the transform), asks the --backend what it can do instead of keeping a list of names, and warns about environment variables the config uses that are not set. It starts no engine and moves no data. Its config hash is now the one the run record keeps: the record used to hash the config after the engine had rewritten its transform, so the two never matched. A config error in plan shows the whole message.
The package says what it is. LICENSE is the full MIT text (it was the single word "MIT"), declared the modern way (license = "MIT" with license-files, which needs setuptools 77). PyPI now shows classifiers for the Python versions CI tests, a description that matches what the engine does now that it is not Spark only, and pandas and lineage among the keywords. Thabang Mashinini-Sekgoto stays the author and Ubunye AI Ecosystems the maintainer and copyright holder.
Spark moved out of the engine core, so the hexagon is real rather than aspirational. The founding rule is "the core never depends on the outside world," but core/write_modes.py and core/catalog.py called Spark directly (df.write, spark.sql("MERGE INTO ..."), USE CATALOG), so the one place the pattern was meant to hold was the one place it leaked. The Spark mechanism now lives in a new Spark adapter (ubunye.adapters.spark.write_exec and ubunye.adapters.spark.catalog); the backend-agnostic decision (validating a write mode against what a connector supports) stays in ubunye.core.write_modes. Behaviour is unchanged and every existing test passes. The old names (write_modes.apply, merge_into, target_exists, dynamic_partition_overwrite, and core.catalog.set_catalog_and_schema) still resolve through a deprecation shim, so third-party connectors keep working; they will be removed in a future major. This is the groundwork for a non-Spark backend (issue #38): the core can be backend-agnostic now because it no longer imports one backend's API.

Fixed¶

A deploy never reports success without the run record it was asked for. Azure's Log Analytics returns lines printed in the same instant in any order, so the record could come back after its end marker: ubunye deploy container-apps stopped waiting at the marker, found nothing between the markers, and still exited 0 without writing --record-out. The record is now found wherever it lands, Container Apps waits for the record itself, and a missing record fails the command when --record-out was given. Found by the final release-candidate run.
A model provider's error reads as its message when it comes in a list. Gemini's OpenAI-compatible endpoint answers errors as [{"error": {...}}]; the port printed the whole list. Found on the first live call to Gemini.
ubunye deploy container-apps runs a second time. The job is created on the first deploy and updated after that, and az containerapp job update refuses --env-vars (it takes --replace-env-vars), so every deploy after the first failed. Found by running the release candidate on Azure Container Apps again.
ubunye lineage list no longer reports an input of 0 rows it never counted. A run record counts and hashes its outputs (ADR 006), not its inputs, and the list added the missing counts up as in:0, which read as an empty input. An uncounted total now shows as in:-.
The Spark backend stops only a session it started. start() attaches to a session that is already running (getOrCreate), and stop() then stopped it anyway, as did the garbage collector through __del__. So run_task(..., backend="spark") in a process that already had a Spark session, a user's own or a notebook's, ended that session when the run finished. A session the backend did not start is now left running. Found by the new backend conformance suite.
ubunye deploy databricks installs what the transform needs. The generated notebook installed the engine alone, so a task written with Narwhals failed on Databricks at its first line. The deploy now reads the transform's imports (the same detection plan uses, ADR 005) and installs Narwhals next to the engine when the transform imports it.
import ubunye works on every pydantic the package accepts. The model transform's model_class field starts with pydantic's reserved model_ prefix: pydantic 2.0 refused it at import, so the package failed to load on the oldest version its requirements allowed, and pydantic 2.1 to 2.9 warned about it. The prefix is switched off for that one config model. Found by CI's new minimum-versions job.
No DeprecationWarning on every plugin lookup. Readers, writers, backends, hooks and artifact stores were found through the dict interface of entry_points(), kept for Python 3.9 and deprecated on 3.10 and 3.11, so every lookup warned there. They now ask entry_points(group=...).
The pandas backend reads quoted CSV values exactly as Spark does. Spark's escape character is a backslash, so a doubled quote inside a quoted value is not an escape: Spark keeps "McGowan, Miss. Anna ""Annie""" as written. pyarrow unescaped it, so 53 of the 891 Titanic names came out different on pandas, with nothing to say so. Files with such quotes (or a stray quote, a backslash, or a line of only spaces, which Spark skips) are now split by a port of Spark's own CSV parser, univocity 2.9.1, and only the lines that need it; every other file is read by pyarrow as before. The port is fuzzed against live Spark for each escape setting, line ending and multiLine. escape: '"' now reads doubled quotes as quotes, as in Spark.
The quickstart works, and a test keeps it working. It used flags that do not exist (run --profile), a lineage list with no task, and a Hive to Delta config no laptop could run. It is rewritten around the ubunye init scaffold, and its commands, in the README and on the Quickstart page, are run by the test suite exactly as written. CONTRIBUTING.md, which the README linked to, now exists (the docs page shows the same file), and stale claims are gone: "Spark native", "288 tests", and run --profile in the engine docs.
The first command in the README works, and what it makes runs. The README and quickstart said ubunye init -d ... -t ..., but the command was ubunye init pipeline ..., so the first thing a new person typed failed. Both forms work now. The default scaffold (--template local) has a small sample CSV next to the task, a config that reads it and writes Parquet into output/ next to it, and a transform (people[people["age"] >= 18]) that means the same in pandas and in Spark: it runs with no Java and on Spark unchanged, and the two runs leave the same data hash (checked against Spark). --template databricks writes the old Unity Catalog scaffold. Configs get a built in {{ task_dir }} variable, the task's own folder, so a path next to the task works from any folder and on either engine (Spark resolves a relative path from where its JVM started, pandas from the current folder).
ubunye models promote honours the model's promotion gates. Gates were read from the training task's config and applied only when that run promoted the model itself; the CLI never saw them, so a model failing every gate could be promoted by hand. The registry now keeps a model's gates (set when the training run registers it), every promotion checks them by default, and a failing gate is named. --force (and force=True in Python) skips them with a warning and marks the version promotion_forced, with the gates it skipped.
A typo in config.yaml says where it is. A YAML syntax error escaped as a raw yaml.ParserError, so run, validate and plan crashed with a traceback. It is now a config error naming the file, the line and the column, and a file whose top level is not a mapping is refused clearly too.
ubunye --help no longer crashes on Windows. A legacy Windows console prints in cp1252, which has no arrow, and a help text and a few messages used one, so ubunye --help died with UnicodeEncodeError on a fresh install. The CLI's text is now plain ASCII where it prints, a test keeps every help text printable on cp1252, and the ubunye command sets its output to replace any character the console cannot show (a file path with an accent, say) instead of crashing. The CI Package job runs --help on windows-latest.
A backend whose packages are missing says so, with the install. After pip install ubunye-engine alone, ubunye backends listed spark and pandas as ready, and choosing one failed deep inside on the first frame. Each backend now declares the packages it needs; choosing it names what is missing and the extra that installs it (pip install 'ubunye-engine[pandas]'), and ubunye backends marks it "not usable here". Listing and inspecting a backend still work anywhere, and Engine() with no backend resolves one only when it first needs it.
The pandas backend works on Windows. pyarrow before 24 cannot find a timezone database on Windows (checked: 19 to 23 fail, even with the tzdata package), so every timestamp read or written failed there. The pandas extra now asks for pyarrow 24 or newer on Windows, and the backend checks at start and says how to fix an older one instead of failing on the first timestamp.
A notebook with lineage=True now leaves a run record. Lineage is recorded around a whole task, and the notebook runs read, transform and write as separate steps, so ubunye.notebook(..., lineage=True) recorded nothing at all. A notebook write now counts as a run: nb.write(...) and nb.run() each leave one record, in the same place and with the same hash as run_task.

[0.5.0] — 2026-07-14¶

Fixed¶

Lineage no longer recomputes your pipeline or risks the driver. Fingerprinting an output used to run the output's entire plan two to three times (a count inside the hasher, an unbounded sample collect, then a second count in the recorder on the same uncached DataFrame), and had a fallback that collected the whole table into the driver. At real scale that is a crashed driver describing a job that had already succeeded. Now: one fingerprint_dataframe() pass, persisted for the duration, one count shared by the hash and the row count, and every collect capped at UBUNYE_LINEAGE_SAMPLE_ROWS (default 1000) whatever the table size.

Added¶

The model registry writes to object storage now. ModelRegistry was built directly on pathlib, so models could only be saved to a normal disk — the one thing AWS and GCP serverless do not offer. Storage is a port now: the scheme of the store path picks the backend. A plain path or file:// stays local and behaves exactly as before. s3://, gs:// and friends go through fsspec (pip install 'ubunye-engine[objectstore]' plus your cloud's filesystem, e.g. s3fs). Any other scheme goes to whatever store someone registers under the new ubunye.artifact_stores entry-point group — one class, one entry point, no engine edits, no inheritance, like every other connector. Models still write real files (a Keras save needs an actual disk), so remote stores stage locally and upload, and get_model downloads before load. Closes #28.
The annotations are a public API now. The package ships a py.typed marker, so type checkers in consuming projects finally see the engine's types (PEP 561 says they must ignore packages without it, so until now every annotation was invisible to users). mypy runs in CI and is green across all 100 source files, and it earned its keep immediately: it caught a Protocol that under-described its own usage, a shadowed variable in the REST reader's pagination, unsound stubs in the S3 lineage store, and thirteen stale type: ignore comments. Reader.read now returns DataFramePort instead of Any, so the most important object in the framework is typed in its own contract.
DataFramePort — the data plane finally gets its port. Specified in the founding design notes and never shipped. The model layer always had its port (UbunyeModel: the engine never imports sklearn or torch); the data layer never did, so every reader returned and every transform received a pyspark.sql.DataFrame by fiat. ubunye.core.ports.DataFramePort is a runtime_checkable Protocol — structural, no inheritance — and a Spark DataFrame satisfies it natively (verified against a real session, not assumed), which is what makes shipping it additive rather than a rewrite. ubunye.adapters.PandasDataFrameAdapter makes pandas fit in thirty lines, exactly as the design promised — including guarding the trap where df.count() exists on pandas and means something else entirely (non-null counts per column).
The plugin contract is structural again. validate_config, SUPPORTED_MODES and SUPPORTS_MERGE are read with getattr and defaults: a connector that declares nothing requires nothing. The previous contract change accidentally demanded inheritance — a duck-typed reader, the exact thing docs/interfaces.md promises will work ("no inheritance needed"), crashed with AttributeError. Declaring is opt-in, not a toll.

Removed¶

Telemetry that could only read zero. The ubunye_rows_total and ubunye_bytes_total Prometheus counters, and the rows= parameter on EventLogger.step_end, were dead on arrival: nothing in the engine ever fed them. A metric that always reads zero is worse than no metric, because somebody builds a dashboard on it and trusts the zero. If row metrics return they will arrive fed by the engine (the lineage fingerprint now computes row counts in one pass, so the plumbing finally exists).
ubunye/compat/analytics_engine_shim.py — a placeholder docstring with a folder around it, imported by nothing.
Stale build artifacts (build/, an 0.2.0 wheel in dist/), mlflow.db and a log file were removed from tracking and ignored.

Changed¶

The CLI stops teaching two systems. run gained --all (validate had it for years, so the two commands taught different habits for the same job). validate accepts -m/--mode as an alias for --profile, and now injects the same template variables as run — a config using {{ mode }} used to run perfectly and fail validation with an undefined variable, which is exactly backwards for a command whose job is to catch problems before the run. Closes #30. The ubunye init and Databricks notebook scaffolds no longer print df.count() per output (a full scan per output, teaching the habit the lineage rework just removed); they print the column list, which is free.
The four control-plane Protocols (AuthBackend, DeployAdapter, LineageBackend, RegistryBackend) are enforced by a conformance test. They were referenced only by docstrings before: a contract nobody had signed. The test checks every shipped implementation member by member, structurally, with no inheritance added.
from ubunye import * works now. __all__ advertised five submodule names that the module never imported, so the star import raised NameError on the package's own public surface.

Breaking¶

The core no longer knows about any connector. It asks.

The engine advertised "adding a connector = write the class, register the entry point". That was not true. Three places in the core held knowledge of specific implementations:

config/schema.py carried an if/elif chain naming hive, jdbc, s3, binary, delta, unity and rest_api, spelling out what each one required.
core/write_modes.py held _MERGE_FORMATS = {"delta"} and decided, on the connector's behalf, which formats were allowed to MERGE.
FormatType was a closed enum, so a correctly-registered plugin was rejected before the registry was ever consulted.

Adding a connector meant editing the engine in three places. That is not open/closed, and it is not what the entry-point system is for.

The plugin contract now carries it:

class Connector(ABC):
    @classmethod
    def validate_config(cls, cfg) -> list[str]: ...   # what I need

class Writer(Connector):
    SUPPORTED_MODES: frozenset = frozenset({"append", "overwrite"})
    SUPPORTS_MERGE: bool = False                      # what I can do
    MERGE_FILE_FORMATS: frozenset = frozenset({"delta"})

An Iceberg or Hudi connector can now declare that it supports MERGE, and be believed. tests/unit/config/test_third_party_connector.py registers a connector the engine has never heard of and asserts it works with zero engine edits.

Breaking: FormatType is deleted. IOConfig.format is a plain str.

The engine no longer changes its behaviour based on where it is running.

Two places sniffed the host and acted on the guess:

writers/unity.py::_is_databricks() string-matched the Spark conf for "databricks" and used the result to decide whether to run OPTIMIZE and VACUUM. It was wrong twice: a connector should ask the target what it can do rather than detect its host, and the guess was factually wrong — OPTIMIZE, ZORDER and VACUUM are Delta features, and open-source Delta has had them for years. Anyone running Delta off Databricks silently did not get the maintenance they had explicitly asked for in their config, and nothing said so. The statements are now attempted, and a target that cannot run them is reported, not predicted in advance and not swallowed.
_internal/auto_detect.py picked the model registry backend from DATABRICKS_RUNTIME_VERSION with no way to override it. The same pipeline, unchanged, registered models to MLflow on Databricks and to a directory anywhere else — and said nothing. UBUNYE_REGISTRY_BACKEND now wins, the host is only a fallback, and it logs which it chose.
Inputs are validated against the READER, outputs against the WRITER.

One shared rule used to validate both roles, so it could not tell them apart: unity as a source may be given sql, but as a sink it must have a table — writing to a SELECT statement is meaningless. The shared rule had to accept sql everywhere, so it silently allowed it. Each side is now checked against the plugin that will actually handle it.

Breaking: IOConfig(format="s3") no longer raises on its own. It cannot: an IOConfig does not know whether it is an input or an output. Validation happens at TaskConfig, where the role exists.

[0.4.0] — 2026-07-13¶

Breaking¶

ENGINE.spark_conf is now applied to a session the engine did not create — and static keys raise instead of being ignored.

Before, the conf was silently discarded whenever a SparkSession already existed. That is always on Databricks, and equally in a notebook, an AWS Glue job, under spark-submit, or in pytest. DatabricksBackend took no conf argument at all, and api.py computed merged_spark_conf(mode) and then threw it away. So every ENGINE.spark_conf and every ENGINE.profiles block did nothing, and nothing said so. The config claimed one thing and the runtime did another.

Spark divides settings into runtime keys (spark.sql.shuffle.partitions) and static ones (spark.master, spark.sql.extensions, spark.driver.memory) that are fixed when the JVM starts. Runtime keys are now applied. Static keys are a request the engine cannot honour, so they raise, naming every offending key at once.

Action required: remove static keys from ENGINE.spark_conf and set them where the session is actually created — a cluster policy, a job conf, --conf on spark-submit. If your pipeline "worked" with them before, it was ignoring them.

A config can no longer override a master the platform already chose.

Under spark-submit — how EMR Serverless and Dataproc Serverless start every job — the platform sets spark.master in the default SparkConf. A task that also set spark.master won, and the job ran entirely in the driver: it ignored every executor, finished, reported success, and billed for a cluster it never touched. Nothing warned. The output was correct; there was just far less of it per minute than there should have been, forever.

SparkBackend now refuses to start when the config's master disagrees with the platform's.

Action required: delete spark.master from ENGINE.spark_conf. The master belongs to whoever launched the session, not to the task.

Added¶

python -m ubunye — an entry point a cloud can actually run.

The engine could only be reached through its console script. AWS EMR Serverless and GCP Dataproc Serverless do not give you a shell: they hand a Python file to spark-submit. An engine reachable only through its CLI cannot run on either of them — and you find that out after wiring up IAM, a bucket and a billing account.

spark-submit --py-files deps.zip -m ubunye       --task-dir s3a://bucket/code/pipelines/sales/etl/daily --mode PROD

It deliberately does not create a SparkSession: spark-submit already made one, with the platform's master and executors, and the engine attaches to it.

A rest extra. requests lived only in the dev extra, so pip install ubunye-engine[spark] shipped a rest_api reader that could not make a request. It worked on Databricks by accident, because the runtime preinstalls it — the bug was invisible from the one platform most people use, and appeared the moment anyone left it, as a bare ModuleNotFoundError from inside a Spark job. A missing requests now says what to install.
format is now an open string, validated against the plugin registry.

The docs said "adding a connector = write the class, register the entry point". That was false: format was a closed enum, so a correctly-registered third-party plugin was rejected by config validation before the registry was ever consulted. The extension story the engine advertises did not work.

Any name the ubunye.readers / ubunye.writers entry points can load is now valid. An unknown name still fails, and the error lists what is actually installed.

Breaking: IOConfig.format is a str, not a FormatType. Code doing cfg.format.value must now use cfg.format.

AmbientSessionBackend — an alias for DatabricksBackend, which contains no Databricks code at all. It attaches to a session somebody else created and declines to stop it, which is equally true of Glue, EMR, Dataproc, a notebook and pytest. The name had convinced people the engine has a Databricks dependency here. It does not.

Fixed¶

{{ dt | default('latest') }} never fell back. ubunye run passes {"dt": ..., "dtf": ..., "mode": ...} unconditionally, and an omitted flag arrives as None. Jinja treats None as defined, so default() did not fire — the template rendered the literal string "None" and pipelines quietly wrote to paths like out/dt=None/. Nothing errored; the data just went somewhere nobody meant. None values are now dropped, so they are genuinely undefined. StrictUndefined still makes a real typo fail loudly.
ENGINE.catalog no longer breaks every non-Databricks Spark. set_catalog_and_schema issued USE CATALOG, which is a Unity Catalog statement — open-source Spark rejects it outright with PARSE_SYNTAX_ERROR. It is now attempted and, if unsupported, logged and skipped: the configs use three-part names anyway, which Spark resolves without it.
ubunye export airflow and ubunye export databricks emitted artifacts that could not run. Both generated ubunye run -c <config> --profile <p> — and there is no -c and no --profile on ubunye run. Every DAG and every job.json this exporter has ever produced would fail on its first task with "no such option". The tests asserted the broken flags, which is why nobody noticed: they checked the exporter still emitted what it always had, rather than something that works.

[0.3.0] — 2026-07-12¶

Breaking¶

The s3 writer now defaults to mode: append, not overwrite. Every other writer already defaulted to append; the same config key meant "insert" on one connector and "destroy and replace" on another. All writers are now consistent, and the default is the mode that cannot lose data by accident.

Action required: any s3 output that omitted mode and relied on the implicit overwrite must now say mode: overwrite explicitly. Nothing fails loudly — the write succeeds, it just appends. Grep your configs for format: s3 outputs with no mode key before upgrading.

Added¶

The delta, hive and binary connectors now exist. All three were declared in FormatType, accepted by config validation, and documented — but had no plugin registered behind them, so any pipeline using them died with ReaderNotFoundError / WriterNotFoundError.

Format	Reader	Writer
`delta`	new — by path, table, or SQL; time travel via `version_as_of` / `timestamp_as_of`	new — all six write modes
`hive`	already existed	new — all six write modes, `partitionBy`
`binary`	new — Spark's `binaryFile` source, one row per file	none: the source is read-only

A test now asserts that every format in FormatType resolves to a registered plugin, so this class of gap fails in CI rather than in a pipeline.

format: binary in CONFIG.outputs is rejected at config load. Spark's binaryFile source cannot write. The failure now happens in ubunye validate, with a reason, instead of after the transform has already run.
Four new write modes. CONFIG.outputs.*.mode previously accepted only overwrite, append and merge — and of those, merge was never implemented. The full set is now:

Mode	Behaviour
`errorifexists` (alias `error`)	Fail if the target exists — Spark's own default
`ignore`	No-op if the target exists
`merge`	Delta MERGE (upsert) on `merge_keys`; creates the target on first run
`overwrite_partitions`	Replace only the partitions in the DataFrame — on Delta (dynamic or `replace_where`), or via `INSERT OVERWRITE` on a non-Delta table

Integration tests on a real Spark session. The write modes are exercised against a real JVM Spark (and a real Delta table) in CI, not just asserted against mocks — ignore really leaves the target alone, merge really upserts, and overwrite_partitions really leaves its neighbouring partitions standing.
ubunye.core.write_modes. Mode semantics live in one module rather than in each connector. Writers declare the modes they support; a mode a connector cannot honour raises SinkWriteError before any rows are written instead of being silently downgraded. JDBC accepts Spark's four native modes only; rest_api is append-only.
partitionBy, merge_keys and replace_where on the s3 writer.

Fixed¶

mode: merge crashed at runtime. It passed config validation and was documented (with merge_keys), but no writer implemented it — the string was handed to df.write.mode("merge"), which Spark rejects with IllegalArgumentException: Unknown save mode. It now performs a real Delta MERGE.
errorifexists and ignore were unreachable. The JDBC and Unity writer docstrings claimed support, but the WriteMode enum rejected both at config load, so no config.yaml could ever set them.
The s3 writer silently dropped options. mergeSchema, overwriteSchema and friends were documented but never applied to the Spark writer.
Stale tests in tests/test_rest_api_plugin.py. Five tests still expected ValueError where the engine has raised typed SourceReadError / SinkWriteError for some time. They had been failing at HEAD unnoticed because CI only runs tests/unit.
overwrite_partitions refuses an unpartitioned target. Dynamic partition overwrite without partitions is a full-table wipe; the engine now fails the config instead of quietly destroying data. The session-level spark.sql.sources.partitionOverwriteMode conf is restored after each write rather than leaking into later writes.
overwrite_partitions now uses the mechanism each target actually supports: Delta's own partitionOverwriteMode write option for Delta, INSERT OVERWRITE (column-aligned, since it is positional) for an existing table, and Spark's session-level partitionOverwriteMode=DYNAMIC for a path. Every route is covered by an integration test against a real Spark session and a real Delta table.

[0.2.0] — 2026-05-20¶

Added¶

Interactive notebook API (ubunye.notebook). New ubunye.notebook() factory returns a NotebookContext for step-by-step task execution in Databricks notebooks. Data scientists can call ctx.read(), ctx.transform(), and ctx.write() in separate cells, inspecting DataFrames between stages. Environment variables referenced via {{ env.VAR }} in config.yaml are auto-resolved from Databricks widgets and secrets — no manual os.environ setup required.
Public step methods on Engine. Engine.read_inputs(), Engine.apply_transforms(), and Engine.write_outputs() expose the pipeline stages individually for interactive and advanced use cases.
extract_env_references() utility. Scans raw YAML text for {{ env.VAR }} patterns before Jinja resolution, enabling automatic env-var discovery.

[0.1.9] — 2026-05-19¶

Added¶

Formal typing.Protocol interfaces for four pluggable seams. DeployAdapter, RegistryBackend, LineageBackend, and AuthBackend in ubunye.interfaces define structural contracts that backends satisfy without inheriting from a base class. Cross-boundary dataclasses (DeployContext, DeployResult, ModelVersionInfo, LineageRecord, Credentials) accompany each protocol so callers never depend on a concrete backend's internal types.
Background metadata worker (Decision 1: non-blocking writes). ubunye._internal.MetadataWorker dispatches lineage and registry metadata writes to a daemon thread with a bounded queue (default 1 000, configurable via UBUNYE_METADATA_QUEUE_SIZE). Queue overflow drops the oldest pending write and logs a warning. Flush timeout configurable via UBUNYE_METADATA_FLUSH_TIMEOUT (default 30 s). Uses a worker-thread pattern — async/await fights Spark's threading model.
Graceful degradation with fallback manifests (Decision 2). When a metadata write fails, the record is appended to ~/.ubunye/fallback/{run_id}/{kind}.jsonl. Pipeline execution continues. Auth failures are excluded — they propagate immediately.
ubunye sync CLI command. Replays fallback manifests against configured backends with idempotent deduplication (key: run_id + task + recorded_at). Sub-commands: ubunye sync lineage, ubunye sync registry. Processed manifests are archived to ~/.ubunye/fallback/synced/.
Entry-point discovery for backend groups. Four new entry-point groups in pyproject.toml: ubunye.deploy_adapters, ubunye.registry_backends, ubunye.lineage_backends, ubunye.auth_backends. Third-party packages register backends by adding entry points under these groups.
Environment-based auto-detection (ubunye._internal.auto_detect). Registry: MLflow on Databricks, filesystem elsewhere. Lineage: Delta when UBUNYE_LINEAGE_TABLE is set, filesystem otherwise. Auth: service principal when both DATABRICKS_CLIENT_ID and DATABRICKS_CLIENT_SECRET are set, token when DATABRICKS_TOKEN is set, raises AuthNotFoundError otherwise.
Schema evolution (Decision 3). Every cross-boundary dataclass separates strict core fields from a flexible metadata: Dict[str, str]. Core fields never change without a major version bump. Every record stamps engine_version. Extra metadata keys survive round-trip through the filesystem lineage store.
Interfaces documentation page (docs/interfaces.md) covering protocols, dataclasses, design principles, discovery, auto-detection, fallback manifests, and how to write a custom backend.
39 conformance and failure-mode tests covering protocol isinstance checks, entry-point discovery, auto-detect logic, lineage/registry backend surfaces, non-blocking write timing, queue overflow, fallback manifest creation, sync dedup, auth propagation, and schema evolution metadata flexibility.
ServicePrincipalAuthBackend — OAuth M2M authentication using DATABRICKS_CLIENT_ID + DATABRICKS_CLIENT_SECRET. Entry point ubunye.auth_backends:service_principal. Takes priority over token auth when both are available (via auto-detect).
MLflowRegistryBackend — model registry that combines filesystem storage with MLflow experiment/run logging. Logs metrics, params, and stage transitions to MLflow when installed; falls back gracefully when MLflow is unavailable. Entry point ubunye.registry_backends:mlflow.
DeltaLineageBackend — lineage backend targeting Delta tables via Spark SQL on Databricks. Falls back to JSONL-based local storage when no active SparkSession is available, enabling unit tests without Spark. Entry point ubunye.lineage_backends:delta.
24 Phase 2 backend tests covering service principal auth (protocol conformance, env var resolution, error cases, entry-point discovery), MLflow registry (full CRUD lifecycle, promotion gates, metadata round-trip), and Delta lineage (record/get/search, metadata preservation, not-found errors).
Performance benchmark suite (benchmarks/bench_engine.py). Nine benchmarks covering config loading (plain and Jinja), entry-point discovery (cached and cold), lineage I/O (write, read, search), metadata worker throughput, and registry registration. Reports ops/sec, p50/p99 latencies, mean, and stdev. Results saved to benchmarks/results.json for before/after comparison.

Changed¶

Filtered entry-point loading. _load_group() in ubunye._internal.discovery now calls importlib.metadata.entry_points(group=group) instead of loading all groups and filtering. Cached discovery is near-zero cost.
O(1) lineage lookup by run ID. FileSystemLineageStore.get_run() uses a run_id→Path index instead of rglob("*.json"). The index is built lazily on first lookup and updated incrementally on writes. ~34% faster reads.
In-memory RunContext cache for lineage search. Parsed RunContext objects are cached on first load and reused by subsequent search() and list_runs() calls, avoiding repeated JSON parsing and disk I/O. ~51% faster searches across 100 records.

Fixed¶

[0.1.8] — 2026-05-19¶

Added¶

ubunye deploy databricks command. End-to-end deployment from a local config.yaml to a running Databricks job — handles auth, file upload to /Workspace/, wrapper notebook generation, and idempotent job creation/update via the Databricks SDK. Uses a two-level targets.yaml lookup (usecase-level defaults, task-level overrides). Supports --dry-run for previewing the job spec without deploying. New optional dependency: pip install ubunye-engine[databricks].
Deploy error types. DeployError base class with AuthNotFoundError, AuthInvalidError, TargetNotFoundError, WorkspaceUploadError, and BundleDeployError — all follow the dual-inheritance pattern (DeployError(UbunyeError, RuntimeError)).
Structured error messages across the engine. Every user-facing exception now inherits from UbunyeError (with optional context dict and hint string) and the stdlib type it replaces (dual inheritance for backward compatibility). New exception classes: TaskNotFoundError, TaskClassMissingError, ReaderNotFoundError, WriterNotFoundError, TransformNotFoundError, TransformOutputError, MonitorNotFoundError, SourceReadError, SinkWriteError, SparkSessionError, ModelLoadError, ModelNotFittedError, VersionExistsError, VersionNotFoundError, PromotionBlockedError, RegistryNotFoundError, LineageRecordNotFoundError, and ConfigProfileError. Existing config errors (ConfigFieldError, ConfigTemplateError) migrated from ubunye.config.loader to ubunye.core.errors. See the new Error Reference for the full hierarchy and examples.
Hook failure logging. HookChain now logs a structured warning when a hook's task() or step() context manager raises, instead of silently swallowing the exception.
CI workflows link to the Databricks job UI after bundle run, so users can find notebook cell output that databricks bundle run does not stream to stdout.
ubunye init github-actions command. Generates a GitHub Actions workflow for any pipeline — validates config and runs tests on PRs, deploys to Databricks via ubunye deploy databricks on merge to main. Supports --no-deploy for CI-only workflows, --extras for pip install extras (auto-includes Java setup when spark is present), and --target for the Databricks deploy target. The init command is now a sub-app: ubunye init pipeline (formerly ubunye init) and ubunye init github-actions.

Fixed¶

Promotion gate failure now propagates. PromotionBlockedError was silently caught by the titanic training task's except ValueError (via dual inheritance). Gate failures now re-raise — CI goes red when a model fails its quality gate instead of reporting green.
typer[all] replaced with plain typer. Typer >= 0.24 dropped the [all] extra, causing pip warnings on install.
merged_spark_conf() / resolved_catalog() / resolved_schema() now raise ConfigProfileError when profiles are defined but the requested profile doesn't match. Previously these methods silently returned the base config, hiding typos in --mode / --profile flags.

Changed¶

ubunye init is now a sub-app with two subcommands: ubunye init pipeline (the previous ubunye init) and ubunye init github-actions. This is a breaking CLI change — update any scripts that call ubunye init -d ... to ubunye init pipeline -d ....
[ml] extra no longer installs torch. Use [ml-torch] for PyTorch workloads. This saves ~1 GB of CUDA wheel downloads in CI for sklearn-only pipelines.
CONFIG.transform.type is now optional. When omitted, the engine defaults to loading the user's Task class from transformations.py — no type: noop declaration needed. Existing configs with type: noop continue to work but emit a DeprecationWarning advising removal. All scaffolded configs, production examples, test fixtures, and docs have been updated to omit the field.
Unknown config fields are now rejected at load time. All Pydantic models (except IOConfig) use extra="forbid". A typo like ENGNE raises ConfigFieldError with a "Did you mean 'ENGINE'?" suggestion via difflib.get_close_matches. IOConfig retains extra="allow" so plugin-specific keys (REST API headers, pagination, etc.) pass through to connectors. Breaking: configs with unknown top-level or nested fields that were previously silently ignored will now fail.
Undefined Jinja template variables fail immediately. The resolver now uses StrictUndefined instead of DebugUndefined. A reference to {{ ds }} without passing ds as a CLI variable raises ConfigTemplateError listing available variables. The | default() filter continues to work.

Added¶

Service principal + OAuth auth for Databricks CI. The four Databricks example workflows (databricks_deploy.yml, jhb_weather_databricks.yml, multitask_databricks.yml, titanic_ml_databricks.yml) now pass DATABRICKS_CLIENT_ID / DATABRICKS_CLIENT_SECRET alongside the existing DATABRICKS_TOKEN env var. The secrets gate accepts either flow: PAT (HOST + TOKEN) or OAuth (HOST + CLIENT_ID + CLIENT_SECRET). The Databricks CLI auto-selects OAuth when both client vars are set. New reference page docs/databricks-auth.md walks through service principal creation, workspace/UC grants, secret rotation, and verification.

[0.1.7] — 2026-04-21¶

Changed¶

MODEL and VERSION are now optional at the top level of config.yaml. MODEL defaults to etl and VERSION defaults to "0.0.0-dev". The semver validator now accepts an optional pre-release suffix (e.g. 1.0.0-rc1, 0.0.0-dev) in addition to plain MAJOR.MINOR.PATCH. Existing configs that set these fields explicitly continue to work unchanged. Rationale: these two lines were boilerplate in every scaffolded pipeline; the defaults cover the common case. Set them explicitly when job type or version is load-bearing (lineage, model registry, orchestrator metadata).

Added¶

Production reference example: Titanic multi-task pipeline (local) — examples/production/titanic_multitask_local/ demonstrates sequential task chaining via ubunye run -t clean_data -t aggregate. Task 1 reads the Titanic CSV, cleans it, and writes intermediate Parquet. Task 2 reads that Parquet and computes survival rates by class and age group. Exercises run_pipeline(), sibling-module isolation between tasks, and cross-task lineage. CI workflow (.github/workflows/multitask_local.yml) runs Spark unit tests, validates both configs, runs the full pipeline, and verifies output. See the example's README.md.
Production reference example: Titanic multi-task pipeline (Databricks) — examples/production/titanic_multitask_databricks/ is the Databricks counterpart of the local multi-task example. Same transformations.py (byte-identical, CI-enforced), but task chaining uses Unity Catalog Delta tables instead of Parquet files. Task 1 writes titanic_cleaned, task 2 reads it and writes survival_summary. Runs via ubunye.run_pipeline() on serverless compute. CI workflow enforces portability diff against the local example and validates/deploys the Asset Bundle.
Production reference example: Titanic ML end-to-end on Databricks — examples/production/titanic_ml_databricks/ demonstrates the full ML lifecycle: UbunyeModel subclass (sklearn RandomForest), MLflow param/ metric/artifact logging, filesystem-backed ModelRegistry on a UC volume, and a PromotionGate on validation AUC. Two serverless jobs share one Asset Bundle — titanic_train registers and auto-promotes; titanic_predict loads the current production/staging model and writes a Unity Catalog Delta predictions table. A one-row training_metrics audit row is appended per training run. CI (.github/workflows/titanic_ml_databricks.yml) runs pandas/sklearn unit tests, diffs the two model.py copies for drift, and validates/deploys the bundle when Databricks secrets are configured. See the example's README.md.

Changed¶

GitHub Actions bumped to Node 24-capable majors. actions/checkout@v4 → @v6, actions/setup-python@v5 → @v6 across all nine workflow files. Resolves the deprecation warning surfaced on 2026-04-16 runs ahead of the Node 20 removal deadline (2026-09-16). No behavioural change.

Fixed¶

Undefined CLI template variables silently leaked into resolved configs. resolve_config pre-checked {{ env.X }} references but not bare {{ var }} identifiers. Jinja2's DebugUndefined left unresolved expressions verbatim, so path: "file:///{{ dt }}" with no dt provided would pass validation and then hand Spark a literal file:///{{ dt }} at runtime. Fix adds a post-render residue scan that names the offending variable and suggests a CLI flag, env var, or | default() filter. Regression tests in tests/unit/config/test_resolver.py. Pre-existing configs under examples/ and pipelines/ all use | default() on CLI-derived vars, so no downstream config needs updating.
Sibling modules leaked between sequential tasks in run_pipeline. _with_task_dir_on_path added the task dir to sys.path but never cleaned up sys.modules on exit. Two tasks that each shipped their own model.py (or utils.py, etc.) would silently run the first task's module when the second task imported it — Python's import cache was keying on the shared short name. Fix evicts only modules whose source file lives under the exiting task dir; stdlib and site-packages are untouched. Regression test in tests/unit/test_task_runner.py. Caught by offline audit on the overnight branch, ahead of the planned multi-task DAG example (tasks/todo/task-04.md).

[0.1.6] — 2026-04-15¶

Changed¶

titanic_databricks DAB switched to serverless + Unity Catalog. Removed the new_cluster block and the DBFS bootstrap; the notebook now provisions a UC volume, downloads the Titanic CSV into it at runtime, and the writer emits a Unity Catalog managed Delta table (workspace.titanic.survival_by_class by default). Validated end-to-end on Databricks Free Edition. The portability contract with titanic_local (byte-identical transformations.py) is unchanged — only the deployment wrapper (config + notebook + DAB) differs.
jhb_weather_databricks DAB switched to serverless compute. Removed the new_cluster block (and the existing_cluster_id escape hatch) from databricks.yml. Notebook tasks with no cluster spec route to serverless on both Free Edition and paid workspaces; the notebook installs ubunye-engine at runtime via %pip. Default weather_catalog changed from main to workspace because Free Edition auto-provisions only the workspace catalog. Paid-workspace users override via --var="weather_catalog=main". README documents the rationale.
databricks_deploy.yml (titanic) skips deploy gracefully when secrets are absent. Mirrors the soft-skip pattern in jhb_weather_databricks.yml so PRs from forks (or repos that have not configured DATABRICKS_HOST/DATABRICKS_TOKEN) still exercise the unit tests and portability diff instead of failing the workflow.
Production examples switched from pandas twins to PySpark tests. Every transformations.py under examples/production/ now exposes a single Spark implementation. Tests use a session-scoped SparkSession fixture (local[1], 512 MB driver, shuffle partitions=1) so the production code is the code under test. Eliminates the dual-maintenance burden and the risk of silent drift between pandas and Spark paths. Unit-test CI steps now install Java 17 + ubunye-engine[spark,dev] on the runner.

Added¶

Production reference example: JHB hourly weather (REST API → Unity Catalog) — end-to-end example at examples/production/jhb_weather_databricks/ demonstrating REST ingestion with the rest_api reader against the free Open-Meteo API (lat/lon for Johannesburg, no auth required), a Spark transform that explodes parallel hourly arrays into a tidy one-row-per-hour DataFrame, and a Unity Catalog Delta writer partitioned by forecast_date. Ships a Databricks Asset Bundle with a scheduled daily job (06:00 Africa/Johannesburg), a notebook wrapper around ubunye.run_task(), seven pandas unit tests over a hand-built fixture response, and a CI workflow (.github/workflows/jhb_weather_databricks.yml) that runs the tests, smoke-checks the endpoint, and validates/deploys the bundle when Databricks secrets are configured. See the example's README.md.
Production reference example: Titanic (local runtime) — end-to-end example at examples/production/titanic_local/ demonstrating a CSV → Parquet pipeline with dev/prod profiles, Jinja-templated paths, pandas unit tests (no Spark), a committed golden Parquet, and a GitHub Actions workflow (.github/workflows/local_pipeline.yml) that validates config, runs the pipeline on a real SparkSession, and diffs the output against the golden. One half of the portability demo — the Databricks counterpart shares transformations.py verbatim. See the example's README.md.
Production reference example: Titanic (Databricks Community Edition) — the Databricks half of the portability demo at examples/production/titanic_databricks/. Ships a Databricks Asset Bundle (databricks.yml) sized for CE's single-node / DBFS / no-UC constraints, a notebook entry (notebooks/run_titanic.py) that calls ubunye.run_task() against the active SparkSession, the same pandas unit tests, and a CI workflow (.github/workflows/databricks_deploy.yml) that installs the Go-based Databricks CLI, validates and deploys the bundle, and enforces the portability contract by diffing transformations.py against the local example. Known CE limitations (no service principals, restricted Jobs API, DBFS deprecation) are documented honestly rather than worked around.
Cross-runtime reference index — examples/production/README.md explains the portability contract, provides a side-by-side config comparison of the two examples, a decision guide for choosing between the local and Databricks runtimes, and a migration table covering what changes when moving from Community Edition to a standard Databricks workspace.
Hook abstraction for observability (ubunye/core/hooks.py) — Hook base class and HookChain multiplexer. Tasks and steps are now wrapped in hook context managers so the Engine no longer imports telemetry modules directly. Built-in hooks shipped under ubunye/telemetry/hooks/: EventLoggerHook, OTelHook, PrometheusHook, LegacyMonitorsHook. Third parties can register custom hooks (Slack alerts, audit logs, drift checks) without modifying the Engine. See docs/patterns/hooks.md.
ubunye.hooks entry point group (pyproject.toml) — third-party packages can register Hook subclasses as entry points and have them auto-discovered by the Engine. The three built-in telemetry hooks (events, otel, prometheus) are registered via this mechanism and gated on UBUNYE_TELEMETRY=1.
Python API (ubunye/api.py) — run_task() and run_pipeline() for running Ubunye tasks from Python code (Databricks notebooks, scripts, tests) without the CLI. Auto-detects and reuses active SparkSessions. Exported from ubunye.__init__.
DatabricksBackend (ubunye/backends/databricks_backend.py) — backend that wraps an existing SparkSession instead of creating one. stop() is a no-op since we don't own the session.
Dev notebook scaffolding — ubunye init now generates notebooks/<task>_dev.ipynb alongside config.yaml and transformations.py. The notebook uses DatabricksBackend, dbutils.widgets, and display(). The Load step is commented out by default.
Deployment docs — docs/deployment.md covering Databricks Asset Bundles pattern, GitHub Actions CI/CD, and Python API on Databricks. DABs belong in the usecase repo, not the engine.
Deploy workflow — .github/workflows/deploy.yml validates configs on PR and runs unit tests. Bundle deployment is handled in the usecase repo.
ubunye test run CLI sub-command — runs tasks with a test profile and reports PASS/FAIL.
Model Registry (ubunye/models/) — library-independent ML lifecycle management.
UbunyeModel abstract contract: train, predict, save, load, metadata, validate.
ModelRegistry — filesystem-backed versioning with stages: development → staging → production → archived.
PromotionGate — configurable metric thresholds (min_*, max_*, require_drift_check).
load_model_class() — dynamic model file importer; mirrors the task-dir import pattern.
ModelTransform plugin (type: model) — train and predict from config YAML.
ubunye models CLI sub-commands: list, info, promote, demote, rollback, archive, compare.
RegistryConfig and ModelTransformParams Pydantic schema additions.
Lineage tracking (ubunye/lineage/) — automatic run provenance.
RunContext, LineageRecorder, FileSystemLineageStore, hash_dataframe.
ubunye lineage CLI sub-commands: show, list, compare, search, trace.
--lineage flag on ubunye run.
REST API connector — paginated HTTP reader and writer.
Pagination strategies: offset, cursor, next_link.
Auth: bearer, api_key (header or query param), basic.
Rate limiting with configurable requests_per_second.
Exponential backoff retry on configurable status codes.
Optional explicit schema declaration.
Config validation — ubunye validate command with full Pydantic v2 schema.
Format-specific field validation in IOConfig.
Jinja2 rendering before Pydantic validation.
Semver validation on VERSION field.
ubunye export airflow|databricks CLI — the AirflowExporter and DatabricksExporter under ubunye/orchestration/ are now reachable from the command line. The command loads the task's config.yaml, pulls defaults from its ORCHESTRATION block, and writes the artifact to --output. Airflow emits a DAG Python file; Databricks emits a Jobs API job.json. Classes are now exported from ubunye.orchestration.__init__.
Test infrastructure — 288 unit tests, all Spark-free in tests/unit/.

Changed¶

databricks_expoter.py renamed to databricks_exporter.py (typo fix). Not previously exported from ubunye.orchestration, so external callers are unaffected.
Unified execution path (ubunye/core/task_runner.py) — api.py, cli/main.py run and cli/test_cmd.py run previously each reimplemented the read → transform → write loop and called load_monitors / safe_call directly. They now delegate to execute_user_task(), which wraps the user's Task.transform() as an ephemeral Transform plugin and runs it through Engine. Single code path, single hook lifecycle, MonitorHook adapts the legacy lineage recorder to a Hook. Engine.__init__ gained extra_hooks= (append to defaults) and manage_backend= (caller-owned vs engine-owned backend lifecycle). run_task / run_pipeline accept hooks= for notebook callers who want to swap in custom hook chains.
Engine runtime refactored — ubunye/core/runtime.py reduced from 374 to 255 lines. Engine.run() body shrank from ~220 lines to ~35 by delegating telemetry plumbing to hooks. The engine no longer imports from ubunye.telemetry.* — only from ubunye.core.hooks. UBUNYE_TELEMETRY and UBUNYE_PROM_PORT env vars still honored; user monitors in CONFIG.monitors continue to work via LegacyMonitorsHook. Engine.__init__ gained an optional hooks= argument.
ubunye/config/schema.py — added RegistryConfig, ModelTransformParams, FormatType.REST_API.
ubunye/__init__.py — exports run_task and run_pipeline.
ubunye/cli/main.py — mounted models_app, lineage_app, test_app Typer sub-apps; added notebook scaffolding to init.
pyproject.toml — added model entry point under ubunye.transforms.

Fixed¶

Unity Catalog writer collapsed plugin dispatch with Spark source format. UnityTableWriter.write() was reading cfg["format"] and passing it to Spark's DataFrameWriter.format(...). But cfg["format"] is the Ubunye plugin selector and is always "unity" by the time the writer runs, so Spark raised [DATA_SOURCE_NOT_FOUND] Failed to find the data source: unity. Switched the writer to read cfg["file_format"] (defaulting to delta), matching the convention already used by the s3 writer. Caught while deploying the jhb_weather_databricks example to a serverless workspace.

[0.1.0] — 2025-09-11¶

Added¶

First alpha release of Ubunye Engine.
Config-first ETL framework built on Apache Spark.
Plugin system for Readers, Writers, and Transforms via Python entry points.
Built-in connectors: Hive, JDBC, Delta, Unity Catalog, S3, binary.
CLI commands: init, run, plan, config, plugins, version.
Orchestration exporters: Airflow DAG Python file, Databricks Jobs API JSON.
Internal ML wrappers: SklearnModel, SparkMLModel, BatchPredictMixin, MLflowLoggingMixin.
Telemetry modules: JSON event log, Prometheus, OpenTelemetry.
Example tasks: fraud_detection/claims/claim_etl, rest_api/customer_sync.
SparkBackend with context manager and safe multiple-start support.