Changelog

All notable changes to Ubunye Engine will be documented here.

The format is based on Keep a Changelog and this project adheres to Semantic Versioning.

---

[Unreleased]

[0.9.3] (2026-11-11)

Fixed

• A run record table with a hyphenated or three-part name is created and written. The Delta run record store gave Spark the table name as written, so a Unity Catalog catalog with hyphens (aws-db-x-catalog.schema.uby_run_records) could not be read, and Delta's table builder refused any catalog.schema.table name on open-source Delta. The job carried on, but no run was recorded. The store now quotes each part once (the | ident rule) and creates the table with CREATE TABLE IF NOT EXISTS ... USING DELTA CLUSTER BY (task_path).

Changed

• The Databricks examples follow the 0.9.2 job notebook. The job notebooks in examples/production (titanic_databricks, jhb_weather_databricks, titanic_multitask_databricks, titanic_ml_databricks) run each task as one with ubunye.notebook(...) as run: block of three commented steps (READ, TRANSFORM, WRITE). Their bundles install ubunye-engine==0.9.2 through the job's environment instead of %pip, switch run records on with the job parameter ubunye_run_records (a table next to the outputs), and pass git_commit, git_branch, deploy_target and ubunye_mode. Their configs quote the catalog and schema with | ident. The business logic and the local examples are unchanged.
• examples/python_api shows the job notebook shape. A new run_as_notebook.py runs clean_data as one ubunye.notebook() with block on local Spark, with a run record in a local folder; the README shows the same block as a Databricks job notebook. The run_task and run_pipeline scripts are unchanged.

[0.9.2] (2026-11-10)

Added

- Job settings read in one place: ubunye.core.settings.job_settings()

returns where run records go (run_records, run_records_fallback), the batch date (dt) and the mode, and where each came from. What the code passes wins, then the job's parameters (on Databricks, the notebook widgets: the only channel on serverless compute, where a job cannot set environment variables), then UBUNYE_* environment variables. It works off Databricks and never fails a run for a platform reason (ADR 015). - Who and where ran a task: ubunye.core.identity.collect_identity() returns the platform, workspace, job, job run, task, user, git commit and branch, and deploy target. Platforms are plugins in the new ubunye.run_identity entry point group: databricks (job parameters and the notebook context, dbutils never imported at import time), ci (GitHub Actions, GitLab, Azure Pipelines) and local (git rev-parse in the task folder).

• Every run record says who and where ran it, under environment["identity"]: platform, workspace, job, job run, task, user, git commit and branch, deploy target. The entry point passes it on the new EngineContext.identity; when that is empty the recorder collects it, so the CLI, run_task, run_pipeline and notebook all record it. The record version stays 2 and older records read as before. environment_hash leaves the identity out, so runs on the same environment still hash the same. The Databricks deployment docs show the job parameters a databricks.yml passes (ubunye_run_records, git_commit: ${bundle.git.commit}, git_branch: ${bundle.git.branch}, deploy_target: ${bundle.target}) in a full example.

- A job notebook is three steps. with ubunye.notebook(task_dir, dt=dt) as run:

then run.read(), run.transform(sources), run.write(outputs). The block is one run (ADR 016): a success and every failure leave a run record, and a failed one says where it failed in the new failed_stage field (setup, read, expectations, transform, write, notebook, or a step named with run.step("score")). Before, a notebook run that failed before write() left no record. run_task records the same stages. - Run records turn on from the job. A run-record location switches them on: run_records=, else the job parameter ubunye_run_records, else $UBUNYE_RUN_RECORDS. A table name goes to the Delta store, a path to files. ubunye.notebook, run_task and run_pipeline all read it; lineage=False turns records off; lineage=True, lineage_dir= work as before. dt and mode left empty (dt="" from an empty job parameter) come from the job parameters dt / ubunye_mode, else $UBUNYE_DT / $UBUNYE_MODE. - Each notebook step prints one line (what it did, how long, or why it failed), and the run's last line says where its record went. quiet=True turns it off. - The notebook checks CONFIG.requires against its run record store when records are on (it never checked it before); without a store it warns.

Changed

• The Delta run record store says what to do when it cannot create or write its table: create it once, or grant CREATE TABLE (MODIFY) on the schema. The job still goes on.
• A run record that could not be saved is kept in a fallback folder only when one is configured (ubunye_run_records_fallback / $UBUNYE_RUN_RECORDS_FALLBACK, usually a Unity Catalog Volume) for runs whose records were switched on by a location; with none, a warning says why. lineage=True keeps ~/.ubunye/fallback.
• Notebook run records now carry the task's code hash (the notebook passes its task folder to the run).

- | ident, a Jinja filter for SQL names (ADR 017). {{ env.TELM_READ_CATALOG | ident }}

gives `aws-x-catalog` whether the value arrives with backticks or without, drops surrounding spaces, and quotes each part of catalog.schema.table once. A Databricks catalog with a hyphen no longer needs the config to guess how the job parameter is spelled. An empty name is refused when the config loads. ubunye validate (and its --json, as sql) and ubunye plan print each input's SQL after templating. ubunye doctor no longer calls a variable required when a filter such as ident sits before its | default(...). - Write mode insert_new (ADR 018), with required merge_keys: only rows whose keys are not in the target are inserted; existing rows never change, so the first write of a key wins and a rerun adds nothing. On Spark it is a Delta MERGE with only WHEN NOT MATCHED THEN INSERT * (null-safe keys), on a table or a path, on delta, unity, and s3 / hive with file_format: delta. The first run creates the target; a path holding non-Delta files is refused, not written over. validate and plan check it; pandas, jdbc and rest_api refuse it with a clear message.

Fixed

• A Task sees the run's batch date. self.dt and self.variables (dt, dtf, mode and every --var) are set on the Task before setup(), from run_task, the CLI and ubunye.notebook. Before, a Task had only self.config, so a backfill run with -dt computed the current month instead of the one asked for.
• The deploy workflow's config check named a sample pipeline that no longer exists and ended in || true, so it could never fail. It now validates fraud_detection/risk_scoring and fails the job when the config is invalid.

Changed

• The home page plays the narrated intro film from YouTube, in privacy mode (youtube-nocookie.com, nothing is loaded until you press play). The silent placeholder video and its poster are gone from the repo.

[0.9.1] (2026-11-10)

Changed

• One path for the model registry. ModelRegistry is now a thin front over the lifecycle layer for every backend, the filesystem included. The filesystem no longer keeps its own copy of the rules (version numbers, gates, one production version, rollback, archive). Same API, same files on disk: a store written by 0.9.0 reads and promotes the same (a fixture test pins it). Reading a version's stage no longer downloads its artifact from a remote store.
• ModelRegistry.promotion_gates gives {} for a model with no versions on the filesystem too (it raised RegistryNotFoundError), like every other backend.
• Metadata given through the FilesystemRegistryBackend protocol (kept nested under registry_metadata on disk, unchanged) now reads back flat from ModelRegistry, as it already did through the backend. A model whose own metadata() uses a version field's name (promoted_by, lineage_run_id, config_hash, the stage times) or registry_metadata reads back with that key in the field, as on every other backend; no shipped model does.

• An unknown stage name given to the filesystem backend raises ValueError; it was written to registry.json as is.

• ubunye deploy databricks defaults to Databricks Runtime 17.3 LTS (17.3.x-scala2.13, Spark 4.0), the runtime ubunye export databricks already used, for a target in targets.yaml that does not set spark_version. The old default, 13.3 LTS, runs Spark 3.4, older than the engine supports (pyspark 3.5 or later). A target that sets spark_version is unchanged.

• A first Delta append of a named batch makes no history lookups (F-100). The two DESCRIBE HISTORY calls around each keyed append, there only to warn that Delta skipped a batch an earlier run had committed, now run only when an earlier run of the batch could have: the run took over a dead run's lease, or a failed run that wrote the batch left a note beside it. Same warning, fewer steps.

• The docs were reworked for clarity. Each topic now has one home page, and the others link to it. New pages: Concepts, Run records, Safe reruns, Troubleshooting, Upgrading from 0.8 and Azure. The README is a short front door. The home page has an intro film. Every page uses the same words (use case, pipeline, task, profile, run record), and internal ticket numbers are gone from user pages. Examples that did not run as written (drift detection, RAG, the registry gates, the hooks setup, the Databricks bundle) now run, and their output is real. The site has top tabs, tap to zoom on diagrams, link preview cards, and colours that match the diagrams.

Fixed

• ubunye run on a task whose transformations.py has no Task class prints one [ERROR] Run refused line, not a traceback. ubunye doctor no longer suggests --backend databricks on a laptop (it only attaches to a cluster's session).

• Saving a Delta run record from a CLI or pandas run is about twice as fast (F-116). With no active SparkSession the Delta run record store starts a local one, and Delta read the run record log in 50 tasks on every save. That session, and only that one, now sets spark.databricks.delta.snapshotPartitions=1 and turns off the Spark UI and console progress bar. Measured on the dev box (16 cores, 8 saves): about 1.35 s to 0.72 s per save. A session the user or the platform started is never changed.

• Each command's help says its default profile plainly (F-115). ubunye export defaults to production (--profile prod for airflow and databricks, -m PROD for spark-pipeline) and ubunye run to DEV. The defaults are unchanged; the help of export spark-pipeline and run now says so, as export airflow and export databricks already did, and docs/cli.md says it once for all.

• python -m ubunye documents only what works (F-114). Its docstring showed spark-submit ... -m ubunye (spark-submit takes a file, not -m) and a --task-dir s3a://... (only a local folder works); docs/deployment/anywhere.md said "path or bucket". Both now show the file path and a local task folder, and point to ubunye deploy glue|dataproc|emr-serverless for running from a bucket.

• A task the backend cannot run is one [ERROR] line in ubunye run (F-113). A BackendCapabilityError (say, a hive input on --backend pandas) printed a full traceback box. It now ends in [ERROR] Run refused for <task>: ... with the hint and exit code 1, like the other refusals. Nothing ran, as before.

• Spark with no usable Java fails in one clean message (F-112). The spark backend now looks for Java (JAVA_HOME, then java on PATH) and its version before it starts a session. No Java, or one older than the installed Spark needs, raises SparkSessionError with the versions that work, and ubunye run prints it as one [ERROR] line instead of pyspark's [JAVA_GATEWAY_EXITED] traceback. A session that is already running, or Spark Connect, is not checked. ubunye doctor and ubunye backends ask the same function, so the spark backend is no longer called usable with no Java. The Java versions are now right everywhere: Spark 4 runs on Java 17 or 21, Spark 3.5 on Java 8, 11 or 17 (the doctor's fix text said 17 or 11).

• ubunye deploy dockerfile --out img/ writes img/Dockerfile (F-111). A path ending in / is a folder: it is made when missing. Before, a missing img/ gave a file named img.

• The workflow from ubunye init github-actions works out of the box (F-110). ubunye deploy databricks --host now reads DATABRICKS_HOST, the secret the workflow sets, so the deploy step needs no committed targets.yaml (a targets.yaml, when there is one, still wins). The manual "Run job" step now starts the job with the new --run flag; before, it deployed again and started nothing. --run starts a run after the deploy and prints its link. The workflow's "Notebook output link" step is gone: the run step prints the link.

• hooks= adds to the engine's hooks, it does not replace them (F-101). ubunye.run_task(..., hooks=[...]), run_pipeline and notebook dropped the run record (lineage=True) and the CONFIG.monitors hook when given hooks. Your hooks now run with them. Engine(hooks=...) is unchanged.

• A unity read that names its table with db_name fails ubunye validate (F-102). It passed validation and failed at run time: the reader never read db_name. The message says what to use: table: 'catalog.schema.table', or catalog + schema + tbl_name (now accepted by validate, as the reader always read it), or sql. ubunye init --template databricks wrote such a read; it now writes table: main.raw_db.<task>_input, and a test checks that every template init writes passes ubunye validate.

• ubunye validate reports a write mode the writer cannot do (F-103), with the message ubunye plan gives: a jdbc output with mode: merge, a merge with no merge_keys. Only plan and the write itself caught these. One check (write_modes.resolve_for) now serves both. A writer that declares no modes is not held to any, as before.

• secret:// in ENGINE.spark_conf is checked and resolved (F-104). It reached Spark as literal text, and a misspelt provider passed ubunye validate. Validate now checks it (base and profiles), and a Spark backend gets the value, fetched when the backend is built, masked like any secret. A backend without Spark does not fetch it.

• The pandas backend reads a JSON file holding one array (F-105), as Spark reads it with multiLine. It read JSON Lines only, so a pretty-printed array failed on its first line. JSON Lines, including an array on each line, reads as before.

• The C01 proving workload runs on pandas without a Narwhals warning (F-106). Its counts and sums were cast inside the group-by, which made Narwhals fall back to a slow per-group apply and warn "Found complex group-by expression". The casts now follow the aggregation. Same outputs, same golden digest.

• Registering with promotion_gates={} clears a model's stored gates on every backend. Before, only the default filesystem path cleared them; MLflow and the others kept the old gates.

• A rollback keeps who promoted the version on every backend. Before, a filesystem backend named by object, and the deprecated MLflowRegistryBackend, set it to empty.

• ubunye deploy and ubunye export end an expected error in one [ERROR] message (F-096), exit code 1, the way ubunye run does. No targets.yaml, no token, a cloud CLI not on PATH, a bad task list or a config that does not validate printed a full traceback; deploy k8s and deploy container-apps also printed their whole launcher script. A missing kubectl or az is now found before anything runs, and a failed launcher is named, never printed. A bug still shows its traceback. A config that does not validate now raises ConfigError (still a ValueError).

• --resume skips a task that only overwrites once it finished the batch (F-098). Only a task with an append output was skipped, though the docs said every finished task is. With --lineage (lineage=True), a task whose latest run record for this batch (same -dt and --var) says success is now skipped with [SKIP], whatever it writes. Without run records an overwrite task runs again, as the docs now say.

• The Delta run record store no longer grows one file per save for ever (F-099). A table it creates is clustered by task_path (liquid clustering) with auto compaction on (delta.autoOptimize.autoCompact), so Delta merges the small files each append leaves: 200 saves leave under 60 files, not 200. An existing table is used as it is; docs/cli.md shows the three statements that turn the same on for it.

• A run record no store could write is kept, whatever the store (F-097). When the filesystem store could not write (its folder under a file), the record was lost with a log line, and ubunye sync lineage found nothing. Now every store's failed save goes one way, through the recorder, to ~/.ubunye/fallback/<run_id>/lineage.jsonl, and the warning names the ubunye sync lineage command that replays it. The Delta store's own copy of that code is gone; its save raises like any store's.

• The docs say what 0.9.1 does. Merging the docs rework dropped some 0.9.1 facts and kept older text they made false. Back in place: deploy databricks --run and --host reading DATABRICKS_HOST (the Databricks guide also lost a stray copy of that text in "What success looks like", and no longer says to commit a targets.yaml for the workflow), --resume skipping an overwrite task with --lineage, the Delta run record table's clustering and the three statements for an existing table, the fallback manifest for every store, and python -m ubunye taking only a local --task-dir. Fixed: hooks= and OTelHook no longer said the run record is dropped, secret:// in spark_conf no longer said it is not read, validate -m no longer said a config with no profiles fails, registry.auto_version is marked deprecated, and the troubleshooting doctor output is the real 0.9.1 output (Java versions).

• The CLI reference shows every command again. A code span broken across two lines put <table> at the start of a line, and Markdown read it as raw HTML: everything after "Where run records are stored" (lineage show, runs, models, prove, sync, deploy, export) showed as plain text. The same fix applies to ADR 008. The example catalog names on the deployment page are now neutral.
• ubunye validate checks the profile the way run does. It uses DEV when no -m or --profile is given, refuses a name the config does not define, and accepts any name when the config has no profiles. Before, a config with profiles dev and prod passed validate and then stopped run. The error now lists the profiles and, when only the case differs, says which one you meant. run prints the same message instead of a traceback.
• ubunye plan and ubunye doctor check -m too. plan -m XYZ said "Plan OK" on a config whose profiles are DEV and PROD, and doctor passed it. Both now refuse a profile the config does not define, with the message run gives, and accept any name on a config with no profiles.
• load_config(profile=...) follows the same rule (and so run_task, run_pipeline and notebook, which pass profile= to it). It refused any profile on a config with no profiles, which run accepts. It now calls UbunyeConfig.check_profile, the one rule, and its error is the one run prints. It is still a ConfigProfileError.
• prove observe before any run is a clean error (exit 2) that says to run the task with --lineage first, not a traceback. An unknown --run-id is too.
• The warning for CONFIG.transform.type no longer says it became an error in 0.8.0. It says what happens (the task still runs transformations.py) and what to do instead.
• The config that ubunye init writes spells {{ task_dir }} in its comment the way the config uses it.

Deprecated

Nothing in Ubunye, its docs or examples uses these. They still work, warn with a DeprecationWarning, and go in 1.0:

• registry.auto_version in a model transform. It never did anything: a version is made whenever registry.version is not set. Remove the key.
• ubunye.core.planning.plan_problems, ubunye.proving.matrix.render_many, ubunye.plugins.ml.adapters.to_numpy_from_pandas and to_numpy_from_spark (use ensure_Xy_numpy), ubunye.cli.backend_choice.capabilities_or_exit, and ubunye.telemetry.otel.init_tracer (use setup).

Removed

• The private module ubunye._internal.auto_detect (detect_registry_backend, detect_lineage_backend, detect_auth_backend). Nothing called it; the registry backend has been chosen by ubunye.models.backends.choose since 0.9.0 (ADR 010), and the lineage store by ubunye.lineage.stores (ADR 011).
• Dead private code: a second copy of the engine version lookup in ubunye plan (it now reads ubunye.__version__), and two attributes that were set and never read (notebook()'s task object, SklearnModel._last_metrics).

[0.9.0] (2026-11-10)

Added

• Job health and upstream checks, from run records (ADR 013). ubunye runs health shows, per task, the last status, when it last succeeded, runs in a row that failed, and runs left running past --stuck-after (likely dead); it exits 1 when a task is failing, stuck, or older than --max-age, so it can be a scheduled check. CONFIG.requires: [{task: uc/pkg/t, variables: [month]}] refuses to start a task until the upstream succeeded for the same batch, with an optional wait; ubunye runs wait-for is the same check as a command. Both read only the run record store (any store, ADR 011), so jobs that share no disk can check each other; no platform plugin. Also ubunye.api.job_health, ubunye.api.wait_for_upstream and the MCP tools health and upstream. A task with no requires runs, and hashes, exactly as before.
• The run record names the model (ADR 012). A ModelTransform train puts the model, the version it registered, the stage it reached and the gate result on the record (model_runs): passed, or blocked with the gate, metric, value and threshold. A predict puts the version and stage it used. A blocked promotion still fails the run, and the failed run's record now says why; registry.on_blocked: record keeps the run going instead. ubunye lineage trace prints a MODELS section; lineage show includes the entries. The lineage conformance suite checks every store keeps them.
• Scored rows can name their model. registry.stamp_columns: true adds model_version and model_stage columns to a ModelTransform predict, the same on pandas and Spark. Off by default.

• Features are checked at predict (ADR 012). A frame without a feature the model version declares is logged and recorded before the model is called, and a model that then fails gets a note naming the missing columns. registry.check_features: true refuses the frame with FeatureContractError; false skips the check. A model that declares no features works as before.

• Model registry backends are plugins, found by name (ADR 010). The ubunye.registry_backends entry points existed but nothing used them, so a third party registry could be installed and never reached. Now ubunye models * takes --backend / -b, a model transform takes registry.backend, and ModelRegistry(store, backend=...) and ubunye.models.resolve_registry() exist. The order is: the name asked for, else UBUNYE_REGISTRY_BACKEND, else filesystem. With neither set, nothing changes. A backend implements six methods; rollback, archive, compare, version numbers, stored promotion gates and one production version at a time are done by Ubunye on top of them, for every backend.

• --backend mlflow now uses the MLflow Model Registry (ADR 014). Before, the mlflow backend kept models in a folder and only logged a best effort MLflow run. Now each version is a registered model version ({use_case}.{model_name}, or the UBUNYE_MLFLOW_MODEL_NAME template), the saved model is the version's artifact, and the stage, metrics, metadata, gates and lineage run id are version tags. staging and production are MLflow aliases, so models:/<name>@production works. The MLflow stage API is never called, so the backend fits Unity Catalog. MLflow is found through MLFLOW_TRACKING_URI / MLFLOW_REGISTRY_URI; --store names the experiment. The old class, MLflowRegistryBackend, still works from Python and warns that it is deprecated.
• ubunye.testing.registry_conformance, the suite a registry backend runs to prove itself. The filesystem and MLflow backends pass it, and so does a third party one installed through a real entry point.
• Run records can live in a shared store (ADR 011). The store is a plugin in ubunye.lineage_backends, chosen by --lineage-backend (on run, test run, lineage *, gate, mcp, sync lineage), lineage_backend= (on run_task, run_pipeline, notebook), LineageRecorder(backend=), then the UBUNYE_LINEAGE_BACKEND variable, then filesystem. With --lineage-backend delta --lineage-dir main.ops.ubunye_runs, jobs that share no disk write their receipts to one Delta table and can read each other's. Nothing changes if you set nothing: records stay in <usecase_dir>/.ubunye/lineage.
• ubunye.testing.lineage_conformance: the tests a run record store must pass (every field read back equal, newest first, limit, status, missing run id, tasks kept apart). The filesystem store passes it in the unit tier, Delta on live Spark.
• RunRecordStore in ubunye.interfaces: the four methods a store provides.

Fixed

• CLI docs and help say what the commands do (F-094). docs/cli.md said python -m ubunye runs the same CLI; it is a separate runner for spark-submit (--task-dir, --mode, --dt), and the page now says so. The version example shows 0.9.0, the middle level is called a pipeline where the page still said package, and ubunye config --help says it validates (it never showed anything).

• export databricks defaults are current (F-095). With no wheel_dbfs_path the job named the wheel ubunye_engine-0.1.0; it now names the installed version's wheel. The default job cluster runtime moves from 13.3.x-scala2.12 (Spark 3.4, past end of support, older than the pyspark>=3.5 the engine needs) to 17.3.x-scala2.13 (Databricks Runtime 17.3 LTS). Set spark_version or wheel_dbfs_path under ORCHESTRATION.databricks to choose your own. export airflow and export databricks help now says --profile defaults to prod, unlike run, which defaults to DEV.

• deploy dockerfile --out makes its folder, and takes a folder (F-093). The documented --out build/Dockerfile failed with FileNotFoundError when build/ did not exist, and --out build failed with IsADirectoryError. Missing folders are now made, and a folder gets <folder>/Dockerfile.

• prove options with fixed values reject a wrong value cleanly (F-092). prove skip --status nonsense and prove observe --cost-basis bogus printed a raw ValueError traceback, and prove observe --kind bogus was kept silently. Each is now a usage error (exit 2) that lists the valid choices.

• models demote can no longer move a version up past its gates (F-091). ubunye models demote --to production put a version in production without checking its promotion gates, and without the promotion_forced mark. Demote now only moves a version down: --to takes development, staging or archived, and staging only from production. The library refuses the same moves (ModelRegistry.demote and every built in registry backend raise RegistryError), and the registry conformance suite checks it for every backend. models promote --to now takes only staging or production, as its help said. A wrong --to is a usage error (exit 2) that lists the choices.

• Rerun safety reads the run record store you chose (ADR 008, ADR 011). With --lineage-backend delta (or any store other than the filesystem), a run that took over a dead run's lease looked for its record in local files, found none, and could take back the appends of a run that had in fact recorded success. It now reads and marks the record through the chosen store, and refuses if the store cannot be read.

• A Delta append of a batch lands once, and a rerun can correct it (F-071). A run killed or failed after its Delta commit was appended again by the rerun, and --rerun appended a finished batch twice. A Delta append (path or table, through the delta, s3, hive or unity writer) of a named batch is now Delta's idempotent write, keyed by task, output and batch: a retry commits nothing, and says so if the table did not change. Set replace_where on the append (for example "month = '{{ dt }}'") and each run replaces that batch's rows in one commit instead, so a retry or --rerun writes the corrected batch. Behaviour change: replace_where on an append was ignored before. See ADR 008. On Spark 3.5 the replace of a table batch uses writeTo(table).overwrite(...), since saveAsTable cannot replace rows there.

• A JDBC append with replace_where lands its batch once (F-072). Spark's JDBC writer commits each partition on its own, so a failed run left part or all of the batch and the rerun inserted it again. With replace_where the batch is written to a staging table, then one database transaction deletes the batch's rows and inserts the staged ones; the staging table is dropped. A JDBC append without it is unchanged.

• The Databricks examples run on serverless again. Four example configs set spark.sql.sources.partitionOverwriteMode, which serverless refuses. Since the engine stopped ignoring refused keys, the scheduled jhb_weather job failed every day from 2026-09-26. The key is removed; serverless never applied it, so the writes behave as before.

Changed

• -p/--pipeline replaces --package. The middle level of use case, pipeline, task is called a pipeline everywhere now. Every CLI command that took -p/--package takes -p/--pipeline ("Pipeline name."). --package still works until 1.0: it is hidden from --help and prints one warning to stderr. Giving both is an error. In Python, run_pipeline() and job_health() take pipeline=; package= still works with a DeprecationWarning, and both is a TypeError. The positional order of run_pipeline() is unchanged. The MCP health tool takes pipeline and still accepts package. Run records keep their package field.
• The Delta lineage store keeps the whole run record. It used to store a LineageRecord, which dropped steps, hashes, timings, expectations and the environment. It now writes the record as JSON plus index columns, only appends, and needs Spark: a record it cannot write goes to the fallback manifest with a warning, never quietly to local files. DeltaLineageBackend.search() returns run records; the old search is search_records(), and search(task=, usecase=, pipeline=) still reaches it with a DeprecationWarning.
• LineageRecorder(store=...) is now LineageRecorder(backend=...); store= still works, with a DeprecationWarning (store="s3" keeps the stub, as before). A failed record write is now logged instead of silently dropped.
• ubunye sync lineage replays whole run records, the last one kept for each run, into the store named by --lineage-backend and --lineage-dir.
• LineageStore.search() takes limit.
• Clearer docs diagrams. Project Structure zooms from a use case into a pipeline (its tasks run in order) and into a task (config.yaml and transformations.py), and colour-matches the folder path to the -u, -p and -t flags. The home page shows a task written once, run unchanged on a laptop, a cloud or Databricks, each run leaving a record that ubunye prove compares. Connectors shows inputs:, your transform and outputs: with the names you write in YAML, and which also work on pandas. They keep the look of the original slides (dark stage, stacked cards, logos) and are drawn as SVG by the scripts in scripts/diagrams/.
• More diagrams, and one on the README. New pictures for the parts of config.yaml (Config Overview), what expectations do with bad rows (Expectations), the run receipt and ubunye gate (Gate Pull Requests, ADR 006) and the model stages (Model Registry). The README shows the home diagram.
• Diagrams for the rest of the key ideas. Rerun safety (CLI, ADR 008), profiles (Engine and Profiles), how a backend is chosen (Execution Backends, ADR 003), ubunye export (Orchestration), agents over MCP, and the RAG and feature store guides, whose text arrows become pictures.
• The Databricks example notebooks use the notebook API. They now open a context with ubunye.notebook() and run ctx.read(), ctx.transform() and ctx.write() in separate cells, so you can see the data between steps. Catalog and schema come from the widgets on their own (no os.environ), and the notebooks install ubunye-engine==0.8.0.
• The registry backend is no longer picked from the host. The private detect_registry_backend() returned mlflow on Databricks; it now follows the ADR 010 order and is deprecated. Set UBUNYE_REGISTRY_BACKEND=mlflow to keep MLflow.
• FilesystemRegistryBackend.register keeps the metadata it is given (it was returned once and dropped), and stores promotion_gates with the model.

[0.8.0] (2026-09-30)

The hardening release. Ubunye was tested against real use: newcomers given only the public docs, real open data (Olist, WFP food prices), a scale ladder to 50 million rows, awkward data of ten shapes, and seven environments. Every fix below has a test that failed before it, and every engine fix was reviewed by an adversarial skeptic. The same three workloads (C01, R1, R2) give identical digests on pandas, Spark, AWS Glue, Azure Container Apps, Databricks, GCP Dataproc and Kubernetes (42 of 42 checks).

Upgrade notes (read these first)

• Digests change for map columns (maps are now unordered: entries sorted by key, null values kept) and for text holding control characters such as U+000B (the escape is written in upper case, as Spark writes it). A ubunye gate against a 0.7 record may report these columns as changed once.
• Delta reads are pinned to the version seen when the input is read, so every output of a run sees one snapshot.
• New refusals, each before anything is written: a Spark task that overwrites a folder it reads; a lease or finished note that exists but cannot be read; charset names Spark 4 refuses (it accepts UTF-8, ISO-8859-1, US-ASCII, UTF-16, UTF-16LE, UTF-16BE and UTF-32); column names that differ only by case, on file reads and writes.
• Input contracts name timestamps by their real type (timestamp or timestamp_ntz), as Spark 3.4 and later do.
• New settings: UBUNYE_HASH_WORKERS, UBUNYE_SOURCE_VERSION_TIMEOUT, UBUNYE_LLM_PROGRESS.
• Recording a run costs 1.9 times the plain job on Spark and 6.4 times on pandas at 5 million rows (it was 2.5 and 9.8 times). The 1.5 times target is not met yet.

It also carries the first fixes from real use. Two newcomers were given only pip install, the public docs and a Kaggle dataset each (Olist e-commerce, 9 tables; Amazon Fine Food Reviews with a local LLM labelling step), and logged every place they got stuck.

Changed

• A Delta read is pinned to one version for the whole run (F-046, skeptic review). A lazy Delta frame read the table's latest version on every action, so with another job appending, two outputs of one input could count 10 and 13 rows. A Delta read that names no version (format: delta, or s3 with file_format: delta) now carries versionAsOf for the version the table is at when it is read, so the transform, every output and the run record see one snapshot. A version in the config (version_as_of, timestamp_as_of, or versionAsOf / timestampAsOf in options, any case) is left as written. It costs one DESCRIBE HISTORY per read (about 0.16 s on the dev box); if that fails, the read is unpinned, as before.
• The run record says whether an input's hash is of what the run read (F-046). On Spark an input is hashed at task end by reading its source again; if another job changed it in between, the hash described a later state and nothing said so. Each input now records its source version when it is read and again after its hash (source_version, source_version_at_hash): a Delta table's version, or a file input's files with sizes and times, taken without reading data. When they differ, source_changed: true and source_note say the hash is of the later state; lineage trace prints the note, lineage compare calls that hash unknown, and ubunye gate warns instead of calling a changed output nondeterministic. The hash and its default are unchanged; old records load as before. After review: without content tags (etags, which S3A and ABFS give) "unchanged" means names, sizes and times only, and the note says so; a failed listing is "not known", never missing files; OPTIMIZE and other commits that change no rows are not a change; no version is taken when inputs are not hashed; a few files are asked one by one (0.005 s for one file of a 10,000 file folder, was 2.6 s); and taking a version stops after UBUNYE_SOURCE_VERSION_TIMEOUT seconds (30).
• The run record's hash treats a map as unordered, and writes its null values (ADR 006). Every map's entries are sorted by key before the row's line is hashed, on Spark (in the expression, only for columns that hold a map) and on pandas, recursively. A null map value is written as null, as Spark's to_json writes it; the pandas side left it out, so a map column with a null value hashed differently on the two engines. Behaviour change: digests of tables with map columns change. Tables without maps keep their digests.
• On pandas, --lineage hashes on several cores (F-038). A table of 500,000 rows or more is hashed by helper processes, one per core, at most 4 by default (UBUNYE_HASH_WORKERS sets the cap; 1 turns it off). Every row is still hashed and the digest is the same: property tests hold the helpers to the one process path and to the row at a time reference over generated tables of every kind. If a helper fails, the table is hashed in the calling process. E-06 job at 5,000,000 rows on the dev box (4 helpers, median of 3): the record's hashing took 6.0 s (was 17.2 s), the whole --lineage run 10.5 s (was 21.8 s), against 4.7 s for the plain job. Each helper holds about 105 to 125 MB resident while it works; see Deploying anywhere for containers. Not yet the 1.5 times bound E-06 asks for: the SHA-256 of each row is still one Python call. Made safe after review: helpers are stopped on Ctrl+C and at a deadline, stdin and stdout are served at once (a program that echoes cannot deadlock), no helpers in a frozen app or when sys.executable is not Python, a helper never starts helpers, a helper that runs other hash code (the file changed on disk) or another pyarrow gives no answer, the cap never exceeds the usable cores (CPU affinity, cgroup v2 quota) and is shared by concurrent hashes in one process, and a fallback prints nothing (one debug log line) and leaks no pipe.
• A Spark run record's hash takes about 45% less time (F-041). The per row SHA-256 lanes are summed as 32 bit halves in long instead of as decimals; the digest is the same. A table past about 2.1 billion rows falls back to the decimal sums.
• On Spark, a task that overwrites the folder it reads is refused before it starts (F-047). Spark deleted the input's files and then failed reading them, so the run failed and the source was gone. ubunye validate --backend spark reports it too. Write to a new path, or use Delta, whose overwrite reads a fixed snapshot. pandas runs are not affected.
• A Spark session the engine creates works in UTC (spark.sql.session.timeZone), like the pandas backend, unless the task sets the key (ADR 007). Before, it took the machine's zone: on a laptop in Johannesburg, truncating 2024-01-02 10:15 UTC to a day gave 2024-01-01 22:00 UTC on Spark and 2024-01-02 00:00 UTC on pandas and on a UTC cloud. Found by the proving ground's first workload (F-021). Tasks that mean local days set the key; a session the engine did not start is never changed.
• The run record says which time zone the run cut time in (time_zone), from the backend. ubunye prove report treats the same code in two zones as two different runs and names the zones as the reason.
• --lineage costs about a third of what it did on pandas (F-014). A 1,000,000 row task with one input and two outputs took 1.6 s without --lineage and 11 to 13 s with it; it now takes 4.4 to 4.8 s. The rows-v1 hash builds each row's canonical line with Arrow compute, hashes it straight from Arrow's buffer and sums the lanes with numpy, instead of building every line in Python: 1,000,000 rows of four columns hash in 1.5 s (was 2.6 s), of eight mixed columns in 2.8 s (was 14 s). It runs in the calling thread and changes no interpreter setting. A frame written to two outputs is hashed once. Values Arrow's text cannot be proven to match (a string with a control character, a double outside 0.001 to 10,000,000, NaN, a timestamp before year 1000, nanoseconds, decimals, binary, nested types) are still written by the Python path, and a string column holding invalid UTF-8 still records its UnicodeDecodeError and no digest. Every digest that was recorded before is unchanged: property tests hold the new path to the row at a time reference, and the C01 and R1 golden digests did not move. Behaviour change: a timestamp column with a zone east of UTC holding instants near 9999-12-31 used to record OverflowError and no digest; it now gets a digest, the same as the same instants in UTC.
• The run record says how long each hash took (hash_seconds on every input and output, hashSeconds in the OpenLineage ubunye_hash facet, and hashed in in ubunye lineage trace). The hash runs after the writes and was in no timing. A step whose frame was already hashed for another step (one frame written to two outputs) says so: hash_reused_from names that step (output:<name>) and hash_seconds is 0.

Fixed

• A failed expectation's hint names only what the rule can do (F-057). Every failure said "change the rule's severity to quarantine or warn", but a reconcile, unique, row_count and columns cannot quarantine (the config refuses it). The hint now depends on what broke: quarantine or warn for a rule on rows, a share (max_lost: "1%") or warn for a reconcile, warn for the others, and "the source probably changed" when max_quarantine_rate is broken.
• ubunye prove observe -t a -t b observes every task (F-056). It took one -t and click kept the last of several, silently: one observation, of the last task, under the name meant for all. Several tasks now give one observation each, named <workload>-<task> (as ubunye-infra's proving/observe.sh names them); one task keeps the workload's name. A task given twice is refused, and so are several tasks with --record or --run-id, which name one run.
• ubunye run stops on a broken expectation without a traceback (F-055). An expectation that fails, an input contract or a reconcile, is the engine doing its job: nothing was written and the message names the rule. The CLI printed that message, then about 60 lines of traceback through the engine, which read as a crash. It now prints [ERROR] Run stopped for <task>: <message> and exits with code 1, as a refused run does. Any other error still shows its traceback. Found writing Tutorial 2 (Olist). After review, only an error that carries rule results (a verdict on the data) stops quietly: an ExpectationError about the engine (narwhals not installed, a reconcile given no inputs) keeps its traceback.
• A failed run's record says why it failed (F-054). The record has an error field; OpenLineage sends it as the failed run's errorMessage, and ubunye prove shows it as the reason. Nothing filled it: every failed run said status: error, error: null. It now holds the error's type and message (ExpectationError: An input broke its expectations...), with the values of secret-looking variables masked, as step locations are. lineage trace prints it (and --json carries it). Found by the Olist example (R2).
• A failed run's error keeps no secret (F-054, skeptic review). The first fix masked only secret-looking --var values word for word; the skeptic's probe found the record, lineage trace, OpenLineage and ubunye prove still kept a password from an environment variable in a JDBC URL, user:password@ in a URL, ?api_key=, Authorization: Bearer, a resolved secret:// value, and a --var secret URL-encoded, in base64 or broken across a line. One function now masks all of them, ubunye.core.secrets.mask_text, and the REST connector's log and error masking (F-050) calls it too, so there is one implementation. The text is masked before any monitor receives it. Values shorter than 4 characters are not masked word for word; see Secrets.
• A failed run's error is kept to about 4 KB (F-054, skeptic review). A transform that raised a 5 MB message wrote a 5 MB run record and would have sent 5 MB to OpenLineage. The record (and OpenLineage's errorMessage) keeps the first 4,096 characters and says ... (N more characters); masking happens before the cut.
• A quarantined row's reasons read the same on pandas and Spark (F-052). On pandas, a row that broke only the first of two quarantine rules got _ubunye_failed_rules = price_between, (a stray comma); Spark wrote price_between. So the quarantine output, and the run's digest, differed between the two engines. Narwhals' concat_str(ignore_nulls=True) on pandas writes a separator after every present value except the last one given. The reasons are now built without it. Behaviour change on pandas only, and only for rows that did not break the last quarantine rule: their reasons lose the trailing comma. Found by the Olist example (R2).
• A Spark append to a folder lands once, however the run ends (F-070). Spark appends were not claimed (ADR 008), so on live Spark a run killed after its append was appended again by the rerun, a run that failed after its append left it, and --rerun appended a finished batch a second time. Spark cannot say its file names before they land (it puts a job UUID of its own in each name), so a path append now goes to a hidden folder inside the output that the run's lease names before it exists (_ubunye-<id>, skipped by Spark, pyarrow, pandas and Ubunye's readers). Once Spark has finished, the files there are claimed in the lease, then moved up in one rename each. A failed or killed run has exactly those files and that folder taken back, and --rerun replaces the batch, as on pandas. The output folder is never listed to decide what is a run's, so a file another job put there is never removed. Applies to parquet, csv, json, orc, avro and text on a path Spark resolves to the local file system (a laptop or a shared disk). Delta, catalog tables, JDBC and object storage (s3a://, abfss://, gs://, dbfs:/) are written as before and named after a crash (F-071, F-072, F-073), and the log says in one line which output was not claimed and why. Live Spark 4.2: 9 of 10 new integration cases fail on the old code, 10 of 10 pass. After the skeptic review: the staging folder moved from beside the output to inside it (an output on another disk, a long folder name, and globs of the parent folder broke); claiming 12,000 files takes 14.5 s (was 76 s, each file re-read the growing lease; the pandas backend gains the same); a Spark append refused or killed before it claimed anything is no longer reported as maybe holding part of the batch; Ctrl+C cancels the Spark write's job group before its folder is removed.
• The awkward-data readers pass mypy (E-09 CI). spark_json.raw_tree used the json module's private WHITESPACE and scanstring; it now uses its own JSON whitespace pattern and the public decoder. The Java-style UTF-8 error handler has the signature codecs.register_error expects. No change in what is read.
• A CSV with a NUL character reads on Python 3.10 too (F-062 follow-up, E-09 CI). Python 3.10's csv module stops at NUL ("line contains NUL" when reading, "need to escape" when writing), where 3.11 and later take it. The pandas CSV reader uses the module for the header, ragged rows and the port of Spark's splitter, so such a file failed on 3.10 only. A private use character stands in for NUL there and is swapped back; the read and its digest are the same on 3.10 and 3.13.
• JSON +INF, -INF and +Infinity read as doubles, as on Spark (F-089, skeptic review). Spark's JSON parser (Jackson, with allowNonNumericNumbers) reads them as infinities; Python's reads only NaN, Infinity and -Infinity, so the pandas read stopped. A line that fails to parse and holds one of them is now read with them spelled the Python way. INF, +NaN and -NaN stay refused, as Jackson refuses them.
• A JSON record that repeats a key is refused, as Spark refuses it (F-088, skeptic review). {"a":1,"a":"x"} read on pandas as one column holding the last value (x), silently. Spark infers one column per occurrence (a:bigint, a:string) and refuses the read ("Found duplicate column(s)"). The pandas reader now refuses it too, naming the key, at any depth.
• A time with a fraction of a second in a daylight saving gap before 1970 moves as Java moves it (F-063 follow-up, skeptic review). pyarrow judged such a time by the next whole second, so 1950-04-30 02:59:59.5 in New York (in the gap) was read an hour early. The whole second now decides and the fraction is added back. Against Java's ZonedDateTime.of at every transition from 1950 to 2037 in 21 zones, the only differences left are where pyarrow's time zone data and Java's differ.
• With nullValue set, an empty CSV field is still null (F-087, skeptic review). univocity hands Spark the nullValue text for an empty field, and Spark reads that as null, so 1,,NA,2 with nullValue: "NA" is an int column 1, null, null, 2. The pandas reader took only NA as null, so the empty field made the column text. A quoted empty field ("") is also null on pandas; Spark keeps it as an empty string.
• Names that differ only by case: refused only where Spark refuses (F-069 follow-up, skeptic review). The refusal sat in the frame-to-Arrow step, so the REST sink and the run record's hash refused such a frame too (the hash then recorded no digest). It now applies to file reads and writes only. And a parquet read with a schema naming a field that matches two file columns (id and ID) read one of them silently; Spark stops ("Found duplicate field(s)"), and so does the pandas backend now.
• JSON records whose names differ only by case merge as on Spark (F-064 follow-up, skeptic review). {"Id":1} then {"id":2} gave two columns on pandas, which the engine then refused (F-069). Spark (with spark.sql.caseSensitive false) merges two structs of the same shape whose names differ only by case into one, with the names seen first (findTypeForComplex), at the top level and nested, widening each field's type; and then reads id as a different name, so that record's value is null. The pandas reader now does the same. Checked against Spark 4.2's own classes on 4,007 generated record sets: all match but one duplicate-key case.
• CSV inferSchema numbers follow Spark's rules exactly, in row order (F-067 follow-up, skeptic review). Checked against Spark 4.2's own CSVInferSchema and UnivocityParser over 12,506 generated columns: every case matches except the looser date and time forms (F-077). Three changes: a number BigDecimal reads with scale 0 is a whole-number decimal (5., 1.5E1, 0E0 are decimal, as 1E5 and 1.5 are double); digits in any script (１２３, ٤٥) read as numbers, as Integer.parseInt reads them; and a decimal's precision depends on the order of the values, as Spark folds them (a bigint before the first decimal widens it to 20 digits, one after it does not). A value Java cannot read as a double in a double column is null, as Spark's PERMISSIVE mode reads it. Performance guard: the plain CSV read is 33% slower than before the port (40 to 54 ms for 60,000 rows), inside the guard's 30% plus 5 ms.
• Broken UTF-8 is replaced exactly as Java replaces it (F-086, E-09). An encoded surrogate (ED A0 80, which some tools write for half an emoji) became three U+FFFD on pandas and one on live Spark 3.5 and 4, so the text differed. The pandas CSV reader now decodes with Java's rule; over 40,000 random byte strings it matches Java 21's new String(bytes, UTF_8) exactly (UTF-8, UTF-16LE, UTF-16BE, US-ASCII).
• encoding takes the names Spark 4 takes (F-085, E-09). Live Spark 4.2 refuses any csv or json encoding but UTF-8, ISO-8859-1, US-ASCII, UTF-16, UTF-16LE, UTF-16BE and UTF-32 (INVALID_PARAMETER_VALUE.CHARSET; cp1252, latin1 and even utf8 are refused). The pandas backend read them, so a task could pass on a laptop and fail on Spark 4. It now refuses them too, before reading, and says which names work. Behaviour change for tasks that used another name: use ISO-8859-1 for Latin-1, or convert the file. Spark 3.5 (and Spark 4 with spark.sql.legacy.javaCharsets) takes any Java name; the pandas backend follows Spark 4.
• A JSON value read into a text column keeps its source text, as Spark 4 reads it (F-084, E-09). Live Spark 4.2 keeps the exact text of a number or object that lands in a text column when it reads JSON lines (1.50 stays 1.50, 1e2 stays 1e2, { "k" : 1 } keeps its spaces and escapes); with multiLine or an encoding option it writes the value back through Jackson (1.5, 100.0, compact, upper case hex). The pandas backend did the second always; it now does what Spark 4 does in each case. Spark 3.5 writes every such value back through Jackson; the pandas backend cannot tell which Spark a task will meet, so it follows Spark 4.
• Text holding a control character hashes as on Spark (F-083, E-09). Spark's to_json (Jackson) writes a control character that has no short escape with upper case hex (\u000B); the pandas side wrote Python's lower case (\u000b), so a column holding one had another rows-v1 digest than live Spark 3.5 and 4. The pandas json writer wrote it the same way; both now write Jackson's. Behaviour change: digests of text holding such a character (U+0000 to U+001F other than \b \t \n \f \r, and only those with a hex letter, such as U+000B or U+001F) change; others do not.
• The smallest doubles and floats are written as Java writes them (F-082, E-09). Where the shortest text of a number has one digit, Java's Double.toString and Float.toString write the closest two digits instead: 4.9E-324, not 5.0E-324; 1.4E-45, not 1.0E-45. The run record's hash and the pandas json and csv writers used Python's shortest text, so a table holding such a number had a different rows-v1 digest on pandas than on live Spark 3.5 and 4. Other digests do not change.
• CSV inferSchema: a whole number past 64 bits with small ones is decimal(19,0), as live Spark types it (F-067 follow-up, E-09). The port widened it to decimal(20,0) whenever a smaller number was present; Spark does that only for a number past the int range.
• The run record hashes a folder of many small files quickly on pandas (F-076, E-09). A read of 5,000 small files keeps one Arrow chunk per file, and the hash paid about a millisecond per chunk: 10,000 rows took 1.9 s to hash, most of a --lineage run. Such a table is now put together first: 0.02 s. Same digest.
• Columns whose names differ only by case are refused on pandas, as on Spark (F-069, E-09). A parquet file with Col and col, JSON keys a and A, or a transform that returns both, ran on pandas and failed on Spark, whose default (spark.sql.caseSensitive false) refuses such data on read and on write. The pandas backend now refuses them too, before anything is written, and says why.
• An empty file or folder read with a schema is zero rows on pandas (F-068, E-09). A zero byte file, or a folder holding only _SUCCESS, read with schema: stopped the run ("Path does not exist or holds no data files"). Spark reads zero rows with that schema, and so does the pandas backend now. A missing path is still an error.
• CSV inferSchema on pandas follows Spark's rules (F-067, E-09). pyarrow chose the types, and differed from Spark on numbers written a little differently: a whole number past 64 bits became a double that lost digits (Spark: an exact decimal), 1, 2 with a space after the comma stayed int (Spark: double), +5 was a double, 1.5d text, inf a double (Spark: text), tRuE text, and timestamps with and without an offset in one column text. Types are now chosen by a port of Spark's CSVInferSchema, over every file of a folder at once (each file was inferred alone before). About 20% slower on a plain file (performance guard: 40 to 48 ms for 60,000 rows).
• Unsigned parquet columns read with Spark's types on pandas (F-066, E-09). A uint8 to uint64 column kept its unsigned type, so the schema and the run record hash differed from Spark's. They now read as Spark reads them: smallint, int, bigint and decimal(20,0), at any depth.
• JSON records with no fields are no longer lost on pandas (F-065, E-09). A file of {} records (or records holding only fields Spark drops) read as 0 rows, and the run succeeded. Spark reads one row per record, with no columns; so does the pandas backend now.
• JSON on pandas is typed by Spark's rules (F-064, E-09). The pandas reader let pyarrow infer the types. Nested fields kept the order they were first seen (Spark sorts them, so the schema hash differed), and a field that was a number in one record and text in the next, an object in one and text in the next, or a whole number past 64 bits stopped the read. The reader now uses a port of Spark's JsonInferSchema: such a field is text (an object keeps its JSON text), a long whole number is a decimal, bigint and double give double, empty names and empty objects are dropped, and a line holding an array of objects is one row per object.
• A timestamp in a daylight saving gap or fold no longer stops a pandas run (F-063, E-09). With a session zone such as America/New_York, the text 2024-11-03 01:30:00 (it happens twice) or 2024-03-10 02:30:00 (it never happens) stopped a CSV read with inferSchema, a read with a TIMESTAMP schema, and a write of a naive timestamp. The pandas backend now follows Java's rule, as Spark does: a time in a gap moves later by the gap (02:30 reads as 03:30), a time in a fold takes the earlier offset. Times outside a change cost nothing extra.
• A CSV value over 1 MB no longer stops a pandas read (F-062, E-09). pyarrow parses in 1 MB blocks and stopped at a longer row ("straddling object straddles two block boundaries"); Python's csv module, used for the header and for ragged rows, stopped at 131,072 characters. Spark has no limit (maxCharsPerColumn is -1). Such a file is now parsed again as one block, and the csv module's limit is lifted for the read.
• A CSV byte that is not valid in its encoding no longer stops a pandas read (F-061, E-09). A Windows (cp1252) export read as UTF-8 failed with "can't decode byte 0xe9". Spark decodes the Java way: such a byte becomes U+FFFD and the read goes on. The pandas backend now does the same. Files that decode cleanly are read as before, at the same cost.
• A CSV header with a repeated or blank name reads as on Spark (F-060, E-09). On the pandas backend a header such as id,amount,amount,,note stopped the read ("duplicate field names"), and a blank name became a column called "". The names are now made as Spark's makeSafeHeader makes them: a blank name (or the nullValue text) becomes _c<position>, and a name that appears more than once, ignoring case, gets its position added to each copy (a,a,,A reads as a0, a1, _c2, A3).
• On Windows, reading a run lease while its heartbeat replaces it no longer fails (F-048). Windows refuses a read for a moment while a file is being replaced. The engine then called a live run's lease unreadable, and a second run of the same batch was refused as "Run ?" instead of naming the run that holds it (11 to 20 times in 3 seconds against a fast heartbeat; now 0). Every lease read, stat, rename and removal now retries a PermissionError for up to 2 seconds through one helper. A missing lease is still missing at once. A flaky lease test is fixed the same way.
• A lease file that exists but cannot be read now refuses the run (F-048 review). A finished note held open for more than 2 seconds (a backup tool, a scanner, a file permission) used to count as no note, so the batch was appended a second time. A live run's lease that could not be read was judged by its age alone, so a run 20 minutes into a long job was taken over while alive. Now the lease, the finished note, a dead run's lease left beside it and a dead run's record are read or the run is refused, naming the file. A damaged finished note is refused too. A run whose own lease cannot be read stops after one 2 second wait, not five.
• The REST connector runs on the pandas backend (F-015). plan and run with --backend pandas refused format: rest_api ("needs spark"), so a laptop could not pull JSON from an API. The HTTP side (session, auth, pagination, retries, rate limit) is now one module shared by the reader and the writer on every backend (ubunye/plugins/rest_http.py). Only the last step is the backend's, through two new Backend methods: frame_from_records (records to a frame) and iter_records (a frame's rows as records), behind a new capability, records, which the spark, databricks and pandas backends declare and the connector now requires instead of spark. On pandas the records are typed by a port of Spark's own createDataFrame rules (ubunye/adapters/pandas_records.py), so both backends give the same columns, order, types and values, and refuse the same records (a field that is 1 in one record and 2.5 in another, or null in all of them). A map's entry order is not promised: Spark's is not stable (it changed between Java 11 and Java 21, and Spark Connect keeps the JSON's), so pandas keeps the JSON's and the parity tests compare maps as sets of entries. Checked against live Spark 4 and 3.5 in tests/integration/test_rest_api_parity.py. The writer POSTs the same payloads on both.
• The REST writer sends NaN, timestamps, dates, decimals and binary (F-051). It crashed on the first such row (requests refuses NaN and cannot encode a datetime or a Decimal), after earlier batches were posted. Now NaN and the infinities are null, an instant is ISO 8601 UTC text with Z, timestamp_ntz has no offset, a date is yyyy-mm-dd, a decimal is its exact text as a string, binary is base64, the same from Spark and pandas (Spark's naive local timestamps are made UTC first). Each batch is encoded whole before any of it is sent, so a row that cannot be sent stops the write with its row and field named and none of its batch posted. Behaviour change for Spark users whose writes used to fail.
• Security: a REST API key no longer appears in logs or errors (F-050). With api_key_query the key is in the URL, and the writer's error log, the reader's HTTPError, a ConnectionError and urllib3's DEBUG request lines all carried the full URL. Every secret the config sends (auth token, key, password, an Authorization or secret-named header, a secret-named query parameter) is now masked as *** in the connector's and urllib3's log lines while it runs, and in every error it raises (same exception class and response).
• The REST docs name the options the connector reads, and an unknown auth type is refused (F-049). The docs printed auth: {type: api_key}, cursor_field and link_field; the code reads api_key_header / api_key_query, cursor_response_key and next_key, and ignored an auth type it did not know, so the documented API key example sent no key. The docs are fixed, the old names are still read with a warning, and an unknown auth type (or api_key with both or neither of header and param) is refused by ubunye validate and before any request.
• A REST field declared double or float takes whole numbers (F-015 review). Spark's schema check refuses a Python int in a double column, so a price field of 1 and 2.5 could not be read at all, even with a schema, on either backend. The reader now turns a whole number in such a column into the same decimal number before the backend sees it, when a double holds it exactly (up to 2**53); a larger one is still refused. Behaviour change on Spark: these records used to fail.
• REST records with no fields are rows on pandas, as on Spark (F-015 review). [{}, {}, {}] gave an empty frame on pandas and three rows on Spark. A pandas frame with rows and no columns now keeps its rows on the way to Arrow too, so its run record counts them and the writer posts one {} per row.
• A REST field with a whole number past 64 bits is refused by name on pandas (F-015 review). It raised a bare OverflowError. Now it is a SourceReadError naming the field, with the fix (declare it string). Spark classic stores null there silently; pandas refuses on purpose, since a lost value should not pass.
• A plugin Spark backend keeps running the REST connector (F-015 review). The F-015 fix made rest_api require the new records feature, so a third party backend that declared spark (as on 0.7) was refused, and RestApiWriter().write with no backend (None) crashed. Now spark implies records, the connector builds and reads records the Spark way when it has a SparkSession, and the writer falls back to the frame's toLocalIterator, as before.
• The pandas backend writes a map column to JSON as Spark does (F-015 review): as an object, keeping its null values ({"m":{"a":null,"b":"x"}}). It wrote Arrow's pairs instead ({"m": [ ["a",null], ["b","x"] ]}), which Spark reads back as a different type. Maps inside lists and structs too.
• On Spark, the run record hashes the rows that were written (F-040), and a checked or recorded output is computed once (F-039, F-043). The record hashed each output at task end by computing it again, so anything that differs per computation (a current_timestamp() column, a UDF that calls a service, a changed source) gave a digest of rows never written: 0 of 3 runs matched the written files, now 3 of 3. The checks had the same flaw, and every check and the write paid for the transform again. Now each output that the record, its expectations or a second output name will act on is computed once (localCheckpoint, memory and disk) and every consumer gets that copy; it is released when the task ends. E-06 job at 5M rows: the source is read 3 times with --lineage (was 5) and 2 times with expectations (was 5). A plain run is unchanged. Each input and output in the record now says hash_basis: materialised (the rows written) or recomputed (computed again: every Spark input, and an output that could not be held, as on Spark Connect). UBUNYE_MATERIALISE_OUTPUTS=0 turns holding off. The cost is executor memory and local disk for the held outputs, and a lost executor fails the run instead of recomputing. ADR 009.
• Expectations give the same verdict on pandas and Spark when a float column holds NaN (F-045). pandas counted NaN as null and Spark did not, so not_null and between broke on different rows and a fail rule could stop a run on one backend only. NaN now counts as missing on both: not_null breaks on it, between and one_of let it pass. Behaviour change on Spark (and on Arrow-backed pandas).
• ubunye deploy k8s stops when the Job fails, not when its timeout runs out. It waited only for Complete, so a failed Job held the command for the whole --timeout (30 minutes by default). It now stops at Complete or Failed and prints the Job's log either way. When kubectl cannot read the Job three times in a row (deleted, RBAC, an expired token) it stops and prints kubectl's error (F-037).
• Expectations on a Spark output no longer need pyarrow. The one row of counts was collected through Arrow, and the images ubunye deploy dockerfile dataproc and container write have no pyarrow, so a task with expectations died on Dataproc, Kubernetes and Container Apps after its transform ran. Spark now gives the counts itself; the numbers are the same (F-036).
• A cloud run's record survives a log line cut inside its marker. Glue's CloudWatch sent UB and UNYE-RECORD-PART 9/14 ... as two lines, so a part lost its header and a run that succeeded gave back no record. The reader now joins a marker cut in two, whatever the cut point and log prefix. A record in parts is now accepted only when its SHA-256 line is read whole and matches (before, a record whose digest line was lost was taken unchecked, and reordered pieces could decode to a scrambled record), and a log holding two different records is refused (F-035).
• A backend that cannot write partition folders is refused before the run when an output sets partitionBy. The check looked for partition_by, which no config uses, so it never fired and the task failed at the write instead (F-032).
• A rerun is safe (ADR 008). A run takes a lease on its task and variables (its batch) under .ubunye/leases/. A second run of a batch that is still running is refused in one line naming the first (was: both appended, 7 of 10 pairs, F-019; or a raw OS error, F-020). A killed run is taken over: its record is marked interrupted (was: running for ever, F-013) and the append files it claimed are removed before the batch runs again (was: the batch landed twice, 16 of 16, F-011). A run that fails removes its own. Only files a backend claims before they land are ever removed (the pandas backend claims each part file; no folder is listed to guess), so another run's files are never touched (Spark claims a path append on a local disk too since F-070). Appends that cannot be claimed (JDBC, catalogs, Delta, Spark on object storage) are named in the log and the dead run's record, never deleted. UBUNYE_RUN_LEASE=off turns it off.
• A batch that finished is not appended twice by accident (F-031). A run of a batch (-dt or a --var) whose task appends, and which a run already finished, is refused in one line (was: it appended the batch again). ubunye run --rerun (rerun=True in Python) replaces the batch instead: once the new run succeeds, the files the finished run claimed are removed, so a rerun that fails loses nothing. Appends that cannot be claimed (JDBC, catalogs, Delta, Spark on object storage) are appended again, and the run says so. A run with no -dt and no --var (a snapshot job) and a task that only overwrites are never refused. ubunye run --resume (run_pipeline(..., resume=True)) finishes a pipeline that stopped half way: tasks that finished the batch are skipped, the rest run. Behaviour change: a scheduler that runs the same appending batch twice now gets exit code 1 the second time; give each run its own variable, or pass --rerun to replace.

• The pandas backend writes and reads partition folders, and does overwrite_partitions (F-012). partitionBy was refused on pandas, so the rerun safe way to write a daily batch (replace the day's partition) needed Spark. Now the same config writes the same folders as Spark 4.2 (dt=2024-01-02/, Spark's % escapes, __HIVE_DEFAULT_PARTITION__ for null, one part-00000-<uuid>.c000 file per partition, partition columns left out of the files) with every save mode, and overwrite_partitions replaces only the partitions the data fills, one staged folder swapped in at a time and put back if a swap fails. Reading a partitioned folder finds the partition columns and infers their types as Spark does (int, bigint, decimal, double, timestamp, date, text; widened across folders). A partitioned append claims every file, so ADR 008 takes back exactly those. Partition columns Spark does not read back as the same type (double, decimal, binary, time) are refused with the reason, as are the layouts Spark refuses. Checked against live Spark: 56 parity cases. E-01 with overwrite_partitions: 10 of 10 right after a kill and a rerun. Behaviour change: reading a folder now follows Spark's partition discovery. A folder holding name=value folders gains their columns, and data files directly in it, beside those folders, are dropped (with a warning naming them), as Spark drops them; the old code read those top level files. Layouts Spark refuses now fail the read where the old code read the top level files: a plain folder beside name=value folders (CONFLICTING_DIRECTORY_STRUCTURES), or folders with different partition columns. Empty (zero byte) files are skipped for a folder, a glob and a single file alike, as Spark skips them. A skeptic review then found and fixed: an old partition deleted when it could not be put back after a failed swap (now kept and named), two timestamps in a daylight saving fold merged into one folder (now refused, as Spark fails), case collisions at an outer level or on a case blind macOS disk or against existing folders (now refused everywhere), .old folders named by number (now by partition path, and the next write warns about any left behind), and a first partitioned append's root _SUCCESS left after its take back (now removed).

• ubunye prove report treats two names of one time zone as one (UTC, Etc/UTC, GMT, Zulu; other zones by their offsets over time). Databricks records Etc/UTC and was reported as a different run while its data matched exactly (F-025).

• A cloud run's record survives the log store. ubunye deploy glue --record-out failed after a successful Glue run because CloudWatch returned the record's one long JSON line cut at about 1,000 characters (and cut again wherever Glue's output buffer flushed). The entry script now prints the record's SHA-256 and the record as short numbered base64 parts that say their length; the reader puts them back by number from anywhere in the log, completes a part cut short from the lines after it, and checks the SHA-256. A record that is not provably whole is refused, never read in part. Records printed by older engines still read. Found by the proving ground (F-023).

• Replay is the recorded run, call for call. A run that sent the same request twice (two identical reviews) to a model that answered each differently replayed one answer for both, so a "free, identical rerun" changed 2 of 300 rows. The nth identical request now replays the nth recorded answer, including across complete_many's parallel calls; a run that asks more often than was recorded fails closed. Replay files from 0.7.1 still replay as before.

• A UTF-8 byte order mark at the start of a CSV file is dropped, as Spark does. The pandas backend kept it in the first column's name, so a join on that column failed. Checked against Spark 4.2 on 16 cases.
• An expectation on a missing column, or on a column of the wrong type, is one ExpectationError naming the output, rule, column and type, not a pyarrow traceback of thousands of characters.
• A variable whose name says it is a secret is masked where it is recorded. A token passed as --var token=... was written to the lineage record, sent in every OpenLineage event and printed by ubunye plan. Names made of words such as password, secret, token, auth, credential, or pairs such as api_key, are now recorded as ***; the run still uses the real value. tokenizer, author, max_token and secret_scope are not masked. The value is also masked wherever it was templated in, and a password written into a URL (user:password@, ;password=, ?token=) is masked in every recorded location. Found by experiment E-04, which also showed secret:// references and {{ env.X }} values never reach any record; an adversarial review found the URL path.
• lineage compare and lineage show take the short run id that lineage list prints. A prefix that fits more than one run is refused; gate used to take the first match silently.

Added

• A long complete_many shows its progress (F-008). A 300 call model step ran for 6 minutes with nothing on screen. Now a line goes to stderr, such as LLM anthropic/claude-haiku-4-5: 90/300 calls (30%), 1m48s, $0.0412 spent of $2, at most every 10 seconds and every 10% of the calls, whichever is rarer, and one at the end. A batch under 20 prompts or under 10 seconds writes nothing. Sequential and concurrent batches both report. Calls that got no answer are named (50 answered, 250 refused by the budget, or failed), not counted as done. The spend shows only when a dollar limit is enforced (UBUNYE_LLM_MAX_USD in a run or outside one, or the port's max_usd=; both when both are set). The line is best effort: a missing or broken stderr drops it and never stops a batch or hides a call's own error. UBUNYE_LLM_PROGRESS=0 turns it off.
• Docs: check each model answer, ask again for the bad ones (F-009). A newcomer asked for SENTIMENT | ASPECT and 20 of 300 answers missed the format. The LLM page now has a short recipe, complete_parsed: parse each answer, ask again only where the parse fails, up to a cap, with an optional reminder. It answers by position (a pandas column's index is not used), treats any error in the parse as "did not parse" and keeps it as the reason, adds the reminder to chat message prompts too, and keeps the answers already paid for when the budget refuses an ask again. It needs no new engine feature: the extra calls count against the budget and are in the run record, and inside a run they replay call for call. A test runs the recipe from the page as written.

• Input contracts: expectations can name an input, checked before the transform (F-018). Its rules run right after it is read. A new rule, columns: {price: float64, qty: int64} (with extra: allow|forbid), checks each column's type by the names the run record uses, on Spark and pandas alike, from the schema alone. Types match exactly (int32 is not int64; list several to accept them); nulls are not part of the type. A mismatch stops the run before the transform, naming each column with the type expected and found. Before, a source that wrote price as text made pandas compute "11.011.0" and the run succeeded; the E-05 cases (column dropped, renamed, retyped) now stop with, for example, "price: expected float64, found string". Results carry side: input in the record. ubunye validate refuses a name that is both an input and an output, an input with a quarantine or reconcile, and an unknown type name (with the right one for double, long and friends). columns also works on an output.

• reconcile: an output can declare that nothing is lost on the way from an input (F-017). Under CONFIG.expectations.<output>, reconcile: [{input: orders, rows: {max_lost: 0}, sum: {column: amount, tolerance: 0.01}}]. Bounds are a number or a share ("1%"); max_gained catches a join that fans out. It is checked with the other expectations, before anything is written, and reported like a rule (rows_from_orders, with what was found in a new detail field) in the error, the run record, ubunye lineage show and ubunye gate. Rows sent to quarantine count as carried over. Before, an inner join that dropped 100 of 1,000 orders succeeded; the E-03 task now stops with "1000 rows read from orders, 900 reached enriched: 100 lost". It costs one pass over each reconciled input (on Spark, a second read of it). ubunye validate refuses a reconcile that names no real input. Engine.write_outputs takes inputs= for it; the notebook passes what it read.

• Reconcile and input contracts, after the skeptic's review (F-017, F-018). The inputs are counted before the transform runs: a pandas transform that dropped rows from its input in place used to make the record say "8 read, 0 lost". Sums are exact: integers no longer wrap past 263 (3 x 262 came out negative) and decimals are no longer compared as floats (0.01 on a 1.2e19 total passed at tolerance 0); on Spark an integer sum is taken as decimal(38,0). Equal infinite totals match. A reconcile item takes an optional name, so a warning at 1% and a stop at 5% can check one input. ubunye validate gives a clear message for a blank tolerance, a list as a bound, a number as a type, and NaN or infinite tolerances (it crashed with a TypeError or accepted them). Nested type names are checked part by part (list<banana> was accepted), and every kind a frame can report (uint8, null, mixed, time64[us]) can be declared. A contract names a timestamp by what it is, timestamp with a zone and timestamp_ntz without, on pandas as on Spark 3.4+; the run record is unchanged. The notebook checks the contract on, and reconciles with, the frames passed to transform(), and tells you to call read() or transform() when it has none. Input contract results are on the input datasets in OpenLineage.

• ubunye deploy runs several tasks in one launch (glue, dataproc, k8s, container-apps, emr-serverless): repeat -t, as in -t clean -t monitor. They run in that order in the one job, so a task can read what the one before it wrote to the job's own disk; the first that fails stops the rest and fails the deploy, as does a task the job never reached. --record-out glue.json then writes one record per task (glue.clean.json, glue.monitor.json), also when the job failed: the records the failed job printed are read, written and reported task by task before the deploy exits 1. A repeated task or a name with /, \ or , is refused before anything starts; several tasks get a job name and bundle folder with a short hash of the list. Images built by an older deploy dockerfile must be rebuilt to run several tasks. One task works as before. Before, a container job ran one task and its disk went with it, so a two step pipeline could not run on Kubernetes or Container Apps without shared storage (F-034).

• A real-world example: a staple food price monitor for African markets (examples/real-world/food_prices_africa). WFP market prices, African retail, one price per kg, a monthly price per country and food, and alerts where a staple rose by half on a year before across at least three markets (on 2025-2026 data: Mali paddy rice +144%, Ethiopia groundnuts +122%, Chad wheat flour +78% across 54 markets). A small WFP sample ships with it, so the first run needs no account and no Java; the two steps give the same rows on pandas and Spark (tested, golden digests).

• A second real-world example: Olist sales, nine tables to an order fact table (examples/real-world/olist_ecommerce), and Tutorial 2 (docs/tutorials/02-olist-multi-table.md). Three steps: clean (input contracts on the nine raw CSV files, typed outputs, categories in English, one point per zip prefix, bad items, payments and reviews quarantined), orders_fact (one row per order, reconciled with its inputs: every order in, one row out, payments and prices add up) and monthly (sales and on-time rate per seller, sales per category). A made up sample shaped like the real files ships with it (the Olist data is CC BY-NC-SA 4.0, so it is downloaded at run time by scripts/fetch_data.sh, never committed). The tutorial breaks the pipeline on purpose to show a contract, a quarantine, a reconcile and a warning each catching its problem. Golden digests on pandas in the unit tier, the same on Spark in the integration tier. The portable transforms guide gains "days between two dates" (F-053). Each step checks for Narwhals 2.9 or later first and says so (the example needs narwhals>=2.9 (Expr.floor)), instead of failing inside the transform. The example needs the release after 0.7.1; until then the tutorial and README install Ubunye from the cloned repository (pip install -e "../../..[pandas]").

• A guide to writing portable transforms (docs/guides/portable-transforms.md): the places the same Narwhals code gave different numbers on pandas and Spark, each found on real data, each with a fix (rounding modes, float sums, a lost cast, str.replace, casting text, time zones, null group keys).

• ubunye prove: the proving ground. Collect one workload's run records from many environments (observe, from a stored run or a record file such as a cloud run's artifact), record the ones that did not run and why (skip), and compare them all with a reference (report): execute, identity (code_hash), inputs, data (rows-v1), schema and rows, each PASS, FAIL, PARTIAL, NOT RECORDED, NOT RUN or UNSUPPORTED, as JSON and a generated table. An expected environment without evidence is NOT RUN, never a pass; a cost says whether it is billed, a provider's estimate or Ubunye's. ubunye.proving has the same in Python. See docs/proving-ground.

• A warning for CSV files that double their quotes (as pandas and Excel write them) read with Spark's default backslash escape. Such files split into wrong rows on Spark and so on pandas too; both newcomers hit it. ubunye plan and the pandas reader now say to set escape: '"'. What is read is unchanged.

• Docs: transform.params (dropped by mistake in 0.7.1), CSV quotes and line breaks on the connector page, replay call for call, and gating on a number (an expectation on a metrics output).
• A docs map in the README, so on PyPI too (F-010). A newcomer guessed docs addresses such as /expectations/ and got 404s: the README linked only the docs home. It now links the main pages directly (install, quickstart, config, expectations, the run record, backends, CLI, Python API, language model steps, examples, connectors, deployment, errors, changelog). A unit test maps every docs link in the README to its page in docs/ and the mkdocs.yml nav, so a renamed page fails the tests instead of turning into a 404.
• Docs: big merges on the pandas backend (F-042). The pandas backend hands a transform Arrow backed frames, and on pandas 3 a merge on Arrow whole number keys is about 2.8 times slower than on NumPy keys (memory goes the other way: 9.1 GB against 12.2 GB peak at 50M rows). The pandas backend page now says so and shows a three line numpy_keys helper for a transform to convert join keys with no nulls (0.97 s merge becomes 0.33 s at 5M rows). The engine keeps Arrow types on purpose: a whole number column with a null must stay whole numbers. A unit test runs the snippet from the page.

Changed

• The budget hint for a model with no price says how to declare a free local model: price=(0, 0).

[0.7.1] (2026-09-27)

Run records you can trust when a run has more than one task, and a registered model that points at the run that trained it.

Fixed

• Each task in a run has its own run id. ubunye run -t a -t b (and --all) and run_pipeline gave every task the same run id. Each task now gets its own, so each has one run record and one OpenLineage run.
• The lineage store never hands back another task's record. It cached records by run id alone, so with two records under one id (as 0.7.0 multi-task runs wrote) it could return task A's record when asked for task B's. It now caches by file.
• A model registered inside a run records that run's id. ModelRegistry.register fills in lineage_run_id from the running task when it is not given, so a model trained in transformations.py links to its run record. Before, it was always empty.
• ModelTransform reads its settings as config.yaml writes them, under params:, as well as flat.

Deprecated

• CONFIG.transform.type in a task's config.yaml. A task always runs its transformations.py; any other type (model, or a plugin) was dropped without a word. ubunye run now warns (FutureWarning), and ubunye plan and ubunye validate report it. It becomes an error in 0.8.0. type: noop is unchanged. The docs showed type: model as working; they now show training in transformations.py.

Changed

• ubunye validate --json gives each task a warnings list.
• ubunye plan checks for transformations.py whatever the transform type says.

[0.7.0] (2026-09-25)

One release where three were planned (0.6, 0.7 and 0.8). The same task folder runs on Spark or on pandas (no Java) and leaves the same run record hash for the same data; that record is now proof: code, environment, input hashes, timings and expectations, sent as OpenLineage and OpenTelemetry, checked in CI by ubunye gate, and deployable in one command to AWS Glue, GCP Dataproc Serverless, Kubernetes, Azure Container Apps and EMR Serverless, or exported to Airflow 2 and 3 and Spark Declarative Pipelines. The same data hashes were written on Glue, Dataproc, Kubernetes (kind), Azure Container Apps and Spark Declarative Pipelines; EMR Serverless is built but not yet run (the AWS free plan blocks EMR). A task can call a language model through one port that the engine sees: every call in the record, the bill capped before it is sent and priced before the run, the whole run replayable for nothing, and the cost exported as FOCUS 1.4 rows. Agents drive it all through ubunye mcp. Python 3.10 to 3.13 on Linux, Windows and macOS.

Not in this release, on purpose: a positional ubunye run ./task (four flags stay the interface), a DuckDB backend (it failed SQL parity with Spark), a tool port and divergence ledger for agents, Dagster and Prefect exporters, and a benchmark suite (a separate project, later).

Added

• ubunye mcp: the engine as an MCP server for agents. Tools tasks, doctor, plan, runs, record, gate and focus only read; run exists only with --allow-run. Tasks are named, never paths, and must sit under -d. An agent's run replays its model calls unless the server starts with --allow-live-llm, and the run's limits apply. A task's prints go to stderr, so stdout stays the protocol's. A failed run is reported with its record, not raised. New mcp extra (the MCP Python SDK 2.x).
• The gate covers model calls; a model step replays in CI. ubunye gate reports a run's model calls (count, replayed, tokens, list-price cost) and gains --require-replay (fail if a call went live) and --max-llm-cost-increase (fail if the list price of the tokens, or the tokens for an unpriced model, grew by more than a share; measured on replayed runs too). The gate Action's new llm-mode input defaults to replay. A new example, examples/production/llm_replay, labels reviews through ubunye.llm with answers recorded from a stub model, and a CI job replays it on Linux, Windows and macOS against a golden hash with nothing listening at the model's address.
• ubunye lineage focus: the model bill as FOCUS 1.4 cost rows. A run's model calls become rows a FinOps tool loads next to the cloud bill: every mandatory FOCUS 1.4 column, one row per provider, model and token direction, cost equal to quantity times unit price, times in UTC, custom columns prefixed x_ (run, task, model, direction, price date, cost basis). Costs are tokens times the list price the run used; the provider's invoice is the authority and every row says so. Replayed calls make no rows; unpriced calls are left out and counted. CSV or JSON lines. Each logged call now carries its provider, service, unit prices and their date.
• ubunye plan shows the bill before the run. For a task that calls a model, the plan prices its recorded calls (the replay file) at today's prices and sets them against UBUNYE_LLM_MAX_USD, with no data read and no model called. It fails on what would stop the run (replay with nothing recorded, a limit that is not a number, a dollar ceiling on an unpriced model) and warns on an estimate over the ceiling or live calls with no ceiling. --json adds an llm section per task.
• The bill is capped before the run. UBUNYE_LLM_MAX_USD, UBUNYE_LLM_MAX_CALLS and UBUNYE_LLM_MAX_SECONDS set one budget per run, shared by every port (and by calls running at the same time); llm.port(max_usd=, max_calls=, max_seconds=) adds a port's own. Before each call the port reserves its worst case (prompt tokens counted high, plus the whole max_tokens); a call that could pass a limit raises LLMBudgetError and is never sent, and the task writes nothing. After the call the reservation becomes the real cost. Prices come from a dated table of Anthropic's current models taken from its pricing page, or from price= or a UBUNYE_LLM_PRICES file; an unpriced model has an unknown cost, not zero, and a dollar ceiling on it fails closed. Each call records cost_usd and estimated_usd; the run record's new llm_budget keeps the limits and the spend, and ubunye lineage trace prints them. No new config field: limits are environment variables and port arguments.
• Record once, replay anywhere: model calls answered from a file, for nothing. UBUNYE_LLM_MODE=record calls the model and keeps each answer in the task folder's llm-replay.jsonl, next to config.yaml, to be committed with the task (or in UBUNYE_LLM_STORE), keyed by the request's hash; prompts are never stored. UBUNYE_LLM_MODE=replay answers from that file with no key, no network and no cost, and fails closed: a request with no recorded answer stops the run with its key, and never falls through to a live call. Every call in the run record now says its source (live, record or replay). A recorded run replays to the same data and row hashes on any machine.
• ubunye.llm: one port for language model calls, seen by the engine. A task calls llm.port("anthropic" | "openai_compatible" | "databricks_serving", model=...) and then complete() or complete_many() (answers in prompt order, a few calls at a time). The three backends use the standard library only; openai_compatible covers OpenAI, Azure OpenAI, vLLM, Ollama and LiteLLM. Keys come from the provider's usual variable or api_key=, which can be a secret:// reference; a missing key fails before any call. Rate limits and server errors are retried, honouring retry-after; other refusals fail at once with the provider's reason and never the key. Every call made during a run is in the run record's new llm_calls (backend, model, tokens, seconds, attempts, status and a sha256: key of the request; never the prompt or the answer), and ubunye lineage trace sums them per model. More backends are plugins in the ubunye.llm_backends group.
• ubunye doctor: what will fail, and why, before a run. One command checks the Python version, every backend (usable, or what it needs and the install command), whether a run without --backend would work, that Java suits the installed Spark, that delta-spark is built for the same Spark major (a mismatch installs cleanly and fails at run time), winutils.exe on Windows, and that every plugin loads. Given tasks (-d -u -p -t), it names the environment variables they use without a default that are not set, and checks their configs load. Warnings are environment problems that matter only for what you use; failures are what makes a run fail, and set exit code 1. --json prints one document. The config resolver gains required_env_references() for the variable check.
• ubunye export spark-pipeline: a task as a Spark Declarative Pipeline. Writes spark-pipeline.yml, a definitions module and a copy of the task, for spark-pipelines run on Spark 4.1+: inputs become temporary views, the task's transform() runs unchanged, outputs become materialized views. What does not carry over (merge modes, output paths, expectations) is reported; secret:// references are refused. The run-anywhere example exported this way wrote the same row hashes on Spark 4.2 as on every other platform.
• ubunye export airflow writes a DAG that runs, on Airflow 2.4+ and Airflow 3. The generated DAG used schedule_interval (removed in Airflow 3), imported BashOperator only from its Airflow 2 home, and passed env= without append_env=True, which replaces the whole environment, so ubunye was not on PATH and every run failed. It now uses schedule=, imports from the standard provider on Airflow 3 and falls back on Airflow 2, keeps the environment, passes Airflow's logical date as -dt {{ ds }}, quotes the command, and is compiled before it is written. New options: --usecase-dir (where the pipelines are on the Airflow workers), --backend, --lineage.
• One command to Kubernetes, Azure Container Apps and EMR Serverless. ubunye deploy k8s runs a task as a Kubernetes Job (kubectl's context, no retries, cleaned up after a day); ubunye deploy container-apps as an Azure Container Apps job pulling with a managed identity, reading the run record back from Log Analytics; ubunye deploy emr-serverless starts a run in an EMR Serverless application (tested as a plan: the free plan blocks EMR). ubunye deploy dockerfile container writes a self-contained image (Java, Spark in local mode, Delta, engine, pipelines). The entry script also reads its arguments from UBUNYE_ENTRY_ARGS, for runtimes whose CLI cannot pass them.
• One command to AWS Glue and GCP Dataproc Serverless. ubunye deploy glue and ubunye deploy dataproc run a task unchanged on either service, through the cloud's own CLI and login: the task goes up as a zip with a small entry script that runs it with a run record and prints the record back, so --record-out saves it and ubunye gate can compare runs across clouds. Glue pip-installs the engine (or an uploaded wheel) and supplies Delta; Dataproc runs in an image whose Dockerfile ubunye deploy dockerfile dataproc writes, following Dataproc's rules. --env, --var, --dry-run, --wait/--no-wait. See Glue and Dataproc.
• ubunye gate and a GitHub Action: the receipt gates pull requests. The gate compares a run record with a baseline and fails when the run failed, a fail expectation broke, an output's data or schema changed without a VERSION bump, an output went missing, or a limit was passed (--max-slowdown, --max-seconds, --max-row-change). Each changed output says what else changed (config, code, environment, which inputs), and a change with none of them is reported as nondeterminism. Runs are named by record file, run id, previous or latest; --json and --summary (Markdown for the job summary). The composite action .github/actions/gate runs a task on a pull request's base and head and gates them. See Gate.
• OpenTelemetry done properly. The OTel hook now follows OpenTelemetry's own configuration (OTEL_EXPORTER_OTLP_ENDPOINT, protocol, headers, OTEL_TRACES_EXPORTER / OTEL_METRICS_EXPORTER, OTEL_SERVICE_NAME), exports over OTLP (http/protobuf or grpc) with the new otel extra, reuses a host application's providers, and flushes when a task ends so short CLI runs export. It used to print every span to the console and ignore the endpoint. Spans now nest steps under the task, carry the run id, backend and config hash, and mark failures as errors with the exception recorded. New metrics: ubunye.task.runs, ubunye.task.duration, ubunye.step.duration, ubunye.rows.read, ubunye.rows.written (rows counted where free, on Spark only with UBUNYE_OTEL_COUNT_ROWS=1). UBUNYE_TELEMETRY is read when a run starts, not at import, so setting it in a notebook works. The Prometheus hook is unchanged; its row and byte counters are fed only by callers of observe_step(), never by the engine, as before. See OpenTelemetry.
• OpenLineage events: the receipt lands in your catalogue. A recorded run sends START and COMPLETE or FAIL (OpenLineage 2-0-2) to any OpenLineage server (Marquez, DataHub, OpenMetadata, Google Dataplex) when OPENLINEAGE_URL is set, and/or to a JSON-lines file (UBUNYE_OPENLINEAGE_FILE). Datasets are named by the OpenLineage conventions with credentials removed; events carry the standard outputStatistics, dataQualityMetrics, dataQualityAssertions and errorMessage facets plus ubunye_evidence and ubunye_hash (schemas in docs/schemas). No new dependency; a server that is down never fails a run. ubunye lineage openlineage exports (and with --send, backfills) stored runs. Every event in the tests is validated against the vendored OpenLineage spec. See OpenLineage.
• Run record v2: the receipt says why two runs differ. Each record now carries a hash of the task's code, the environment (Python, platform and the versions of the packages that can change a result, plus one hash of them), every input's row hash and count (like the outputs), per-step timings, and every expectation's result; timings and expectation results are kept when a run fails. ubunye lineage compare reports code, environment and inputs as changed or unchanged and names the packages whose versions moved; trace prints them. Input hashing costs one more scan per input and can be turned off (LineageRecorder(hash_inputs=False)). Monitors get the new evidence only if their task_end accepts it, so existing monitors are unaffected; v1 records still load. See the addendum to ADR 006.
• The pandas content hash is 2.4 times faster (60,000 rows: 828 ms to 350 ms, the same hash). Each column's text is now built once, with the column name encoded once instead of on every row; a property test holds the fast path to the row-at-a-time reference byte for byte. It pays for hashing inputs in run record v2: a recorded run now hashes its input and its output in less time than it used to hash its output alone.
• Secrets by reference: secret://<provider>/<reference>. A config names a secret (password: "secret://aws-sm/prod/db#password") and the engine fetches it only into the copy of the config a connector receives, at the moment it reads or writes. The config hash, plan, config, run records and logs keep the reference, so a secret cannot leak through them, and rotating it does not change the config hash (tested). Providers are plugins (ubunye.secrets): env, file, databricks, aws-sm, gcp-sm, azure-kv, with #field for JSON secrets and new aws, gcp, azure extras for the cloud SDKs. validate refuses an unknown provider with the closest name; doctor names the package a task's provider is missing; neither fetches anything. See Secrets.
• CONFIG.expectations: what an output must look like, checked before anything is written. Declared rules per output (not_null, unique, between, one_of, matches, row_count), each with a severity: fail stops the run with nothing written, quarantine moves the breaking rows to a named output with a _ubunye_failed_rules column listing every rule each row broke, warn logs. max_quarantine_rate fails a run when too much is set aside. A missing value passes every rule but not_null, as in SQL. The rules run through Narwhals, so the same config gives the same verdicts on Spark and pandas (tested on both, and at the oldest supported versions, where pandas 2.2 and pandas 3 disagree about a missing value in a pattern match). Narwhals becomes a dependency (narwhals>=2.0, pure Python, no dependencies of its own). Every rule's result, passed or not, is on the error and in the run state for the run record. See Expectations.

• A typo inside a connector block fails validation, with a suggestion. Input and output blocks accept keys the engine does not know, because each connector reads its own settings, so paht: data/in.csv validated and was ignored. Connectors can now declare CONFIG_KEYS; for one that does, any other key is an error that names the closest real key (inputs.src: 'paht' is not a setting of 's3'; did you mean 'path'?), reported before the "requires" error it usually causes. Every built-in reader and writer declares its keys; format and options are allowed everywhere, and mode, merge_keys and replace_where on every output. A connector that declares nothing (most third-party ones) accepts any key, as before. The s3 and unity writers each accept the other's key (table, path) so one output block can serve both, as the run-anywhere example does. All 53 task configs in the engine and examples repos and two downstream projects still validate.

• A conformance suite every backend must pass, shipped for yours. ubunye.testing.backend_conformance is the set of tests that says a backend keeps the engine's promises: it is registered under its name and declares what it can do, gives transforms its own frames and the engine a port, reads a CSV file exactly as Spark does and so leaves the same run record hash as every other engine (checked against a reference built from plain Python values), reads back what it writes, and honours the write modes it claims. Subclass it in a backend's tests. The pandas backend passes it in the unit tier, and both Spark backends in the integration tier. ADR 006 now lists the type names the hash uses, which a port that is not Spark or pandas must report.

• One transform for every engine, written with Narwhals (ADR 005). A transform written with the Spark API runs only on Spark, and one written with pandas only on pandas. Two ways to write it once were tested on the Titanic example's own logic against Spark: Narwhals gave the same data hash on both engines (with a sum cast to a 64 bit integer, which Spark does and pandas does not), and SQL did not (DuckDB and Spark type the same aggregate differently), so Narwhals ships and SQL waits for a DuckDB backend, not in this release. A transform may return a Narwhals frame; the engine unwraps it without importing Narwhals. There is no config field for portability: ubunye plan reads the imports in transformations.py and says what it is written for (pyspark, narwhals, pandas), in text and in --json, and warns when that is not what --backend gives it. The example in the docs is run on both engines by the test suite.
• --json for scripts and agents. plan, validate, backends, every lineage command and models list/info/compare print exactly one JSON document on stdout with --json, errors included ({"ok": false, "error": ...}), and keep their exit codes. lineage compare --json reports each data hash as unchanged, changed, unknown or not comparable, the same verdict the text form prints.
• The pandas backend understands Spark's mode option, and backends check their IO details before a run. Every Titanic example reads its CSV with mode: "FAILFAST", which the pandas backend refused, so none of them could run there. mode now works for CSV and JSON exactly as Spark 4.2 does it (checked against Spark): FAILFAST stops at a bad row, DROPMALFORMED skips it, and PERMISSIVE, Spark's default, cuts a row with too many fields and pads one with too few with null. A backend can also check an input's or output's details (options, schema) before anything runs (Backend.check_io); the pandas backend uses it, so plan, validate --backend and the run's own preflight report an option it cannot honour instead of failing on open.
• --var key=value, documented for years, now works. It was in the README and three docs pages and was never implemented, so every example that used it failed. It now works on every command that renders a config (run, validate, plan, config, test run), and the Python API takes the same thing as variables= on run_task, run_pipeline and notebook. Names must be valid template names, env is reserved and mode has its own flag; --var dt=... works like -dt, and the same name with two values is refused instead of one silently winning. The variables a run used are kept in its run record. On the way, validate stopped setting dtf to the timestamp's value (it now has -dtf), and test run now renders with mode (its profile), as run does.
• The run record's data hash now means "these exact rows" (ADR 006). It read a 1 percent sample, changed with row order on Spark, and on pandas quietly recorded the schema hash as the data hash. The new rows-v1 hash reads every row in the same pass as the count, ignores row and column order, changes when any cell changes, tells null from NaN, does not depend on the timezone, and is the same on Spark and pandas for the same data (Spark computes it in one aggregation on the cluster). When rows cannot be read the record says why instead of inventing a hash. Records also carry the Ubunye version, the backend, the run variables and each output's hash_method, and runs from run_task and run_pipeline are stored under the same folder and name as CLI runs, so ubunye lineage list finds them. lineage compare calls two missing hashes "unknown" (it said "unchanged") and a pre-0.7 hash "not comparable". sample_fraction is ignored and kept so old configs load.
• A pandas transform gets a plain pandas DataFrame (ADR 004). It used to get an adapter and had to write sources["x"].native to reach the DataFrame. The Backend port gains to_native and to_port, and the engine converts at the edges: transforms get and may return native frames; writers, hooks and lineage get the port, where count() means rows (a raw pandas count() counts non-nulls per column). run_task, run_pipeline and the notebook's read() and transform() return native frames, so on pandas use len(frame) for rows. Both methods default to doing nothing, so Spark tasks and older backends are unchanged.
• Backends are plugins, and say what they can do (ADR 001, 002). Spark, Databricks and pandas now register in a new ubunye.backends entry point group, exactly as a third party engine would, so adding an engine is a package with one entry point and no edit to Ubunye. Each backend declares its capabilities (features such as a SparkSession or path IO, file formats, write modes, distributed, needs Java) and each connector declares what it requires. Before anything starts, the engine checks every input and output against the backend and lists every problem at once, so a task that cannot run stops in the first second instead of halfway through. The core no longer imports SparkBackend (a test now reads every import in ubunye/core and fails on any engine). Backend.is_spark still works, read from the capabilities, and is deprecated. Backends and connectors written before 0.7.0 declare nothing and behave exactly as before.
• Choose a backend by name everywhere, with one resolution order (ADR 003). --backend NAME on ubunye run, ubunye test run and ubunye validate; backend="pandas" (or an instance) on run_task, run_pipeline and notebook. With no choice: the platform's session if there is one (on Databricks, the notebook's), else Spark. The CLI now follows that order too: run inside a process that already has a SparkSession, it attaches to it instead of stopping it at the end. New ubunye backends (and --json) lists what is installed and what each backend can do; ubunye validate --backend pandas checks a task can run there without starting anything. An unknown or broken backend gives a clear error with the installed names or the pip install that fixes it. New page: Execution Backends; new section: Architecture Decisions.
• A pandas backend, so ubunye run --backend pandas runs a task on a laptop with no Spark and no JVM (issue #38). Backend was a port with a single kind of adapter (Spark), and a port with one adapter has never really been tested as a port. It has a second one now. PandasBackend reads and writes the generic path formats (csv, parquet, json) with pandas; lakehouse formats and managed tables stay Spark's job, and their connectors say so, so an unsupported pairing fails with a clear message instead of a stray AttributeError. This works because the data plane gained a small read/write seam on Backend (read_frame / execute_write): a path connector like s3 asks the backend to do the IO instead of naming Spark, so the same task and the same config.yaml run on either backend. The proof is an integration test that runs one passthrough task through the engine on Spark and on pandas and asserts the output is identical. Install the extra with pip install 'ubunye-engine[pandas]'.
• The pandas backend reads data exactly as Spark does. The first version used pandas' own defaults, so the same CSV gave different columns and types on the two backends: pandas assumed a header Spark does not, guessed types Spark leaves as text, read JSON as one array where Spark reads one object per line, and could not read a folder Spark had written. Reads now follow Spark: no header by default (_c0, _c1), text unless inferSchema, Spark's inferred types (int when every value fits, else bigint; an all empty column is text), JSON Lines with columns sorted by name, folders of part files, globs, explicit schema: strings, and timestamps read in spark.sql.session.timeZone (UTC when unset). Columns are Arrow backed, so a whole number column with nulls stays whole numbers. Options it cannot honour, nested schema types and remote paths are refused by name instead of ignored. Needs pandas 2.2 and pyarrow 14 or newer.
• The pandas backend writes data exactly as Spark does, so each can read the other's output. It used to write one file where Spark writes a folder, so Spark could not read pandas output as a table, and append re-read and rewrote the whole file every run. It now writes Spark's layout (a folder of part-* files and _SUCCESS); append adds a part file; overwrite is staged and swapped in only when complete, so a failed write keeps the old data. The text formats match Spark byte for byte on the cases tested: CSV without a header by default, minimal quoting with a backslash escape, text trimmed, Java style numbers, Spark's timestamp text; JSON Lines without null fields. Parquet timestamps are written as UTC microseconds (Spark cannot read nanoseconds). A named index (what groupby leaves) is kept as columns. partition_by, unknown write options, nested values in CSV and non pandas frames are refused by name.
• Connectors that need Spark say so on the pandas backend. hive, jdbc, delta, unity, binary and rest_api build on a SparkSession. On the pandas backend they used to fail with AttributeError: 'PandasBackend' object has no attribute 'spark', and rest_api only after fetching every page. They now stop first, before any network call, with a message naming the connector and the way out (--backend spark, or csv/parquet/json paths with format: s3).
• Proof that the two backends agree, run against real Spark. The old parity test compared one passthrough task as strings. The new suite runs Spark 4.2 beside the pandas backend and compares Arrow types and Python values, never strings: eight reader cases (CSV with and without header, inferSchema, explicit schema, separators and null values; JSON Lines, multiLine and schema), a folder Spark wrote, each engine reading what the other wrote for parquet, CSV and JSON, CSV and JSON files byte for byte, and one task end to end on both backends. It runs in a timezone other than UTC so a timezone slip cannot hide. It found two gaps, now fixed: explicit TIMESTAMP schemas could not read text without an offset, and JSON was not written the way Spark writes it.

---

Changed

• The Titanic examples run on Spark and on pandas, with the same receipt. Their three transforms (survival by class; clean, then aggregate) are written once with Narwhals, identically in the local and Databricks examples. CI runs each local example on Spark, then on the pandas backend into the same output, checks the golden output again, and requires the two run records to carry the same config hash and data hash (scripts/same_receipt.sh, which uses only ubunye lineage list and lineage compare). On the real 891-row file the pandas run takes 0.4 s against Spark's 4 s. The example tests run every transform on both engines. The Databricks notebooks install Narwhals. The docs now name the two known Spark and pandas differences under Narwhals: the type of a sum (cast first) and rounding exactly halfway.

• Python 3.10 to 3.13, tested on Linux, Windows and macOS. Python 3.9 is no longer supported (it reached end of life in October 2025). CI now runs the unit tier on every supported Python on all three systems (Windows and macOS ran none of it before), the Spark tier on Spark 4 with Python 3.10 and Java 17 and with Python 3.13 and Java 21, and on Spark 3.5 with Java 11.

• The oldest versions the package accepts are tested, and two were raised. A new CI job installs exactly the declared minimum of every dependency and runs the unit tier, and a test keeps that job and pyproject.toml in step. It found that typer>=0.12 could not work with today's click (the CLI could not start), so the minimum is now typer>=0.15.4, the first that works. The spark extra now asks for pyspark>=3.5 (it said 3.3, which was never tested); the whole Spark tier passes on Spark 3.5.

• ubunye plan is a real dry run, and exits 1 when it finds a problem. It printed the config's names back and always exited 0. Built on the September work, it now checks each local input exists or is written by an earlier task in the same plan (the September version failed every multi task pipeline on that), loads the transform class, resolves every write mode (so merge without merge_keys is caught before a cluster runs the transform), asks the --backend what it can do instead of keeping a list of names, and warns about environment variables the config uses that are not set. It starts no engine and moves no data. Its config hash is now the one the run record keeps: the record used to hash the config after the engine had rewritten its transform, so the two never matched. A config error in plan shows the whole message.

• The package says what it is. LICENSE is the full MIT text (it was the single word "MIT"), declared the modern way (license = "MIT" with license-files, which needs setuptools 77). PyPI now shows classifiers for the Python versions CI tests, a description that matches what the engine does now that it is not Spark only, and pandas and lineage among the keywords. Thabang Mashinini-Sekgoto stays the author and Ubunye AI Ecosystems the maintainer and copyright holder.
• Spark moved out of the engine core, so the hexagon is real rather than aspirational. The founding rule is "the core never depends on the outside world," but core/write_modes.py and core/catalog.py called Spark directly (df.write, spark.sql("MERGE INTO ..."), USE CATALOG), so the one place the pattern was meant to hold was the one place it leaked. The Spark mechanism now lives in a new Spark adapter (ubunye.adapters.spark.write_exec and ubunye.adapters.spark.catalog); the backend-agnostic decision (validating a write mode against what a connector supports) stays in ubunye.core.write_modes. Behaviour is unchanged and every existing test passes. The old names (write_modes.apply, merge_into, target_exists, dynamic_partition_overwrite, and core.catalog.set_catalog_and_schema) still resolve through a deprecation shim, so third-party connectors keep working; they will be removed in a future major. This is the groundwork for a non-Spark backend (issue #38): the core can be backend-agnostic now because it no longer imports one backend's API.

Fixed

• A deploy never reports success without the run record it was asked for. Azure's Log Analytics returns lines printed in the same instant in any order, so the record could come back after its end marker: ubunye deploy container-apps stopped waiting at the marker, found nothing between the markers, and still exited 0 without writing --record-out. The record is now found wherever it lands, Container Apps waits for the record itself, and a missing record fails the command when --record-out was given. Found by the final release-candidate run.

• A model provider's error reads as its message when it comes in a list. Gemini's OpenAI-compatible endpoint answers errors as [{"error": {...}}]; the port printed the whole list. Found on the first live call to Gemini.

• ubunye deploy container-apps runs a second time. The job is created on the first deploy and updated after that, and az containerapp job update refuses --env-vars (it takes --replace-env-vars), so every deploy after the first failed. Found by running the release candidate on Azure Container Apps again.

• ubunye lineage list no longer reports an input of 0 rows it never counted. A run record counts and hashes its outputs (ADR 006), not its inputs, and the list added the missing counts up as in:0, which read as an empty input. An uncounted total now shows as in:-.

• The Spark backend stops only a session it started. start() attaches to a session that is already running (getOrCreate), and stop() then stopped it anyway, as did the garbage collector through __del__. So run_task(..., backend="spark") in a process that already had a Spark session, a user's own or a notebook's, ended that session when the run finished. A session the backend did not start is now left running. Found by the new backend conformance suite.
• ubunye deploy databricks installs what the transform needs. The generated notebook installed the engine alone, so a task written with Narwhals failed on Databricks at its first line. The deploy now reads the transform's imports (the same detection plan uses, ADR 005) and installs Narwhals next to the engine when the transform imports it.
• import ubunye works on every pydantic the package accepts. The model transform's model_class field starts with pydantic's reserved model_ prefix: pydantic 2.0 refused it at import, so the package failed to load on the oldest version its requirements allowed, and pydantic 2.1 to 2.9 warned about it. The prefix is switched off for that one config model. Found by CI's new minimum-versions job.
• No DeprecationWarning on every plugin lookup. Readers, writers, backends, hooks and artifact stores were found through the dict interface of entry_points(), kept for Python 3.9 and deprecated on 3.10 and 3.11, so every lookup warned there. They now ask entry_points(group=...).
• The pandas backend reads quoted CSV values exactly as Spark does. Spark's escape character is a backslash, so a doubled quote inside a quoted value is not an escape: Spark keeps "McGowan, Miss. Anna ""Annie""" as written. pyarrow unescaped it, so 53 of the 891 Titanic names came out different on pandas, with nothing to say so. Files with such quotes (or a stray quote, a backslash, or a line of only spaces, which Spark skips) are now split by a port of Spark's own CSV parser, univocity 2.9.1, and only the lines that need it; every other file is read by pyarrow as before. The port is fuzzed against live Spark for each escape setting, line ending and multiLine. escape: '"' now reads doubled quotes as quotes, as in Spark.
• The quickstart works, and a test keeps it working. It used flags that do not exist (run --profile), a lineage list with no task, and a Hive to Delta config no laptop could run. It is rewritten around the ubunye init scaffold, and its commands, in the README and on the Quickstart page, are run by the test suite exactly as written. CONTRIBUTING.md, which the README linked to, now exists (the docs page shows the same file), and stale claims are gone: "Spark native", "288 tests", and run --profile in the engine docs.
• The first command in the README works, and what it makes runs. The README and quickstart said ubunye init -d ... -t ..., but the command was ubunye init pipeline ..., so the first thing a new person typed failed. Both forms work now. The default scaffold (--template local) has a small sample CSV next to the task, a config that reads it and writes Parquet into output/ next to it, and a transform (people[people["age"] >= 18]) that means the same in pandas and in Spark: it runs with no Java and on Spark unchanged, and the two runs leave the same data hash (checked against Spark). --template databricks writes the old Unity Catalog scaffold. Configs get a built in {{ task_dir }} variable, the task's own folder, so a path next to the task works from any folder and on either engine (Spark resolves a relative path from where its JVM started, pandas from the current folder).
• ubunye models promote honours the model's promotion gates. Gates were read from the training task's config and applied only when that run promoted the model itself; the CLI never saw them, so a model failing every gate could be promoted by hand. The registry now keeps a model's gates (set when the training run registers it), every promotion checks them by default, and a failing gate is named. --force (and force=True in Python) skips them with a warning and marks the version promotion_forced, with the gates it skipped.
• A typo in config.yaml says where it is. A YAML syntax error escaped as a raw yaml.ParserError, so run, validate and plan crashed with a traceback. It is now a config error naming the file, the line and the column, and a file whose top level is not a mapping is refused clearly too.
• ubunye --help no longer crashes on Windows. A legacy Windows console prints in cp1252, which has no arrow, and a help text and a few messages used one, so ubunye --help died with UnicodeEncodeError on a fresh install. The CLI's text is now plain ASCII where it prints, a test keeps every help text printable on cp1252, and the ubunye command sets its output to replace any character the console cannot show (a file path with an accent, say) instead of crashing. The CI Package job runs --help on windows-latest.
• A backend whose packages are missing says so, with the install. After pip install ubunye-engine alone, ubunye backends listed spark and pandas as ready, and choosing one failed deep inside on the first frame. Each backend now declares the packages it needs; choosing it names what is missing and the extra that installs it (pip install 'ubunye-engine[pandas]'), and ubunye backends marks it "not usable here". Listing and inspecting a backend still work anywhere, and Engine() with no backend resolves one only when it first needs it.
• The pandas backend works on Windows. pyarrow before 24 cannot find a timezone database on Windows (checked: 19 to 23 fail, even with the tzdata package), so every timestamp read or written failed there. The pandas extra now asks for pyarrow 24 or newer on Windows, and the backend checks at start and says how to fix an older one instead of failing on the first timestamp.
• A notebook with lineage=True now leaves a run record. Lineage is recorded around a whole task, and the notebook runs read, transform and write as separate steps, so ubunye.notebook(..., lineage=True) recorded nothing at all. A notebook write now counts as a run: nb.write(...) and nb.run() each leave one record, in the same place and with the same hash as run_task.

---

[0.5.0] — 2026-07-14

Fixed

• Lineage no longer recomputes your pipeline or risks the driver. Fingerprinting an output used to run the output's entire plan two to three times (a count inside the hasher, an unbounded sample collect, then a second count in the recorder on the same uncached DataFrame), and had a fallback that collected the whole table into the driver. At real scale that is a crashed driver describing a job that had already succeeded. Now: one fingerprint_dataframe() pass, persisted for the duration, one count shared by the hash and the row count, and every collect capped at UBUNYE_LINEAGE_SAMPLE_ROWS (default 1000) whatever the table size.

Added

• The model registry writes to object storage now. ModelRegistry was built directly on pathlib, so models could only be saved to a normal disk — the one thing AWS and GCP serverless do not offer. Storage is a port now: the scheme of the store path picks the backend. A plain path or file:// stays local and behaves exactly as before. s3://, gs:// and friends go through fsspec (pip install 'ubunye-engine[objectstore]' plus your cloud's filesystem, e.g. s3fs). Any other scheme goes to whatever store someone registers under the new ubunye.artifact_stores entry-point group — one class, one entry point, no engine edits, no inheritance, like every other connector. Models still write real files (a Keras save needs an actual disk), so remote stores stage locally and upload, and get_model downloads before load. Closes #28.

• The annotations are a public API now. The package ships a py.typed marker, so type checkers in consuming projects finally see the engine's types (PEP 561 says they must ignore packages without it, so until now every annotation was invisible to users). mypy runs in CI and is green across all 100 source files, and it earned its keep immediately: it caught a Protocol that under-described its own usage, a shadowed variable in the REST reader's pagination, unsound stubs in the S3 lineage store, and thirteen stale type: ignore comments. Reader.read now returns DataFramePort instead of Any, so the most important object in the framework is typed in its own contract.

• DataFramePort — the data plane finally gets its port. Specified in the founding design notes and never shipped. The model layer always had its port (UbunyeModel: the engine never imports sklearn or torch); the data layer never did, so every reader returned and every transform received a pyspark.sql.DataFrame by fiat. ubunye.core.ports.DataFramePort is a runtime_checkable Protocol — structural, no inheritance — and a Spark DataFrame satisfies it natively (verified against a real session, not assumed), which is what makes shipping it additive rather than a rewrite. ubunye.adapters.PandasDataFrameAdapter makes pandas fit in thirty lines, exactly as the design promised — including guarding the trap where df.count() exists on pandas and means something else entirely (non-null counts per column).

• The plugin contract is structural again. validate_config, SUPPORTED_MODES and SUPPORTS_MERGE are read with getattr and defaults: a connector that declares nothing requires nothing. The previous contract change accidentally demanded inheritance — a duck-typed reader, the exact thing docs/interfaces.md promises will work ("no inheritance needed"), crashed with AttributeError. Declaring is opt-in, not a toll.

Removed

• Telemetry that could only read zero. The ubunye_rows_total and ubunye_bytes_total Prometheus counters, and the rows= parameter on EventLogger.step_end, were dead on arrival: nothing in the engine ever fed them. A metric that always reads zero is worse than no metric, because somebody builds a dashboard on it and trusts the zero. If row metrics return they will arrive fed by the engine (the lineage fingerprint now computes row counts in one pass, so the plumbing finally exists).

• ubunye/compat/analytics_engine_shim.py — a placeholder docstring with a folder around it, imported by nothing.

• Stale build artifacts (build/, an 0.2.0 wheel in dist/), mlflow.db and a log file were removed from tracking and ignored.

Changed

• The CLI stops teaching two systems. run gained --all (validate had it for years, so the two commands taught different habits for the same job). validate accepts -m/--mode as an alias for --profile, and now injects the same template variables as run — a config using {{ mode }} used to run perfectly and fail validation with an undefined variable, which is exactly backwards for a command whose job is to catch problems before the run. Closes #30. The ubunye init and Databricks notebook scaffolds no longer print df.count() per output (a full scan per output, teaching the habit the lineage rework just removed); they print the column list, which is free.

• The four control-plane Protocols (AuthBackend, DeployAdapter, LineageBackend, RegistryBackend) are enforced by a conformance test. They were referenced only by docstrings before: a contract nobody had signed. The test checks every shipped implementation member by member, structurally, with no inheritance added.

• from ubunye import * works now. __all__ advertised five submodule names that the module never imported, so the star import raised NameError on the package's own public surface.

Breaking

• The core no longer knows about any connector. It asks.

The engine advertised "adding a connector = write the class, register the entry point". That was not true. Three places in the core held knowledge of specific implementations:

• config/schema.py carried an if/elif chain naming hive, jdbc, s3, binary, delta, unity and rest_api, spelling out what each one required.
• core/write_modes.py held _MERGE_FORMATS = {"delta"} and decided, on the connector's behalf, which formats were allowed to MERGE.
• FormatType was a closed enum, so a correctly-registered plugin was rejected before the registry was ever consulted.

Adding a connector meant editing the engine in three places. That is not open/closed, and it is not what the entry-point system is for.

The plugin contract now carries it:

class Connector(ABC):
    @classmethod
    def validate_config(cls, cfg) -> list[str]: ...   # what I need

class Writer(Connector):
    SUPPORTED_MODES: frozenset = frozenset({"append", "overwrite"})
    SUPPORTS_MERGE: bool = False                      # what I can do
    MERGE_FILE_FORMATS: frozenset = frozenset({"delta"})

An Iceberg or Hudi connector can now declare that it supports MERGE, and be believed. tests/unit/config/test_third_party_connector.py registers a connector the engine has never heard of and asserts it works with zero engine edits.

Breaking: FormatType is deleted. IOConfig.format is a plain str.

• The engine no longer changes its behaviour based on where it is running.

Two places sniffed the host and acted on the guess:

• writers/unity.py::_is_databricks() string-matched the Spark conf for "databricks" and used the result to decide whether to run OPTIMIZE and VACUUM. It was wrong twice: a connector should ask the target what it can do rather than detect its host, and the guess was factually wrong — OPTIMIZE, ZORDER and VACUUM are Delta features, and open-source Delta has had them for years. Anyone running Delta off Databricks silently did not get the maintenance they had explicitly asked for in their config, and nothing said so. The statements are now attempted, and a target that cannot run them is reported, not predicted in advance and not swallowed.

• _internal/auto_detect.py picked the model registry backend from DATABRICKS_RUNTIME_VERSION with no way to override it. The same pipeline, unchanged, registered models to MLflow on Databricks and to a directory anywhere else — and said nothing. UBUNYE_REGISTRY_BACKEND now wins, the host is only a fallback, and it logs which it chose.

• Inputs are validated against the READER, outputs against the WRITER.

One shared rule used to validate both roles, so it could not tell them apart: unity as a source may be given sql, but as a sink it must have a table — writing to a SELECT statement is meaningless. The shared rule had to accept sql everywhere, so it silently allowed it. Each side is now checked against the plugin that will actually handle it.

Breaking: IOConfig(format="s3") no longer raises on its own. It cannot: an IOConfig does not know whether it is an input or an output. Validation happens at TaskConfig, where the role exists.

---

[0.4.0] — 2026-07-13

Breaking

• ENGINE.spark_conf is now applied to a session the engine did not create — and static keys raise instead of being ignored.

Before, the conf was silently discarded whenever a SparkSession already existed. That is always on Databricks, and equally in a notebook, an AWS Glue job, under spark-submit, or in pytest. DatabricksBackend took no conf argument at all, and api.py computed merged_spark_conf(mode) and then threw it away. So every ENGINE.spark_conf and every ENGINE.profiles block did nothing, and nothing said so. The config claimed one thing and the runtime did another.

Spark divides settings into runtime keys (spark.sql.shuffle.partitions) and static ones (spark.master, spark.sql.extensions, spark.driver.memory) that are fixed when the JVM starts. Runtime keys are now applied. Static keys are a request the engine cannot honour, so they raise, naming every offending key at once.

Action required: remove static keys from ENGINE.spark_conf and set them where the session is actually created — a cluster policy, a job conf, --conf on spark-submit. If your pipeline "worked" with them before, it was ignoring them.

• A config can no longer override a master the platform already chose.

Under spark-submit — how EMR Serverless and Dataproc Serverless start every job — the platform sets spark.master in the default SparkConf. A task that also set spark.master won, and the job ran entirely in the driver: it ignored every executor, finished, reported success, and billed for a cluster it never touched. Nothing warned. The output was correct; there was just far less of it per minute than there should have been, forever.

SparkBackend now refuses to start when the config's master disagrees with the platform's.

Action required: delete spark.master from ENGINE.spark_conf. The master belongs to whoever launched the session, not to the task.

Added

• python -m ubunye — an entry point a cloud can actually run.

The engine could only be reached through its console script. AWS EMR Serverless and GCP Dataproc Serverless do not give you a shell: they hand a Python file to spark-submit. An engine reachable only through its CLI cannot run on either of them — and you find that out after wiring up IAM, a bucket and a billing account.

spark-submit --py-files deps.zip -m ubunye       --task-dir s3a://bucket/code/pipelines/sales/etl/daily --mode PROD

It deliberately does not create a SparkSession: spark-submit already made one, with the platform's master and executors, and the engine attaches to it.

• A rest extra. requests lived only in the dev extra, so pip install ubunye-engine[spark] shipped a rest_api reader that could not make a request. It worked on Databricks by accident, because the runtime preinstalls it — the bug was invisible from the one platform most people use, and appeared the moment anyone left it, as a bare ModuleNotFoundError from inside a Spark job. A missing requests now says what to install.

• format is now an open string, validated against the plugin registry.

The docs said "adding a connector = write the class, register the entry point". That was false: format was a closed enum, so a correctly-registered third-party plugin was rejected by config validation before the registry was ever consulted. The extension story the engine advertises did not work.

Any name the ubunye.readers / ubunye.writers entry points can load is now valid. An unknown name still fails, and the error lists what is actually installed.

Breaking: IOConfig.format is a str, not a FormatType. Code doing cfg.format.value must now use cfg.format.

• AmbientSessionBackend — an alias for DatabricksBackend, which contains no Databricks code at all. It attaches to a session somebody else created and declines to stop it, which is equally true of Glue, EMR, Dataproc, a notebook and pytest. The name had convinced people the engine has a Databricks dependency here. It does not.

Fixed

• {{ dt | default('latest') }} never fell back. ubunye run passes {"dt": ..., "dtf": ..., "mode": ...} unconditionally, and an omitted flag arrives as None. Jinja treats None as defined, so default() did not fire — the template rendered the literal string "None" and pipelines quietly wrote to paths like out/dt=None/. Nothing errored; the data just went somewhere nobody meant. None values are now dropped, so they are genuinely undefined. StrictUndefined still makes a real typo fail loudly.

• ENGINE.catalog no longer breaks every non-Databricks Spark. set_catalog_and_schema issued USE CATALOG, which is a Unity Catalog statement — open-source Spark rejects it outright with PARSE_SYNTAX_ERROR. It is now attempted and, if unsupported, logged and skipped: the configs use three-part names anyway, which Spark resolves without it.

• ubunye export airflow and ubunye export databricks emitted artifacts that could not run. Both generated ubunye run -c <config> --profile <p> — and there is no -c and no --profile on ubunye run. Every DAG and every job.json this exporter has ever produced would fail on its first task with "no such option". The tests asserted the broken flags, which is why nobody noticed: they checked the exporter still emitted what it always had, rather than something that works.

---

[0.3.0] — 2026-07-12

Breaking

• The s3 writer now defaults to mode: append, not overwrite. Every other writer already defaulted to append; the same config key meant "insert" on one connector and "destroy and replace" on another. All writers are now consistent, and the default is the mode that cannot lose data by accident.

Action required: any s3 output that omitted mode and relied on the implicit overwrite must now say mode: overwrite explicitly. Nothing fails loudly — the write succeeds, it just appends. Grep your configs for format: s3 outputs with no mode key before upgrading.

Added

• The delta, hive and binary connectors now exist. All three were declared in FormatType, accepted by config validation, and documented — but had no plugin registered behind them, so any pipeline using them died with ReaderNotFoundError / WriterNotFoundError.

Format	Reader	Writer
delta	new — by path, table, or SQL; time travel via version_as_of / timestamp_as_of	new — all six write modes
hive	already existed	new — all six write modes, partitionBy
binary	new — Spark's binaryFile source, one row per file	none: the source is read-only

A test now asserts that every format in FormatType resolves to a registered plugin, so this class of gap fails in CI rather than in a pipeline.

• format: binary in CONFIG.outputs is rejected at config load. Spark's binaryFile source cannot write. The failure now happens in ubunye validate, with a reason, instead of after the transform has already run.

• Four new write modes. CONFIG.outputs.*.mode previously accepted only overwrite, append and merge — and of those, merge was never implemented. The full set is now:

Mode	Behaviour
errorifexists (alias error)	Fail if the target exists — Spark's own default
ignore	No-op if the target exists
merge	Delta MERGE (upsert) on merge_keys; creates the target on first run
overwrite_partitions	Replace only the partitions in the DataFrame — on Delta (dynamic or replace_where), or via INSERT OVERWRITE on a non-Delta table

• Integration tests on a real Spark session. The write modes are exercised against a real JVM Spark (and a real Delta table) in CI, not just asserted against mocks — ignore really leaves the target alone, merge really upserts, and overwrite_partitions really leaves its neighbouring partitions standing.

• ubunye.core.write_modes. Mode semantics live in one module rather than in each connector. Writers declare the modes they support; a mode a connector cannot honour raises SinkWriteError before any rows are written instead of being silently downgraded. JDBC accepts Spark's four native modes only; rest_api is append-only.

• partitionBy, merge_keys and replace_where on the s3 writer.

Fixed

• mode: merge crashed at runtime. It passed config validation and was documented (with merge_keys), but no writer implemented it — the string was handed to df.write.mode("merge"), which Spark rejects with IllegalArgumentException: Unknown save mode. It now performs a real Delta MERGE.

• errorifexists and ignore were unreachable. The JDBC and Unity writer docstrings claimed support, but the WriteMode enum rejected both at config load, so no config.yaml could ever set them.

• The s3 writer silently dropped options. mergeSchema, overwriteSchema and friends were documented but never applied to the Spark writer.

• Stale tests in tests/test_rest_api_plugin.py. Five tests still expected ValueError where the engine has raised typed SourceReadError / SinkWriteError for some time. They had been failing at HEAD unnoticed because CI only runs tests/unit.

• overwrite_partitions refuses an unpartitioned target. Dynamic partition overwrite without partitions is a full-table wipe; the engine now fails the config instead of quietly destroying data. The session-level spark.sql.sources.partitionOverwriteMode conf is restored after each write rather than leaking into later writes.

• overwrite_partitions now uses the mechanism each target actually supports: Delta's own partitionOverwriteMode write option for Delta, INSERT OVERWRITE (column-aligned, since it is positional) for an existing table, and Spark's session-level partitionOverwriteMode=DYNAMIC for a path. Every route is covered by an integration test against a real Spark session and a real Delta table.

---

[0.2.0] — 2026-05-20

Added

• Interactive notebook API (ubunye.notebook). New ubunye.notebook() factory returns a NotebookContext for step-by-step task execution in Databricks notebooks. Data scientists can call ctx.read(), ctx.transform(), and ctx.write() in separate cells, inspecting DataFrames between stages. Environment variables referenced via {{ env.VAR }} in config.yaml are auto-resolved from Databricks widgets and secrets — no manual os.environ setup required.

• Public step methods on Engine. Engine.read_inputs(), Engine.apply_transforms(), and Engine.write_outputs() expose the pipeline stages individually for interactive and advanced use cases.

• extract_env_references() utility. Scans raw YAML text for {{ env.VAR }} patterns before Jinja resolution, enabling automatic env-var discovery.

---

[0.1.9] — 2026-05-19

Added

• Formal typing.Protocol interfaces for four pluggable seams. DeployAdapter, RegistryBackend, LineageBackend, and AuthBackend in ubunye.interfaces define structural contracts that backends satisfy without inheriting from a base class. Cross-boundary dataclasses (DeployContext, DeployResult, ModelVersionInfo, LineageRecord, Credentials) accompany each protocol so callers never depend on a concrete backend's internal types.

• Background metadata worker (Decision 1: non-blocking writes). ubunye._internal.MetadataWorker dispatches lineage and registry metadata writes to a daemon thread with a bounded queue (default 1 000, configurable via UBUNYE_METADATA_QUEUE_SIZE). Queue overflow drops the oldest pending write and logs a warning. Flush timeout configurable via UBUNYE_METADATA_FLUSH_TIMEOUT (default 30 s). Uses a worker-thread pattern — async/await fights Spark's threading model.

• Graceful degradation with fallback manifests (Decision 2). When a metadata write fails, the record is appended to ~/.ubunye/fallback/{run_id}/{kind}.jsonl. Pipeline execution continues. Auth failures are excluded — they propagate immediately.

• ubunye sync CLI command. Replays fallback manifests against configured backends with idempotent deduplication (key: run_id + task + recorded_at). Sub-commands: ubunye sync lineage, ubunye sync registry. Processed manifests are archived to ~/.ubunye/fallback/synced/.

• Entry-point discovery for backend groups. Four new entry-point groups in pyproject.toml: ubunye.deploy_adapters, ubunye.registry_backends, ubunye.lineage_backends, ubunye.auth_backends. Third-party packages register backends by adding entry points under these groups.

• Environment-based auto-detection (ubunye._internal.auto_detect). Registry: MLflow on Databricks, filesystem elsewhere. Lineage: Delta when UBUNYE_LINEAGE_TABLE is set, filesystem otherwise. Auth: service principal when both DATABRICKS_CLIENT_ID and DATABRICKS_CLIENT_SECRET are set, token when DATABRICKS_TOKEN is set, raises AuthNotFoundError otherwise.

• Schema evolution (Decision 3). Every cross-boundary dataclass separates strict core fields from a flexible metadata: Dict[str, str]. Core fields never change without a major version bump. Every record stamps engine_version. Extra metadata keys survive round-trip through the filesystem lineage store.

• Interfaces documentation page (docs/interfaces.md) covering protocols, dataclasses, design principles, discovery, auto-detection, fallback manifests, and how to write a custom backend.

• 39 conformance and failure-mode tests covering protocol isinstance checks, entry-point discovery, auto-detect logic, lineage/registry backend surfaces, non-blocking write timing, queue overflow, fallback manifest creation, sync dedup, auth propagation, and schema evolution metadata flexibility.

• ServicePrincipalAuthBackend — OAuth M2M authentication using DATABRICKS_CLIENT_ID + DATABRICKS_CLIENT_SECRET. Entry point ubunye.auth_backends:service_principal. Takes priority over token auth when both are available (via auto-detect).

• MLflowRegistryBackend — model registry that combines filesystem storage with MLflow experiment/run logging. Logs metrics, params, and stage transitions to MLflow when installed; falls back gracefully when MLflow is unavailable. Entry point ubunye.registry_backends:mlflow.

• DeltaLineageBackend — lineage backend targeting Delta tables via Spark SQL on Databricks. Falls back to JSONL-based local storage when no active SparkSession is available, enabling unit tests without Spark. Entry point ubunye.lineage_backends:delta.

• 24 Phase 2 backend tests covering service principal auth (protocol conformance, env var resolution, error cases, entry-point discovery), MLflow registry (full CRUD lifecycle, promotion gates, metadata round-trip), and Delta lineage (record/get/search, metadata preservation, not-found errors).

• Performance benchmark suite (benchmarks/bench_engine.py). Nine benchmarks covering config loading (plain and Jinja), entry-point discovery (cached and cold), lineage I/O (write, read, search), metadata worker throughput, and registry registration. Reports ops/sec, p50/p99 latencies, mean, and stdev. Results saved to benchmarks/results.json for before/after comparison.

Changed

• Filtered entry-point loading. _load_group() in ubunye._internal.discovery now calls importlib.metadata.entry_points(group=group) instead of loading all groups and filtering. Cached discovery is near-zero cost.

• O(1) lineage lookup by run ID. FileSystemLineageStore.get_run() uses a run_id→Path index instead of rglob("*.json"). The index is built lazily on first lookup and updated incrementally on writes. ~34% faster reads.

• In-memory RunContext cache for lineage search. Parsed RunContext objects are cached on first load and reused by subsequent search() and list_runs() calls, avoiding repeated JSON parsing and disk I/O. ~51% faster searches across 100 records.

Fixed

---

[0.1.8] — 2026-05-19

Added

• ubunye deploy databricks command. End-to-end deployment from a local config.yaml to a running Databricks job — handles auth, file upload to /Workspace/, wrapper notebook generation, and idempotent job creation/update via the Databricks SDK. Uses a two-level targets.yaml lookup (usecase-level defaults, task-level overrides). Supports --dry-run for previewing the job spec without deploying. New optional dependency: pip install ubunye-engine[databricks].

• Deploy error types. DeployError base class with AuthNotFoundError, AuthInvalidError, TargetNotFoundError, WorkspaceUploadError, and BundleDeployError — all follow the dual-inheritance pattern (DeployError(UbunyeError, RuntimeError)).

• Structured error messages across the engine. Every user-facing exception now inherits from UbunyeError (with optional context dict and hint string) and the stdlib type it replaces (dual inheritance for backward compatibility). New exception classes: TaskNotFoundError, TaskClassMissingError, ReaderNotFoundError, WriterNotFoundError, TransformNotFoundError, TransformOutputError, MonitorNotFoundError, SourceReadError, SinkWriteError, SparkSessionError, ModelLoadError, ModelNotFittedError, VersionExistsError, VersionNotFoundError, PromotionBlockedError, RegistryNotFoundError, LineageRecordNotFoundError, and ConfigProfileError. Existing config errors (ConfigFieldError, ConfigTemplateError) migrated from ubunye.config.loader to ubunye.core.errors. See the new Error Reference for the full hierarchy and examples.

• Hook failure logging. HookChain now logs a structured warning when a hook's task() or step() context manager raises, instead of silently swallowing the exception.

• CI workflows link to the Databricks job UI after bundle run, so users can find notebook cell output that databricks bundle run does not stream to stdout.

• ubunye init github-actions command. Generates a GitHub Actions workflow for any pipeline — validates config and runs tests on PRs, deploys to Databricks via ubunye deploy databricks on merge to main. Supports --no-deploy for CI-only workflows, --extras for pip install extras (auto-includes Java setup when spark is present), and --target for the Databricks deploy target. The init command is now a sub-app: ubunye init pipeline (formerly ubunye init) and ubunye init github-actions.

Fixed

• Promotion gate failure now propagates. PromotionBlockedError was silently caught by the titanic training task's except ValueError (via dual inheritance). Gate failures now re-raise — CI goes red when a model fails its quality gate instead of reporting green.

• typer[all] replaced with plain typer. Typer >= 0.24 dropped the [all] extra, causing pip warnings on install.

• merged_spark_conf() / resolved_catalog() / resolved_schema() now raise ConfigProfileError when profiles are defined but the requested profile doesn't match. Previously these methods silently returned the base config, hiding typos in --mode / --profile flags.

Changed

• ubunye init is now a sub-app with two subcommands: ubunye init pipeline (the previous ubunye init) and ubunye init github-actions. This is a breaking CLI change — update any scripts that call ubunye init -d ... to ubunye init pipeline -d ....

• [ml] extra no longer installs torch. Use [ml-torch] for PyTorch workloads. This saves ~1 GB of CUDA wheel downloads in CI for sklearn-only pipelines.

• CONFIG.transform.type is now optional. When omitted, the engine defaults to loading the user's Task class from transformations.py — no type: noop declaration needed. Existing configs with type: noop continue to work but emit a DeprecationWarning advising removal. All scaffolded configs, production examples, test fixtures, and docs have been updated to omit the field.

• Unknown config fields are now rejected at load time. All Pydantic models (except IOConfig) use extra="forbid". A typo like ENGNE raises ConfigFieldError with a "Did you mean 'ENGINE'?" suggestion via difflib.get_close_matches. IOConfig retains extra="allow" so plugin-specific keys (REST API headers, pagination, etc.) pass through to connectors. Breaking: configs with unknown top-level or nested fields that were previously silently ignored will now fail.

• Undefined Jinja template variables fail immediately. The resolver now uses StrictUndefined instead of DebugUndefined. A reference to {{ ds }} without passing ds as a CLI variable raises ConfigTemplateError listing available variables. The | default() filter continues to work.

Added

• Service principal + OAuth auth for Databricks CI. The four Databricks example workflows (databricks_deploy.yml, jhb_weather_databricks.yml, multitask_databricks.yml, titanic_ml_databricks.yml) now pass DATABRICKS_CLIENT_ID / DATABRICKS_CLIENT_SECRET alongside the existing DATABRICKS_TOKEN env var. The secrets gate accepts either flow: PAT (HOST + TOKEN) or OAuth (HOST + CLIENT_ID + CLIENT_SECRET). The Databricks CLI auto-selects OAuth when both client vars are set. New reference page docs/databricks-auth.md walks through service principal creation, workspace/UC grants, secret rotation, and verification.

---

[0.1.7] — 2026-04-21

Changed

• MODEL and VERSION are now optional at the top level of config.yaml. MODEL defaults to etl and VERSION defaults to "0.0.0-dev". The semver validator now accepts an optional pre-release suffix (e.g. 1.0.0-rc1, 0.0.0-dev) in addition to plain MAJOR.MINOR.PATCH. Existing configs that set these fields explicitly continue to work unchanged. Rationale: these two lines were boilerplate in every scaffolded pipeline; the defaults cover the common case. Set them explicitly when job type or version is load-bearing (lineage, model registry, orchestrator metadata).

Added

• Production reference example: Titanic multi-task pipeline (local) — examples/production/titanic_multitask_local/ demonstrates sequential task chaining via ubunye run -t clean_data -t aggregate. Task 1 reads the Titanic CSV, cleans it, and writes intermediate Parquet. Task 2 reads that Parquet and computes survival rates by class and age group. Exercises run_pipeline(), sibling-module isolation between tasks, and cross-task lineage. CI workflow (.github/workflows/multitask_local.yml) runs Spark unit tests, validates both configs, runs the full pipeline, and verifies output. See the example's README.md.

• Production reference example: Titanic multi-task pipeline (Databricks) — examples/production/titanic_multitask_databricks/ is the Databricks counterpart of the local multi-task example. Same transformations.py (byte-identical, CI-enforced), but task chaining uses Unity Catalog Delta tables instead of Parquet files. Task 1 writes titanic_cleaned, task 2 reads it and writes survival_summary. Runs via ubunye.run_pipeline() on serverless compute. CI workflow enforces portability diff against the local example and validates/deploys the Asset Bundle.

• Production reference example: Titanic ML end-to-end on Databricks — examples/production/titanic_ml_databricks/ demonstrates the full ML lifecycle: UbunyeModel subclass (sklearn RandomForest), MLflow param/ metric/artifact logging, filesystem-backed ModelRegistry on a UC volume, and a PromotionGate on validation AUC. Two serverless jobs share one Asset Bundle — titanic_train registers and auto-promotes; titanic_predict loads the current production/staging model and writes a Unity Catalog Delta predictions table. A one-row training_metrics audit row is appended per training run. CI (.github/workflows/titanic_ml_databricks.yml) runs pandas/sklearn unit tests, diffs the two model.py copies for drift, and validates/deploys the bundle when Databricks secrets are configured. See the example's README.md.

Changed

• GitHub Actions bumped to Node 24-capable majors. actions/checkout@v4 → @v6, actions/setup-python@v5 → @v6 across all nine workflow files. Resolves the deprecation warning surfaced on 2026-04-16 runs ahead of the Node 20 removal deadline (2026-09-16). No behavioural change.

Fixed

• Undefined CLI template variables silently leaked into resolved configs. resolve_config pre-checked {{ env.X }} references but not bare {{ var }} identifiers. Jinja2's DebugUndefined left unresolved expressions verbatim, so path: "file:///{{ dt }}" with no dt provided would pass validation and then hand Spark a literal file:///{{ dt }} at runtime. Fix adds a post-render residue scan that names the offending variable and suggests a CLI flag, env var, or | default() filter. Regression tests in tests/unit/config/test_resolver.py. Pre-existing configs under examples/ and pipelines/ all use | default() on CLI-derived vars, so no downstream config needs updating.

• Sibling modules leaked between sequential tasks in run_pipeline. _with_task_dir_on_path added the task dir to sys.path but never cleaned up sys.modules on exit. Two tasks that each shipped their own model.py (or utils.py, etc.) would silently run the first task's module when the second task imported it — Python's import cache was keying on the shared short name. Fix evicts only modules whose source file lives under the exiting task dir; stdlib and site-packages are untouched. Regression test in tests/unit/test_task_runner.py. Caught by offline audit on the overnight branch, ahead of the planned multi-task DAG example (tasks/todo/task-04.md).

---

[0.1.6] — 2026-04-15

Changed

• titanic_databricks DAB switched to serverless + Unity Catalog. Removed the new_cluster block and the DBFS bootstrap; the notebook now provisions a UC volume, downloads the Titanic CSV into it at runtime, and the writer emits a Unity Catalog managed Delta table (workspace.titanic.survival_by_class by default). Validated end-to-end on Databricks Free Edition. The portability contract with titanic_local (byte-identical transformations.py) is unchanged — only the deployment wrapper (config + notebook + DAB) differs.
• jhb_weather_databricks DAB switched to serverless compute. Removed the new_cluster block (and the existing_cluster_id escape hatch) from databricks.yml. Notebook tasks with no cluster spec route to serverless on both Free Edition and paid workspaces; the notebook installs ubunye-engine at runtime via %pip. Default weather_catalog changed from main to workspace because Free Edition auto-provisions only the workspace catalog. Paid-workspace users override via --var="weather_catalog=main". README documents the rationale.
• databricks_deploy.yml (titanic) skips deploy gracefully when secrets are absent. Mirrors the soft-skip pattern in jhb_weather_databricks.yml so PRs from forks (or repos that have not configured DATABRICKS_HOST/DATABRICKS_TOKEN) still exercise the unit tests and portability diff instead of failing the workflow.
• Production examples switched from pandas twins to PySpark tests. Every transformations.py under examples/production/ now exposes a single Spark implementation. Tests use a session-scoped SparkSession fixture (local[1], 512 MB driver, shuffle partitions=1) so the production code is the code under test. Eliminates the dual-maintenance burden and the risk of silent drift between pandas and Spark paths. Unit-test CI steps now install Java 17 + ubunye-engine[spark,dev] on the runner.

Added

• Production reference example: JHB hourly weather (REST API → Unity Catalog) — end-to-end example at examples/production/jhb_weather_databricks/ demonstrating REST ingestion with the rest_api reader against the free Open-Meteo API (lat/lon for Johannesburg, no auth required), a Spark transform that explodes parallel hourly arrays into a tidy one-row-per-hour DataFrame, and a Unity Catalog Delta writer partitioned by forecast_date. Ships a Databricks Asset Bundle with a scheduled daily job (06:00 Africa/Johannesburg), a notebook wrapper around ubunye.run_task(), seven pandas unit tests over a hand-built fixture response, and a CI workflow (.github/workflows/jhb_weather_databricks.yml) that runs the tests, smoke-checks the endpoint, and validates/deploys the bundle when Databricks secrets are configured. See the example's README.md.

• Production reference example: Titanic (local runtime) — end-to-end example at examples/production/titanic_local/ demonstrating a CSV → Parquet pipeline with dev/prod profiles, Jinja-templated paths, pandas unit tests (no Spark), a committed golden Parquet, and a GitHub Actions workflow (.github/workflows/local_pipeline.yml) that validates config, runs the pipeline on a real SparkSession, and diffs the output against the golden. One half of the portability demo — the Databricks counterpart shares transformations.py verbatim. See the example's README.md.

• Production reference example: Titanic (Databricks Community Edition) — the Databricks half of the portability demo at examples/production/titanic_databricks/. Ships a Databricks Asset Bundle (databricks.yml) sized for CE's single-node / DBFS / no-UC constraints, a notebook entry (notebooks/run_titanic.py) that calls ubunye.run_task() against the active SparkSession, the same pandas unit tests, and a CI workflow (.github/workflows/databricks_deploy.yml) that installs the Go-based Databricks CLI, validates and deploys the bundle, and enforces the portability contract by diffing transformations.py against the local example. Known CE limitations (no service principals, restricted Jobs API, DBFS deprecation) are documented honestly rather than worked around.

• Cross-runtime reference index — examples/production/README.md explains the portability contract, provides a side-by-side config comparison of the two examples, a decision guide for choosing between the local and Databricks runtimes, and a migration table covering what changes when moving from Community Edition to a standard Databricks workspace.

• Hook abstraction for observability (ubunye/core/hooks.py) — Hook base class and HookChain multiplexer. Tasks and steps are now wrapped in hook context managers so the Engine no longer imports telemetry modules directly. Built-in hooks shipped under ubunye/telemetry/hooks/: EventLoggerHook, OTelHook, PrometheusHook, LegacyMonitorsHook. Third parties can register custom hooks (Slack alerts, audit logs, drift checks) without modifying the Engine. See docs/patterns/hooks.md.

• ubunye.hooks entry point group (pyproject.toml) — third-party packages can register Hook subclasses as entry points and have them auto-discovered by the Engine. The three built-in telemetry hooks (events, otel, prometheus) are registered via this mechanism and gated on UBUNYE_TELEMETRY=1.

• Python API (ubunye/api.py) — run_task() and run_pipeline() for running Ubunye tasks from Python code (Databricks notebooks, scripts, tests) without the CLI. Auto-detects and reuses active SparkSessions. Exported from ubunye.__init__.

• DatabricksBackend (ubunye/backends/databricks_backend.py) — backend that wraps an existing SparkSession instead of creating one. stop() is a no-op since we don't own the session.

• Dev notebook scaffolding — ubunye init now generates notebooks/<task>_dev.ipynb alongside config.yaml and transformations.py. The notebook uses DatabricksBackend, dbutils.widgets, and display(). The Load step is commented out by default.

• Deployment docs — docs/deployment.md covering Databricks Asset Bundles pattern, GitHub Actions CI/CD, and Python API on Databricks. DABs belong in the usecase repo, not the engine.

• Deploy workflow — .github/workflows/deploy.yml validates configs on PR and runs unit tests. Bundle deployment is handled in the usecase repo.

• ubunye test run CLI sub-command — runs tasks with a test profile and reports PASS/FAIL.

• Model Registry (ubunye/models/) — library-independent ML lifecycle management.

• UbunyeModel abstract contract: train, predict, save, load, metadata, validate.
• ModelRegistry — filesystem-backed versioning with stages: development → staging → production → archived.
• PromotionGate — configurable metric thresholds (min_*, max_*, require_drift_check).
• load_model_class() — dynamic model file importer; mirrors the task-dir import pattern.
• ModelTransform plugin (type: model) — train and predict from config YAML.
• ubunye models CLI sub-commands: list, info, promote, demote, rollback, archive, compare.

• RegistryConfig and ModelTransformParams Pydantic schema additions.

• Lineage tracking (ubunye/lineage/) — automatic run provenance.

• RunContext, LineageRecorder, FileSystemLineageStore, hash_dataframe.
• ubunye lineage CLI sub-commands: show, list, compare, search, trace.

• --lineage flag on ubunye run.

• REST API connector — paginated HTTP reader and writer.

• Pagination strategies: offset, cursor, next_link.
• Auth: bearer, api_key (header or query param), basic.
• Rate limiting with configurable requests_per_second.
• Exponential backoff retry on configurable status codes.

• Optional explicit schema declaration.

• Config validation — ubunye validate command with full Pydantic v2 schema.

• Format-specific field validation in IOConfig.
• Jinja2 rendering before Pydantic validation.

• Semver validation on VERSION field.

• ubunye export airflow|databricks CLI — the AirflowExporter and DatabricksExporter under ubunye/orchestration/ are now reachable from the command line. The command loads the task's config.yaml, pulls defaults from its ORCHESTRATION block, and writes the artifact to --output. Airflow emits a DAG Python file; Databricks emits a Jobs API job.json. Classes are now exported from ubunye.orchestration.__init__.

• Test infrastructure — 288 unit tests, all Spark-free in tests/unit/.

Changed

• databricks_expoter.py renamed to databricks_exporter.py (typo fix). Not previously exported from ubunye.orchestration, so external callers are unaffected.

• Unified execution path (ubunye/core/task_runner.py) — api.py, cli/main.py run and cli/test_cmd.py run previously each reimplemented the read → transform → write loop and called load_monitors / safe_call directly. They now delegate to execute_user_task(), which wraps the user's Task.transform() as an ephemeral Transform plugin and runs it through Engine. Single code path, single hook lifecycle, MonitorHook adapts the legacy lineage recorder to a Hook. Engine.__init__ gained extra_hooks= (append to defaults) and manage_backend= (caller-owned vs engine-owned backend lifecycle). run_task / run_pipeline accept hooks= for notebook callers who want to swap in custom hook chains.

• Engine runtime refactored — ubunye/core/runtime.py reduced from 374 to 255 lines. Engine.run() body shrank from ~220 lines to ~35 by delegating telemetry plumbing to hooks. The engine no longer imports from ubunye.telemetry.* — only from ubunye.core.hooks. UBUNYE_TELEMETRY and UBUNYE_PROM_PORT env vars still honored; user monitors in CONFIG.monitors continue to work via LegacyMonitorsHook. Engine.__init__ gained an optional hooks= argument.
• ubunye/config/schema.py — added RegistryConfig, ModelTransformParams, FormatType.REST_API.
• ubunye/__init__.py — exports run_task and run_pipeline.
• ubunye/cli/main.py — mounted models_app, lineage_app, test_app Typer sub-apps; added notebook scaffolding to init.
• pyproject.toml — added model entry point under ubunye.transforms.

Fixed

• Unity Catalog writer collapsed plugin dispatch with Spark source format. UnityTableWriter.write() was reading cfg["format"] and passing it to Spark's DataFrameWriter.format(...). But cfg["format"] is the Ubunye plugin selector and is always "unity" by the time the writer runs, so Spark raised [DATA_SOURCE_NOT_FOUND] Failed to find the data source: unity. Switched the writer to read cfg["file_format"] (defaulting to delta), matching the convention already used by the s3 writer. Caught while deploying the jhb_weather_databricks example to a serverless workspace.

---

[0.1.0] — 2025-09-11

Added

• First alpha release of Ubunye Engine.
• Config-first ETL framework built on Apache Spark.
• Plugin system for Readers, Writers, and Transforms via Python entry points.
• Built-in connectors: Hive, JDBC, Delta, Unity Catalog, S3, binary.
• CLI commands: init, run, plan, config, plugins, version.
• Orchestration exporters: Airflow DAG Python file, Databricks Jobs API JSON.
• Internal ML wrappers: SklearnModel, SparkMLModel, BatchPredictMixin, MLflowLoggingMixin.
• Telemetry modules: JSON event log, Prometheus, OpenTelemetry.
• Example tasks: fraud_detection/claims/claim_etl, rest_api/customer_sync.
• SparkBackend with context manager and safe multiple-start support.