Skip to content

Deploying on Kubernetes

This page shows how to run pipelines on a Kubernetes cluster. It is for anyone with a cluster and kubectl pointed at it.

There are two ways. Each builds its image differently:

Way Image built with Status
ubunye deploy k8s (the engine's command) ubunye deploy dockerfile container Proven on kind (a small Kubernetes that runs inside Docker)
The companion repo's manifests (this page) the companion repo's platforms/docker/Dockerfile Run in that repo's CI on a throwaway cluster

The shortest way is the engine's own command: ubunye deploy dockerfile container writes an image's Dockerfile, and ubunye deploy k8s runs a task as a Kubernetes Job in kubectl's current context. See One command to the cloud. The proving ground runs its workloads this way on a kind cluster.

The rest of this page uses the scripts and manifests in the examples repository (platforms/docker/, platforms/k8s/), not this one. They are exercised in CI there against a disposable cluster. Run the commands below from a clone of it.

The shape

A pipeline on Kubernetes is a container image plus a Job manifest. The image carries Spark, Delta, the JDBC driver and the engine. Your task folders are copied in unchanged. Nothing in any pipeline knows it is inside a container.

1. Build the image

This path uses the companion repo's Dockerfiles, not ubunye deploy dockerfile.

docker build -f platforms/docker/Dockerfile -t ubunye-portable:latest .

For machine learning pipelines there is a second image that adds the open source models with the weights already inside, so pods need no internet:

docker build -f platforms/docker/Dockerfile.ml \
  --build-arg BASE=ubunye-portable:latest -t ubunye-ml:latest .

Push whichever you need to the registry your cluster pulls from.

2. Run a task as a Job

platforms/k8s/job-template.yaml takes any task:

sed -e "s|{{NAME}}|my-pipeline|" \
    -e "s|{{ARGS}}|\"examples/02_ingest_rest_api\", \"weather\", \"ingestion\", \"hourly_forecast\"|" \
    platforms/k8s/job-template.yaml | kubectl apply -f -

The environment variables in the manifest are the whole configuration surface: where the data lives, what kind of table to write, and any secrets your connectors need. Mount them from Kubernetes Secrets in the usual way.

3. Run it on a schedule

Wrap the same spec in a CronJob:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: ubunye-nightly
spec:
  schedule: "0 2 * * *"
  jobTemplate:
    spec:
      # the same template as the Job, unchanged

Two honest notes

Storage. A pod's disk vanishes with the pod. Point UBUNYE_DATA_ROOT at object storage or a persistent volume, and give the model registry a mounted path or an s3:// style location, which it supports directly since 0.5.0.

One pod or many. The manifests run Spark in local mode inside one pod, which is the honest shape for most scheduled jobs. To fan out across executor pods instead, set SPARK_MASTER to your cluster's API address (k8s://https://kubernetes.default.svc), give the job a service account that may create pods, and set spark.kubernetes.container.image. The task folders do not change: the launcher owns the master, as the spark.master rule says.