Skip to content

Language models

These examples live in the companion repository ubunye-ai-ecosystems/ubunye-examples. Run the commands from a clone of it (setup per place).

Working with LLMs inside ordinary pipelines. Two examples: retrieval augmented answers with sources cited, and teaching a small model you own from a large model's labels.

Both run in two ways, chosen by one environment variable. On Databricks they use the workspace's hosted model endpoints. Everywhere else, MODEL_BACKEND=local swaps in open source models that run on a CPU with no endpoint and no API key. The pipelines do not change. Which model answers is business logic, and it lives in one Python file.

05 · RAG, answers with receipts

Chunk documents, embed them, find the chunks relevant to a question, and answer using only what was found, citing the sources. Needs example 03's chunks first.

Run 03 first, then the two notebooks under examples/05_rag_documents/notebooks/, or:

databricks bundle run rag_documents --target dev

pip install sentence-transformers transformers torch --extra-index-url https://download.pytorch.org/whl/cpu
MODEL_BACKEND=local \
platforms/run_task.sh examples/05_rag_documents rag knowledge embed_chunks answer_questions

The first run downloads two small models from Hugging Face, about one gigabyte total. After that it is fully offline.

The ml image has the weights baked in, so this works with no internet:

docker run --rm --entrypoint /app/platforms/models_demo.sh ubunye-ml:ci

kubectl apply -f platforms/k8s/job-models.yaml
kubectl logs -f job/ubunye-rag

06 · Fine-tune an open LLM

Distillation, end to end. A large model labels customer reviews once, then a small DistilBERT you own learns from those labels, gets judged on held-out data, and must pass a quality gate before it is registered. The result costs nothing per call and no one can rate limit it.

databricks bundle run finetune_llm --target dev

Fine-tuning on CPU takes about ten minutes. That is the slowest example in the repository.

MODEL_BACKEND=local STUDENT_MODEL_STORE=/tmp/ubunye/model_store \
platforms/run_task.sh examples/06_finetune_llm reviews nlp label_reviews finetune_classifier

docker run --rm -e WITH_DISTILLATION=true \
  --entrypoint /app/platforms/models_demo.sh ubunye-ml:ci

platforms/k8s/job-models.yaml runs example 05 only. To add the distillation step, add this entry to the container's env: list in that file (next to MODEL_BACKEND):

            - name: WITH_DISTILLATION
              value: "true"

Then apply it and follow the logs:

kubectl apply -f platforms/k8s/job-models.yaml
kubectl logs -f job/ubunye-rag

Give the pod time: the student trains on CPU. A Job's pod settings cannot be changed after it is created, so if ubunye-rag already exists, delete it first with kubectl delete job ubunye-rag.