kobe 101: lease a Kubernetes cluster instead of creating one

Ask who owns a given test cluster and you often get a pause. It was created for a pipeline that finished weeks ago, or by somebody who has since moved teams, and it is still running because deleting it is a decision nobody wants to get wrong. Tooling competes hard on how fast you can get a cluster, and much less on what happens to it afterwards, which usually comes down to whoever remembers.

There is a real difference between a cluster you create and a cluster you borrow. Create one and you own it, along with everything that follows: deleting it, noticing when it leaked, remembering what it was for. kobe approaches the problem from the other end. You ask for a cluster, you get one that already exists, and it arrives with an expiry that the system enforces whether or not your job exits cleanly.

kobe is an operator you run inside a host Kubernetes cluster, written in Rust on kube-rs and axum. Its job is to have spare clusters standing by before anybody asks for one, reachable through a small HTTP API and a CLI. We build it alongside Kunobi and run our own CI on it. From the caller's side it is a single command:

kobe lease ci-small --ttl 30m

That prints a lease ID and an expiry, writes a kubeconfig to disk, and the cluster is usable by the time the command returns.

Kobe cattle are the most pampered animals on earth, roughly the opposite of how a test cluster should be treated. That inversion is where the name comes from: the old operations line is cattle, not pets. A cluster in the pool isn't something you nurse back to health, it's something you replace. The pool is what makes replacing one cheaper than saving it.

Pools and leases

A ClusterPool is a description of the pool you want to exist. It says which backend to use, which Kubernetes version, how big each cluster is, how many should be sitting ready, and how long a lease may last by default.

apiVersion: kobe.kunobi.ninja/v1alpha1
kind: ClusterPool
metadata:
  name: ci-small
spec:
  size: 3                 # warm clusters to keep ready
  ttl: "1h"               # default lease duration
  backend:
    type: k3s             # k3s | k0s | vcluster | capi
  cluster:
    version: "v1.31.3+k3s1"
    servers: 1

That pool holds a fixed three. Adding a scaling block instead replaces size entirely: the pool then tracks minReady and grows toward maxClusters as leases queue up, and size is ignored from that point on.

The operator's job is to keep reality matching it, continuously. This is the reason kobe is an operator rather than a provisioning script: nothing about a pool is a one-shot action. A pool is a target the controller keeps converging on, so drift away from it (a cluster that died, a member that never came up, a burst of demand) is corrected by the same loop that filled the pool in the first place.

Asking for a lease does not trigger a build. kobe binds the caller to an instance already sitting in Ready, mints a kubeconfig for it, and returns. The provisioning happened in the background minutes earlier, which is why the handoff takes seconds.

Release is where the model parts company with most pooling. Handing a cluster back does not put it into circulation again; the operator tears it down and starts building another one to take its place. That costs more than reuse would, and what it buys is that nothing from one lease can survive into the next, so there is no state left over for the following caller to trip over.

None of that is hidden state. The pool publishes its counters as printer columns, so a pool of three at rest looks like this:

kubectl get clusterpools

NAME       PHASE     READY   LEASED   CREATING   FAILURES   AGE
ci-small   Healthy   3       0        0          0          6d

Take a lease and the same command shows the pool one member down and already refilling:

NAME       PHASE     READY   LEASED   CREATING   FAILURES   AGE
ci-small   Healthy   2       1        1          0          6d

ClusterInstance defines no printer columns of its own, so the per-member view has to ask for the phase:

kubectl get clusterinstances -o custom-columns=NAME:.metadata.name,PHASE:.status.phase

NAME              PHASE
pool-ci-small-0   Leased
pool-ci-small-1   Ready
pool-ci-small-2   Ready
pool-ci-small-3   Creating

pool-ci-small-3 is the replacement, and it is being built while the lease is still in use rather than after it ends. When the lease finishes, pool-ci-small-0 is destroyed instead of going back into the pool.

Expiry is enforced by the operator

Every lease carries a TTL, and the operator is what enforces it. The usual alternatives all depend on the client behaving: a cleanup step at the end of a pipeline, or a cron job that deletes namespaces older than a day. Because the countdown runs server-side instead, a killed runner or a cancelled pipeline makes no difference to whether the cluster is destroyed on time.

Leases can be extended, but not indefinitely. Extensions are capped per policy (maxExtensions, two by default), so "just give it one more hour" cannot quietly promote a lease into a long-lived cluster. Health checks work in the same direction: a warm member that fails its probe as many times as the pool's failureThreshold allows goes straight to recycling rather than being handed to the next caller. Beyond the counters above, pool status also tracks recycling, unhealthy and queueDepth.

What warm actually means

Skipping the boot is the obvious saving, though booting is rarely the expensive part of getting a test environment ready. What a test needs is almost never an empty cluster. It is a cluster with your GitOps controller running, your CRDs installed and your operator's dependencies present. If every run installs that stack itself, the job spends the time it saved on cluster creation waiting for addons instead.

A BootstrapConfig is a reusable bundle stored in the host cluster: manifests applied in lexical order, or a Job that runs the real tooling (flux install, helm upgrade --install, kustomize build | kubectl apply). A pool references the bundles it wants, and an instance does not reach Ready until they have completed. Pool members therefore warm up all the way to the state you care about, and then wait there.

Our own CI pool is built that way. Members come up with Flux, its sync configuration, Argo CD and a demo Application already applied, so a lease arrives with GitOps running rather than blank. For heavier stacks, where installing the addons is genuinely slow, kobe can keep a Velero backup of a golden cluster and restore new members from it, which pulls the creating phase back from minutes to seconds.

Who is allowed to lease

The usual CI story is a kubeconfig or a service account token sitting in a secret. It was created once, scoped to whatever seemed reasonable that day, shared with whoever needed it, and rotated when someone remembered.

kobe authenticates the caller instead. An AccessPolicy configures one authentication method, either OIDC (GitHub Actions and any other provider), SSH agent keys, a bearer token from a Secret, or an in-cluster ServiceAccount, and then turns the resulting identity into authorization: which pools that identity may use, the maximum TTL it may request, how many leases it may hold at once, how many times it may extend.

apiVersion: kobe.kunobi.ninja/v1alpha1
kind: AccessPolicy
metadata:
  name: github-ci
spec:
  auth:
    oidc:
      issuer: https://token.actions.githubusercontent.com
      audience: ["https://kobe.example.com"]
  identity: "{repository}:{ref}"
  rules:
    - match:
        claim: repository
        value: my-org/my-repo
      pools: ["ci-*"]
      maxTtl: 1h
      maxConcurrentLeases: 3

The kubeconfig handed back is minted per lease, with a client certificate that expires when the lease does. No long-lived credential is issued at any point, so there is none to leak, rotate or accidentally commit. For CI that means no cluster credential in the repository at all: the OIDC token the platform already mints is the credential, and the policy names the repository and ref allowed to use it.

One fleet layer, several runtimes

The lease API is the stable surface. What actually provisions the cluster underneath is deliberately pluggable, and pools, admission, TTL enforcement and observability behave the same regardless of which backend is in play.

Need	Backend	Status
Nested real clusters	k3s, k0s	Production path today
High density, low overhead	vcluster	Adopted
Provider-managed real infrastructure	capi	Available
Full isolation and API fidelity	pkobe	In design

Being pluggable does not make the backends equivalent. The vcluster backend inherits vcluster's fidelity boundaries, so cloud identity such as IRSA, syncer coverage and version skew all behave as they do there. Tenants that need full cluster semantics belong on a nested real cluster today, and on pkobe once it lands. We also had our own proxy-based syncer runtime in the tree, kobe-sync, and put its development on hold: maintaining translation fidelity for the entire Kubernetes resource surface is a product in itself, and the fleet layer is where we think kobe's value actually is.

Where we run it

Kunobi's end-to-end suite leases a cluster for every CI run. The workflow calls kobe-action, which authenticates with GitHub's OIDC token against a policy scoped to the repository and ref, waits until the cluster's nodes report ready rather than merely until the lease is bound, and releases the lease from its post hook on every exit path, success, failure and cancellation alike. Nothing has to be torn down afterwards, including when somebody cancels a job halfway through.

What that replaced was booting a cluster inside the runner on every job, which also meant running Docker-in-Docker on the runner purely to host a control plane. Both of those are gone now. The control plane lives on infrastructure that was already up, and the runner no longer has to host one.

It is not free of operational edges. A pool member occasionally fails to reach ready inside the window we allow it, and the workflow handles that by leasing a fresh member instead of failing the run. It is worth knowing that before writing a suite which assumes the first lease always succeeds.

When kobe is not the answer

kobe runs inside a host Kubernetes cluster, so the problem it addresses is a fleet-sized one. A single developer who needs a single cluster in front of them is well served by kind or k3d and does not need any of this. It starts to pay off when several people or several pipelines want clusters at the same time and somebody has to own the supply.

A nested cluster is a real Kubernetes cluster, though it does not reproduce everything a managed one gives you. Anything that depends on provider identity, provider-specific CSI, or the exact behavior of a managed control plane should run on real infrastructure through the CAPI backend rather than on a nested pool member.

And leases are for environments that end. A staging environment that lives for a quarter should be managed as a cluster, with the ownership and lifecycle that implies, rather than squeezed into a lease and extended forever.

Getting started

The source lives on GitHub under Apache-2.0. You install the operator from its Helm chart, distributed as an OCI artifact, and the CLI is published on crates.io as kobectl, which installs a binary called kobe. Prebuilt release binaries, the AUR, Scoop and Chocolatey all carry it too. The documentation covers pool configuration, the authentication methods and the full API, and the product page has the shorter version. The quick start goes from an installed operator to a first lease if you want to try it directly.