Ceph clusters as first-class resources in Kunobi

Plenty of teams run Ceph next to their Kubernetes clusters. It stores the block volumes their pods mount, the objects their apps write, and the shared filesystems in between. Checking its health, or how close it is to full, means leaving Kubernetes and going somewhere else.

The state you need is spread across three tools. Cluster health and capacity live in the Ceph Manager Dashboard. The per-disk and per-pool detail comes fastest from the ceph CLI. Alerts are in Prometheus and Alertmanager. None of that is in the Kubernetes resource browser most platform engineers keep open all day, so Ceph ends up as a separate tab you check by hand.

The Ceph extension pulls that state into Kunobi. It reads the Manager Dashboard's REST API and renders the result through the same drilldown tables we already use for Pods and Deployments, so clusters, OSDs, pools and block images become resource kinds you browse like any other.

What Ceph is

If you have not run it: Ceph is an open-source distributed storage system. It runs on ordinary servers, pools their disks into one cluster, and serves three kinds of storage from them:

• block devices (RBD)
• S3-compatible object storage
• a shared filesystem (CephFS)

Data is broken into objects, replicated or erasure-coded for durability, and placed across the disks by an algorithm called CRUSH, so there is no central lookup table to become a bottleneck or a single point of failure. A few terms recur through the rest of this post:

• OSD: the daemon that manages one disk
• Pool: a logical bucket of data with its own replication rules
• Monitors: they keep the authoritative map of where everything lives

Connecting your clusters

Everything starts on the Sources screen, under Ceph Clusters. A connection is a name, the Manager Dashboard URL, and a username and password. The password goes into the operating system keychain, not a config file on disk. Self-signed certificates get a per-connection toggle, since most on-cluster deployments serve them.

The Ceph Clusters screen is a list, not a single connection.

You can register a production cluster, a lab cluster, and one per region, and they all live here, with a keystroke to switch which one is active. The Test action checks the credentials against the dashboard before anything is saved, so a wrong URL or a rotated password shows up straight away instead of as an empty screen later. The Status column reflects the live health of whichever cluster is active.

Adding one opens a form that gathers the same pieces in one place: the credential fields, the untrusted-certs checkbox, and a Test Connection button sitting next to Add.

Health and capacity, at a glance

With a connection active, the Cluster view answers the first thing you want to know. Two things on the row carry it:

• Health: Ceph's own verdict, either HEALTH_OK, HEALTH_WARN or HEALTH_ERR
• Used, Total and Usage: capacity, with bytes shown in binary units alongside a percentage

The FSID is on the row too, which uniquely identifies the cluster when two of them share a similar name and you need to be sure which one you are about to change. In the above example capacity sits at 36%, and Health still shows HEALTH_WARN. At least that rules out free space as the cause. To find the root cause, you drill down.

OSDs: where the bytes actually live

A cluster-wide capacity figure is an average, and the average looked fine. Ceph spreads data across OSDs, one per storage device, and behind that 36% there is a single OSD, osd.6, at 93%. Once an OSD fills up it stops accepting writes, however much room the rest of the cluster has. The OSD view lists all of them.

Each row has the OSD id, its host, its up/in state, and its own used, total and usage. Up means the daemon is running; in means Ceph is currently placing data on it. They are tracked separately because an OSD that is up but not in has usually been drained for maintenance, while one that is down but still in is a fault the cluster is routing around. Sort by usage and the disk closest to trouble comes to the top.

Pools: layout, durability, and cost

A pool holds Ceph's durability and cost settings, and the Pool view puts them on a single line each.

Reading across a row: the pool type, replicated or erasure; its applications; the placement-group count; the replication factor (Size, 3× or 6×) and the minimum copies required for writes (Min size); the autoscale mode; compression; usage; and the CRUSH rule and erasure-code profile that decide where the data lands. Here rbd runs aggressive compression, rgw.buckets is erasure-coded with a jerasure profile, and the other three are three-way replicated. A pool built with two replicas instead of three would show up in the Size column without any digging.

Pools are editable from here as well. Creating one, changing its parameters, and removing it are all available without leaving the desktop app.

Block storage: thin-provisioned by default

The block side of Ceph is RBD, the RADOS Block Device, and it backs most of the Kubernetes persistent volumes and VM disks that run on Ceph. The RBD Images view lists images across pools.

RBD images are thin-provisioned, so Used and Total rarely match, and the difference is worth keeping an eye on. k8s-pv-registry holds 640 GB inside a 1 TB image; vm-disk-01 holds 90 GB inside 200 GB. The provisioned size is space the cluster has committed to even where nothing is written yet, so it is useful to see it next to real usage. Each row also shows the pool, the enabled RBD features (layering, exclusive-lock, object-map and so on), the format, the snapshot count, and the parent image for clones. Snapshots, the trash, and namespaces have their own views, and the usual image and snapshot operations (create, resize, clone, trash, restore) run from here instead of the CLI.

Alerts, next to the cluster they came from

When a Ceph cluster exports its alerts through Prometheus, the Manager Dashboard picks them up and the extension reads them from its API, so they turn up in Kunobi rather than only in Alertmanager.

Each alert carries:

• its name and the instance it fired on
• its severity
• its state: firing, pending or inactive
• a short summary

This is where the earlier HEALTH_WARN gets specific: CephOSDNearFull is firing on osd.6, the same disk that read 93% in the OSD view, alongside a predicted device failure still pending on another node. The firing versus pending state says whether an alert has passed its for duration or is still inside it, a detail a plain notification tends to drop. The alerting rules and silences have their own views, so what would fire and what has been muted are visible too.

What it adds up to

Mostly the extension is a read view of Ceph, rendered as sortable tables in the drilldown you already use for Kubernetes. It surfaces cluster health, capacity, OSDs, pools, block images and alerts. It also handles a handful of writes: the pool and RBD operations common enough to be worth keeping within reach. It ships as a Kunobi Pro feature, and it leans read-only by default.

The practical effect is one less place to look. Storage health views live in the same window as the workloads that depend on it.