Where Does My Traffic Go? Services, Selectors, and Readiness in Kubernetes

We have a Service. kubectl get svc shows it with a ClusterIP and a port, the Deployment behind it reports the right number of replicas, every pod is Running, and nothing in the cluster is throwing an error. Requests to the Service still get nothing back. No connection refused, no 503 from an upstream, no crash loop to grep for. Just a request that hangs or returns empty, against a Service that looks completely healthy.

This is one of the more disorienting failures in Kubernetes, and the reason is worth sitting with. A Service is the object most people treat as the thing traffic goes to. So when traffic does not arrive, the Service is the first place we look, and the Service is exactly where the problem refuses to show itself. The object is fine. What is broken is a relationship between the Service and its pods, and that relationship does not live on the Service object at all.

This post is about that relationship: how a Service decides which pods receive traffic, why a perfectly valid Service can route to nothing, and how to look at the set of pods behind a Service directly instead of inferring it.

How a Service finds its pods

The first thing to unlearn is that a Service holds a list of pods. It does not. A Service holds a label query, called a selector, and Kubernetes evaluates that query continuously to decide which pods sit behind the Service at any given moment. If you label three pods app=web and give the Service the selector app=web, those three pods are the backends. Start a fourth pod with the same label and it joins the set on its own. Delete one and it drops out. Nobody edits the Service to make this happen.

The match itself is deliberately blunt. A Service selector is a flat set of equality conditions, every one of which a pod must satisfy to be included, with no "or", no negation, and no ranges. That bluntness is the point. It keeps the Service decoupled from the identity of any single pod: the Service never names a pod, never stores a pod IP, and does not notice that pods are being created and destroyed underneath it. It describes a shape, and whatever currently fits the shape is what receives traffic. Pod IP addresses change every time a pod is rescheduled, which is exactly why the Service is defined in terms of labels rather than addresses in the first place.

The result of that query has to be stored somewhere the network layer can read it, and that somewhere is a separate object. For each Service with a selector, Kubernetes maintains a set of EndpointSlices listing the IP addresses currently backing the Service. This is the list kube-proxy reads when it programs the routing rules on every node. The older Endpoints object holds the same information in a single, simpler shape, and newer clusters shard it across EndpointSlices for scale. You can inspect either one with kubectl get endpoints or kubectl get endpointslices, and for understanding the behaviour the difference does not matter. What matters is that the routing list is computed from the selector and kept somewhere other than the Service.

The selector is also optional, and that turns out to be a useful thing to know. You can create a Service with no selector at all and then write the backing addresses yourself, which is how people point a stable in-cluster name at a database living outside the cluster, or at a fixed external IP. That this is even possible tells you the routing list and the Service are genuinely separate mechanisms rather than two views of one thing. When a selector is present, a controller in the control plane does the writing: it watches pods, evaluates the selector, and reconciles the EndpointSlices so they always reflect the pods matching right now. When the selector is absent, you have taken that controller's job.

Once you see it this way, the silent failure stops being mysterious. The ClusterIP a Service hands out is a virtual address that no pod owns and no process listens on directly. It means something only because kube-proxy, running on every node, turns it into a rule that rewrites traffic toward one of the real pod addresses in the routing list. The Service name and its IP are stable; the list they resolve to is rebuilt from labels continuously. A selector that matches no pods leaves kube-proxy with an empty list and nothing to rewrite toward, so connections aimed at a perfectly healthy-looking ClusterIP have nowhere to land. Everything about the object reads healthy because nothing about the object is wrong.

Readiness decides who gets traffic

Matching the selector gets a pod considered, not selected. A pod only enters the routing list once it is Ready, and until then it is matched but deliberately held out. This is the whole purpose of a readiness probe. While a pod is still starting up, or while it is reporting itself unhealthy, Kubernetes keeps it out of rotation so that traffic skips it. The pod is not deleted and not restarted. It is simply not in the list of addresses that receive requests.

This is also where two probes that look alike do very different jobs. A liveness probe decides whether the container should keep running at all, and failing it gets the container killed and restarted. A readiness probe decides only whether the pod should receive traffic at this moment, and failing it changes nothing about the pod's lifecycle. The pod keeps running, keeps its place in the set, and waits to be let back in. Reaching for the wrong one is a common way to turn a few seconds of unreadiness into a restart loop, or to keep aiming requests at a pod that has already told the cluster it cannot serve them.

The part that catches people is what this looks like from the outside. An unready pod is still in the Running phase. It still appears in kubectl get pods. It still carries the label the selector is looking for. By every signal you would normally glance at, the pod is present and serving. The routing layer has quietly removed it, and that fact is recorded on the EndpointSlice, not on the pod and not on the Service. The information you need to explain the missing traffic is one object removed from any of the objects you were looking at.

Each endpoint also carries a little more state than a single Ready flag. An EndpointSlice tracks separate conditions for whether an endpoint is ready, whether it is still serving, and whether it is terminating, which is what lets a pod that is shutting down stop taking new traffic while it finishes the requests already in flight. And because that state has to propagate to the kube-proxy on every node before it takes effect, there is always a small lag between a probe flipping and the routing rules catching up. That lag is why a rollout can produce a handful of connection errors even when nothing is misconfigured: for a moment, some node is still holding an address that is no longer Ready.

Let's break it

Rather than walk through a catalogue of failures, we will take one Service through a single session and watch the set of pods behind it change as we make a few ordinary mistakes and fix them. The cluster is a local kind cluster, the app is three nginx pods, and we will watch the whole thing in Kunobi.

We start by deploying the app and creating a Service, except the Service selector has a typo in it. The pods are labelled app=web and the Service is looking for app=web-typo. Nothing complains. The Service comes up with a ClusterIP and a port, and its overview looks exactly like a working Service would.

The selector is right there in the overview, which is the first useful thing: we can read what the Service is looking for without going to fetch the YAML. To find out what that query actually returns, we follow the Service to its pods. Kunobi runs the Service's selector and lands us on the matching set, and the matching set is empty.

No pod is broken. The Deployment is healthy and its three pods are Running. The Service is healthy. The query connecting them matches nobody, and because that is the only thing wrong, it is the one thing none of the individual objects will tell you. We fix it by lining the selector up with the pods' real label, app=web, follow the Service again, and this time land on the three running pods, each Ready.

Now we introduce the second, subtler problem. We add a readiness probe that never passes, an HTTP check against a path nginx serves a 404 for. The pods keep running, the probe keeps failing, and the pods settle at 0/1. Following the Service still lists them, because they still match app=web, but the Ready column tells the real story.

This is the case from the theory section made concrete. The pods are matched and excluded at the same time, and if we look at the Endpoints object backing the Service it has no addresses to hand out: the pods exist, they match, and the routing list is empty because none of them are Ready.

Finally, we make the pods healthy again and scale the Deployment. The set behind the Service is a live view rather than a snapshot we asked for, so as the Deployment adds replicas they appear in the Service's pod set the moment they go Ready.

Scale back down and they drop out of the set just as directly, without us refreshing or re-running anything.

Every problem in that session was invisible on the Service object itself, and every one of them became obvious the moment we could ask a single question: which pods are behind this Service right now, and are they Ready? That question spans three objects in the API. Being able to answer it in one step is most of the work.

Closing

A Service is a stable name in front of a set that Kubernetes rebuilds constantly out of two inputs: which pods match the selector, and which of those pods are Ready. When traffic disappears and the Service looks fine, the Service is usually not where the answer is. The answer is in who currently passes that query, and the habit worth building is to go look at that set directly rather than reason about it from the pieces.