Service discovery

Service discovery is how a caller turns a service’s name, such as restaurant, into the address and port of a healthy instance it can call right now. With autoscaling and containers, instances start, move and die all day, so the current list lives in a registry that updates itself, and either the caller or a proxy in front of it reads that registry on every call.

Picture it first: the co-working space front desk

A big co-working space in Bengaluru hosts 300 people on hot desks. People switch desks and come and go all day, so a seating chart printed on Monday is wrong by Tuesday.

Reception keeps a live board instead: each person’s name and current desk. Some visitors read the board and walk over; others ask the receptionist, who takes them there. Members update the board by signing in at a kiosk, or the building updates it from badge swipes. Someone who leaves by the fire exit stays “in” until the 6 pm sweep clears everyone who hasn’t swiped for hours.

Map it back:

• The board is the service registry.
• Reading it yourself is client-side discovery; asking the receptionist is server-side discovery.
• The kiosk is self-registration; badge swipes are third-party registration.
• The evening sweep is a TTL. Until it runs, visitors walk to empty desks.

Why hard-coded addresses break

In a Swiggy-style delivery backend, the order service calls the restaurant service on every checkout. The restaurant service runs as 40 pods, and each pod gets a fresh IP when it starts.

Five rolling deploys a day replace 5 × 40 = 200 addresses. The autoscaler adds 40 pods for the dinner rush and removes them after midnight: 80 more. That is close to 300 address changes a day for one service, in a backend with a hundred services. No hand-edited config file keeps up. Service discovery automates three jobs:

1. Register each instance’s address, port and metadata once it is ready.
2. Look up the current healthy instances for a name.
3. Keep it fresh: remove instances that stop, crash or turn unhealthy before many calls are wasted.

The service registry

A service registry is a small database whose keys are service names and whose values are instance lists. One entry might look like this:

service: restaurant
instances:
  - id: restaurant-7f9c-x2k4
    address: 10.4.2.9
    port: 8080
    zone: ap-south-1b
    version: v42
    status: UP            # or STARTING, DOWN, OUT_OF_SERVICE
    lease_expires_in: 25s

Three properties matter more than the storage format:

• It must stay up. Nearly every request depends on it, so it runs as three or five replicated servers, and every client caches its answers so an outage freezes the list instead of emptying it.
• It must announce change quickly. Callers either poll or hold a watch that pushes changes: ZooKeeper watches, etcd watch streams, Consul blocking queries, the Kubernetes watch API.
• It carries metadata. Zone, version and weight let callers prefer nearby instances or send 5% of traffic to a canary.

Some platforms hide it: as the microservices.io pattern notes, an AWS load balancer’s target list is a registry you never install.

Getting into the registry

Self-registration: the instance signs itself in

The instance registers when it starts, then renews a lease on a timer. If renewals stop, the lease expires and the entry disappears; on a clean shutdown the instance deregisters at once. Every system has this shape:

• Eureka: renew every 30 s; the server evicts after 90 s of silence. Shutdown sends a cancel.
• etcd: attach a key such as /services/restaurant/10.4.2.9:8080 to a lease with a TTL and stream keep-alives. When the lease expires, the key is deleted and watchers get a DELETE event.
• ZooKeeper: an ephemeral znode lives exactly as long as the client’s session.
• Consul: a local agent registers the service and runs its health checks (HTTP, TCP, gRPC, or a TTL the app must refresh).

You gain a richer state model: an instance warming its cache can say STARTING, then UP. You pay with a registry client in every language, and a blind spot: a wedged process keeps renewing its lease happily, so pair renewals with a real health check. Sizing the intervals is covered in heartbeats.

Third-party registration: someone watches for you

A separate registrar watches the platform and updates the registry; the service’s code knows nothing about it. In Kubernetes, the EndpointSlice controller writes ready pods into EndpointSlices. AWS Auto Scaling registers each new instance with its load balancer as it launches.

You gain zero discovery code and one mechanism for every language. You pay by trusting the platform’s health signal, which is only as good as your probe, and by depending on one more critical controller.

The lesson: planned exits should never rely on the TTL. Most churn is planned (deploys, scale-in), so deregistering on shutdown removes most stale traffic; the TTL is the safety net for crashes.

Finding an instance: client-side or server-side

The two patterns differ in who reads the list and picks an instance.

Client-side discovery. The caller reads the list from a local cache, kept fresh by polling or a watch, and picks an instance itself: round robin, fewest outstanding requests, same zone first. Netflix paired the Eureka registry with the Ribbon client library; today gRPC clients can take the list from an xDS control plane, and a Kubernetes headless Service hands out raw pod IPs. You gain one hop and smarter balancing, because the client sees its own errors and latencies. You pay with a library in every language, upgrades across every team, and clients only as fresh as their cache.

Server-side discovery. The caller sends every request to one stable address (a load balancer or virtual IP) that knows the list and forwards. AWS load balancers and Kubernetes ClusterIP Services work this way. You gain simple clients in any language. You pay with an extra hop, a router that must itself scale and stay up, and coarser balancing if it works per connection.

A service mesh blurs the line: a proxy beside every caller balances each request (client-side logic), while the app just calls a local address (server-side simplicity).

DNS-based discovery

Every language already speaks DNS, so many registries expose a DNS face. A and AAAA records map a name to IPs, but the caller must know the port. SRV records (RFC 2782) add a port, priority and weight per target: clients try the lowest priority first and, within it, pick at random in proportion to weight.

; name                                TTL  class type prio weight port target
_grpc._tcp.restaurant.example.com.    30   IN    SRV  10   60     8443 r1.example.com.
_grpc._tcp.restaurant.example.com.    30   IN    SRV  10   40     8443 r2.example.com.
_grpc._tcp.restaurant.example.com.    30   IN    SRV  20   0      8443 dr1.example.com.

Here r1 gets about 60% of new connections and r2 about 40%; standby dr1 is used only if neither priority-10 target answers.

• Consul answers restaurant.service.consul on the local agent’s port 8600, returning only instances that pass health checks, shuffled, with a TTL of 0 so nothing is cached.
• Kubernetes names each Service restaurant.prod.svc.cluster.local, resolving to its ClusterIP. A headless Service (no ClusterIP) resolves to every ready pod’s IP instead, for clients that balance themselves.
• AWS Cloud Map serves only healthy registered instances, through an API or DNS inside a VPC.

DNS’s weaknesses are mostly about caching:

• TTLs are not always obeyed. Resolvers and runtimes cache past the TTL, and some apps resolve once at startup. Kubernetes’ documentation gives this as the reason Services use a virtual IP instead of round-robin DNS.
• No load signal. A client can’t tell that one address is overloaded.
• Size limits. Consul warns that a service with over 500 instances may not fit in one response.
• Negative caching. A cached “no healthy instances” keeps a recovered service looking down.
• Search paths. Kubernetes pods use ndots:5, so api.example.com is first tried against each cluster search domain, wasting lookups unless you add a trailing dot.

Kubernetes: discovery built in

Kubernetes bundles third-party registration, a registry, DNS and server-side discovery into one object, the Service:

1. A Service with selector app: restaurant gets a ClusterIP, a virtual IP no machine owns, and a DNS name pointing at it.
2. The EndpointSlice controller writes matching pods’ IPs, ports and zones into EndpointSlices of at most 100 endpoints by default. Our 40 pods fit in one; 250 pods need ⌈250 ÷ 100⌉ = 3. Every change is sent to every node, so replacing one pod should rewrite one small slice, not one giant list.
3. Readiness gates membership. With the default 10 s probe period and failure threshold of 3, a failing pod leaves rotation after about 10 × 3 = 30 s, without a restart.
4. kube-proxy on every node watches EndpointSlices and programs kernel rules (iptables by default, nftables as the faster successor; IPVS mode is deprecated) that send each new connection for the ClusterIP to a random ready pod.
5. The caller just connects. The kernel on its own node forwards; there is no central balancer to scale or fail.

Rolling deploys without dropped requests

When a pod is deleted, two things start at the same moment: the kubelet starts shutdown (any preStop hook, then SIGTERM), and the control plane marks the endpoint terminating. That change must still reach every node and client cache, so a process that exits on SIGTERM instantly drops connections routed a moment earlier. The fix:

• A preStop hook that sleeps a few seconds, so routing updates land first.
• On SIGTERM, refuse new connections, finish in-flight ones, and exit well inside the 30 s default grace period.
• Retry idempotent calls on another instance for the few that slip through.

AWS load balancers build this in: a leaving target drains for a deregistration delay of 300 s by default.

Registries and their consistency choices

Four standalone registries dominate, and they disagree about network splits. ZooKeeper, etcd and Consul’s servers replicate through majority-quorum consensus, so the minority side of a partition stops accepting writes. Eureka skips consensus entirely.

Consul’s agents find each other and detect dead nodes by gossip, so every node need not ping the servers; HashiCorp recommends three or five servers and up to 5,000 agents per datacenter.

A stale list beats no list

Discovery data is a hint the caller checks the moment it connects. A wrong entry costs one failed attempt and a retry; an unreachable registry costs every call. That asymmetry is the argument of Knewton’s widely shared 2014 post, “Eureka! Why You Shouldn’t Use ZooKeeper for Service Discovery”: better to know which servers were up five minutes ago than nothing at all. In CAP terms, the phone book should be AP.

Eureka is built that way. A dead server triggers no election; clients just use another. During a partition every server keeps accepting registrations. Clients cache the whole registry, so lookups survive even if every server is down. And in self-preservation mode, if renewals fall below 85% of the expected rate over 15 minutes, a server stops evicting anyone: 15% of a fleet going quiet at once is far likelier a broken network than mass death. What you pay: in a real mass outage it hands out dead instances, so its wiki tells clients to time out fast and retry elsewhere.

Others land in the same place. Consul’s DNS serves stale reads by default (maximum staleness: 10 years), so it answers even with no Raft leader. Envoy states that discovery doesn’t need full consistency: a host missing from discovery data but passing health checks keeps getting traffic. Kubernetes stores EndpointSlices in etcd, a CP store, but if the control plane becomes unreachable, kube-proxy’s already-programmed rules keep forwarding; only updates stop.

Service mesh: discovery moves into a sidecar

A mesh moves client-side discovery out of your code into a proxy, usually Envoy, beside every instance. The app calls a local address; the proxy does the rest.

• A control plane (Istio, Consul and others) watches the registry and streams config to every proxy over xDS, Envoy’s discovery APIs. Its Endpoint Discovery Service (EDS) carries instance lists with weights, zones and canary flags that DNS cannot hold.
• The sidecar balances per request, retries, encrypts with mTLS, and ejects bad hosts by outlier detection. Envoy’s defaults eject a host after 5 consecutive 5xx or connection failures, for 30 s (longer on each repeat), never more than 10% of a cluster at once.

You gain per-request balancing, retries and mTLS in every language without code changes. You pay with two extra proxy hops per call, a proxy’s CPU and memory in every pod, and a critical control plane. gRPC libraries can consume xDS directly and skip the sidecar, but only for gRPC. More in sidecar, ambassador and service mesh.

Freshness: the staleness window

Every setup has a window in which callers still route to a dead instance:

staleness window ≈ detection (TTL or probe threshold) + propagation (push or poll) + caller cache (refresh interval, DNS TTL, JVM cache)

Worked example: heartbeats every 10 s, a 30 s TTL, clients refreshing every 30 s. Pod R7 crashes just after a heartbeat. The registry expires it about 30 s later, and a client that refreshed just before keeps it another 30 s: up to 30 + 30 = 60 s. The order service sends 1,200 req/s across 40 pods, so R7 gets 1,200 ÷ 40 = 30 req/s, and up to 30 × 60 = 1,800 calls fail or hang. Eureka’s defaults are slower: 90 s to evict plus a 30 s fetch is about 2 minutes, which its wiki confirms.

Shorter TTLs cost: 2,000 instances renewing every 10 s is 200 registry writes a second; every 2 s is 1,000, and GC pauses start evicting healthy instances. Better levers:

• Deregister on graceful shutdown, which removes planned exits, most of the churn, in one propagation delay.
• Push, don’t poll: watches and xDS deliver changes in about a message’s travel time.
• Retry idempotent calls on another instance with a short connect timeout (timeouts and retries, idempotency).
• Eject on failure. Live traffic is the fastest probe. With 20 order pods, each sends R7 1,200 ÷ 20 ÷ 40 = 1.5 req/s, so 5 straight failures take about 3.3 s, and wasted calls fall from 1,800 to about 5 × 20 = 100.
• Check health actively to catch a wedged process that still renews its lease (health checks done right).

Registry outages and thundering herds

• Keep the last good list. Never treat “registry unreachable” as “zero instances”.
• Spread out reconnects. After a registry restart, thousands of clients re-register and re-fetch at once. Add jitter; Consul adds up to 1/16 of the wait at random to blocking queries so watchers don’t wake together.
• Never serve a partial list. A Eureka server that cannot copy the registry from a peer at startup waits 5 minutes for registrations, because a half-empty list would pile all traffic onto a few instances.

Going deeper: zones, canaries, regions and trust

Zone-aware routing

Crossing availability zones adds latency, and on AWS costs $0.01 per GB in each direction. At 1,200 calls a second of 10 KB each, the order service pulls 12 MB/s, about 1 TB a day. Spread over three zones, two-thirds crosses a boundary: about 690 GB × $0.02 ≈ $14 a day for one pair of services.

So prefer same-zone instances and spill over only when the zone runs short. Kubernetes offers trafficDistribution: PreferSameZone, falling back cluster-wide when the zone has no ready endpoints; Envoy and Eureka clients use zone metadata the same way. What you pay: a zone with few pods gets hammered, so spread pods evenly.

Metadata for versions and canaries

With labels such as version: v43, a mesh or smart client can send 5% of traffic to the canary. Netflix rolled back releases by marking every instance of the bad version OUT_OF_SERVICE in Eureka, returning traffic to the old one in one step. See deployment strategies.

Multiple clusters and regions

Run a registry per region; a consensus cluster stretched across oceans pays cross-region round trips on every registration. Consul keeps a separate catalog per datacenter; federation or peering lets one query another, and prepared queries can fail over to a remote datacenter when no local instance is healthy. See multi-region and active-active.

Who is allowed to register?

A registry is a routing table: anything that registers itself as payments receives payment calls. Authorise writes (Consul ACL tokens; in Kubernetes, restrict who may edit EndpointSlices), and authenticate the instance, not the address: with mTLS the caller checks the callee’s certificate identity, so a spoofed entry fails the handshake. See service-to-service auth and mTLS.

Choosing an approach

Cheat sheet

How to say it in the interview

“The order service shouldn’t know any IPs. On Kubernetes, the restaurant service gets a Service: the EndpointSlice controller registers pods once their readiness probes pass, DNS maps the name to a virtual IP, and kube-proxy forwards each connection to a ready pod. That’s third-party registration with server-side discovery. A virtual IP balances connections, not requests, so for gRPC I’d use a headless Service with client-side round robin, or a mesh. Discovery data is a hint, so I favour availability: clients keep the last good list, retry idempotent calls elsewhere, and eject an instance after five straight failures. A preStop sleep and graceful drain make deploys lossless.”

Why not just put every instance in DNS? For slow-changing services, you can. But caches ignore TTLs (the JVM holds answers 30 s by default), answers carry no load signal, big lists overflow a response, and balancing is per connection at best. For churning services, use watches or a virtual IP that hides the churn.

What happens when the registry goes down? Very little at first, if you designed for it: clients keep their last good list, kube-proxy’s rules keep forwarding, Envoy keeps healthy hosts. New instances stay invisible, so scale-ups stall. Never treat “registry unreachable” as “no instances”.

Client-side or server-side? Server-side by default: any language works and the platform provides it. Go client-side, or mesh, when you need per-request balancing over long-lived connections, zone awareness or weighted canaries.

How long can callers hit a dead instance? Detection plus propagation plus cache: up to 60 s with a 30 s TTL and 30 s refresh; about 2 minutes with Eureka’s defaults. Explicit deregistration removes the planned cases; retries and outlier ejection cover crashes within a few failed calls.

Would you use ZooKeeper or etcd as the registry? They work (Kubernetes keeps endpoints in etcd), but they are CP: the minority side of a partition can’t update registrations, and an expired session deletes ephemeral entries. So every client must cache the last good list. As a standalone registry I’d pick something AP-leaning: Eureka, or Consul with its default stale DNS reads.

Where to go next

• Heartbeats: how registries decide an instance is dead, and how to size the timeout.
• Gossip protocol: how Consul’s agents track membership and failures without a central monitor.
• Consensus and leader election: why ZooKeeper, etcd and Consul’s servers use majority quorums.
• ZooKeeper and etcd: the two classic coordination stores in practice.
• Load balancing: L4 vs L7: per-connection versus per-request balancing.
• Domain Name System (DNS): resolution, caching and TTLs.
• Sidecar, ambassador and service mesh: the proxy pattern behind meshes.
• Health checks done right: readiness, liveness and deep checks.
• API gateway: discovery at the edge, where outside traffic enters.

References and further reading

1. Pattern: Service registry (microservices.io)microservices.ioThe registry, client-side and server-side discovery, and self versus third-party registration as named patterns, with their trade-offs.
2. Kubernetes documentation: Virtual IPs and Service Proxieskubernetes.ioHow kube-proxy turns a ClusterIP into per-node iptables or nftables rules, why Kubernetes avoids round-robin DNS, terminating endpoints and PreferSameZone.
3. Kubernetes documentation: EndpointSliceskubernetes.ioThe 100-endpoint default per slice, the ready, serving and terminating conditions, and zone data per endpoint.
4. Kubernetes documentation: DNS for Services and Podskubernetes.ioService names, headless Services that return pod IPs, SRV records for named ports and the ndots:5 search path.
5. Netflix Eureka wiki: Understanding Eureka client/server communicationgithub.comRenew every 30 s, eviction after 90 s, delta fetches every 30 s, explicit cancel on shutdown and up to 2 minutes of propagation lag.
6. Eureka! Why You Shouldn’t Use ZooKeeper for Service Discovery (Peter Kelley, Knewton, 2014)medium.comThe argument that discovery should favour availability: stale information beats no information during a partition.
7. Consul documentation: Perform static DNS queriesdeveloper.hashicorp.comservice.consul lookups, health-filtered and shuffled answers, SRV records and the 500-instance DNS size warning.
8. RFC 2782: A DNS RR for specifying the location of services (DNS SRV)rfc-editor.orgThe SRV record format and the priority and weighted-random selection rules clients must follow.
9. Envoy documentation: Service discoveryenvoyproxy.ioStrict and logical DNS, the Endpoint Discovery Service, and the eventually consistent design where health checks overrule discovery data.
10. etcd documentation: API (leases and watches)etcd.ioLeases with a TTL and keep-alives, keys deleted when a lease expires, and watch streams that deliver PUT and DELETE events.