Service-to-service auth and mTLS

Service-to-service authentication is how one piece of software proves to another, on every call, which workload it is, without relying on where the call came from. The modern default is mutual TLS (mTLS): each workload holds a short-lived certificate naming it, both ends prove they hold the matching private key during the handshake, and the receiver then decides whether that identity may make this call. The end user travels separately, in a short-lived token.

Picture it first: a badge reader on every door

Picture a large hospital. In the old days one guard stood at the main gate, and anyone who got past him in scrubs could open the pharmacy, the records room and the ICU.

Now every door has a reader, and every employee carries a chipped photo badge that works for one shift. The reader checks two things: that the chip is genuine and belongs to its holder, then whether the door’s own list admits that person. When a nurse collects medicine for a patient, the slip names both: nurse Kavya, for bed 12.

Map it back: the main gate is the perimeter firewall, and scrubs are an internal IP address. The chip badge is a workload certificate; proving the chip is yours is proving you hold the private key. Whoever issues the badges is your internal certificate authority (CA), and shift-long badges are short-lived certificates. Each door’s list is an authorisation policy, and the slip is the end user’s token travelling with the call.

Why the network is not an identity

Many systems assume that anything able to reach a service is one of us: plain HTTP inside the VPC, perhaps an IP allowlist, and one firewall at the edge. That fails in three ways:

• One foothold reaches everything. Compromise any workload, say through a vulnerable library, and you can call every service and database directly; on a flat network this lateral movement is free.
• Addresses are not names. Pods churn and IPs are reused, so an allowlist entry may describe a different workload tomorrow.
• Plaintext is readable by anything on the path, including a compromised node.

Zero trust is the alternative. NIST SP 800-207 (August 2020) defines it as granting no implicit trust based solely on network location; its tenets include securing all communication regardless of location and granting access per session with least privilege. For services: authenticate and authorise every call as if it came from the internet. Segmentation stays, as an extra layer, not as the identity.

Five ways for a service to prove who it is

From simplest to strongest; most systems use two or three.

Shared API keys

The caller sends a random string in a header (X-Api-Key: sk_live_…), and the receiver compares it with a stored hash. It is trivial to build and crosses any proxy. But the secret travels on every request and leaks into logs, keys rarely rotate because both sides must change together, and a shared key makes its callers indistinguishable. Keep API keys for partners and webhooks, one per caller.

HMAC request signing

Instead of sending the secret, the caller signs with it. In AWS Signature Version 4, the client builds a canonical string from the method, path, query, chosen headers and a SHA-256 hash of the body, then computes HMAC-SHA256 over it with a key derived for one service, region and day. The receiver recomputes and compares, and AWS rejects most requests more than five minutes past their timestamp. An internal scheme modelled on it signs:

POST
/v1/payouts

content-type:application/json
host:payouts.internal
x-date:20261027T081500Z

content-type;host;x-date
b3d890b7ad7dd9f44ee06a0f3ccbf1c669ffd2b5eb449a5ab93d606f7edf7ab7

The last line hashes the body {"amount":50000,"currency":"INR"}. You gain a secret that never travels, tamper detection and a short replay window. You pay with brittle canonicalisation (a proxy that re-encodes a path breaks signatures), replay inside the window, and a symmetric key: a compromised verifier can sign as the caller.

Bearer JWTs from client credentials

The service uses the OAuth client credentials grant (OAuth 2.0 and OpenID Connect) to get a short-lived JWT for itself (subject orders, audience payments, scope charges:create) and sends it as Authorization: Bearer …; receivers validate it with cached public keys. You gain scoped, audience-restricted tokens that cross any HTTP proxy. You pay because a bearer token works for whoever holds it until it expires, and fetching it needs a client secret, so the “secret zero” problem of distributing a long-lived credential returns.

Mutual TLS with workload certificates

In mutual TLS the server also asks for a certificate, so both sides prove who they are in the handshake. Each workload’s private key never leaves its host, and its certificate, signed by your internal CA, names it with a URI in the Subject Alternative Name (SAN) field. You gain an identity proof that cannot be replayed, plus encryption, with no secret on the wire. You pay with running a PKI, and TLS is hop-by-hop: a proxy that terminates it ends the proof.

Workload identity with SPIFFE

SPIFFE (Secure Production Identity Framework For Everyone) standardises the ID format, the identity documents (SVIDs: X.509 certificates or JWTs) and a local Workload API that delivers them with no pre-shared secret. You gain one scheme across Kubernetes, VMs and clouds, and an answer to secret zero; you pay with running SPIRE or a SPIFFE-aware mesh.

How mTLS works in TLS 1.3

TLS 1.3 (RFC 8446, August 2018) completes a full handshake in one round trip:

1. ClientHello: a random nonce, cipher suites and a Diffie–Hellman key share.
2. ServerHello: the server’s choices and key share. Both sides derive handshake keys, so everything after this is encrypted, certificates included; TLS 1.2 sent the client certificate in the clear.
3. CertificateRequest: the message that makes TLS mutual, listing acceptable signature algorithms and, optionally, trusted CAs.
4. Certificate and CertificateVerify from the server: its chain, then a signature with its private key over the transcript hash so far.
5. Certificate and CertificateVerify from the client. The signature covers the string “TLS 1.3, client CertificateVerify” and a transcript that includes both nonces, so it is valid for this handshake only.
6. Finished: MACs over the transcript confirm both sides saw the same handshake. The client’s first request rides with its Finished, after one round trip.

Each side then checks that the chain reaches a CA in its local trust bundle within validity dates, and that the identity is the expected one. The client confirms the server’s SAN belongs to the service it dialled, or any valid workload could pose as payments; the server hands the client’s SAN to authorisation.

A resumption ticket lets a later connection skip the certificates and signatures and inherit the original identities. RFC 8446 caps tickets at 7 days; keep yours below your certificate lifetime, so a session cannot outlive its certificate.

Running an internal PKI

Public CAs certify DNS names, not workload identities, so you run your own CA, usually in three levels:

• Root CA: offline, its key in a hardware security module (HSM), signing only intermediates, valid for years. Its certificate goes in every trust bundle.
• Intermediate CA: online, its key in a KMS or HSM, issuing all day, valid about a year. One per region or cluster means a compromise replaces one intermediate, not every bundle.
• Leaf certificates: one per workload instance, valid for hours.

Why hours? Because revocation is classic PKI’s weak spot: revocation lists (CRLs) must be downloaded by every verifier, OCSP adds a query per connection, and slow responders make clients fail open. A stolen 12-hour certificate is worth at most 12 hours, and you cut off a compromised workload by not renewing it. Renewal happens at half-life: an agent generates a new key and certificate signing request (CSR) and swaps the new pair in before expiry, while open connections finish on the old one.

The issuance load is small. For 5,000 pods with 12-hour certificates renewed every 6 hours:

• Steady state: 5,000 ÷ (6 × 3,600 s) = 5,000 ÷ 21,600 ≈ 0.23 certificates per second.
• A full rolling restart in 10 minutes: 5,000 ÷ 600 s ≈ 8.3 per second.
• CPU: on one Apple M1 core, openssl speed (OpenSSL 3.6) makes an ECDSA P-256 signature in about 18 µs, so the peak costs 8.3 × 18 µs ≈ 0.15 ms of CPU per second.

The real constraint is the CA’s availability. Renewing at half-life keeps at least half a lifetime on the newest certificate, so a broken CA has 6 hours before anything expires. With 1-hour certificates, issuance rises to 5,000 ÷ 1,800 ≈ 2.8 per second and the grace falls to 30 minutes: a smaller theft window, a more critical CA.

To rotate a root without downtime:

1. Add the new root to every trust bundle beside the old one, and wait until every verifier has it (the highlighted band).
2. Issue from an intermediate under the new root. Both chains now verify everywhere.
3. Drop the old root after the last leaf under it has expired, plus a margin.

The SPIFFE federation standard says the same across organisations: publish new keys well before signing with them, and remove old ones only when nothing valid depends on them. Skipping the wait in step 1 is the classic outage.

SPIFFE and SPIRE: identity without a secret to start with

A SPIFFE ID is a URI, spiffe://trust-domain/path. The trust domain is a root of trust, and SPIFFE advises separate ones for environments with different security practices, such as staging and production. The path is yours, for example /ns/payments/sa/ledger. The identity document is an SVID:

• An X.509-SVID is a certificate whose SAN holds exactly one URI, the SPIFFE ID. Verifiers must reject one with more, and never accept a CA certificate as a workload.
• A JWT-SVID carries the ID in sub and must have aud and exp. It is a bearer token, so SPIFFE prefers X.509-SVIDs and keeps JWT-SVIDs for paths through an L7 proxy.

The Workload API solves secret zero. A workload asks a local agent over a Unix domain socket “who am I?”, presenting nothing. The agent works out the answer itself, returns the SVID, private key and trust bundle, and streams renewals before expiry. SPIRE implements this with a server, the signing authority and registry for one trust domain, and an agent on every node.

1. Node attestation: the agent sends proof only its machine could have, such as an AWS instance identity document or a Kubernetes service account token.
2. The server verifies the proof with the cloud or Kubernetes API.
3. The server issues the agent’s own SVID and the registration entries it may serve.
4. A workload calls the Workload API with no credentials.
5. Workload attestation: the agent gets the caller’s process ID from the kernel and asks local authorities about it: its Unix user and group, or, through the kubelet, its pod’s namespace and service account.
6. On a match with a registration entry, the agent returns the X.509-SVID, key and bundle.
7. Workloads run mTLS with their SVIDs.

Attestation is the real security boundary. Provider-signed documents, TPM-backed keys and Kubernetes tokens are hard to forge; SPIRE’s fallback join tokens are pre-shared, so they expire after one use. Root on a node yields every identity scheduled there.

Federation links trust domains without merging them. Each publishes its bundle at an HTTPS bundle endpoint that the others poll at the publisher’s refresh hint (the spec suggests about five minutes when none is set) and keep separate, because a merged bundle would let one domain forge the other’s identities.

Service mesh: mTLS without touching code

A mesh moves all of this into infrastructure. In Istio’s sidecar mode, an Envoy proxy beside each pod intercepts its traffic; the app speaks plain HTTP to localhost and the proxies run mTLS between pods. An agent beside each proxy sends a CSR to istiod’s CA and rotates the certificate before expiry. The identity is a SPIFFE ID from the pod’s service account, spiffe://cluster.local/ns/<namespace>/sa/<account>, which policies write as a principal such as cluster.local/ns/checkout/sa/checkout. Ambient mode swaps per-pod sidecars for a per-node proxy, ztunnel. Clients also enforce secure naming: the identity in the server’s certificate must be allowed to run the service dialled, which defeats DNS hijacking. More on meshes in sidecar, ambassador and service mesh.

PeerAuthentication has three modes: PERMISSIVE accepts mTLS and plaintext, STRICT only mTLS, and DISABLE turns mTLS off. A mesh-wide policy with no mode behaves as PERMISSIVE. STRICT on day one breaks every caller outside the mesh, so migrate in steps.

Istio’s advice is to pass through PERMISSIVE, confirm with telemetry that every workload has switched, then apply the final mode, namespace by namespace:

apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: default
  namespace: payments
spec:
  mtls:
    mode: STRICT

PERMISSIVE left on forever is a common audit finding: it still admits plaintext from anything outside the mesh.

The mesh has a price. Istio’s benchmarks for version 1.24, at 1,000 requests per second of 1 KB, measured about 0.20 vCPU and 60 MB per sidecar and 0.06 vCPU and 12 MB per ztunnel, and every request crosses two proxies. At 20,000 internal requests per second that is 20 × 2 × 0.20 = 8 vCPU of sidecars, or 20 × 2 × 0.06 = 2.4 vCPU of ztunnels for L4-only mTLS; across 5,000 pods, sidecar memory reaches 5,000 × 60 MB = 300 GB. Latency grows by two proxy hops per call, so measure with your own payloads.

Authorisation: who may call what

mTLS answers who is calling, not may it. A compromised reports service still proves it is reports, and if nothing stops reports calling the ledger, you have rebuilt the flat network with better encryption. Key policies on the caller’s identity, enforced at the receiver:

• Default deny. In Istio, a workload with no AuthorizationPolicy allows everything; once an ALLOW policy applies, unmatched requests are denied. Start each namespace with an allow-nothing policy.
• Least privilege per method: “checkout may POST /v1/charges”, not “checkout may call payments”.
• Deny rules win. Istio evaluates CUSTOM, then DENY, then ALLOW, so a broad ALLOW cannot bypass a DENY.
• Every receiver protects itself, because a gateway check does nothing for east–west calls.

apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: ledger-writes
  namespace: payments
spec:
  selector:
    matchLabels:
      app: ledger
  action: ALLOW
  rules:
  - from:
    - source:
        principals: ["cluster.local/ns/payments/sa/upi-adapter"]
    to:
    - operation:
        methods: ["POST"]
        paths: ["/v1/entries"]

Rules needing business context can go to an external engine, such as Open Policy Agent, through Istio’s CUSTOM action; keep it close and cached, as it sits on every request. Tenant checks belong to multi-tenancy and isolation.

Carrying the end user across hops

When the ledger gets a call from orders, it must also know which customer the call is for. Otherwise it becomes a confused deputy: a service with broad powers tricked into using them for someone else. If orders may read any balance and just passes along customer_id=42, one bug or one compromised orders pod reads anybody’s balance.

Bad: a plain header. The gateway sets X-User-Id: 42 and every service trusts it, but any workload that can reach a service can set it too.

Okay: forward the user’s token. It is signed, but its audience is the public API, not the ledger. Services that accept tokens meant for someone else let any holder replay them anywhere, with the user’s full scope.

Good: the gateway mints an internal token, say a 60-second JWT signed with an internal key: subject u_42, audience orders, narrowed scope. You pay with one powerful signing key, and audiences that must be re-minted for deeper hops.

Great: token exchange per hop, on top of mTLS. With OAuth 2.0 Token Exchange (RFC 8693), orders presents the token it received as the subject_token and asks a token service for one for its next hop. The service checks that orders, identified by its mTLS certificate or an actor_token, may act for this user towards payments, and returns a token with the user as sub, payments as aud, a narrower scope and an act claim naming orders:

{
  "iss": "https://sts.internal",
  "sub": "u_42",
  "aud": "spiffe://prod.example/ns/payments/sa/payments",
  "scope": "charges:create",
  "exp": 1793095260,
  "act": { "sub": "spiffe://prod.example/ns/orders/sa/orders" }
}

Payments checks two facts that must agree: the mTLS peer is orders, and the token says orders acts for u_42. Longer chains nest act claims with the current actor outermost; RFC 8693 says only top-level claims and the current actor count for access control, and earlier actors are audit history.

Certificate-bound tokens

RFC 8705 ties a token to a certificate. A token issued over mTLS carries the SHA-256 thumbprint of the client’s certificate as cnf: { "x5t#S256": "…" }, and the resource server rejects a request whose mTLS certificate does not match with 401 invalid_token, so a token copied from a log is useless without the private key. The catch, which the RFC notes: rotating the certificate invalidates its bound tokens, so keep them to minutes and fetch a new one after each rotation.

Going deeper: failure modes and operations

Certificate expiry

An expired certificate is a predictable outage: at a known second, every handshake that depends on it fails. Long-lived certificates make this rare and catastrophic: the one expiring after two years, often a CA, is the one nobody remembers owning. Short-lived ones make expiry an hourly non-event, forcing automation and keeping failures small and loud.

Clock skew

Validity is checked against the verifier’s clock, so a node running 10 minutes slow rejects a certificate issued a minute ago. Run NTP, alert on drift, back-date notBefore a few minutes.

CA compromise and blast radius

Whoever holds the issuing key can mint any identity in the trust domain. Keep the root offline and intermediates in a KMS or HSM that will not export keys (encryption and key management), separate intermediates or trust domains per environment and region, and rehearse replacing an intermediate. Don’t lean on name constraints: the X.509-SVID standard warns that libraries support URI name constraints poorly.

Trust bundle lag

Bundles spread slowly: a pod started from a cached bundle, a VM refreshing hourly, a client that read its CA file at boot. Measure that every verifier has the new root before issuing under it.

Load balancers that terminate TLS

A load balancer that terminates TLS ends the mTLS proof at itself, so the backend sees the balancer, not the caller. Three fixes:

• Pass through at L4, keeping the handshake end to end but losing L7 routing on that hop (load balancing: L4 vs L7).
• Re-originate mTLS: the balancer authenticates the caller, opens its own mTLS connection, and passes the caller on in a signed token.
• Forward the certificate in a header, such as Envoy’s x-forwarded-client-cert. The edge must strip that header from outside traffic and backends must trust it only from the proxy, or any client can type in someone else’s identity.

RFC 8705 leaves this hand-off out of scope; JWT-SVIDs exist largely for this case.

Debugging a failed handshake

# Does the server ask for a client certificate, and does ours pass?
openssl s_client -connect payments.internal:8443 -servername payments.internal \
  -cert orders.crt -key orders.key -CAfile bundle.pem -brief

# Which identity does this certificate carry, and when does it expire?
openssl x509 -in orders.crt -noout -ext subjectAltName -enddate

Usual culprits: an unknown CA (stale bundle), an expired or not-yet-valid certificate (renewal or clock), the wrong SAN (policy or secure naming), or no client certificate loaded at all.

Handshake cost

On the same M1 core, openssl speed measured about 18 µs per ECDSA P-256 signature, 56 µs per verification and 32 µs per X25519 operation. A full mTLS handshake costs each side about two X25519 operations, one signature and three verifications (the peer’s CertificateVerify and two chain signatures): 2 × 32 + 18 + 3 × 56 = 250 µs of CPU per side. A client opening a new connection for each of 5,000 requests per second burns 5,000 × 0.25 ms ≈ 1.3 cores on handshakes; a pool of long-lived connections makes the cost round to zero.

A worked example: a UPI-facing payments platform

A Mumbai payments company runs 300 microservices, one of which talks to the UPI switch:

• Separate trust domains and roots for production and staging, so a staging pod can never hold a production identity.
• STRICT mesh mTLS, one service account per service, and default deny: only upi-adapter may POST to the ledger’s /v1/entries.
• Token exchange at every hop, issuing 60-second tokens with act naming the caller.
• The external switch link uses the certificates the payment network mandates, keys in an HSM, trusted through a separate bundle.

Cheat sheet

How to say it in the interview

“I don’t trust the internal network, so every workload gets its own identity: a short-lived certificate with a SPIFFE ID from its service account, issued by an internal CA and renewed at half-life. Services talk over mutual TLS; a mesh or SPIRE handles issuance and rotation, and I move from permissive to strict once metrics show no plaintext. Authentication isn’t authorisation, so each service has default-deny, per-method policies keyed on the caller. For the end user, each hop exchanges the user’s token for a short-lived one addressed to the next service, with the caller in the act claim.”

mTLS or JWTs between services? Both, for different questions. mTLS proves which workload is calling and can’t be replayed. A JWT carries the end user, or the service identity across an L7 proxy or company boundary. With only one inside a cluster, mTLS.

How do you rotate certificates without downtime? Short lifetimes and automation: renew at half-life, and open connections finish on the old certificate. For a root, add the new one to every bundle, wait until all verifiers have it, switch issuance, and drop the old root after its last leaf expires.

What does SPIFFE add over plain mTLS? A standard identity format, and a first credential without a secret: the Workload API attests the calling process through the kernel and orchestrator. Plus federation across trust domains, and JWT-SVIDs for L7 hops.

How does service B know which user it is acting for? From a token, never a bare header. Each hop exchanges the user’s token for a short-lived one with the user as sub, B as aud and the caller in act; B checks that the mTLS peer matches act.

What breaks mTLS behind a load balancer? TLS termination: the backend sees the balancer, not the caller. Use L4 passthrough, re-originate mTLS and pass the caller in a signed token, or forward the certificate in a header the edge strips from outside traffic.

Where to go next

• Authentication: sessions, tokens and JWT: validating JWTs.
• OAuth 2.0 and OpenID Connect: the client credentials grant, scopes and DPoP.
• Encryption and key management: where CA and signing keys live.
• Multi-tenancy and isolation: authorisation bugs that survive good identity.
• Sidecar, ambassador and service mesh: how meshes intercept traffic.
• HTTP and HTTPS: the TLS basics underneath.
• API gateway: where user tokens are checked.
• Load balancing: L4 vs L7: why terminating TLS changes who the backend sees.
• Service discovery: how callers find the services they then authenticate.

References and further reading

1. NIST SP 800-207 — Zero Trust Architecture (August 2020)csrc.nist.govThe definition of zero trust as no implicit trust based on network location, and the tenets that all communication is secured regardless of location and access is granted per session with least privilege.
2. RFC 8446 — The Transport Layer Security (TLS) Protocol Version 1.3rfc-editor.orgThe full handshake message flow, CertificateRequest, the CertificateVerify signature over the transcript with its context strings, encrypted certificates, the server’s options on an empty client certificate, and PSK resumption with its 7-day ticket cap.
3. AWS Signature Version 4 for API requestsdocs.aws.amazon.comHow SigV4 signs a canonical request (method, path, query, headers, payload hash) with HMAC-SHA256 under a key scoped to one service, region and day, and the five-minute timestamp window against replay.
4. SPIFFE conceptsspiffe.ioWorkloads, SPIFFE IDs, trust domains per environment, SVIDs, the Workload API that needs no bootstrap secret, trust bundles, and the advice to prefer X.509-SVIDs because tokens can be replayed.
5. SPIFFE standards: SPIFFE ID, X.509-SVID, JWT-SVID and Federationgithub.comExactly one URI SAN per X.509-SVID, leaf-versus-CA rules, poor library support for URI name constraints, mandatory aud and exp in JWT-SVIDs, and federation bundle endpoints with advance publication of new keys and no merged bundles.
6. SPIRE conceptsspiffe.ioServer and agent roles, the step-by-step life of an SVID, node attestors (cloud identity documents, Kubernetes service account tokens, single-use join tokens) and workload attestors (Unix uid/gid, Kubernetes namespace and service account).
7. Istio documentation — Security conceptsistio.ioCertificate issuance by istiod via CSRs, secure naming, PeerAuthentication PERMISSIVE/STRICT/DISABLE and the PERMISSIVE default, migration advice, and AuthorizationPolicy semantics (allow-all without policies, CUSTOM then DENY then ALLOW).
8. Istio documentation — Performance and scalability (Istio 1.24)istio.ioMeasured proxy cost at 1,000 requests per second of 1 KB: about 0.20 vCPU and 60 MB per sidecar, 0.06 vCPU and 12 MB per ztunnel, and two proxy traversals per request.
9. RFC 8693 — OAuth 2.0 Token Exchangerfc-editor.orgDelegation versus impersonation, subject_token and actor_token, and the act claim with nested actors, of which only the current actor counts for access control.
10. RFC 8705 — OAuth 2.0 Mutual-TLS Client Authentication and Certificate-Bound Access Tokensrfc-editor.orgThe cnf claim with an x5t#S256 certificate thumbprint, the 401 invalid_token rule on mismatch, tokens invalidated when the certificate changes, and TLS termination left out of scope.