Failure modes
0-day streakReview

Your progress

Level 1

0 XP100 XP to level 2
0day streak
0best streak
0topics started
0cards reviewed
    Review cards

    Progress is stored only in this browser. No account is required.

    Layer 3 · Go deeper · 7 min

    Go deeper

    Size it, build it, then make the calls an interviewer will push you on.

    Back of the envelope

    Size it up

    Tap through each step of the arithmetic.

    5% of requests hang. How many of all requests fail?

      How many fanned-out requests touch one silently failing switch?

        The look-aside cache is flushed. How much load can you recover from?

          Build-up · 5 steps

          Watch a 10-second blip become an outage

          The retry example from Bronson et al.: a database that copes below about 300 req/s, an app sending 280, and one retry after a 1-second timeout.

          Step 1 of 5

          Vulnerable, but efficient

          The database answers in under 100 ms below about 300 req/s, and ten times slower above. The app sends 280 and retries once after 1 s. Many systems run this close because it’s efficient.

          4 interview moments

          You’re the candidate

          The interviewer pushes on the hard parts. Pick an approach and see how it holds up.

          1. Deep dive 1 of 4

            Interviewer

            “Replica 3’s orders time out on an empty connection pool, yet its health check returns 200. What should the check exercise?”

            Your move. Pick an approach:

          2. Deep dive 2 of 4

            Interviewer

            “Your service tipped into a crash cascade at 11,000 req/s. Load is back to 9,000, yet servers keep crash-looping. What now?”

            Before the cascade it was healthy at 10,000 req/s.

            Your move. Pick an approach:

          3. Deep dive 3 of 4

            Interviewer

            “Replicas crash one after another, each just after receiving the same request. The load balancer retries errors on another replica. What do you change?”

            Your move. Pick an approach:

          4. Deep dive 4 of 4

            Interviewer

            “Your DNS resolver times each upstream’s round trip to pick the fastest. How do you measure the duration?”

            Your move. Pick an approach: