43 TB
of raw logs every day
1M lines a second, 3M at peak, 500 bytes each. Over 99% are never read, so every byte must be cheap to keep, yet quick to find.
Where do logs go first?
Bad: To the backend
Slow backend, blocked request threads: a logging outage becomes a checkout outage.
Good: To a local file
A node agent tails it, buffering to disk. The app never waits.
The log store is down for 40 minutes. What’s lost?
Kafka keeps 72 hours, so 2.4 billion lines simply wait, then writers drain them at twice the normal rate. Search is stale; nothing is lost.
12×
more disk to index every word
A full-text index is about as big as the raw logs, so 30 days with a replica is 2.6 PB of SSD. Sorted, compressed columns need 219 TB.
One team logs status as 200, another as “OK”. The first type wins; the other’s lines are rejected.
The trick
Keep name and value arrays per type: a new field or clash is one more entry.
Checkout crash-loops and logs 100× normal. Who else suffers?
400,000 lines a second, 40% of the platform’s average load. On a shared pipe, writers fall behind and all 400 teams’ logs go stale.
The fix: layered limits that keep the errors
- Agent: collapse repeated traces, cap each container
- Gateway: drop DEBUG, sample INFO, keep errors
- Kafka: separate topics shield audit logs
- Showback: each team sees its rupee cost
Remember
Logs are written constantly and rarely read. Protect the write path with disk buffers and Kafka; make every byte cheap with sorted columns and tiers.