Lucene never edits a file. So what is a shard?
Each batch becomes a complete, write-once mini-index, a segment. A small file, segments_7, lists the ones that count: the commit point.
Refreshed means safe?
Bad: Yes, it’s a segment
It sits in the OS page cache, never fsynced. Recovery discards it.
Good: No: the translog is
Fsynced before every ack, then replayed after a crash.
Power cut 8 s after the last commit. What survives?
request fsyncs before every ack: nothing lost. async syncs every 5 s, so later acked writes are lost on that copy.
A seller changes one price. Lucene can’t edit segment _3.
Instead
It flips the old copy’s live-docs bit and re-indexes the whole document into a new segment. The dead copy stays until a merge.
60 GB
of fresh segments a day for Bazaar
Assume 50M products at 2 KB: 100 GB a copy. Say 30M price and stock changes daily: 60% of the index, before any merging.
Three ways to tame Bazaar’s churn
- Index
in_stock, not 37 → 36; check exact stock at checkout - Coalesce per product: five reprices a minute → one reindex
- Refresh every 5 s: a fifth as many tiny segments
A 64 GB node. Why only ≈ 30 GB of heap?
The index lives in the OS page cache, whatever the heap leaves. A cached read takes ≈ 100 ns; a disk seek ≈ 8 ms.
Remember
Segments are write-once. Acks wait for an fsynced translog; a refresh makes writes searchable in ≈ 1 s; a flush commits them. Updates re-add whole documents, merges clean up, and the page cache serves reads.