Disk I/O and durability for search

A search index lives on disk as write-once files called segments. New documents wait in an in-memory buffer, become searchable when a refresh writes them out as a new segment (about once a second), and stay safe through a write-ahead translog until a commit fsyncs the segments. Background merges keep the segment count down, and the operating system’s page cache keeps reads fast.

This is part 3 of the search engines series. Part 2 built the inverted index, doc values and BKD trees, and ended on a catch: sorted, packed structures cannot be edited in place. Here we deal with the consequences on disk: segments, merging, caching and crash safety. The mechanics are Lucene’s, with Elasticsearch 9.x settings (current docs, October 2026) as the example; OpenSearch and Solr share them.

Picture it first: a wholesaler’s order books

A cloth wholesaler takes phone orders all day.

• The clerk writes each order on a loose slip and adds one line to a rough book kept in the steel cupboard.
• Every few minutes she staples the slips into a thin booklet for the counter, where salesmen look things up. Booklets are never edited.
• A cancelled order is not torn out; its number is ticked on a cancellation sheet clipped to its booklet.
• At closing, the munim locks the booklets in the cupboard and signs a contents page listing which ones count. Old rough-book pages can go.
• Once a month someone recopies twenty thin booklets into one thick register, skipping cancelled orders.

If the shop floods overnight, the counter booklets are ruined, but the last contents page and the rough book let the clerk rewrite every order since.

Map it back: the slips are the indexing buffer, the rough book is the translog, stapling a booklet is a refresh (searchable, not safe), the cancellation sheet is the live-docs bitset, the signed contents page is a commit (fsync plus a segments_N file), recopying is a merge, and rewriting orders after the flood is translog replay.

Why segments are write-once

Part 2 gap-encoded postings into packed blocks of 256 doc IDs and put the term dictionary behind a compact prefix index (a trie since Lucene 10.3, an FST before). Slotting one doc ID into a packed block would mean rewriting it, its skip data and everything after it. So Lucene never edits an index file. It writes each batch of documents as a complete mini-index, a segment, and a Lucene index (one Elasticsearch shard) is “a collection of segments plus a commit point”, in the Elasticsearch docs’ words. One shard on disk, with names from Lucene’s file-format docs:

segments_7               commit point: which segments make up the index
write.lock               only one IndexWriter may write here
_a.si  _a.cfs  _a.cfe    a small segment packed into one compound file
_b.si  _b.tim  _b.tip    term dictionary and its index
_b.doc _b.pos            postings: doc IDs and frequencies, positions
_b.dvd _b.kdd  _b.fdt    doc values, BKD points, stored fields
_b_3.liv                 live-docs bitset for segment _b, generation 3

File names are never reused: the next commit writes segments_8 beside segments_7.

Immutability buys compactness (no free space reserved, no page splits), lock-free reads (a searcher holds a fixed, point-in-time set of segments), cache friendliness (unchanging files stay valid in the OS page cache, and per-segment filter caches never need invalidating) and cheap copying (snapshots and replica recovery copy only new files). You pay three ways: changing a document is indirect, every query visits every segment, and merging rewrites data repeatedly.

That is an LSM tree’s bargain: immutable sorted runs, a write-ahead log, background compaction. A B-tree updates pages in place instead. Reads and writes: how databases store and fetch data compares the two, and Write-ahead logging covers the log. What is special about search: each segment is a whole searchable index, so queries fan out across segments, and freshness gets its own knob, refresh.

The write path: buffer, refresh, commit

Follow one Bazaar price change into one shard.

1. Index into the buffer. The document is analysed (part 2) and added to Lucene’s IndexWriter buffer. Elasticsearch shares a buffer of 10% of the heap across active shards (indexing-speed guide). Not searchable yet.
2. Append to the translog. After Lucene processes the operation and before acknowledging it, Elasticsearch appends it to the shard’s translog and, by default, fsyncs it on the primary and every allocated replica.
3. Acknowledge. The write is durable, but invisible to search.
4. Refresh. Every index.refresh_interval (default 1 s) the buffer is written out as a new segment and opened by a near-real-time reader, DirectoryReader.open(writer). The IndexWriter flush underneath “moves all in-memory segments to the Directory, but does not commit (fsync) them”, so the files sit in the OS page cache. Cheap, and not crash-safe.
5. Flush, which is a Lucene commit. Periodically Elasticsearch fsyncs the new segment files, writes a new segments_N and starts a new translog generation, deleting old ones. A commit, says the javadoc, lets updates “survive an OS or machine crash or power loss”.
6. Merge, in the background, when segments pile up.

Controlling when writes become visible

The 1-second refresh only runs on shards searched in the last 30 seconds (index.search.idle.after). An idle shard refreshes when the next search arrives, and that search waits. Setting index.refresh_interval explicitly opts out, -1 disables refresh, and Elastic Cloud Serverless defaults to 5 s (index settings).

Per request, the refresh parameter takes false (the default), wait_for, which holds the response until the next refresh, or true, which forces one and creates tiny segments you pay for when they are written, searched and merged. A GET by ID is real-time anyway, and a bulk reindex runs fastest with refresh_interval: -1 and no replicas, restored afterwards.

For Bazaar, a new price is searchable about a second after the indexer sends it; most staleness comes earlier, in the database-to-index pipeline (part 7).

The translog: durability between commits

Why not commit after every write? A commit fsyncs every new segment file, “a costly operation” in Lucene’s words. So Elasticsearch uses a write-ahead log: appending one operation to one file is a short sequential write, one fsync makes it durable, and the commit can wait. The translog settings:

• index.translog.durability. request, the default, fsyncs after every index, delete, update or bulk request, on the primary and every allocated replica, before reporting success. async fsyncs every index.translog.sync_interval (default 5 s, minimum 100 ms), so writes acknowledged since the last sync can be lost.
• index.translog.flush_threshold_size. Flush when operations not yet in a commit reach this size: 10 GB in the current Elasticsearch docs, while OpenSearch keeps 512 MB. The translog never exceeds 1% of the disk. A bigger threshold means fewer commits but a longer replay after a crash.

What does an fsync per request cost? Assume about 1 ms per fsync on a decent SSD, a rough figure; devices vary widely. A client sending one document per request to one shard, waiting for each reply, gets at most about 1,000 acknowledged writes a second; a bulk request of 1,000 documents shares one fsync. Hence Elastic’s advice to use bulk requests and parallel workers, which “help reduce the cost of each fsync”.

async instead risks up to 5 seconds of acknowledged writes on a copy that crashes. A replica on another node still holds them, unless it fails at the same moment, as in a cluster-wide power cut or an index with no replicas. It suits logs and metrics you can re-send; keep request for anything a user was told is saved.

Crash recovery

When a node restarts, each shard:

1. Opens the last commit: the newest segments_N and the files it lists. Segments refreshed after that commit are not listed, so they are discarded.
2. Replays the translog: Elasticsearch “replays any unflushed operations from the transaction log into the Lucene index” (flush API).
3. Refreshes and serves. Replicas then catch up from the primary.

An fsync is only as honest as the hardware: Lucene warns that some devices cache writes even during an fsync and report success early. Use drives with power-loss protection, and count on replicas, not one disk.

Deletes and updates: tombstones, not erasers

A segment cannot shrink, so a delete flips the document’s bit in that segment’s live-docs bitset, the .liv file: one bit per doc ID, and doc IDs restart at 0 in every segment (part 2). The change is saved as a new generation of that small file (_b_3.liv, then _b_4.liv). Queries still find the document in the postings, check the bit and skip it. Its bytes are reclaimed only by a merge.

An update is a delete plus an add. Lucene’s updateDocument “just deletes and then adds the entire document”, atomically for readers. Elasticsearch’s update API fetches the stored _source, applies your change and reindexes the result, so changing one price re-analyses the title, the description and every other field.

Elasticsearch also uses soft deletes: the old version is marked in a hidden doc-values field and kept for a while, so a replica that was briefly offline can replay the operations it missed instead of copying the whole shard. The retention leases that hold this history expire after 12 hours by default; the sequence numbers behind it are part 5’s subject.

Why Bazaar’s price and stock churn hurts

Assume Bazaar’s 50 million products take about 2 KB each on disk, counting postings, doc values and the stored source: about 100 GB per copy, held as four 25 GB primary shards, inside Elastic’s 10–50 GB guidance. Sellers reprice and every order changes stock: say 30 million changes a day, peaking at 5,000 a second during a sale.

• New data: 30 million × 2 KB = 60 GB of fresh segments a day, 60% of the index, before merges rewrite any of it.
• Tiny segments: at 5,000 updates a second, every shard writes a new small segment at every 1-second refresh.
• Dead weight: the merge policy tolerates 20% deleted documents. At that ceiling, 100 GB of live data sits in 125 GB of segments: about 25 GB of dead copies that use disk and cache and are skipped by every query.
• ID lookups: each update must first find the old document by ID in every segment, a check that “gets even more costly as the index grows”, per the indexing-speed guide.

What helps:

• Index coarse facts. Search needs in_stock: true, not that 37 became 36, and most stock changes never flip the flag. Keep exact counts in the database or a cache and check them when rendering results and at checkout, as part 2 suggested.
• Coalesce. Key the change stream by product ID and keep only the latest change per product every few seconds: five reprices in a minute become one reindex.
• Drop no-ops. The update API skips unchanged documents by default.
• Refresh less often if your freshness target allows: every 5 s means a fifth as many tiny segments.
• Use in-place updates where they exist. Lucene can rewrite a numeric doc-values field without reindexing the document (updateNumericDocValue), and Solr exposes this as in-place updates for non-indexed, non-stored, single-valued numeric doc-values fields. Elasticsearch always reindexes.

Merging: paying down the segment debt

A shard refreshed every second would collect tens of thousands of segments a day, each costing lookups, file handles and heap, with deleted documents piling up inside. Merging copies the live documents of several segments into one. A merge policy chooses what to merge; a merge scheduler runs the merges on background threads.

The tiered merge policy

Lucene’s default, TieredMergePolicy, “merges segments of approximately equal size, subject to an allowed number of segments per tier”. When the index exceeds its segment budget, it picks the cheapest merge: similar sizes, a small total and many deletes reclaimed. Unlike the older LogByteSizeMergePolicy, it can merge segments that are not adjacent; Mike McCandless introduced it in 2011. The Lucene 10.5 defaults:

• 5 GB maximum merged segment; normal merges stop growing a segment there.
• 8 segments per tier, and at most 10 merged at once.
• 16 MB floor: smaller segments count as 16 MB, so tiny refresh segments are merged early.
• 20% deletes allowed. Lower values reclaim space sooner but raise write amplification; below 5% the javadoc warns of almost continuous merging.

Elasticsearch uses the same values, except that it merges up to 16 segments at once. For time-based indices, those with a @timestamp field such as data streams, it switches to LogByteSizeMergePolicy, with segments up to 100 GB and 32 merged at a time. It merges only adjacent segments, so each segment covers one contiguous time range, and a time filter can skip whole segments.

Write amplification

Write amplification is the bytes written by merging divided by the final size of the index. In Fig. 4’s ideal case, merging about ten similar segments at a time copies each byte once per tier, and 16 MB to 5 GB is log₁₀(5,000 ÷ 16) ≈ 2.5, so three tiers: three merge copies, about four writes per byte counting the flush. McCandless measured 6.19 while indexing English Wikipedia (67.30 GB merged for a 10.87 GB index) and 13.6 to 14.5 in an update-heavy near-real-time run. Both used 2011-era Lucene; treat them as orders of magnitude.

For Bazaar, 60 GB of new segments a day at a factor of about 5 is 300 GB of merge writes, only 300 GB ÷ 86,400 s ≈ 3.5 MB/s on average. At the sale peak, 5,000 × 2 KB = 10 MB/s of new data becomes about 50 MB/s of merge I/O. An SSD copes, but merges churn the page cache, pushing out hot search data.

Keeping merges in check

Elasticsearch runs merges on a dedicated thread pool, smallest first, with automatic I/O throttling; if merging still falls behind, it throttles indexing on that shard (merge settings). index.merge.scheduler.max_thread_count defaults to half the processors; the docs suggest 1 for spinning disks.

Merges need free disk: a new segment is written before the old ones go, and readers, snapshots and the last commit can keep old files alive. A force merge to one segment can need twice the index size free, three times with compound files. Elasticsearch stops starting merges when disk runs low and, at the 95% flood-stage watermark, blocks writes to every index with a shard on that node.

Force merge, for indices that are done

An index that no longer changes, such as last month’s logs, can be force-merged down to one segment per shard. Searches then visit one segment, deleted documents are gone, and “more efficient data structures” become usable, per the search-speed guide.

Reads: the page cache does the heavy lifting

Lucene does not load the index into the Java heap. It reads files through the operating system, which keeps recently read pages in its page cache: all the RAM no process is using.

MMapDirectory memory-maps the index files, so each file looks like a byte array in the process’s address space. A cached page is a plain memory access, with no system call and no copy; an uncached page faults, and the kernel reads it from disk. As Uwe Schindler explained in 2012, mapping reserves address space, not memory, which is why top shows a virtual size far above RAM; Lucene has used mmap by default on 64-bit Linux since version 3.3. Elasticsearch’s default store type, hybridfs, maps the term dictionary, norms and doc values and reads the other files with ordinary I/O; both paths go through the page cache. Every mapped file takes entries under Linux’s vm.max_map_count, which Elasticsearch asks you to raise.

Heap versus cache

Elasticsearch’s JVM guidance caps the heap at 50% of RAM and below the compressed-oops threshold, where the JVM can still use 4-byte object pointers (26 GB is safe on most systems, up to 30 GB on some). Past it, pointers double to 8 bytes, so a slightly bigger heap can hold less. The heap is for query state, aggregations, caches, indexing buffers and segment metadata. The index lives in the page cache, “probably Elasticsearch’s number one performance factor” according to the search-speed guide. More heap means less cache and longer garbage-collection pauses; Schindler’s 2012 rule of thumb was a quarter of RAM.

When the hot set stops fitting

Rough per-read costs: a memory fetch takes about 100 ns and a disk seek about 8 ms in Peter Norvig’s table; AWS designs EBS io2 Block Express, its fastest network SSD, for an average under 500 µs per 16 KiB operation, and local NVMe is faster still. A query needing 200 uncached pages, read one after another, waits about 200 × 8 ms = 1.6 s on a spinning disk and up to 200 × 0.5 ms = 100 ms on that SSD, against 0.02 ms from the cache. Parallel I/O narrows the gap, but not the orders of magnitude.

In Fig. 5, Bazaar’s 200 GB of primaries and replicas sit on two such nodes, and each node’s hot 30 GB just fits its 32 GB cache. Double the catalogue or add a large field, and queries start faulting; p99 latency jumps, often suddenly. Fixes, roughly in order:

• Add RAM or nodes, so each node holds less index.
• Shrink the index: drop doc values or indexing on fields you never sort, filter or search, and try best_compression for stored fields (up to about 28% smaller on logs, per the index settings).
• Use local SSDs, which Elastic says generally beat remote storage.
• Keep readahead at 128 KiB, so random reads don’t drag unwanted pages into the cache.
• Warm up after restarts: index.store.preload loads chosen files eagerly, but preloading more than fits in RAM makes search slower.

Rarely searched data can move to cheaper nodes, the hot-warm-cold tiers of part 5.

Durability beyond one disk

An fsynced translog survives a process crash or a power cut, not a dead disk, a dead node or a bad delete. Those need copies elsewhere.

• Replicas. In Elasticsearch’s primary-backup model, the primary forwards each operation to every in-sync replica and acknowledges only when all have applied it (reading and writing documents); with request durability, each copy fsyncs its translog first. Two copies tolerate one failure, where a quorum system needs three. The protocol itself is part 5’s subject.
• wait_for_active_shards. By default only the primary must be active. You can require more active copies before a write starts, but it is a pre-check, not a guarantee.
• Peer recovery. A new copy fetches segment files from the primary (or a snapshot, where configured), then replays recent operations kept by soft deletes. At the default 40 MiB/s per node, one of Bazaar’s 25 GB shards takes 25 × 1,024 ÷ 40 ≈ 640 s, about 11 minutes, with one copy fewer meanwhile.
• Snapshots. Because segments never change, a snapshot to an object store copies only the segments created since the last one. Elastic calls snapshots “the only reliable and supported way to back up a cluster”; copying data directories is not a backup.
• The source of truth. Bazaar’s product database is the truth, and the index is a copy fed by change data capture. If all else fails you rebuild from it; a snapshot restore is just faster than reindexing 50 million documents.

Refresh, flush, fsync and merge compared

Cheat sheet

How to say it in the interview

“Each shard is a Lucene index of immutable segments. A write goes into an in-memory buffer and the translog, and by default the translog is fsynced on every copy before we acknowledge, so acknowledged writes survive a crash. About once a second a refresh writes the buffer as a new segment: searchable, not durable. A periodic flush is the Lucene commit that fsyncs segments and trims the translog, and recovery is last commit plus translog replay. Updates are delete plus re-add, so for Bazaar I keep exact stock out of the index and coalesce price changes. Merges clean up behind us, and I size nodes so the hot index fits in the page cache, with the heap at most half the RAM.”

Why is Elasticsearch near-real-time rather than real-time? Documents become searchable only when a refresh writes them into a new segment, once a second by default. Refreshing per write would create a tiny segment each time, costly to write, search and merge. GET by ID is real-time.

If a refresh writes a segment, why isn’t the data durable? The refresh writes into the OS page cache without an fsync. The fsynced translog carries durability until a flush commits the segments.

What is lost if a node loses power? With request durability, nothing acknowledged. With async, up to 5 s of acknowledged writes on that copy, unless a replica elsewhere still holds them. Uncommitted segments are discarded and rebuilt from the translog.

Why do update-heavy workloads hurt? Each update rewrites the whole document into a new segment and leaves a dead copy until a merge: more segments, write amplification measured at 6–14×, page-cache churn and dead documents in every query’s path.

How would you size memory on a 64 GB search node? Heap at most half and under the compressed-oops limit, about 26–30 GB; the rest is page cache, which must hold the node’s hot index files. Add nodes or shrink the index before tuning anything else.

Where to go next

• Next in the series: Choosing a search engine: the options compared, from PostgreSQL full-text search to hosted engines.
• Elasticsearch in depth: shards, replicas, sequence numbers and the distributed layer above these segments.
• Search data structures: what is inside each segment.
• Reads and writes: how databases store and fetch data: B-trees, LSM trees and the page cache in general.
• Write-ahead logging: the idea behind the translog.
• Disaster recovery: backups, restores and recovery objectives.
• Elasticsearch: when to reach for it in an interview.

References and further reading

1. Elasticsearch documentation: Near real-time searchelastic.coSegments, the commit point, refresh writing to the filesystem cache, and the 1-second refresh that only runs on shards searched in the last 30 seconds.
2. Elasticsearch documentation: Translog settingselastic.coWhy commits are too expensive per operation, request versus async durability, the 5 s sync interval and the 10 GB flush threshold.
3. Elasticsearch source: MergePolicyConfig.javagithub.comElasticsearch’s merge defaults: tiered policy with 5 GB segments, 8 per tier, 16 at once, 16 MB floor, 20% deletes; LogByteSize policy for time-based data.
4. Elasticsearch documentation: Tune for search speedelastic.coAt least half the memory for the filesystem cache, readahead, warming, and force-merging read-only indices.
5. Elasticsearch documentation: JVM settingselastic.coHeap at most 50% of RAM and below the compressed-oops threshold (26 GB safe, up to about 30 GB).
6. Elasticsearch documentation: Snapshot and restoreelastic.coSnapshots copy only new segments, and copying data directories is not a supported backup.
7. Apache Lucene 10.5 API: IndexWriterlucene.apache.orgFlush versus commit (fsync and segments_N), update as delete plus add, and the free space a force merge needs.
8. Apache Lucene 10.5 API: TieredMergePolicylucene.apache.orgHow merges are chosen, the default limits, and the warning about write amplification at low delete percentages.
9. Visualizing Lucene’s segment merges (Michael McCandless, 2011)blog.mikemccandless.comWrite amplification measured at 6.19 for append-only Wikipedia and about 14 for an update-heavy near-real-time run.
10. Use Lucene’s MMapDirectory on 64bit platforms, please! (Uwe Schindler, 2012)blog.thetaphi.deHow mmap reads index files straight from the OS cache, why it reserves address space rather than memory, and why the heap should stay small.