Peer-to-peer (P2P) architecture

In a peer-to-peer (P2P) system every participant, a peer, is both a client and a server: it downloads from others and uploads to them. The bandwidth, storage and compute come from the participants themselves, so capacity grows as users join, and no central server sits on the data path. The price is that you must find data without a central index, survive peers that come and go, get through home routers and mobile NATs, and trust strangers.

Picture it first: notes before exams in a hostel

The night before an exam, 500 hostel students want the topper’s notes. If the topper hands each student a photocopy, they spend the whole night at the copier and most students get nothing. Now suppose every student who gets a copy makes copies for two others. The topper copies a handful; the copies spread through the hostel in a few rounds, and the more students there are, the more copiers there are.

Details appear quickly. A notice board lists who has which chapter, so you don’t knock on every door. Pages are numbered and stamped, so a forged page is easy to spot. Some students take copies and never make any; people stop sharing with them. Some rooms only allow visitors who were expected, so two students in such rooms can only swap pages through the warden.

Map it back. Each student is a peer, the topper is the original seed, the notice board is a tracker or index, the stamps are piece hashes, the students who never share are free-riders, the strict rooms are NATs, and the warden passing pages is a relay.

What makes a system peer-to-peer

In client-server architecture, clients ask and servers answer, and the server pays for every byte. In P2P, connections are symmetric: BEP 3, the BitTorrent specification, says messages in both directions look the same and data can flow either way. Three properties follow:

• Resources come from users. Every peer brings upload bandwidth, disk or CPU, so supply grows with demand.
• No central data path. Content moves directly between peers, so no single server carries it all.
• Peers are unreliable. They join, leave, crash, sit behind NATs and may lie, and the design must assume this.

Pure P2P has no special node at all; early Gnutella-style file sharing worked this way. Hybrid P2P keeps a small central service for coordination (a tracker, an index, a signalling server or a login) and moves the heavy traffic between peers. Almost every successful system is hybrid: BitTorrent with trackers, video calls with signalling servers, Windows updates with a cloud service that tells devices which peers to use. Central coordination is cheap because it handles kilobytes; the peers carry the gigabytes.

Why it scales: the distribution arithmetic

Distribute one file of size F to N users. The server uploads at u_s; each peer uploads at u_i and downloads at up to d_min.

Client-server. The server must send N full copies, so the time is at least N × F ÷ u_s, and no user can finish faster than F ÷ d_min.

Peer-to-peer. The server must still send at least one full copy (F ÷ u_s), the slowest downloader still needs F ÷ d_min, and the N copies must be uploaded by someone, using the combined upload of the server and every peer: N × F ÷ (u_s + Σu_i). The time is at least the largest of these three. As N grows, Σu_i grows with it, so the last term levels off instead of growing without limit.

Work it for a 2 GB game patch (16 Gbit) going to 1 million players, with a 10 Gbit/s server and players who each upload at 5 Mbit/s and download at 50 Mbit/s:

• Client-server: 10⁶ × 16 Gbit ÷ 10 Gbit/s = 1.6 million seconds ≈ 18.5 days.
• P2P: the peers add 10⁶ × 5 Mbit/s = 5,000 Gbit/s of upload, so 1.6 × 10⁷ Gbit ÷ 5,010 Gbit/s ≈ 3,200 s ≈ 53 minutes. The other terms are 1.6 s and 320 s, so the upload term decides.

This is a lower bound that assumes perfect scheduling; real swarms are slower because of uneven links and churn. But the gap is the point: to match 53 minutes from servers you would need about 5 Tbit/s of server or CDN capacity, which the players provide for free.

The same arithmetic explains the costs. That 5,000 Gbit/s only exists while peers stay online and actually upload, so P2P has to cope with churn (Cohen’s BitTorrent paper notes peers rarely stay connected for more than a few hours, often only minutes), free-riders who download without uploading, NATs that stop peers accepting connections, and trust, because every byte comes from a stranger. The rest of this page is how real systems handle each.

Finding things: index, flooding or DHT

Before peers can exchange data, a peer must learn who has it. There are three classic answers.

A central index. Peers register what they hold with a server, and searches go to that server, as in Napster-style file sharing and today’s BitTorrent trackers. Lookups take one round trip and the index sees everything. But it is a single point of failure, a scaling bottleneck and a single place to shut down.

Unstructured flooding. Peers connect to a few random neighbours, and a query is passed from neighbour to neighbour with a hop limit, the TTL. The Dynamo paper describes this first generation (Freenet, Gnutella) as arbitrary overlay links with queries flooded through the network. It needs no special node, but it is expensive: with 4 neighbours and a TTL of 7, one query sends up to 4 + 12 + 36 + 108 + 324 + 972 + 2,916 = 4,372 messages (each peer forwards to its 3 other neighbours), and a rare file outside that radius is never found.

Structured overlays: distributed hash tables (DHTs). Give every node and every key an ID in the same space, typically a hash, and make each key the responsibility of the nodes whose IDs are closest to it. This is the same idea as consistent hashing, with one difference: no node knows the whole ring, so a lookup is routed through a few nodes, each closer to the key. Chord, Pastry and Kademlia are the famous designs; Kademlia won in practice and powers the BitTorrent DHT.

Kademlia: XOR distance and k-buckets

Kademlia, by Petar Maymounkov and David Mazières (2002), uses 160-bit IDs for nodes and keys, and defines the distance between two IDs as their bitwise XOR, read as a number. Two IDs that share a long leading prefix are close. XOR is symmetric (d(x, y) = d(y, x)), so the nodes that send you queries are exactly the kind you want in your routing table, and every message teaches you something useful.

Each node keeps a list, a k-bucket, for each distance range 2ⁱ to 2ⁱ⁺¹, holding up to k contacts. So a node knows many nodes near its own ID and only a few far away. When a bucket is full and a new node appears, Kademlia pings the least-recently-seen contact and keeps it if it answers. The reason is measured: in the Gnutella traces the authors used, the longer a node had been up, the more likely it was to stay up another hour. As a bonus, an attacker can’t flush your table by flooding the network with fresh nodes.

To look up a key, a node asks the α closest nodes it knows (α = 3 in the paper) in parallel for nodes closer to the key, then asks the closer ones it learns about, until no one closer turns up. Each step reaches a node sharing at least one more leading bit with the key, so the paper proves lookups take about ⌈log₂ n⌉ + c steps. For 10 million nodes, log₂ 10⁷ ≈ 23.3, so the bound is about 24 steps. Parallel queries and well-filled tables usually finish sooner, and the paper’s optimised variant, routing b bits per hop, cuts the bound to about log₂ n ÷ b: with b = 5, about 5 hops.

BitTorrent, end to end

BitTorrent is the most successful P2P protocol and a good model answer for “distribute a big file to many clients”. BEP 3, its specification, states the goal: when many people download the same file at once, they upload to each other, so the source can serve very large numbers of downloaders with only a modest increase in its own load.

1. Describe the content. The publisher splits the file into fixed-size pieces, almost always a power of two and most commonly 256 KiB, and writes a metainfo (.torrent) file containing the tracker’s URL and an info dictionary with the file name, the piece length and the SHA-1 hash of every piece. The SHA-1 of the encoded info dictionary is the info hash, the torrent’s identity everywhere. A magnet link carries just that info hash.

2. Find peers. The client announces itself to the tracker with the info hash, its peer ID, its port and how much it has left, and gets back a list of peers. Without a tracker, it asks the DHT (BEP 5): a Kademlia network where node IDs live in the same 160-bit space as info hashes, buckets hold K = 8 nodes, and each node effectively becomes a small tracker for the info hashes closest to its ID. To announce, a node must present a token it recently received from the node it is announcing to, so a malicious host can’t sign up other people’s addresses.

3. Swap pieces. Peers exchange bitfields of the pieces they have, announce each new piece with a have message, and request data in 16 KiB blocks. A peer reports a piece only after its SHA-1 matches the hash in the metainfo, so a corrupt or malicious block is detected and fetched again from someone else.

4. Choose pieces wisely. Cohen’s 2003 paper describes the policy. Start with a random first piece, so you quickly have something to share. Then go rarest first: download the pieces the fewest of your peers have, which spreads rare pieces before the peers holding them leave and keeps you useful to others. At the very end, endgame mode requests the last blocks from everyone and cancels the duplicates as they arrive, so one slow peer can’t hold up completion.

5. Reward uploaders: tit-for-tat. Each peer uploads to a few peers at a time and chokes (refuses) the rest. BEP 3 describes the deployed algorithm: four upload slots, reassigned every 10 seconds to the interested peers you download from fastest (Cohen’s paper uses a rolling 20-second average). One slot is an optimistic unchoke, given regardless of rate and rotated every 30 seconds, so newcomers get a first piece and better partners can be discovered. A peer that hasn’t received a block from someone for a minute treats itself as snubbed and stops uploading to them except optimistically. Once a download completes, the peer becomes a seeder and unchokes the peers it can upload to fastest.

Getting through NATs: STUN, TURN and ICE

Most peers cannot accept an inbound connection. A home router or a carrier’s network address translator (NAT) shares one public IP address among many devices and drops packets nobody inside asked for. Many mobile operators, in India and elsewhere, put subscribers behind carrier-grade NAT, so a phone usually has no inbound path at all. Two such peers can’t simply connect, and P2P systems use a toolkit from the IETF to get around it:

• STUN (RFC 8489) lets an endpoint ask a server on the internet which public IP address and port its NAT allocated, check connectivity and keep NAT bindings alive. The RFC is explicit that STUN is a tool, not a NAT traversal solution by itself.
• Hole punching. Both peers send packets to each other’s public address at about the same time. Each outgoing packet opens a mapping in the sender’s own NAT, so the other side’s packets are let in. It works when a NAT keeps the same public port whatever the destination, and fails with NATs that pick a new port for each destination.
• TURN (RFC 8656) is the fallback: a relay on the internet that forwards every packet. It is very likely to work, but the RFC warns it comes at a high cost to whoever runs it, so use it only when no direct path exists. A relayed 1:1 video call at 1.5 Mbit/s each way pushes 3 Mbit/s into the relay and 3 Mbit/s out; 10,000 such calls need 30 Gbit/s of relay egress.
• ICE (RFC 8445) ties them together. Each peer gathers candidates: host addresses, server-reflexive addresses from STUN and relayed addresses from TURN. The peers exchange them through a signalling channel and run connectivity checks on candidate pairs in priority order; the recommended type preferences are 126 for host, 100 for server-reflexive and 0 for relayed. ICE searches for a direct path first and uses the relay only if none works.

WebRTC is this stack built into browsers and mobile SDKs: ICE with STUN and TURN for connectivity, encrypted media streams and data channels between peers. It leaves signalling to the application, typically a WebSocket server (real-time connections). A 1:1 call sends media peer to peer. A group call is different: in a full mesh every sender uploads a copy to everyone else, so with 6 people each uploads 5 streams, about 7.5 Mbit/s at 1.5 Mbit/s per stream, more than many home and mobile uplinks can carry. Group calls therefore use a selective forwarding unit (SFU): each participant uploads once to a server that forwards streams to the others. That is the hybrid answer again (design Zoom).

P2P ideas inside server systems

You rarely build a file-sharing network, but P2P techniques run inside many server-side systems.

Gossip. Dynamo, Amazon’s key-value store, lists symmetry (every node has the same responsibilities) and decentralisation (favour peer-to-peer techniques over central control, which had caused outages) among its design principles. Membership spreads by gossip: each node contacts a random peer every second and they reconcile their views of the ring. A few seed nodes known to everyone prevent the ring splitting into groups that don’t know about each other. Dynamo calls itself a zero-hop DHT: every node knows enough to route a request straight to the right node, avoiding multi-hop routing’s latency at high percentiles. Cassandra, which borrowed heavily from Dynamo, uses gossip for membership too (gossip protocol, Cassandra and DynamoDB).

Content addressing. IPFS names data by a content identifier (CID) derived from a cryptographic hash of the content, not by where it is stored. Any peer can serve it, and the receiver checks the hash on arrival, so you can fetch from strangers without trusting them. BitTorrent’s info hash and piece hashes apply the same idea.

Update distribution. Windows Delivery Optimization downloads update packages from other devices on the network and from caches, falling back to Microsoft’s HTTP servers. Microsoft’s own IT department reports that more than 76% of update content came from peer devices instead of the internet, a real-world version of the arithmetic above.

Blockchains are P2P networks too: nodes gossip transactions and blocks, and a consensus protocol makes untrusted nodes agree on one history. That agreement is expensive, so reach for it only when there is genuinely no party everyone trusts.

Security and trust

Every peer is a stranger, and the system must work even when some of them lie.

• Poisoning. Peers can serve corrupt or fake data. Hashes defeat corruption: BitTorrent checks every piece against the metainfo, and content addressing checks every block. Index poisoning, fake entries pointing at junk, still wastes time, so prefer trusted sources for the root hash.
• Sybil attacks. One attacker creates thousands of identities, for example node IDs chosen to sit next to a target key in a DHT, and then controls its lookups. Defences make identities costly: limits per IP address or subnet, proof of work, or a central sign-up.
• Eclipse attacks. The attacker fills a victim’s routing table so that every peer it sees is malicious. Kademlia’s preference for long-lived contacts helps, because a flood of new nodes can’t push out old ones that still answer.
• Free-riding. Users download without uploading. Tit-for-tat makes uploading the way to download fast; private communities add sharing ratios.
• Privacy and abuse. Peers see each other’s IP addresses, and WebRTC’s host candidates can expose local addresses. With no central operator, there is no one to moderate content or answer a takedown request, which is a legal risk as well as a design choice.

Client-server or P2P?

When to propose P2P in an interview

P2P fits when the expensive thing is moving the same large bytes to many places, or when the participants are few and want a direct path:

• Large static content to many clients: game patches, OS updates, app installs on a campus or office network, often as an offload beside a CDN with HTTP as the fallback.
• Real-time media between a few participants: 1:1 calls and small meetings over WebRTC, with an SFU once groups grow.
• Local and offline-first sync: devices on the same LAN or nearby sharing files without the internet.
• Decentralised systems where no single party is trusted or allowed to control the data.

It is the wrong tool when you need strong consistency or transactions, central control (moderation, audit, instant revocation), predictable latency, or when users pay for upload: a phone on a metered plan uploading for strangers is a bad user experience. At small scale, a server or a CDN is simply cheaper to build and run.

Going deeper: failure modes

Churn is the normal state. Peers often stay minutes, not days. Replicate every DHT entry on several nodes (k contacts per bucket, K = 8 in BitTorrent’s DHT), refresh routing tables (BEP 5 treats a node as good only if it was heard from in the last 15 minutes) and republish what you announce.

Bootstrap nodes are a soft centre. A new node needs at least one known peer to join. Those well-known bootstrap nodes, like Dynamo’s seeds, are a small point of centralisation: run several, and let clients save their routing tables between runs, as BEP 5 recommends.

NAT forces relays. When both sides are behind restrictive NATs, traffic falls back to TURN and P2P quietly becomes client-server with your bandwidth bill. Measure the relay rate and budget for it.

Long tails disappear. Popular content has thousands of seeders; rare content may have one, and when it leaves, the content is gone. Rarest-first helps inside a swarm; for availability guarantees, keep a server-side seed or a CDN copy.

Upload is the scarce resource. Consumer links usually upload far less than they download: in the patch example, 5 Mbit/s up against 50 down. Swarm capacity is the sum of uploads, so size your plans on upload, not download.

Cheat sheet

How to say it in the interview

“For distributing a 2 GB patch to a million players, one 10 Gbit/s origin would need about 18 days, and a CDN fast enough would cost us terabits a second. So I’d make it hybrid peer-to-peer. Our servers publish a manifest with the piece hashes and run a tracker; clients download 256 KB pieces from each other, rarest first, verify every piece against its hash, and reward peers that upload back. The CDN stays as a fallback for rare pieces and for clients that can’t connect. Behind NATs we use STUN and hole punching, with TURN only when needed, and we let users cap their upload, especially on mobile data.”

Why doesn’t P2P need more servers as users grow? Each new user brings upload capacity as well as demand. Total upload is the server’s plus the sum of all peers’, so the time to serve N users levels off instead of growing linearly with N.

How does a peer find who has a piece? Through a tracker that lists the swarm’s peers, or trackerless through a DHT: hash the content to a 160-bit key and look it up in Kademlia, which reaches the nodes storing that key in about log₂ n steps. Peers then trade bitfields to learn exactly which pieces each has.

What if both peers are behind NAT? Each learns its public address from STUN, the two swap candidates through a signalling server, and ICE tries hole punching. If the NATs don’t allow it, traffic goes through a TURN relay, which works but costs server bandwidth.

How do you stop free-riders? Make uploading pay: BitTorrent’s tit-for-tat uploads to the peers that upload to you fastest, re-chosen every 10 seconds, with one optimistic slot rotating every 30 seconds. Peers that never upload get choked by everyone and download slowly.

How is a DHT different from consistent hashing in a database? Both map keys to the nodes closest on a hash ring or space. A database cluster like Dynamo is a zero-hop DHT: every node knows the whole ring and routes in one hop. An internet DHT can’t know millions of churning nodes, so each node knows only O(log n) contacts and lookups take about log n hops.

Where to go next

• Client-server architecture: the model P2P is contrasted with.
• Choosing an architecture style: where P2P fits among the styles.
• Consistent hashing: the key-to-node mapping under every DHT.
• Gossip protocol: membership and failure detection the P2P way.
• Content delivery network: the server-side alternative for large content.
• Design Zoom: WebRTC, SFUs and where P2P stops.
• TCP vs UDP: why NAT traversal and DHTs live on UDP.

References and further reading

1. BEP 3: The BitTorrent Protocol Specificationbittorrent.orgMetainfo files, info hash, pieces with SHA-1 hashes (256 KiB most common), 16 KiB requests, the tracker protocol, choking every 10 s with four unchoked peers, the 30-second optimistic unchoke and endgame mode.
2. BEP 5: DHT Protocolbittorrent.orgThe Kademlia-based trackerless DHT: 160-bit node IDs, XOR distance, buckets of K = 8, iterative lookups, announce tokens and the 15-minute good-node rule.
3. Incentives Build Robustness in BitTorrent (Bram Cohen, 2003)bittorrent.orgChurn (peers rarely stay more than a few hours), random-first then rarest-first piece selection, endgame mode, tit-for-tat choking, optimistic unchoking and anti-snubbing.
4. Kademlia: A Peer-to-peer Information System Based on the XOR Metric (Maymounkov and Mazières, 2002)scs.stanford.eduThe XOR metric, k-buckets that prefer long-lived contacts, parallel lookups with α = 3, ⌈log n⌉ + c lookup steps, and routing b bits per hop.
5. RFC 8489: Session Traversal Utilities for NAT (STUN)rfc-editor.orgHow an endpoint learns the IP address and port its NAT allocated, checks connectivity and keeps NAT bindings alive.
6. RFC 8656: Traversal Using Relays around NAT (TURN)rfc-editor.orgRelaying when no direct path exists, its high bandwidth cost to the provider, and its use only after ICE fails to find a direct path.
7. RFC 8445: Interactive Connectivity Establishment (ICE)rfc-editor.orgGathering host, server-reflexive and relayed candidates, connectivity checks, and the recommended type preferences of 126, 100 and 0.
8. Dynamo: Amazon’s Highly Available Key-value Store (DeCandia et al., 2007)allthingsdistributed.comSymmetry and decentralisation as design principles, a zero-hop DHT, once-a-second gossip for membership, seed nodes, and the unstructured versus structured P2P generations.
9. Content Identifiers (CIDs) (IPFS documentation)docs.ipfs.techContent addressing: identifiers derived from a cryptographic hash of the content, not its location.
10. What is Delivery Optimization? (Microsoft Learn)learn.microsoft.comWindows updates downloaded from peers and caches with HTTP fallback, and Microsoft IT getting more than 76% of content from peer devices.