Serverless architecture

Serverless is an operating model in which you hand the cloud provider your code and your data, and it runs, scales and bills them for you. Functions such as AWS Lambda run your handlers only when an event arrives, managed services such as object storage and key-value databases hold your state, there are no servers for you to size or patch, idle costs fall to zero, and you pay per request. There are still servers, of course; they just stop being your problem.

Picture it first: app cabs versus owning a car

You can own a car or book app cabs. The car costs EMI, insurance and parking whether you drive or not, but it is always outside your door, and if you drive 80 km every day it is the cheaper choice. A cab costs nothing while you sit at home and you never service it, but on a Diwali night you may wait for one, every ride is billed by the kilometre, you can’t leave your luggage in it overnight, and nobody books a cab for a 12-hour road trip.

Map it back: owning the car is running servers or containers that you size and pay for around the clock. Booking cabs is serverless: pay per use and scale to zero. The wait is the cold start, the per-kilometre fare is per-request billing that overtakes ownership at steady heavy use, the luggage you can’t leave is state, and the road trip is a long-running job.

What serverless actually means

The 2019 Berkeley paper on serverless computing gives the cleanest definition: serverless = FaaS + BaaS. Function as a Service runs your code on demand: AWS Lambda, Azure Functions, Google Cloud Run functions, Cloudflare Workers. Backend as a Service covers the managed pieces that hold state and move data: object storage, key-value databases, queues and event buses. To count as serverless, the authors say, a service must scale automatically with no explicit provisioning and be billed based on usage.

They list three differences from “serverful” computing:

1. Compute and storage are decoupled. Functions are stateless; state lives in a separate managed service, and each scales and is priced on its own.
2. You run code without allocating resources. You hand over a function, not a server size.
3. You pay for what you use, not what you reserve. Billing follows execution time, not the number of machines.

So the honest summary is an operating model: no servers to manage, automatic scaling including to zero, and pay per use. The same idea now spans serverless containers (Cloud Run runs any container image this way) and edge runtimes.

The execution model

A function is a handler that runs when an event source fires:

• HTTP: a request through an API gateway or a function URL.
• Queues and streams: messages from a queue, records from Kinesis or Kafka, change events from a database.
• Object events: a file landing in blob storage, such as a scanned answer sheet to process.
• Schedules: a cron rule such as “every 5 minutes”.

How the event arrives decides what failure looks like. Synchronous invocation (HTTP) makes the caller wait for the result; Lambda allows 6 MB each way. Asynchronous invocation (object events, many pub/sub rules) drops the event on the platform’s internal queue and returns at once; payloads are capped at 1 MB, and on a function error Lambda retries twice by default. Poll-based sources (queues and streams) are read by a platform-run poller that invokes your function with batches.

Execution environments, cold and warm

Lambda runs each request in an execution environment, a small isolated sandbox that handles one request at a time. When a request arrives and no idle environment exists, Lambda creates one and runs the Init phase: it starts extensions and the runtime, then runs your code outside the handler (imports, SDK clients, configuration). Init is capped at 10 seconds. Then the Invoke phase runs your handler, and the environment is frozen until the next request.

That first request pays a cold start. AWS says cold starts typically hit under 1% of invocations, take from under 100 ms to over a second, and are billed. The Berkeley paper notes that loading libraries can dwarf starting the function itself: tens of seconds for a heavy stack.

Because environments are reused, anything created outside the handler survives between requests. Use that on purpose.

The cold-start fixes, from cheap to dear: a smaller package and lazy imports; SnapStart, which runs Init when you publish a version and resumes new environments from a snapshot; provisioned concurrency, which keeps a set number of environments initialised and billed while they wait; or a platform without the problem, such as Cloudflare Workers, which runs code in V8 isolates inside an already running process.

Scaling is measured in concurrency

Functions don’t scale in servers; they scale in concurrent executions:

concurrency = requests per second × average duration

At 2,000 requests a second and 250 ms each, that is 2,000 × 0.25 = 500 environments busy at once. Two Lambda numbers then matter:

• The regional quota. An account gets 1,000 concurrent executions per Region by default, shared by every function, and raisable to tens of thousands.
• The ramp rate. Each function can add at most 1,000 execution environments every 10 seconds. Requests beyond what it can create get a 429 throttle.

Two controls sit on top. Reserved concurrency guarantees a function a slice of the quota and also caps it, which protects whatever it calls downstream. Provisioned concurrency keeps environments initialised ahead of time and is billed while they wait.

Picture an exam-results portal. On a normal day it peaks at 300 requests a second; at 10:00 on results day, 30,000 a second arrive, a 100-fold jump. At 100 ms per request, concurrency must leap from 300 × 0.1 = 30 to 30,000 × 0.1 = 3,000. First, the default quota of 1,000 would cap the whole account, so you must raise it weeks ahead. Even then, at 1,000 environments per 10 seconds the function reaches 3,000 after about (3,000 − 30) ÷ 100 per second ≈ 30 seconds.

The shaded triangle is ½ × 29.7 s × 2,970 environments ≈ 44,100 environment-seconds, and each environment would have served 10 requests a second: about 440,000 throttled requests in the first half-minute. The fixes are known in advance, which is the point: put the results behind a CDN so most requests never reach a function, or schedule provisioned concurrency. Keeping 3,000 environments of 512 MB warm from 09:45 to 11:00 costs 3,000 × 0.5 GB × 4,500 s × $0.0000041667 ≈ $28. Serverless removes capacity planning for normal days, not for your biggest day. See autoscaling done right.

Hard limits, and how platforms differ

Every function platform trades limits for convenience. Lambda’s, per its quotas page:

Other platforms draw the lines elsewhere:

The column that matters most is requests per instance. A Lambda environment busy waiting 90 ms on a database is a whole environment you pay for; a Cloud Run instance at concurrency 80 shares that wait across 80 requests. Lambda itself now also offers Managed Instances, which run functions on EC2 instances you choose, billed at the EC2 price plus a 15% management fee and the usual request charge: closer to server pricing, for steady load.

The cost model, and where it crosses over

Lambda bills two things, with the same prices in Mumbai and North Virginia:

• Requests: $0.20 per million.
• Duration: memory × time, in GB-seconds: $0.0000166667 per GB-second on x86, $0.0000133334 on Arm (20% less).
• Free tier: 1 million requests and 400,000 GB-seconds a month.

Take a 512 MB function that runs for 100 ms. Each request uses 0.5 GB × 0.1 s = 0.05 GB-s, so it costs 0.05 × $0.0000166667 + $0.0000002 ≈ $0.00000103: about $1.03 per million requests.

The results portal. Off-season it serves 3 million requests a month. That is 3M × 0.05 = 150,000 GB-s, inside the free tier, plus 2 million billable requests × $0.20 per million = $0.40 a month, about ₹35 at an assumed ₹88 to the dollar. In results month it serves 150 million: (7.5M − 0.4M) GB-s × $0.0000166667 ≈ $118, plus 149 × $0.20 ≈ $30, so about $148. The year costs roughly 11 × $0.40 + $148 ≈ $152 (₹13,400).

The always-on alternative. One Fargate container with 1 vCPU and 2 GB costs $0.04048 + 2 × $0.004445 = $0.04937 an hour, or about $36 a month; two for availability cost $72 a month, $865 a year, and they still have to scale out by dozens of tasks on results day.

The crossover. Two containers cost the same as $72.08 ÷ $0.00000103 ≈ 70 million requests a month, an average of 70M ÷ 2.63M seconds ≈ 27 requests a second. On Arm the break-even moves to about 32 a second. Above that, steady traffic favours the containers: an API averaging 500 requests a second would cost 500 × 2.63M × $0.00000103 ≈ $1,360 a month on Lambda, against about $144 for four containers (assuming each handles 200 requests a second, plus a spare): roughly 9 times more.

The function bill is rarely the whole bill. The API gateway charges per request too, logs are billed per gigabyte ingested, and functions inside a private network may need NAT gateways. Count them all before you claim savings; more in cost-aware design.

State, connections and retries

Functions are stateless by design (stateful vs stateless design), so every byte that must outlive a request goes to a managed store. Key-value databases such as DynamoDB fit naturally: they speak HTTP, need no connection pool and scale per request (Cassandra and DynamoDB). Relational databases are where serverless designs most often break.

The connection storm. Each environment opens its own database connection. At 1,000 concurrent executions that is 1,000 connections, against a PostgreSQL max_connections that is typically 100 by default. The first 100 succeed and the rest fail with “too many clients”, at exactly the moment traffic peaks. Fixes, best first:

• Put a pooling proxy in between, such as Amazon RDS Proxy or PgBouncer, so thousands of short-lived clients share a small pool of real connections (connection pooling).
• Cap the function with reserved concurrency, say 200, so it can never open more connections than the proxy and database can take.
• Open the connection during Init and reuse it across invocations, never one per request.
• Use an HTTP data API or a store designed for many short clients.

Retries mean duplicates. Asynchronous invocations are retried twice after a function error, throttled events are retried for up to 6 hours, and queue sources deliver at least once. So every handler must be idempotent (idempotency). Take a UPI-style payment-success webhook: the gateway resends it if you don’t answer quickly, so the function writes the order update with a conditional put keyed on the transaction ID, and a second delivery changes nothing. Events that fail every attempt should land in a dead-letter queue or an on-failure destination that someone watches, not vanish.

Orchestration, not chains

The tempting design has function A call function B synchronously, which calls C. Each caller is billed while it waits, the timeouts stack, a retry at A re-runs B and C, and no one place knows where a request got to. Instead:

• Hand off through a queue when the next step can happen later.
• Use a workflow engine such as AWS Step Functions when the steps, retries, timeouts and compensations matter; it records each execution’s state (multi-step workflows, workflow engines).
• For long jobs, split the work into checkpointed steps, or run it on a container or batch service. A 2-hour job does not fit Lambda’s 15 minutes; Lambda’s newer durable functions can checkpoint and run for up to a year (long-running jobs).

The event-driven architecture article covers choreography versus orchestration in depth.

Observability, testing and lock-in

A request may cross an API gateway, three functions, a queue and two managed stores, so debug it with a correlation ID on every log line and distributed tracing across the hops (observability). Track cold-start rate and init duration, throttles, concurrency, iterator age for streams, and dead-letter queue depth. Local emulators are partial; run integration tests against real services in a sandbox account.

Lock-in is real but uneven. Your business logic ports easily; the event shapes, triggers, permissions and managed services around it do not. Keep handlers thin over a plain core library, use standard formats where you can, and if portability matters, prefer serverless containers: the same image runs on Cloud Run, on a Kubernetes cluster with Knative (which can also scale to zero), or on a plain VM.

A real case: Prime Video’s monitoring service

In 2023 the Prime Video team described one internal service, which inspects live streams for defects such as frozen video and audio drift. The first version was built from serverless parts: Step Functions orchestrated a media converter and defect detectors on Lambda, with video frames passed through S3. The team chose it to build quickly. It hit a hard scaling limit at about 5% of the expected load: the workflow made several state transitions for every second of every stream, which hit account limits and is billed per transition, and the frames meant a very high number of S3 calls.

The team kept the same components but ran them in one process on ECS, passing frames in memory. Infrastructure cost fell by over 90%, and the service now monitors thousands of streams. Their own conclusion was that serverless and microservices work at high scale, but the choice is case by case.

When not to use serverless

The Berkeley paper names four limits of today’s platforms: inadequate storage for fine-grained operations, no fine-grained coordination between functions, poor performance for communication patterns such as broadcast and shuffle, and unpredictable performance (cold starts and varying hardware). In practice, avoid functions for:

• Steady, high throughput. Above the crossover, about 27 requests a second in the example, you pay a premium on every request.
• Long-running jobs past the platform limit: video encodes, big exports, model training.
• Latency-critical paths sensitive to tail latency, unless you pay for provisioned concurrency.
• Heavy in-memory state: large caches, game rooms, anything that must stay warm and shared.
• Long-lived connections at scale, such as WebSockets with chatty traffic (real-time connections).
• Chatty, data-heavy pipelines between steps, as in the Prime Video case.

It shines for spiky or unpredictable traffic, event glue (file processing, webhooks, scheduled jobs), new products with unknown load, and the long tail of internal tools that would otherwise idle on servers.

Choosing a compute model

Going deeper: failure modes

One noisy function starves the rest. The regional quota is shared, so a runaway batch function can throttle your checkout API. Give critical functions reserved concurrency, and cap the rest.

Retry storms. A downstream outage makes every invocation fail; async retries and client retries multiply the load just as the dependency tries to recover. Back off with jitter, cap retries, and send failures to a dead-letter queue (retry storm, timeouts and backoff).

Poison batches. With queue and stream sources, one bad record can fail a whole batch and be retried forever, blocking a stream shard. Report partial batch failures and route repeat offenders to a dead-letter queue.

Cost explosions. Unlimited scale is also unlimited spend: a loop, a crawler or a load test can run up thousands of dollars overnight. Set concurrency caps and budget alarms on day one.

Timeout mismatches. If the gateway gives up before the function finishes, the client sees an error while the function keeps working and may commit the write anyway. Keep each timeout shorter than the one in front of it, and make the work idempotent.

Cheat sheet

How to say it in the interview

“Serverless is an operating model: functions plus managed services, no servers to run, scale to zero and pay per request. For this results portal it fits well: it idles most of the year, so it costs almost nothing, and I don’t size servers for a spike I see once a year. The day still needs planning: 30,000 requests a second at 100 ms is 3,000 concurrent executions, so I raise the account quota, pre-warm with provisioned concurrency and serve results through a CDN. Results live in a key-value store over HTTP, so there are no connection limits to hit, and anything touching Postgres goes through a proxy with a concurrency cap. Async work is idempotent with dead-letter queues. If traffic became steady above a few dozen requests a second, I’d move the hot path to containers.”

How do you deal with cold starts? First measure: they usually hit under 1% of requests. Shrink the package, move heavy imports out of the hot path, and create clients once in Init. For latency-critical paths, use provisioned concurrency for the expected peak, or SnapStart where supported; or choose a platform where one instance serves many requests.

How do you stop Lambda exhausting your Postgres connections? Connect through a pooling proxy such as RDS Proxy or PgBouncer, cap the function with reserved concurrency below what the database can hold, and open the connection once per environment. Better still, keep high-fan-out data in a store built for HTTP access.

When is serverless more expensive? At steady, high utilisation. A 512 MB, 100 ms function costs about $1 per million requests; two always-on containers cost about $72 a month, so above about 27 requests a second, averaged over the month, the containers win, and at 500 a second Lambda costs about nine times more. Add gateway, log and NAT charges before comparing.

How do you handle a 2-hour job? Not in one function: Lambda stops at 15 minutes. Split it into checkpointed chunks driven by a workflow engine or a queue, use durable functions that checkpoint between steps, or run it as a container or batch job that a function merely starts and monitors.

Where to go next

• Event-driven architecture: the events that trigger most functions.
• Monolith vs microservices: boundaries, cost and the Prime Video story.
• Choosing an architecture style: where serverless fits among the styles.
• Autoscaling done right: scaling signals, warm-up and capacity for spikes.
• Cost-aware design: counting the whole bill.
• Connection pooling: why proxies save databases.
• Multi-step workflows and long-running jobs: beyond one function.
• Idempotency: making retries safe.

References and further reading

1. Cloud Programming Simplified: A Berkeley View on Serverless Computing (Jonas et al., 2019)arxiv.orgServerless = FaaS + BaaS, the three distinctions from serverful computing, and the four limitations: storage, coordination, communication patterns and predictable performance.
2. AWS Lambda Developer Guide: Lambda quotasdocs.aws.amazon.com1,000 concurrent executions per Region by default, 1,000 new execution environments per function every 10 seconds, 15-minute timeout, 128–10,240 MB memory, 6 MB and 1 MB payloads, /tmp up to 10 GB.
3. AWS Lambda Developer Guide: Understanding the execution environment lifecycledocs.aws.amazon.comInit, Invoke and Shutdown phases, the 10-second Init limit, environment reuse, cold starts in under 1% of invocations and billed, SnapStart, and durable functions that run up to a year.
4. AWS Lambda pricingaws.amazon.com$0.20 per million requests, $0.0000166667 per GB-second on x86 and $0.0000133334 on Arm, the monthly free tier, provisioned concurrency and Lambda Managed Instances pricing.
5. AWS Lambda Developer Guide: Error handling and retries for asynchronous invocationdocs.aws.amazon.comTwo retries by default after a function error, up to 6 hours of retries after throttling, and dead-letter queues or on-failure destinations.
6. AWS Fargate pricingaws.amazon.comPer-second billing for vCPU and memory; the always-on container prices used in the break-even example.
7. Google Cloud Run: Maximum concurrent requests per instancecloud.google.comContainer instances that serve up to 1,000 concurrent requests, a default of 80 per vCPU, and why concurrency drives cost.
8. Azure Functions: Flex Consumption planlearn.microsoft.comScale to zero, per-function scaling, always-ready instances against cold starts, private networking and instance memory sizes.
9. Cloudflare Workers: Limitsdevelopers.cloudflare.comCPU-time limits (10 ms free; 30 seconds by default and up to 5 minutes paid) and 128 MB per isolate, where one isolate serves many requests.
10. Scaling up the Prime Video audio/video monitoring service and reducing costs by 90% (Prime Video Tech, 2023, archived)web.archive.orgWhy one stream-monitoring service moved from Step Functions and Lambda to a single process on ECS, and what that did to cost and scale.