Encryption and key management

Encryption turns data into ciphertext that is useless without a key; key management decides who holds that key, where it lives, how often it changes and how it dies. The cipher is the easy part: AES-256-GCM from a vetted library is one function call. The design work is choosing where to encrypt, so the attacker you actually fear sees only ciphertext, and building a key hierarchy (data keys wrapped by key-encryption keys in a KMS) that makes rotation, per-tenant control and deletion cheap.

Picture it first: a bank’s locker room

A bank keeps your valuables in a locker with its own small key. That key is sealed in a tamper-evident envelope taped to the locker door, and only one machine, in the manager’s office, can open such envelopes. The machine holds one master key per customer, never lets it out, checks the clerk’s ID before every opening and writes each one in a register.

Now count the thieves. Someone who carries off a locker, or photographs every envelope, gets nothing. A clerk allowed to use the machine can open lockers, but the register shows exactly which ones. And when a customer closes their account, the bank destroys that customer’s master key: every one of their envelopes, including copies in the off-site warehouse, can never be opened again.

Map it back: the locker is ciphertext, the small key a data encryption key (DEK), the envelope the wrapped DEK stored beside the data, the machine a KMS backed by HSMs, each master key a key-encryption key (KEK), the register the audit log, the clerk your app server, and destroying a master key crypto-shredding.

Three states of data

Data spends its life in three states, and each has its own attacker and its own tool.

In transit means bytes on a network, exposed to anyone on the path: rogue Wi-Fi, a compromised router, a sniffer inside your VPC. Use TLS 1.3 at the edge and mutual TLS between services; the handshake, internal PKI and certificate rotation are covered in HTTP and HTTPS and service-to-service auth and mTLS. The one rule here: no plaintext hops, including load balancer to backend and database replication.

At rest means bytes on storage: database files and the write-ahead log, object storage, backups and snapshots, and the places teams forget, such as logs, search indexes, cache snapshots and analytics exports. The attacker is whoever gets a copy.

In use means plaintext in memory while the CPU works on it. Every ordinary design decrypts here, so an attacker with code execution on the app server, or a memory dump, sees plaintext. Confidential computing narrows the gap: trusted execution environments (TEEs) such as AMD SEV-SNP, Intel TDX and AWS Nitro Enclaves encrypt a VM’s or enclave’s memory in hardware, and remote attestation lets a KMS release a key only to code whose measurement it recognises. You pay with harder debugging, some overhead and trust in the chip vendor. Use it for narrow, high-value jobs such as key handling or analytics across organisations that don’t trust each other.

Four tools that get confused

“Encrypted”, “hashed” and “encoded” are different tools with different guarantees, and interviewers notice when they are mixed up.

Passwords are a fifth case: store them with a deliberately slow hash such as Argon2id, never with encryption; see authentication, sessions and JWTs.

Symmetric encryption (AES, ChaCha20) uses one key to encrypt and decrypt. It is fast enough for bulk data, but both sides must already share the key. Asymmetric encryption (RSA, elliptic curves) lets anyone encrypt to your public key while only you decrypt, which solves key distribution but is slow and tiny: AWS KMS will RSA-encrypt at most 190 bytes with a 2,048-bit key and OAEP-SHA-256.

So real systems are hybrid: asymmetric cryptography agrees on or wraps a fresh symmetric key, and the symmetric key does the bulk work. TLS 1.3 does exactly this per connection, HPKE (RFC 9180) does it per message, and envelope encryption, below, does it per object.

Authenticated encryption: the only kind to use

Old modes such as CBC or CTR hide data but do not detect tampering: flip a bit of CTR ciphertext and the same plaintext bit flips. Authenticated encryption with associated data (AEAD) fixes that. Encryption outputs ciphertext plus a tag, and decryption refuses to return anything if a single bit of the ciphertext, the tag or the associated data has changed. The two AEADs you will meet:

• AES-256-GCM: the default almost everywhere. CPUs have AES instructions, and AWS KMS uses AES-GCM with 256-bit keys inside its HSMs.
• ChaCha20-Poly1305 (RFC 8439): fast in pure software, so it suits phones and older chips without AES hardware.

Nonces: the one rule you must not break

Both take a nonce (number used once), 96 bits for GCM in its standard form, alongside the key. The rule: never encrypt two messages with the same key and nonce. NIST’s GCM standard requires the chance of that ever happening to be at most 2⁻³² and calls the rule “almost as important as the secrecy of the key”. A repeat breaks two things at once. Confidentiality: both messages share a keystream, so XORing the ciphertexts gives the XOR of the plaintexts. Authenticity: the attacker can likely recover GCM’s internal authentication subkey and forge messages that decrypt as valid.

Random 96-bit nonces are fine, but only up to a point. The chance that any two of n random nonces collide is about n² ÷ 2⁹⁷, so at n = 2³² it is 2⁶⁴ ÷ 2⁹⁷ = 2⁻³³, just inside NIST’s bound. That is why SP 800-38D caps a key used with random IVs at 2³² encryptions (about 4.3 billion). It sounds huge until you divide: a UPI-style payments service encrypting 20,000 records per second under one key reaches it in 4,294,967,296 ÷ 20,000 ≈ 214,748 s, about 2.5 days.

Three ways out: use many keys (envelope encryption gives each object or tenant its own); use a nonce-misuse-resistant mode such as AES-GCM-SIV, which, if a nonce repeats, leaks only that two messages were equal; or use XChaCha20-Poly1305, whose longer nonce can safely be random for far more messages.

Associated data binds ciphertext to its place

Associated data (AD) is authenticated but not encrypted, and it must match exactly at decryption. Use it to pin each ciphertext to where it belongs: tenant ID, table, row ID and column. Without it, an attacker with write access to the database can copy your encrypted PAN (tax ID) or Aadhaar-like number into their row, and the app will happily decrypt and display it. With AD, the tag check fails. AWS KMS calls the same idea encryption context.

import tink
from tink import aead

aead.register()
# In production, load a keyset that is itself wrapped by your KMS key.
keyset = tink.new_keyset_handle(aead.aead_key_templates.AES256_GCM)
cipher = keyset.primitive(aead.Aead)

ad = b"tenant=acme-bank|table=customers|row=8812|col=pan"
blob = cipher.encrypt(b"ABCDE1234F", ad)  # Tink picks a fresh random nonce
plain = cipher.decrypt(blob, ad)          # raises if blob is moved to another row

Never roll your own

Every failure above is a usage mistake, not a broken cipher. OWASP’s advice on custom algorithms is two words long: “Don’t do this.” Use a misuse-resistant library such as Google Tink or libsodium, which generates nonces, puts a key ID in each ciphertext so rotation works, and offers no unauthenticated mode to pick by accident.

Which layer stops which attacker

“Is the data encrypted at rest?” has four very different yes answers. Each layer decrypts at a different point, and protects only against attackers who sit beyond that point.

• Full-disk or volume encryption (LUKS, BitLocker, cloud block-storage encryption): the storage layer decrypts for the operating system, so it stops someone who steals or recycles a drive, and nothing else.
• Transparent data encryption (TDE): the engine encrypts data files, logs and file backups and decrypts for every authenticated session, so copied files are safe but a DBA, a stolen password or a SQL injection sees plaintext.
• Application or field-level encryption: your code encrypts chosen fields before the database sees them, with keys in a KMS. Replicas, backups, DBAs and SQL injection dumps see only ciphertext, but you lose SQL filtering, sorting and indexing on those fields.
• Client-side or end-to-end encryption: the user’s device encrypts and only devices hold keys. Even a compromised server sees ciphertext, but it can no longer search, rank, scan or recover anything.

The table interviewers want is the same picture, read by threat:

Envelope encryption

Why not send every record to the KMS and let it encrypt? Three reasons.

• Size. AWS KMS Encrypt accepts at most 4,096 bytes of plaintext with a symmetric key. A scanned discharge summary is megabytes.
• Latency and quota. Each call is a network round trip to a shared, rate-limited service (numbers below).
• Blast radius and exposure. One key for everything means one leaked key exposes everything, and the KMS would see all your data.

Envelope encryption fixes all three. Your service encrypts data locally with a data encryption key (DEK), asks the KMS to encrypt (wrap) only that DEK under a key-encryption key (KEK) that never leaves the KMS’s HSMs, and stores the wrapped DEK right next to the ciphertext. Here is a health-claims platform, whose tenants are insurers, storing one claim PDF:

On write (steps 1–4), the service calls GenerateDataKey on the tenant’s KEK with the tenant and object ID as encryption context, gets a plaintext DEK plus a wrapped copy, encrypts locally, stores both and wipes the DEK. On read (steps 5–8), it sends only the wrapped DEK to Decrypt; the KMS checks policy and context, logs the call and returns the DEK.

You gain a KMS that never sees data, an audit record of every decryption, and a small blast radius: one leaked DEK exposes one object. A DEK per object is the default (Tink’s envelope AEAD makes a new one for every encryption) and keeps each key far from GCM’s limits. A DEK per tenant or per day means fewer KMS calls, but a bigger blast radius and a usage counter to respect. Never share a DEK across tenants.

Caching data keys, and the KMS quota

The KMS is now on your read path, and it is rate-limited. As of October 2026, AWS KMS allows each account 10,000 symmetric cryptographic requests per second per Region by default, including in Mumbai (ap-south-1); a few large Regions get 20,000 or 100,000. Encrypt, Decrypt, GenerateDataKey and ReEncrypt all draw from that one shared bucket, across every service in the account.

Now take that claims platform at peak, reading 12,000 encrypted health records per second with a DEK per record. That is 12,000 Decrypt calls per second, over the default quota before any writes are counted. The quota is adjustable, but the lasting fix is coarser DEKs plus a cache: one DEK per tenant per day, cached unwrapped in each pod for 5 minutes. With 40 pods and 500 active tenants: 40 × 500 ÷ 300 s ≈ 67 KMS calls per second.

You gain latency, cost and headroom. You pay with plaintext keys sitting in 40 pods’ memory for longer, a “disable key” that only bites after the cache TTL, and more data under each DEK. That is why the AWS Encryption SDK makes a new data key per encryption by default, and caps its optional cache by age, messages and bytes per key. The general TTL trade-off is in caching 101.

Key hierarchy, lifecycle and rotation

Envelope encryption naturally becomes a tree, and the tree is what makes key management tractable:

• Root keys live inside HSMs and never leave them in plaintext. AWS KMS runs on HSMs validated to FIPS 140-3 Security Level 3, a level that requires the hardware to detect and respond to physical tampering.
• KEKs sit in the KMS, one per tenant, service or data class. They are used rarely (one call per DEK) and are never exported.
• DEKs are plentiful and short-lived, stored only in wrapped form next to the data they protect.

Separate duties too: key administrators should not be able to use keys, and services that use them should not be able to change their policies.

NIST SP 800-57 gives every key a lifecycle: pre-activation (generated, not yet authorised), active, suspended, deactivated (may only decrypt or unwrap what it protected earlier), compromised and destroyed. Its suggested cryptoperiods for both data-encryption and key-wrapping keys are an encrypting period of under 2 years, plus up to 3 more years for decrypting; a key encrypting large volumes quickly, such as a link key, should retire after a day or a week.

What rotation means with envelope encryption

Rotating a KEK does not re-encrypt your data. It adds a new version: new wraps use the new version, old versions move to “deactivated” and stay available for unwrapping, and every wrapped DEK records which version wrapped it. AWS KMS works this way: automatic rotation every 365 days by default, configurable from 90 to 2,560 days, plus on-demand rotation; it keeps all old key material and picks the right version on decrypt.

That leaves two optional, very different jobs:

• Re-wrap DEKs (cheap). Unwrap each DEK with the old version and wrap it with the new one; the data is untouched. With 200 million wrapped DEKs and half the 10,000-per-second quota: 200,000,000 ÷ 5,000 = 40,000 s ≈ 11 hours, moving a few hundred bytes per call.
• Re-encrypt data (expensive). Decrypt and rewrite every byte with a new DEK: every object, every replica and, where possible, every backup. Do it only when a DEK itself may have leaked or an algorithm is retired.

Tenant keys, BYOK and crypto-shredding

A SaaS platform that serves banks will hear “we want our own keys” in its first security review. The foundation is a KEK per tenant: every tenant’s DEKs are wrapped by that tenant’s KEK, so key policies, audit trails and deletion are all scoped to one customer, even in a shared pooled database. Who controls that KEK is a ladder:

Be honest about what this buys: a service that processes plaintext can read it, so customer keys give a bank a kill switch, an audit trail and verifiable deletion, not secrecy from your running code.

Crypto-shredding is the deletion part. Erasing a tenant from the primary database is easy; erasing it from 35 daily snapshots, a cross-region replica and a data-lake export is not. Destroy the tenant’s KEK and every DEK it wrapped, and so every ciphertext in every copy, becomes noise. That supports erasure duties such as those in India’s DPDP Act 2023 (erase personal data once consent is withdrawn or the purpose is served, unless a law requires keeping it) and the GDPR’s right to erasure. Regulators have not uniformly equated shredding with deletion, so agree the method with your legal team. Four traps:

• Waiting periods. AWS KMS makes you wait 7 to 30 days (default 30) before a key is really deleted. Disable it on day one; it is deleted at the end.
• Other copies of the key. Multi-Region replica keys can still decrypt, and BYOK customers hold the original material. Delete them all.
• Plaintext that escaped. Search indexes, caches, logs and analytics extracts are not under the key.
• Per-user shredding. To erase one policyholder, the destroyed key must not sit in the backups: a per-user DEK stored in the backed-up database can still be unwrapped from yesterday’s snapshot. Keep per-user keys in a separate key store, or let backups expire on a documented schedule.

Secrets management

Secrets (database passwords, payment-gateway API keys, webhook signing secrets) leak through code, images, CI logs and environment variables dumped by a debug page. The rules:

• Never in code or images. Scan repositories and builds for them.
• One store: HashiCorp Vault or a cloud secrets manager, encrypted under a KMS key, with every read audited.
• Identity, not a password, to fetch secrets: the workload proves who it is with its cloud role or workload certificate, which solves the “secret zero” problem.
• Short-lived dynamic credentials where possible. Vault’s database secrets engine creates a database user per lease and drops it when the lease ends, so with a one-hour lease a leaked password dies within the hour, and logs show which service did what.
• Rotate static secrets with overlap: add the new value, deploy, then remove the old.

Searching encrypted data

Randomised AEAD means the same Aadhaar-like number encrypts differently every time, so WHERE national_id = ? no longer works. Your options trade queries for leakage:

Deterministic encryption on a low-entropy column is a trap: encrypt a state column with 36 possible values and frequency analysis tells Maharashtra from Goa without any key.

A blind index stores HMAC-SHA-256(index_key, normalised_value) beside the randomised ciphertext and puts an ordinary index on it. The index key must be separate from the encryption key and live in the KMS, because the input space is small: a 12-digit identifier has 10¹² values, and an attacker who steals the index key and computes a billion HMACs per second needs 10¹² ÷ 10⁹ = 1,000 s, about 17 minutes, to reverse every entry. Truncating the index adds deliberate false positives (the app decrypts and filters candidates) and leaks less.

Tokenization goes furthest for card numbers. A vault swaps the card number for a random token that only the vault can map back, so systems that see only tokens can fall out of PCI DSS scope. A UPI-style app that saves cards should keep card-network tokens, not card numbers; in India, RBI’s card-on-file tokenisation rules forbid anyone in the payment chain except card issuers and networks from storing the real card number.

Going deeper: the KMS on the hot path

Once reads need a KMS call, the KMS is part of your availability budget.

• Outages and throttling. A cold read cannot fail open: without the DEK there is no plaintext. Cache DEKs within your thresholds, retry with jitter, use a circuit breaker, and let writes queue rather than fail.
• Multi-Region keys. For active-active or disaster recovery, replicate the KEK (AWS multi-Region keys share key material and ID) so the second Region decrypts locally. The price: more copies to delete when shredding.
• The most dangerous button. Disabling or scheduling deletion of a busy KEK is a global outage. Restrict it in key policy, require two people, and alarm on it.

Going deeper: when a key leaks

Scope the response by what leaked:

• A plaintext DEK (from a memory dump): only its objects are exposed. Re-encrypt them under new DEKs.
• The ability to call the KMS (stolen app credentials): the KEK never left the HSM, so the question is what the attacker unwrapped, and the KMS audit log records every Decrypt. Revoke the credential, then re-encrypt under new DEKs every object whose DEK was unwrapped in that window. Rotating the KEK changes nothing for DEKs already exposed.
• Key material itself (a hard-coded key in Git, a leaked BYOK backup): mark it compromised in SP 800-57’s terms, stop encrypting with it, and re-encrypt everything it protected.

Write and rehearse the re-wrap and re-encrypt jobs before you need them.

Going deeper: the post-quantum clock

A large quantum computer would break RSA and elliptic-curve cryptography, which handle key exchange and signatures. AES-256 and HMAC-SHA-256 are expected to stay safe, so symmetric KEKs in a KMS are not the urgent problem. The urgent one is harvest now, decrypt later: traffic recorded today, or data wrapped under RSA keys, can be read once such a machine exists. That matters most for data that stays sensitive for decades, such as health records and national identifiers.

NIST finalised FIPS 203 (ML-KEM) for key establishment in August 2024, alongside FIPS 204 and 205 for signatures. The migration pattern is hybrid: combine a classical exchange with ML-KEM in TLS, so the session stays safe if either one holds. Inventory where asymmetric crypto protects long-lived data, and keep key and algorithm IDs in every ciphertext so the next switch is configuration.

Cheat sheet

How to say it in the interview

“I cover three states. In transit it’s TLS 1.3 at the edge and mTLS inside. At rest, storage encryption only protects against stolen disks and backups, so for card data, Aadhaar-like IDs and health records I add field-level envelope encryption: a fresh AES-256-GCM data key per record or tenant, with tenant and row ID as associated data, wrapped by a per-tenant KEK in an HSM-backed KMS. The KEK rotates yearly by version; old versions only decrypt. Banks can bring their own key, and offboarding a tenant means destroying its KEK, which shreds every backup too. Lookups use an HMAC blind index, cards are tokenized, and services get short-lived credentials.”

Isn’t the database already encrypted at rest? Yes, and that stops someone who walks off with a disk or a backup file. A DBA, a stolen password or a SQL injection still sees plaintext, because the engine decrypts for any valid session. Field-level encryption moves decryption into the app.

Why not just call the KMS for every record? AWS KMS accepts at most 4,096 bytes per Encrypt, adds a network round trip, and shares a default quota of 10,000 requests per second per account in most Regions. Envelope encryption sends it only small data keys, and a bounded cache cuts calls further.

How do you rotate keys without downtime or re-encrypting everything? Rotate the KEK by version: new DEKs are wrapped by the new version, and old versions stay for decryption. Re-wrapping old DEKs is optional and cheap; re-encrypting data is for a leaked DEK or a retired algorithm.

A customer leaves and asks for deletion, including backups. What do you do? Disable their tenant KEK at once and delete it, with every replica, after the waiting period; purge plaintext copies in indexes, caches and exports. Every ciphertext under it, in every backup, becomes unreadable.

Where to go next

• Service-to-service auth and mTLS: TLS 1.3, internal PKI and certificate rotation.
• Authentication, sessions and JWTs: password hashing and token-signing keys.
• Multi-tenancy and isolation: the access-control half of tenant isolation.
• Multi-tenant data: where per-tenant keys fit.
• Blob storage and design an object store like S3: server-side encryption at scale.
• Disaster recovery: restores need keys too.
• Design a payment system: tokenized cards and PCI scope in practice.

References and further reading

1. NIST SP 800-38D — Recommendation for Block Cipher Modes of Operation: GCM and GMAC (2007)csrc.nist.govThe IV uniqueness requirement (collision probability no greater than 2⁻³²), the 2³² invocation limit per key for random IVs, and Appendix A on how a repeated IV enables forgeries.
2. NIST SP 800-57 Part 1 Rev. 5 — Recommendation for Key Management: General (2020)csrc.nist.govKey states (pre-activation, active, suspended, deactivated, compromised, destroyed) and the suggested cryptoperiods for data-encryption and key-wrapping keys.
3. NIST: Post-quantum cryptography FIPS approved (August 2024)csrc.nist.govFIPS 203 (ML-KEM, from CRYSTALS-Kyber) for key establishment, and FIPS 204 and 205 for signatures.
4. AWS KMS API Reference — Encryptdocs.aws.amazon.comThe 4,096-byte plaintext limit for symmetric keys, the much smaller RSA-OAEP limits, and encryption context that must match exactly on decrypt.
5. AWS KMS Developer Guide — Cryptography essentialsdocs.aws.amazon.comAES-256-GCM inside FIPS 140-3 Security Level 3 HSMs, and envelope encryption up to a root key that never leaves the HSM in plaintext.
6. AWS KMS Developer Guide — Rotate AWS KMS keysdocs.aws.amazon.comAutomatic rotation (default 365 days), retained key material for decryption, and the note that rotation never re-encrypts data or rotates data keys.
7. AWS KMS Developer Guide — Request quotasdocs.aws.amazon.comShared per-account quotas for symmetric cryptographic operations by Region (10,000, 20,000 or 100,000 per second), 1,800 per second per custom key store, and data key caching as the remedy.
8. AWS KMS Developer Guide — Delete an AWS KMS keydocs.aws.amazon.comDeletion makes ciphertext unrecoverable, the mandatory 7–30 day waiting period, and the multi-Region replica exception.
9. Google Tink — Authenticated Encryption with Associated Data (AEAD)developers.google.comAEAD properties, associated data that binds ciphertext to a user ID, AES-GCM’s 2³²-message limit and catastrophic failure mode, and AES-GCM-SIV as the nonce-misuse-resistant option.
10. OWASP Cryptographic Storage Cheat Sheetcheatsheetseries.owasp.orgWhere to encrypt and what each level protects against, separating keys from data, DEKs and KEKs, rotation before mode limits, and no custom algorithms.