Designing Data-Intensive Applications, Honestly Reviewed

Every engineer has a book they bought because it kept showing up in other people’s recommendations.

For me, that was Designing Data-Intensive Applications by Martin Kleppmann. I expected a slog through database internals. What I got was something rarer: a book that puts precise names to problems I had already half-noticed in production, then explains why they happen.

This is not a chapter-by-chapter summary. Kleppmann already wrote the book, and he is better at it. Instead, this is what actually stuck, filtered through the work I do: data pipelines, ML systems, and the glue between them.

The ideas I use every week

Describe the load before you design for it. Kleppmann opens by insisting that “scalability” is meaningless until you can state your load parameters: reads versus writes, fan-out, working set size, cache hit rate, data volume, request rate, and latency expectations.

That sounds obvious, and almost nobody does it carefully enough.

It changed the first question I ask when someone says a pipeline is “slow”: slow at what, exactly? Half the time, writing down the load parameters answers the question before any redesign starts.

Percentiles, not averages. An average response time is a comfortable lie. The p99 is where many real user complaints live, because the user with the most data is often the one hitting the slow path.

I now reach for percentiles by default. That applies to latency dashboards, but also to less obvious places: model inference times, batch-job stragglers, queue processing delays, and feature pipeline freshness.

Storage engines explain your database’s personality. The chapter on LSM-trees versus B-trees is the one I recommend most.

Once you know that PostgreSQL updates pages in place while Cassandra appends and compacts, a whole class of vendor documentation stops reading like incantation. You can predict which operations are likely to be cheap before you benchmark, and you stop being surprised that write-optimised stores often make you pay at read time.

Schema evolution is a deployment problem, not just a data problem. Forward and backward compatibility are what make rolling deploys possible: new code reads old data, and old code survives new data.

Anyone who has broken a consumer by “just adding a field” to a message schema knows this pain. The encoding chapter is the best justification I have read for tolerating the ceremony of Avro or Protobuf instead of throwing JSON over the wall and hoping everyone keeps up.

Replication lag is a bug class, not an edge case. If you have ever seen a user save a change, refresh, and watch it vanish, you have met read-after-write inconsistency.

Before this book, I would have treated that as a mysterious one-off. Now I recognise the whole family: stale reads, monotonic-read violations, causality violations, and the small UX choices that can hide or expose them.

Most of my work is derived data. Part 3 reframed my job.

Feature tables, embeddings, search indexes, caches, aggregated dashboards: none of these are systems of record. They are views derived from some upstream truth, and they should be rebuildable.

The chunk-embed-index pipeline in my MkDocs RAG project is a textbook derived-data system. Once you see it that way, the design priorities become obvious:

• Keep the source immutable or versioned.
• Make every transformation idempotent.
• Track lineage between source records and derived outputs.
• Treat “can I reprocess this from scratch?” as a hard requirement.

That mindset has saved me more than once when a transformation bug meant rebuilding a downstream table.

What to skim

The MapReduce material is now mostly a history lesson. It is worth reading for the join algorithms, which outlived the framework, but most engineers will not be writing MapReduce jobs directly.

The Byzantine fault sections are skippable unless you work on peer-to-peer systems, blockchain infrastructure, or similar adversarial environments.

The transactions chapter is excellent, but it goes deeper into isolation-level anomalies than many application engineers will need day to day on managed databases. I would still read it, just slowly, and ideally with a real system in mind.

What has aged well

The final chapter, on the unbundling of databases, has aged in the opposite direction from the MapReduce material. It reads like a prediction of the modern data stack, written before the modern data stack had fully settled into its current shape.

Specialised storage engines, log-centric integration, stream processors stitching systems together, and different derived views optimised for different access patterns: that is a fair sketch of what many teams now operate.

This is one reason the book remains useful. It is not anchored to a single vendor or framework. It gives you the vocabulary underneath the tools.

The honest part

I use managed services, like almost everyone.

I have never operated a leaderless Dynamo-style cluster in anger or debugged a Raft implementation at 3am, and reading about consensus is not the same as carrying a pager for it. Some of Part 2 will stay theoretical for me until the day it suddenly is not.

But that is precisely the value.

The managed service is doing these things on my behalf, and the book tells me which questions to ask of it:

• What happens during failover?
• What are the replication guarantees?
• Is this timestamp actually monotonic?
• What consistency model is exposed to the application?
• What happens when a consumer falls behind?
• Can I rebuild this derived view from source?

Knowing the machinery exists, and roughly how it fails, is the difference between reading vendor docs and merely believing them.

Why it matters for ML and data work

People often frame this as a backend or distributed systems book. It is that, but it is also deeply relevant to data and ML engineering.

Modern ML systems are full of data-intensive problems:

• Training data is derived from operational systems.
• Feature stores need freshness and reproducibility.
• Batch and streaming pipelines need clear consistency expectations.
• Embedding indexes need rebuilding and versioning.
• Model outputs often become inputs to downstream workflows.
• Analytics tables need to preserve business meaning across schema changes.

If your work touches data pipelines or production ML, the book gives you a stronger mental model for the systems around the model.

Should you read it?

Yes, with a strategy.

Read Part 1 carefully. It pays for the whole book.

Take Part 2 slowly, ideally with a real system in mind. Do not attempt it on a deadline.

Read Part 3 for the worldview. That is where the book connects most directly to modern data platforms, ML pipelines, search systems, and derived data.

Then put it on the shelf where you can reach it. It is a reference book wearing a textbook’s jacket, and the returns compound each time a production problem sends you back to a chapter you thought you had finished.

Nearly a decade after publication, nothing has replaced it. That alone tells you something.

Reference

Kleppmann, M. (2017). Designing Data-Intensive Applications. O’Reilly Media.

Portfolio page: Designing Data-Intensive Applications, Honestly Reviewed