After seeing Designing Data-Intensive Applications by Martin Kleppmann recommended countless times across blog posts and engineering discussions, I decided to take the plunge. This book provides a comprehensive look into the principles and practicalities of building robust, scalable, and maintainable data systems at scale—exactly the kind of deep technical knowledge that kept appearing in “must-read” lists.

Below is my attempt to summarize the key takeaways from each chapter. While this review probably doesn’t do justice to Kleppmann’s depth, I don’t think I could invest the necessary time to do so. I hope it serves as a useful guide for anyone considering the same journey, and as future reference for myself when I inevitably need to revisit these concepts.

Overview

The book is divided into three parts:

Part 1: Foundations of Data Systems This part covers the fundamental concepts that underpin data systems. It explores what it means for a system to be reliable, scalable, and maintainable. It delves into different data models (Relational, Document, Graph), query languages, and the internal workings of storage engines like Log-Structured Merge-Trees (LSM-Trees) and B-Trees. Finally, it discusses data encoding formats and the importance of schema evolution.

Part 2: Distributed Data This section tackles the complexities of distributed systems. It starts with replication and partitioning, two essential techniques for achieving scalability and high availability. It then moves on to transactions, exploring ACID properties and the challenges of maintaining consistency in a distributed environment. The final chapters of this part discuss the inherent difficulties of distributed systems, such as network faults and clock synchronization, and introduces the concept of consensus to solve them.

Part 3: Derived Data The final part of the book focuses on systems that derive data from other sources. It covers batch processing, with a focus on paradigms like MapReduce, and contrasts it with stream processing for real-time data. It explores how to build systems that can handle event streams and discusses the future of data systems, including the “unbundling” of databases and the move towards more flexible and specialized data processing tools.

Rather than writing one long post we will break it down into individual pieces.

Part 1: Foundations of Data Systems

Reliable, Scalable, and Maintainable Systems

This foundational chapter establishes the three pillars of good system design. Reliability means systems continue working correctly even when hardware fails, software bugs occur, or humans make mistakes. Key strategies include redundancy, fault tolerance, and graceful degradation. Scalability is about handling increased load gracefully - whether that’s more users, more data, or more complex operations. Kleppmann emphasizes describing load with specific parameters (requests/second, read/write ratio, cache hit rate) and measuring performance with percentiles rather than averages. Maintainability focuses on three aspects: operability (easy for ops teams to keep running), simplicity (easy for engineers to understand), and evolvability (easy to adapt for new requirements).

A key insight is that there’s no one-size-fits-all architecture - systems must be designed for their specific load characteristics and requirements.

Data Models and Query Languages

This chapter explores how the choice of data model profoundly affects how we think about problems. Relational models excel at representing many-to-many relationships and enforcing consistency through ACID properties. Document models (like JSON) are great for one-to-many hierarchical data and offer schema flexibility, but struggle with complex relationships. Graph models shine when everything is potentially related to everything else.

Kleppmann discusses the “impedance mismatch” between object-oriented programming and relational databases, and how document databases can reduce this friction. However, he warns that as applications evolve, data tends to become more interconnected, potentially negating initial simplicity gains.

The chapter also contrasts declarative languages (like SQL) with imperative approaches, highlighting how declarative languages can be optimized and parallelized more effectively.

Storage and Retrieval

This chapter dives deep into the fundamental data structures powering databases. Hash indexes provide O(1) lookups but require all keys to fit in memory and don’t support range queries efficiently. SSTables and LSM-Trees (used in Cassandra, HBase, LevelDB) excel at write-heavy workloads by maintaining sorted string tables and performing periodic compaction. B-Trees (used in most relational databases) provide balanced read/write performance with in-place updates.

The chapter distinguishes between OLTP (Online Transaction Processing) systems optimized for small queries and OLAP (Online Analytical Processing) systems designed for aggregating large datasets. For analytics, column-oriented storage dramatically improves performance by storing each column separately, enabling better compression and vectorized processing.

Key insight: The choice of storage engine should match your access patterns - write-heavy workloads benefit from LSM-trees, while read-heavy workloads often prefer B-trees.

Encoding and Evolution

Schema evolution is crucial for long-running systems. This chapter compares encoding formats: JSON/XML are human-readable but verbose and ambiguous about numbers; Protocol Buffers/Thrift provide efficient binary encoding with strong schema evolution support; Avro is particularly good for schema evolution in distributed systems.

The chapter emphasizes forward and backward compatibility: new code should read old data, and old code should read new data (ignoring unknown fields). This enables rolling deployments and gradual system updates.

Three modes of dataflow are discussed: through databases (writer encodes, reader decodes), through service calls (request/response encoding), and through message passing systems (asynchronous encoding).

Part 2: Distributed Data

Replication

Replication serves three purposes: reducing latency (geographic proximity), increasing availability (redundancy), and scaling read throughput. Single-leader replication is simple but creates a bottleneck; multi-leader replication enables better performance across datacenters but introduces conflict resolution complexity; leaderless replication (Dynamo-style) provides high availability but can return stale data.

Critical challenges include replication lag (eventual consistency), which can cause read-after-write inconsistencies, monotonic read violations, and causality violations. The chapter details practical solutions like reading from leaders for recent writes and using version vectors for conflict resolution.

A sobering insight: “Something is always broken” in large distributed systems, so designs must assume and plan for partial failures.

Partitioning

Partitioning (sharding) is essential for scaling beyond single-machine limits. Key-range partitioning enables efficient range queries but risks hot spots; hash partitioning distributes load evenly but makes range queries expensive.

Secondary indexes complicate partitioning significantly. Document-partitioned indexes (local indexes) require scatter-gather queries; term-partitioned indexes (global indexes) make writes more complex but enable efficient reads.

Rebalancing (redistributing data as nodes are added/removed) must be automatic but not too eager to avoid unnecessary data movement. Fixed partition counts work well when you can predict growth, while dynamic partitioning adapts to actual data volume.

Transactions

Transactions provide crucial safety guarantees through ACID properties: Atomicity (all-or-nothing), Consistency (maintaining invariants), Isolation (concurrent transactions don’t interfere), and Durability (committed data survives failures).

The chapter details isolation levels: Read Committed prevents dirty reads/writes; Snapshot Isolation prevents nonrepeatable reads using MVCC; Serializable prevents all anomalies but comes with performance costs.

Weak isolation is common in practice due to performance concerns, but can lead to subtle bugs like lost updates, write skew, and phantom reads. The chapter provides practical strategies for each scenario, emphasizing that application developers must understand these trade-offs.

The Trouble with Distributed Systems

This sobering chapter catalogs everything that can go wrong in distributed systems. Networks are unreliable - packets get lost, delayed, or corrupted, and timeouts don’t distinguish between slow responses and failures. Clocks are unreliable - they drift, jump backwards during NTP sync, and provide false confidence in ordering.

Byzantine faults (nodes behaving maliciously) are mostly theoretical for internal systems but crucial for peer-to-peer networks like Bitcoin. The chapter introduces failure detectors and fencing tokens as practical solutions for detecting and handling failures.

A key lesson: distributed systems must be designed assuming partial failures, network partitions, and clock skew will occur.

Consistency and Consensus

Linearizability makes a distributed system appear as if there’s only one copy of data, with atomic operations. It’s stronger than eventual consistency but comes with significant performance costs and availability limitations (CAP theorem).

Causal consistency is weaker but more practical - it preserves cause-and-effect relationships without requiring global coordination. Lamport timestamps provide causal ordering, while vector clocks handle concurrent operations.

Consensus algorithms like Paxos and Raft enable distributed systems to agree on values despite failures. They’re fundamental for leader election, atomic commit protocols, and maintaining replicated state machines. Two-phase commit provides atomicity across multiple databases but blocks on coordinator failures.

Part 3: Derived Data

Batch Processing

Batch processing handles large volumes of data by processing it in chunks. MapReduce popularized the pattern of moving computation to data rather than data to computation. The map phase extracts key-value pairs, and the reduce phase aggregates values by key.

Join algorithms are crucial for batch processing: sort-merge joins work well for large, similarly-sized datasets; broadcast hash joins optimize for small-large dataset joins; partitioned hash joins require pre-partitioned inputs.

Modern dataflow engines like Spark and Flink improve on MapReduce by avoiding unnecessary disk I/O, supporting more flexible computation graphs, and enabling iterative algorithms. They provide better performance while maintaining fault tolerance through lineage tracking or checkpointing.

Stream Processing

Stream processing handles unbounded data streams for real-time insights. Unlike batch processing, stream processing must handle late arrivals, reordering, and exactly-once semantics. Event sourcing stores all changes as an immutable log, enabling temporal queries and system reconstruction.

Windowing strategies include tumbling (fixed, non-overlapping), hopping (fixed, overlapping), sliding (continuous), and session windows (dynamic based on activity). Stream joins are complex due to timing - you can join stream-to-stream (temporal), stream-to-table (enrichment), or table-to-table (changelog joins).

Fault tolerance approaches include microbatching (Spark), checkpointing with barriers (Flink), and idempotent operations. The chapter emphasizes that exactly-once processing often requires end-to-end design considerations.

The Future of Data Systems

Kleppmann concludes by envisioning the “unbundling” of databases - separating storage, indexing, querying, and processing into composable services. This microservices approach to data could enable more specialized and efficient systems.

Data ethics becomes increasingly important as data systems grow in power and influence. Privacy, algorithmic bias, and data ownership require technical and social solutions.

The chapter discusses lambda and kappa architectures for combining batch and stream processing, differential dataflow for incremental computation, and the need for end-to-end correctness rather than just database consistency.

An intriguing prediction: future data systems might resemble spreadsheets more than traditional databases, with reactive updates propagating through dependency graphs.

Reference:

Kleppmann, M. (2017). Designing Data-Intensive Applications. O’Reilly Media.

Honest Reflection

Reading “Designing Data-Intensive Applications” has been both enlightening and humbling. While I gained valuable insights into the principles of distributed systems and data architecture, I must admit this book functions more as a comprehensive reference than something I’ve fully internalized.

The depth of algorithmic detail—from consensus protocols like Paxos and Raft to the intricacies of LSM-trees and B-tree implementations—requires hands-on experience to truly master. I can’t yet explain these algorithms verbatim or debug their edge cases, which Kleppmann would likely say is expected for someone who hasn’t built systems at this scale.

What I did gain:

• A mental framework for thinking about trade-offs in distributed systems
• Understanding of when and why certain architectural patterns emerge
• Appreciation for the complexity hidden behind “simple” database operations
• Vocabulary to better understand system design discussions

The real value lies in knowing these concepts exist and where to look when encountering real-world problems. As I progress in my career and work with larger, more complex systems, I hope to return to specific chapters as practical guides rather than theoretical exercises, even if the opportunity to do so does not arise immediately.

Perhaps the book’s greatest strength is its ability to bridge the gap between academic computer science and industry practice, serving as both teacher and reference manual. However, fully grasping the depth of its teachings would require a significant investment of time and effort.