Central Thesis
Data systems architecture involves fundamental trade-offs between consistency, availability, latency, and correctness, and there is no one-size-fits-all solution; architects must navigate these trade-offs with clear-eyed awareness of what is gained and sacrificed with each decision.
Summary

Designing data-intensive applications requires navigating fundamental trade-offs between consistency, availability, latency, and correctness, as no single architecture can optimally serve all use cases. At its core, data systems architecture involves making deliberate decisions about data models, storage engines, replication strategies, and processing paradigms, each with its own set of trade-offs, such as those between correctness and performance in transactions, or between query flexibility and write throughput in sharding. By understanding these trade-offs and leveraging unifying abstractions like immutable, append-only logs, architects can design systems that balance competing demands and meet specific needs. As a result, readers gain a nuanced understanding of the complexities involved in building reliable, efficient, and ethical data systems, and develop the ability to make informed decisions that weigh the benefits and drawbacks of different design choices. With this knowledge, they are better equipped to architect data systems that are tailored to their specific use cases.

Key Ideas
→ No single data systems architecture can optimally serve both transactional and analytical workloads.
The tension between transactional (OLTP) and analytical (OLAP) workloads drives the separation of systems, with Hybrid Transactional/Analytical Processing (HTAP) systems attempting to unify both modes.
Chapter 1. Trade-Offs in Data Systems Architecture
→ The choice of data model and query language significantly impacts the types of questions that can be asked of the data.
Different data models (relational, document, graph, RDF, array, event-sourced) and query languages (SQL, Cypher, GQL, SPARQL) are suited to different types of data and use cases.
Chapter 3. Data Models and Query Languages
→ Storage engine design involves trade-offs between read, write, and space amplification.
Log-structured merge-trees (LSM-trees) and B-trees optimize for different trade-offs, with LSM-trees efficient for write-heavy workloads and B-trees offering strong read performance.
Chapter 4. Storage and Retrieval
→ Replication strategies involve trade-offs between consistency, availability, and latency.
Optimistic replication allows any node to accept writes and resolves conflicts later, while pessimistic replication uses consensus protocols to enforce a strict ordering of updates.
Chapter 6. Replication
→ Distributed systems are fundamentally unreliable due to partial failure and the breakdown of assumptions about time and ordering.
Clocks drift, network packets can be delayed or corrupted, and nodes can fail or behave maliciously, requiring careful design to handle these failures.
Chapter 9. The Trouble with Distributed Systems
→ Consistency models and consensus algorithms are critical for reliable distributed systems.
Linearizability ensures operations appear atomic, while consensus algorithms like Paxos and Raft order operations without relying on physical time, but involve trade-offs in performance and fault tolerance.
Chapter 10. Consistency and Consensus
→ Batch processing treats input as immutable and regenerates output from scratch, enabling human fault tolerance and cost efficiency.
Batch workflows allow easy rollback and rerun, minimizing irreversibility and accelerating feature development, and can run on low-priority spot instances for cost savings.
Chapter 11. Batch Processing
→ Stream processing has evolved into a central paradigm for handling real-time data, driven by the recognition that logs and queues are foundational abstractions.
Systems like Kafka and RabbitMQ support both publish/subscribe messaging and durable storage, and stream processing enables real-time data handling with fault tolerance and exactly-once semantics.
Chapter 12. Stream Processing
→ Data systems must be designed with ethical considerations in mind to avoid amplifying existing discrimination and injustices.
Predictive analytics and AI systems trained on biased data can codify injustices, and the accountability gap allows people to evade ethical obligations, requiring moral imagination and ethical safeguards in system design.
Chapter 14. Doing the Right Thing
→ Transactions involve fundamental trade-offs between correctness and performance.
ACID guarantees are critical for reliable database systems, but stricter isolation levels like serializability can impose a severe performance penalty, and weaker isolation levels permit anomalies.
Chapter 8. Transactions
→ Sharding involves trade-offs between query flexibility, write throughput, and operational simplicity.
The choice of sharding key and strategy is critical, and no single sharding strategy is universally optimal, with different approaches suited to different use cases.
Chapter 7. Sharding
→ Immutable, append-only logs provide a unifying abstraction for real-time data and enable systems to replay history and recover from faults.
The log becomes the source of truth, decoupling producers from consumers and enabling fault tolerance and exactly-once semantics in stream processing systems.
Chapter 13. A Philosophy of Streaming Systems