Home on codedump notes

The Charm of Distributed Systems

Thu, 25 Jun 2026 00:00:00 +0000

This is a story about “how to build reliable systems out of unreliable components.”

When you open your phone on a rainy night to order food delivery, the system behind it is undergoing complex collaboration: your request passes through a load balancer and gets routed to an instance of the order service; the order data is persisted to the master node of the database and synchronized to two replica nodes across datacenters in milliseconds; at the same time, the inventory service in another cluster is deducting stock, and the payment system is calling a third-party gateway. If any step fails, a series of compensation logics will automatically trigger. The whole process spans dozens of servers, unreliable network links, and hardware that might strike at any time—but in most cases, your food still arrives on time.

Chapter 1: Overview of Distributed Systems

Thu, 25 Jun 2026 00:00:00 +0000

In the evolution of modern software engineering, the transition from single-node applications to distributed architectures represents a critical watershed. This leap is not merely a matter of hardware stacking or code migration, but a profound transformation involving shifts in mindset, design philosophy, and even a renewed understanding of the laws of physics.

As the opening chapter of this book, it will lead readers out of the comfort zone of single-node systems and confront the real challenges of distributed environments. We will first clarify the core definition of distributed systems and analyze their essential differences and advantages over centralized systems. Next, we will focus on the unavoidable technical problems in this field—from unreliable network communication to fragmented global clocks, and the uncertainty brought by partial failures. Finally, this chapter will elaborate on the mindset shifts architects must undergo during this transition: moving from pursuing absolute certainty to seeking a balanced trade-off between consistency and availability. Understanding these fundamental concepts and paradigms is a prerequisite for mastering the complex algorithms and consistency models in subsequent chapters, and it is also the cornerstone of building highly reliable and scalable systems.

Chapter 2: Models of Distributed Systems

Thu, 25 Jun 2026 00:00:00 +0000

In the previous chapter, we compared single-node systems with distributed systems. A single-node system communicates through shared memory, has a globally unique clock, and exhibits deterministic behavior when errors occur, making programming on such systems relatively straightforward. In contrast, a distributed system consists of multiple nodes that communicate via messages, which makes distributed systems significantly more complex. Therefore, before designing a distributed system, we need a theoretical framework to describe how the system operates. Designers must clarify the assumptions and conditions of the runtime environment at design time, as different conditions entail varying implementation challenges. Models of distributed systems are the theoretical frameworks used to describe and analyze the behavior, properties, and design of distributed systems. They provide an abstraction for studying communication, computation, failures, and synchronization in distributed systems, helping designers and researchers understand system behavior and solve practical problems in complex environments.

Chapter 3: Time and Order in Distributed Systems

Thu, 25 Jun 2026 00:00:00 +0000

In distributed systems, multiple nodes work together. Client requests are sent to different nodes for processing, and these requests become events on individual nodes. As we will see, the system’s state is formed by executing these events one by one in a specific order. Therefore, the order of events is particularly important. Different nodes may see different sequences of events, which can lead to different states on these nodes.

From this, it is evident that when processing multiple events, the order in which they occur affects the state of nodes. Thus, how to measure the order of events becomes a core problem in distributed systems. A natural idea is to sort events according to their physical time. In the following sections, we will learn about the principles of physical clocks and, unfortunately, discover that in a distributed system, comparing physical time across multiple nodes to determine the order of events is not precise and can sometimes even lead to errors. Once the order of events is disrupted, the state of the entire distributed cluster becomes corrupted.

Chapter 4: Replication

Fri, 26 Jun 2026 00:00:00 +0000

In distributed systems, data replication is one of the core design strategies. Its primary purpose is to improve system reliability, availability, performance, and fault tolerance by redundantly storing identical copies of data:

High Availability: If only a single node provides service, a single-node failure will render the service unavailable. By replicating data across multiple nodes, even if one node fails, other nodes can continue to provide service.
Fault Tolerance and Disaster Recovery: Hardware failures, network partitions, or data center disasters can lead to permanent data loss. Multi-copy storage (e.g., cross-rack or cross-region replication) ensures that data is recoverable.
Reduced Latency: The physical distance between users and data centers causes access latency (e.g., when accessing cross-border services). Replicating data to geographically distributed nodes allows users to access the nearest replica.
Read Performance Optimization: A single node can become a bottleneck for read requests. Multiple replicas can distribute the read load to improve read performance.

Based on whether a primary node is involved in replication, replication is divided into primary-backup replication and leaderless replication. Primary-backup replication means there is a central node in the system responsible for coordinating write operations and replicating them to other copies. Correspondingly, leaderless replication is a decentralized architecture.

Chapter 5: Distributed Consensus Algorithms

Fri, 26 Jun 2026 00:00:00 +0000

In the preceding chapters, we discussed how replication can prevent data loss caused by single points of failure, how partitioning can handle massive amounts of data, and how distributed transactions can ensure the atomicity of cross-node operations.

However, when we try to combine these mechanisms into a truly production-grade system that runs 24/7 with automatic fault tolerance, we discover an obvious gap: how can surviving nodes reach agreement on the system state when some nodes have failed?

Chapter 6: Partitioning

Fri, 26 Jun 2026 00:00:00 +0000

So far, in the systems we have discussed, we have assumed that every machine stores all data. In the earlier chapter on replication, when the primary node receives a write request from a client, it saves the entire dataset both locally and on other replica nodes. This storage approach has the following problems:

Scalability: If data replication is done in a primary-backup manner, all the work falls on the primary node. Under heavy load, the primary node quickly becomes a bottleneck.
Single-node bottleneck: A single node, due to the physical limits of its hardware (disk, memory, CPU, etc.), will always hit an upper bound on processing capacity.
Failure isolation: If a node hosting a specific portion of the data fails, that data becomes unavailable, reducing the overall availability of the system. For example, if data from different cities were stored on different nodes, a service outage in one region would not affect data in other regions.

When scaling a system, engineers typically choose between two primary strategies, as illustrated below:

Chapter 7: Transactions

Fri, 26 Jun 2026 00:00:00 +0000

So far, we have introduced replication and partitioning. Replication (including the consensus algorithms) improves system fault tolerance, while partitioning improves system scalability; these two techniques address the physical problems of data. In addition, data access in distributed systems often faces “logical problems”, which are solved by the transaction technology introduced in this chapter:

Replication: The main goal is high availability and data redundancy. By storing identical copies of data on different nodes, when one node fails, the system can continue to provide service from other replicas. It answers the question: “Will my data be lost or inaccessible because one machine goes down?”
Partitioning (Sharding): The main goal is scalability. When a single node’s storage or computing capacity cannot handle all the data and requests, we horizontally split the data across multiple nodes. It answers the question: “How does my system handle ever-growing data volumes and access pressure?”
Transaction: The main goal is the correctness of data operations. It packages a series of operations into an indivisible logical unit, ensuring that these operations either all succeed or all fail, and that they do not interfere with each other when executed concurrently. It answers the question: “How can I ensure that a series of related operations maintain data correctness under any circumstances (concurrency, failures)?”

Why are replication and partitioning alone not enough? Let us look at a few typical scenarios:

Rockraft: A Strongly-Consistent KV Storage Framework Based on OpenRaft and RocksDB

Sun, 12 Apr 2026 09:59:41 +0800

Motivation #

The Redis protocol has become the de facto standard for key-value storage protocols. Beyond the official Redis implementation, we have seen various compatible alternatives:

Valkey: The Linux Foundation’s official fork of Redis 7.2, licensed under BSD. It is the community’s true open-source alternative after Redis switched to SSPL, fully compatible with the Redis protocol and persistence mechanisms.
Dragonfly: A modern multi-threaded in-memory database pursuing extreme performance, offering up to 25x the throughput of Redis with lower tail latency, but licensed under BSL (transitioning to Apache 2.0 after 4 years).
KeyDB: A multi-threaded Redis fork maintained by Snapchat, adding Active Replication and Flash storage extensions on top of 100% Redis API compatibility, though updates have slowed down.
Kvrocks: An Apache top-level project, a distributed KV storage based on RocksDB that persists data to disk, supporting dozens of terabytes at 1/5–1/10 the cost of in-memory solutions, suitable for large-capacity, low-cost scenarios.

However, none of the above implementations have gone further on strong consistency in distributed systems. In this dimension, they all adopt the eventual consistency of the native Redis implementation.

How to Read Code

Mon, 15 Feb 2021 09:46:47 +0800

The ability to read source code is considered to be one of the underlying fundamental programmer skills, and the reason why this ability is important is that

inevitably need to read or take over other people’s projects. For example, researching an open source project, such as taking over a project from someone else.
Reading good project source code is one of the important ways to learn from other people’s good experience, which I know myself.

Reading code and writing code are two not quite the same skills, the reason is that “writing code is expressing yourself, reading code is understanding others”. Because of the many projects, the authors of the project have their own style, it takes a lot of energy to understand.

About

Thu, 20 Sep 2018 23:08:27 +0800

Hi, I’m lichuang, also known as codedump (not “coredump”). I picked the name because I like digging into how code works under the hood — dumping its internals, so to speak.

I’m a software engineer based in Guangzhou, China (UTC+8), building distributed systems and infrastructure tools, mostly in Rust. I’m comfortable working asynchronously across time zones.

I’m currently open to remote Rust roles in:

Distributed systems
Databases / storage engines
Developer tooling

Featured Projects #

rockraft — A library-first distributed consensus engine built on Raft and RocksDB, providing clean abstractions for strongly consistent systems.
coredb — A Redis-compatible distributed database with Raft-based strong consistency and automatic cluster failover.
replited — Replicate SQLite to cloud storage (S3, FTP, WebDAV, Google Drive, Dropbox, etc.) without heavyweight infrastructure.
ironcode — A terminal-native, AI-powered CLI coding assistant written in Rust.
Lua-Source-Internal — A deep dive into the Lua VM, compiler, and runtime.

Open Source Contributions #

databend — Modern cloud data warehouse
openraft — Advanced Raft consensus in Rust
arrow-rs — Official Rust implementation of Apache Arrow
lance — Open lakehouse format for multimodal AI

Books #

Distributed Systems Illustrated — A visual, engineering-focused guide to distributed systems.
《Lua设计与实现》(Lua Design and Implementation) — A walkthrough of the Lua 5.1 source code.

Find Me #

About

Thu, 20 Sep 2018 23:08:27 +0800

Netizen name “codedump,” not “coredump.” I chose this name because I enjoy exploring the principles of code, hence the name “code dump.” Areas of personal interest:

Distributed storage (hobbyist).
Systems programming (some knowledge).
Storage engines (passion).
Programming language implementation (amateur enthusiast).

网名“codedump”，不是“coredump”。取这个名字，是因为喜欢研究代码的原理，所以就取名“code dump”了。

个人感兴趣领域：

分布式存储（业余）。
系统编程（略懂）。
存储引擎（爱好）。
编程语言的实现（民科）。

“发表是最好的记忆” – 侯捷

找到我 #

博客文章也会更新到公众号上（不一定同步），扫描下面的二维码关注我：

qrcode

正式出版作品 #

《Lua设计与实现》

支持与赞赏 #

可以通过下面的微信赞赏来支持我

Mon, 01 Jan 0001 00:00:00 +0000

Home on codedump notes

The Charm of Distributed Systems

Chapter 1: Overview of Distributed Systems

Chapter 2: Models of Distributed Systems

Chapter 3: Time and Order in Distributed Systems

Chapter 4: Replication

Chapter 5: Distributed Consensus Algorithms

Chapter 6: Partitioning

Chapter 7: Transactions

Rockraft: A Strongly-Consistent KV Storage Framework Based on OpenRaft and RocksDB

Motivation #

How to Read Code

About

Featured Projects #

Open Source Contributions #

Books #

Find Me #

About

找到我 #

正式出版作品 #

支持与赞赏 #

周刊目录 #

2022年 #