In-Depth Guide to Redis Persistence Mechanisms

AOF Persistence

Redis is memory-based, so if the Redis server crashes, data will be lost. To prevent data loss, Redis provides two persistence mechanisms: RDB and AOF. Let’s first introduce AOF.

AOF (Append Only File) persistence logs every write operation in an append-only manner to the end of an AOF file. By default, AOF is disabled in Redis. Upon restart, Redis re-executes the commands in the AOF file to restore data. It mainly addresses the issue of real-time data persistence.

AOF logs are written only after a command is executed. Why not log the command first and then execute it? This is because Redis does not perform syntax checks when writing to the AOF log. If it logs the command first and then executes it, there’s a risk of logging invalid commands, which could cause errors when Redis attempts to restore data from the AOF log.

Because logging happens after command execution, it does not block the current write operation. However, there are two risks:

If the system crashes after the command is executed but before the log is written, data will be lost.
AOF doesn't block the current command, but it may block the next operation.

The best way to mitigate these risks is to make use of the three AOF write strategies provided by the appendfsync setting:

always: synchronous writeback. After each command execution, the log is immediately written to disk.
everysec: after each command, the log is written to the AOF memory buffer and synced to disk every second.
no: the log is only written to the AOF memory buffer, and the operating system decides when to flush it to disk.

The always strategy ensures minimal data loss but has lower performance. The no strategy performs better but risks more data loss. Generally, everysec is a good compromise.

If more commands are received over time, the AOF file will keep growing, which can lead to performance issues. What if the log file becomes too large? That’s where AOF rewrite comes in! Over time, the AOF file accumulates redundant commands, such as invalid or expired ones. The AOF rewrite mechanism merges these into a single command (like a batch command) to reduce and compress file size.

Does AOF rewriting cause blocking? The AOF log is written by the main thread, but rewriting is different — it's performed by a background subprocess called bgrewriteaof.

Advantages of AOF: higher data consistency and integrity, with data loss limited to seconds.
Disadvantages: for the same dataset, AOF files are larger than RDB files. Recovery is also slower.

RDB Persistence

Because AOF persistence may result in very slow recovery when there are many operation logs, is there a faster way to recover after a crash? Yes — RDB!

RDB works by saving in-memory data to disk in the form of snapshots. Unlike AOF, which records each operation, RDB records the data state at a certain moment.

What is a snapshot? You can think of it as taking a photo of the current data and saving that image.

RDB persistence means that, at specified intervals and after a specified number of write operations, Redis writes a snapshot of the dataset in memory to disk. This is Redis's default persistence method. After completing the operation, a dump.rdb file is generated in the specified directory. When Redis restarts, it loads the dump.rdb file to restore the data. RDB can be triggered in several ways:

save: manually triggered, synchronous save, blocks the current Redis server.
bgsave: manually triggered, asynchronous save — the Redis process forks a child process.
save m n: automatically triggered — if the dataset is modified n times within m seconds, it automatically triggers bgsave.

By executing full snapshots using the bgsave command, Redis avoids blocking the main thread. The bgsave command forks a child process to create the RDB file, while the server process continues handling commands.

Can data still be modified during the snapshot? Redis uses the operating system's Copy-On-Write (COW) mechanism, so it can continue handling writes while taking the snapshot.

Although bgsave doesn't block the main thread, frequent full snapshots still incur performance overhead. For instance, creating a child process via fork is required, which does block the main thread during creation. Incremental snapshots could be a better approach in this case.

Advantages of RDB: Compared to AOF, it allows faster recovery for large datasets and is suitable for large-scale data recovery scenarios such as backups and full replication.
Disadvantages: It cannot achieve real-time or second-level persistence.

Starting from Redis 4.0, hybrid persistence of RDB and AOF is supported — memory snapshots are taken at regular intervals, and between snapshots, all command operations are recorded using AOF.

How to Choose Between RDB and AOF

If data loss is unacceptable, use a combination of RDB and AOF.
If Redis is used solely as a cache and can tolerate a few minutes of data loss, using only RDB is sufficient.
If you choose to use only AOF, it's recommended to use the everysec writeback strategy.