ZWAN: Leveraging Zone Random Write Area (ZRWA) for Avoiding WAL Tax in LSM Tree
ZWAN is an LSM-tree key-value store for ZNS SSDs that writes WAL records through the Zone Random Write Area instead of Zone Append. Each record is committed as soon as it arrives, without group commit, and consecutive records are packed into the same block by overwriting it in place inside the ZRWA window. This removes the batching delay of group commit and the sub-block write amplification of per-record commit at the same time.
ZWAN is built on WALTZ (https://github.com/SNU-ARC/WALTZ), which replaced the RocksDB WAL path with per-record Zone Append on top of an SPDK-based ZenFS. The WALTZ WAL path is still available and is used as the baseline in the evaluation scripts.
plugin/zenfs/fs/wal_zrwa.cc, wal_zrwa.h
plugin/zenfs/fs/zbd_zenfs.cc, io_zenfs.cc, fs_zenfs.cc
db/log_writer.cc, db/log_reader.cc, db/log_format.h
db/db_impl/db_impl_open.cc
monitoring/waf_stats.cc
tools/db_bench_tool.cc
The ZRWA WAL engine lives in wal_zrwa.cc. It hands out ZRWA window slots to writer threads through a ring with a small priority window so that pages close to the write pointer are filled first, issues the writes and flushes on a dedicated I/O thread, and retires a full zone in two steps, a ZRWA flush followed by a zone finish.
The ZenFS sources handle ZRWA zone allocation and reuse, track the WAL extent of a file while it is still inside the window, and check at mount time that the file system was formatted for the WAL mode the process runs with.
log_writer.cc sends WAL records to the engine when ZRWA mode is on. Each WAL page carries a CRC, and log_reader.cc reads the window back in page order during recovery and replays the records in sequence order. db_impl_open.cc prints the WAL replay time, its breakdown, and the number of bytes read from the device during replay.
db_bench gains the flags below, the benchmarks resetwafstats, printwafstats_prefill and printwafstats_workload that print write amplification counters, and simulate_crash_and_reopen for recovery tests.
| flag | meaning |
|---|---|
| --waltz_wal_mode=append or zrwa | WAL path: Zone Append (WALTZ) or ZRWA (ZWAN) |
| --waltz_prio_window=N | priority ring window in pages, 0 disables it (default 4) |
| --zrwa_exp_flush=true or false | explicit ZRWA flush commands, or implicit flush only |
| --zenfs_skip_meta_sync | skip the metadata zone persist on fsync |
| --waltz_pcie_addr=BDF | PCIe address of the ZNS SSD |
- Install SPDK v22.01.2 at the plugin directory
$ cd plugin
$ git clone https://github.com/spdk/spdk
$ cd spdk
$ git checkout v22.01.2
$ git submodule update --init
$ sudo ./scripts/pkgdep.sh
$ ./configure --with-shared --without-isal
$ make -j4
$ cd ../..- Install prerequisite library
$ sudo apt install libgflags-dev- Build ZWAN
$ cd script
$ ./rel_build.sh
$ cd ..- Specify the ZNS SSD
The scripts refer to a device by an alias, samsung or wd. The PCIe BDF address and the block device name of each alias are kept in the DEV_PCIE and DEV_NAME maps at the top of format_mkfs.sh, script/bench.sh and script/recovery_test.sh. Edit them to match your machine.
The device must support ZRWA. ZWAN was tested on a Samsung PM1731a, whose ZRWA window is backed by device DRAM, and on a WD ZN540, whose window is NAND-backed. On the ZN540 the evaluation script runs ZRWA mode with --zrwa_exp_flush=false.
- Format the device
$ sudo ./format_mkfs.sh samsung zrwaThe second argument selects the WAL mode written into the file system, append or zrwa. It has to match the --waltz_wal_mode given to db_bench later; otherwise the mount fails with a WAL mode mismatch error.
- Run the benchmark scripts
script/test.sh runs the WALTZ and ZenFS comparison from the WALTZ repository. script/eval.sh runs the evaluation of the ZWAN paper: the microbenchmarks, the MixGraph benchmarks and the thread and value size sensitivity runs, for stock ZenFS, WALTZ and ZWAN. It formats the device before every run and writes a summary table under script/results.
$ sudo ./script/eval.sh samsung allscript/bench.sh is the single-run driver used by both:
$ cd script
$ sudo ./bench.sh TESTMODE TESTTYPE VALUESIZE KEYRANGE KEYTEST [WALMODE] [THREADS] [DEVICE]TESTMODE 0 enables the WALTZ WAL path and WALMODE picks append or zrwa; TESTMODE 1 runs stock ZenFS.
- Run the crash-recovery test
script/recovery_test.sh formats the device, fills it, kills db_bench once a target number of operations has been written, reopens the database and records the WAL replay time, its phase breakdown and the WAL bytes read from the device. script/recovery_sweep.sh repeats it with random targets and script/recovery_phase_aggregate.sh sums up the phase breakdown of a sweep.
$ sudo -E env TARGET_OPS=2000000 bash script/recovery_test.sh samsung 0 append zrwa
$ sudo -E bash script/recovery_sweep.sh
$ script/recovery_phase_aggregate.sh script/results/<sweep directory>