Freezer corruption after inode exhaustion: hashes.0000.rdat torn on disk-full write

[ShisheerBitgo]

Environment

• nitro version: v3.11.3
• network: Robinhood (Arbitrum L2, mainnet)
• host: bare-metal Kubernetes, xfs PVC

Describe the bug

When the disk fills up completely (inodes and device space), the Freezer begins failing writes each freeze cycle with no space left on device but the node continues running. After the disk is expanded and the node restarts, it finds the hashes freezer table corrupted and refuses to open, entering a permanent crash-restart loop with no path to recovery (need a full resync)

ERROR error initializing database
  err="failed to open database: freezer table(path: /data/nitro/l2chaindata/ancient/chain, name: hashes, num: 0) is corrupted"

The error does not indicate whether the .rdat index file, the .cdat data files, or both are affected, only that the table is unreadable. Perhaps the .rdat index file?

Sequence of events

Phase 1: inode exhaustion, Freezer writes failing (18:43-18:48):

ERROR[08-01|18:43:52.449] Error in block freeze operation          err="can't write receipts to Freezer: write /data/nitro/l2chaindata/ancient/chain/receipts.0023.cdat: no space left on device"
ERROR[08-01|18:44:54.121] Error in block freeze operation          err="write /data/nitro/l2chaindata/ancient/chain/headers.0005.cdat: no space left on device"
ERROR[08-01|18:45:52.522] Error in block freeze operation          err="can't write receipts to Freezer: write /data/nitro/l2chaindata/ancient/chain/receipts.0023.cdat: no space left on device"
ERROR[08-01|18:46:52.934] Error in block freeze operation          err="write /data/nitro/l2chaindata/ancient/chain/hashes.0000.rdat: no space left on device"
ERROR[08-01|18:47:52.372] Error in block freeze operation          err="can't write body to Freezer: write /data/nitro/l2chaindata/ancient/chain/bodies.0024.cdat: no space left on device"
ERROR[08-01|18:48:52.276] Error in block freeze operation          err="can't write body to Freezer: write /data/nitro/l2chaindata/ancient/chain/bodies.0024.cdat: no space left on device"

Phase 2: after disk expansion + pod restart (19:07 onward):

ERROR[08-01|19:07:28.009] error initializing database              err="failed to open database: freezer table(path: /data/nitro/l2chaindata/ancient/chain, name: hashes, num: 0) is corrupted"

I saw this error on repeat

Root cause

In chain_freezer.go (see here), SyncAncient() is only called on the success path after freezeRange

So when freezeRange fails mid-write (say, due to inodes or disk exhaustion), the batch is partially written but never flushed. The node keeps running, accumulating more failed freeze attempts. On the next restart, the Freezer finds a torn table and refuses to open

The freezer_batch.go documents explicitly say (see here):

"Note index file isn't fsync'd after the file write, the recent write
can be lost after the power failure."
Which is why I think we care about the .rdat index file?

Also, chain_freezer.go confirms fsync only occurs every freezerBatchLimit = 30000 blocks (see here), which means up to 30,000 blocks of writes can accumulate before the next fsync. Seems like a pretty large window

Expected behaviour

Either:

1. At write time (prevention): When freezeRange fails, roll back the partial write immediately, so the file is never left torn in the first place
2. At startup (recovery): If the file IS torn (from a previous crash), detect it on open and truncate to the last valid entry instead of refusing to start so the node can recover from historical corruption without a full resync

Related

• Unlcean shutdown corrupts the database entirely #3334: same failure mode via unclean shutdown
• error initializing database no space left on device ( still 4.4TB free ) #4103, Arbitrum archive no space left on device with several TB free #4301: same inode exhaustion trigger, different affected file