hamma_scrub.select_target_drive: Windows artifact dirs defeat the most-recent-hour heuristic, can select a read-only drive

[pbitzer]

Summary

scripts/hamma_scrub.py::select_target_drive picks the recovery target as "the drive containing the most recent hourly directory", but the heuristic is defeated on any drive prepared under Windows, because it does not exclude the Windows artifact directories.

The bug

for entry in os.listdir(drive):
    if entry == "compressed":
        continue
    full = os.path.join(drive, entry)
    if os.path.isdir(full) and entry > most_recent:
        most_recent = entry

Only compressed is excluded. Both $RECYCLE.BIN and System Volume Information are directories and both sort above any 2026-… hourly directory in ASCII (S = 0x53 beats 2 = 0x32).

So every Windows-prepared drive yields the same sort key:

drive	sort key
DATA80	System Volume Information
DATA81	System Volume Information

The keys tie. Python's sort is stable, so sorted(glob.glob(pattern)) order decides — i.e. the alphabetically first drive by name, not the one with the most recent data. The hour-directory heuristic is completely inert on such drives.

Observed consequence

On mjolnir54 this handed recovery to DATA80 — which was mounted read-only — while DATA81 was the writable drive. Recovery then failed on every trigger:

Recovery: 1417 attempted, 0 succeeded, 1417 failed
FAILED: ags2026-08-06_00-08-16-939.bin trigger #34 —
  [Errno 30] Read-only file system: '/media/pi/DATA80/.tmp_recover_r9bp_ys8.bin'

See sensor-log #110 for the full incident.

Suggested fix

Exclude the Windows artifacts alongside compressed, and — more robustly — only consider entries that actually match the hourly-directory shape rather than blacklisting known-bad names:

HOURLY_DIR_RE = re.compile(r"^\d{4}-\d{2}-\d{2}T\d{2}$")
...
if os.path.isdir(full) and HOURLY_DIR_RE.match(entry) and entry > most_recent:

A whitelist is preferable here: the blacklist has already been wrong twice (compressed was added reactively, the Windows dirs were missed), and any future stray directory re-breaks it.

Worth deciding separately what should happen when no drive has any hourly directory — currently they all tie at "" and the choice again falls to glob order. On a freshly replaced drive that is exactly the situation.

Related

• sensor-log Track slow-channel noise alongside fast #110 — mjolnir54, read-only DATA80 selected for recovery
• HAM-190 — mj54 disk replacement

[pbitzer]

This flaw has a twin in _scan_mj_incremental, and that one is a correctness bug rather than a drive-selection preference. Same root cause — a single-name blocklist plus a max-by-name — so whoever fixes this should fix both. It is live on mj03 right now.

The twin

select_target_drive (this issue) filters one name and takes the max:

for entry in os.listdir(drive):
    if entry == "compressed":
        continue
    full = os.path.join(drive, entry)
    if os.path.isdir(full) and entry > most_recent:
        most_recent = entry

_scan_mj_incremental does the identical thing at hamma_scrub.py:558-567:

    if name == "compressed" or not os.path.isdir(subdir):
        continue
    if since and name < since:
        dirs_skipped += 1
        continue
    subdirs.append((name, subdir))
newest = subdirs[-1][0] if subdirs else None  # names are sorted

System Volume Information and FOUND.000 both sort above any date (S, F > 2) and both survive the --since comparison for the same reason, so newest becomes the artifact directory. Checked rather than assumed:

sorted: ['$RECYCLE.BIN', '2026-09-25T19', '2026-10-03T02', 'FOUND.000', 'System Volume Information', 'compressed']
survive `name < since` with since='2026-09-25': ['2026-09-25T19', '2026-10-03T02', 'FOUND.000', 'System Volume Information', 'compressed']
newest -> 'System Volume Information'

($RECYCLE.BIN is harmless here — $ sorts below digits.)

Why that matters more than drive selection

newest exists to enforce safety rule 1, which the function's own docstring states:

the newest hourly dir per drive -- the one brokkr is actively appending to -- is ALWAYS re-read (never cache-hit)

With newest pointing at an artifact directory, the genuinely-active hourly directory satisfies name != newest at :577 and becomes cache-hit eligible. The rule is silently off on that drive.

The hidden case the rule exists for, traced through source:

1. _read_dir_headers counts a file whose size is < HEADER_SIZE as skipped and reads no header from it — i.e. a record mid-write contributes nothing to the header set.
2. sig = (os.stat(subdir).st_mtime, len(bin_names)) at :572. After brokkr finishes appending that record in place, len(bin_names) is unchanged, and vfat's 2-second mtime granularity means st_mtime can be unchanged too.
3. Signature matches, cache hits, and the completed record's header never enters headers.

Consequence, stated conservatively: that record reads as absent from the Pi, so compare_headers treats its AGS trigger as a recovery candidate on every subsequent run, and identify_purgeable_files will not confirm the AGS file holding it — so the AGS file is never purged and the space is not reclaimed. This is not a data-loss path: a purge requires positive confirmation, and deleting files changes len(bin_names), so a stale cache cannot manufacture a false confirmation.

I verified the mechanism in source and the directory-ordering live, but I did not reproduce a specific record going missing end to end. Treat the consequence as derived, the cause as confirmed.

Which units are affected

Checked the reachable fleet today:

unit	drive	last entry when sorted
mj03	DATA51 (active)	System Volume Information
mj03	DATA52	System Volume Information
mj02	DATA81	2026-10-03T23 — clean
mj04	DATA71	2026-08-26T23 — clean
mj08	DATA19	System Volume Information (observed earlier today; tunnel refusing since)

So mj03 — the unit we deliberately use for testing — is running with rule 1 disabled on its live data drive, and mj08 was too. mj03's deployed scrub is --recover --purge --since auto with the default --mj-cache, so it takes the incremental path every run.

Suggested fix

A blocklist will keep losing this race; $RECYCLE.BIN, System Volume Information and FOUND.000 are just the ones we have seen. Match the hourly-directory pattern positively instead — YYYY-MM-DDTHH is fixed-width and already assumed to be lexicographically sortable everywhere in this file:

RE_HOUR_DIR = re.compile(r"^\d{4}-\d{2}-\d{2}T\d{2}$")

Applying that in both places fixes this issue and the twin, and it also makes --since's lexicographic comparison sound by construction rather than by luck. Anything that is not an hourly directory is then skipped regardless of what a filesystem check or a Windows mount leaves behind.

Worth a test that puts System Volume Information in a fixture drive and asserts both that select_target_drive ignores it and that newest resolves to the real latest hour — the current suite has neither, which is why this survived.

Source read at 9bbd33f.