Working on the shared NPU

Working on the shared NPU

Several agents and sessions use strixhalo's one NPU at the same time. Most of the failures we chased on 2026-09-22/23 were one job taking down everyone else's. These rules are what that cost.

1. Never reload the driver while others are running. sudo modprobe -r amdxdna (for example npu-infer/tests/run_validation_matrix.sh --reset) kills every process's contexts. They see No such device with index '0'.
2. Wait for an idle NPU before a measured run. fuser /dev/accel/accel0 prints nothing when it is free. Record the holders during the run. A number taken next to a foreign context is not a measurement.
3. Jobs that wait for idle start together. An idle gate lowers the odds of contention, but it does not remove them. If a run fails at dispatch #1, check the kernel log for a foreign IO_PAGE_FAULT in the same second.
4. A kernel that can fault belongs in a probe loop with a timeout, not in a tight retry. One loop that faulted every run caused a watchdog reset about every 45 s for 10 minutes and killed every other job on the device.
5. Kill only your own processes, by exact name (pgrep -x, then kill -9). pkill -f <pattern> also kills other people's SSH shells.
6. Never keep the only copy of working code or kernels in /tmp. It is a tmpfs, and a reboot wipes it. Working ternary kernels for Bonsai were lost this way. Commit them, or keep them under ~/.cache/<project>/.
7. Serialise measured runs with one lock. Since 2026-09-24 every agent wraps each NPU (or GPU benchmark) run in flock ~/.cache/lax-decode/box.lock <cmd>, one batch per hold. Builds need no lock, but they load the host: host load alone moved a 76 ms runlist to 90 ms, so note /proc/loadavg with every timing.
8. Use your own data directories. A shared pool directory was overwritten with a different packing layout mid-session; pack to your own --pool-dir, or pack at load time.
9. Blame a kernel only after reading the watchdog dump. See NPU#reading-an-ert-timeout.