fix: retry version resolution, and stop examples passing on exit 0 (#121, #102)

.github/instructions/examples.instructions.md

@@ -14,7 +14,7 @@ against a CHR, showing both the **CLI and the library API**.
 |---|---|---|
 | `<name>.ts` | **Primary** — runnable Bun script, library API. `#!/usr/bin/env bun`; `if (import.meta.main) main()`; `try…finally` teardown; `process.exitCode = 1` on failure. | always (except `grounding`) |
 | `<name>.sh` | CLI version — POSIX `sh`, sources `../common.sh`. | most |
-| `<name>.ps1` | PowerShell CLI mirror, dot-sources `../common.ps1`. | **all new** examples; existing where the CLI flow is simple |
+| `<name>.ps1` | PowerShell CLI mirror. Sets `$ErrorActionPreference = 'Stop'` **before** dot-sourcing `../common.ps1` (see "Exit 0 is not evidence"). | **all new** examples; existing where the CLI flow is simple |
 | `<name>.py` | Python CLI driver, run with `uv run` (stdlib only). | where a non-TS audience adds value |
 | `<name>.test.ts` | `bun:test` — only when assertions ARE the documentation. | `grounding` only |
 | `README.md` | from `_template/README.md`. | always |
@@ -50,6 +50,47 @@ in [`COVERAGE.md`](../../examples/COVERAGE.md) (mark docs/test-only with a reaso
   run in extended verification, across the supported-OS matrix. `trial-license`
   is manual-only (rate limits).
 
+## Exit 0 is not evidence ([#102](https://github.com/tikoci/quickchr/issues/102))
+
+An example that exits 0 and prints *something* is not a passing example. Two
+instances shipped green for months:
+
+- A `ParserError` in `common.ps1` took out the whole file, so every helper in
+  `quickstart.ps1` was undefined. No quickchr command ran; the script printed one
+  line and exited 0.
+- `mndp.py`'s "no announcement received" branch used a bare `return`, which leaves
+  `main()` through the `finally` and never reaches its `sys.exit(rc)` — so the
+  failure path exited 0.
+
+Three rules, each holding one end of that:
+
+- **The smoke harness asserts output markers**, not just `code === 0` and
+  `out.length > 0`. Each entry in `examples-smoke.test.ts`'s `RUNNABLE` names
+  substrings only a working run produces — including one from the END of the script
+  and one the CLI/library emitted rather than the example's own `echo`.
+- **`.ps1` sets `$ErrorActionPreference = 'Stop'` before dot-sourcing `common.ps1`**,
+  duplicating what `common.ps1` sets. Load-bearing, not stylistic: if `common.ps1`
+  fails to load, its own preference never takes effect and each undefined helper is
+  a non-terminating `CommandNotFound`. Reproduced 2026-08-03 (pwsh 7.4.6, Intel
+  macOS) against the real files with #102's defect reinstated: rc=0 without the
+  guard, rc=1 with it. Enforced by `scripts/validate-examples.ts`.
+- **A failure path must exit non-zero.** In Python `raise SystemExit(msg)`, never a
+  bare `return` past a trailing `sys.exit(rc)`. Same run: rc=0 before, rc=1 after.
+- **A `Start-Job` block sets both preferences itself.** `Start-Job` runs in a
+  separate process and inherits no preference variables, so the caller's and
+  `common.ps1`'s copies do not reach inside it. Without
+  `$ErrorActionPreference = 'Stop'` **and**
+  `$PSNativeCommandUseErrorActionPreference = $true` in the block, a `quickchr` call
+  that exits non-zero *without writing to stderr* is invisible: `Receive-Job`
+  returns, the script continues, the example exits 0 having booted nothing. Measured
+  (pwsh 7.4.6): a silent `exit 3` in a job gives **rc=0** unset, **rc=1** with both
+  set. A job whose command *does* write to stderr happens to propagate — do not rely
+  on that, it is the stderr-to-error-record mapping, not the exit code.
+
+`lint-powershell.yml` parses every `.ps1` with PowerShell's own parser before
+PSScriptAnalyzer, because a `Severity = @('Error','Warning')` filter does not
+surface `ParserError` — a file that cannot be parsed at all passed that gate.
+
 ## A failing example is a quickchr bug until proven otherwise
 
 Examples are **canaries**, not chores. The reason each one boots a real CHR is to catch

CHANGELOG.md

@@ -99,6 +99,45 @@ Even minor versions (0.2.x, 0.4.x) are releases; odd minors (0.3.x, 0.5.x) are p
 
 ### Fixed
 
+- **One refused connection to `upgrade.mikrotik.com` failed a whole `quickchr start`**
+  (#121). `resolveVersion()` had no retry at all: `fetchResilient()` is one attempt per
+  transport (system resolver, then a public-DNS IPv4 fallback), and if the fallback also
+  threw, the error propagated. #116/#119 gave the *download* path a retry policy and left
+  version resolution — which sits on the critical path of `start()`, the wizard and the
+  CLI — with a single shot. Observed on a `platforms=all` run: `Integration
+  (windows/x86_64 · testing)` went red on one `ConnectionRefused`, on a leg where nothing
+  about RouterOS, QEMU or the platform was involved. `resolveVersion()` now retries a
+  connection-class failure up to 3 times with the same backoff shape as `downloadToFile`,
+  and reports the attempt count plus the host when it exhausts them. An HTTP status is
+  still terminal on the first attempt — a 404 for a bad channel stays fast.
+- **The version-resolution error named an IP address nobody configured** (#121).
+  `fetchResilient()` discarded the direct attempt's error whenever the IPv4 fallback
+  itself threw, so the surfaced message read `path: "https://159.148.147.251/routeros/…"`
+  with no hostname — leaving "the system resolver was broken", "the host refused us" and
+  "public DNS handed back a bad address" indistinguishable, which is exactly the question
+  the log has to answer. That fallback fires routinely on hosted runners (their stub
+  resolver fails slowly, 2–26 s), so its error was often the only one visible. Both
+  failures are now raised together as a `ResilientFetchError` naming the URL, each
+  transport's error, the address the fallback used and which resolvers produced it, with
+  the direct failure kept as `cause`.
+- **Every PowerShell example was a silent false green** (#102). A `ParserError` in
+  `examples/common.ps1` took out the entire file — PowerShell parses `$LASTEXITCODE:` as
+  a scope-qualified variable reference — so every helper it defines was undefined for the
+  caller and `quickstart.ps1` ran no quickchr command at all. It exited 0 having printed
+  one line, and the smoke harness (`code === 0` and `out.length > 0`) reported `(pass)`
+  for months. The parse itself was fixed in #101; the false-green *class* is closed here:
+  the smoke harness now asserts per-example output markers — substrings only a working
+  run produces, including one from the end of the script and one the CLI itself emitted —
+  and each `.ps1` sets `$ErrorActionPreference = 'Stop'` **before** dot-sourcing
+  `common.ps1`, so a `common.ps1` that fails to load can no longer leave the example
+  running past it. Reproduced locally against the real files (pwsh 7.4.6, Intel macOS):
+  with #102's defect reinstated, rc=0 unguarded and rc=1 guarded. `validate-examples`
+  enforces the guard.
+- **`examples/mndp/mndp.py` exited 0 when no MNDP announcement arrived.** The failure
+  branch used a bare `return`, which leaves `main()` through the `finally` and never
+  reaches the trailing `sys.exit(rc)` — the same false green as #102, in a different
+  language. It now raises `SystemExit`; cleanup still runs. Measured with the identity
+  match forced to fail: rc=0 before, rc=1 after.
 - **A healthy large download was reported as a timeout** (#116). Both download paths
   bounded a transfer by *total duration*: `images.ts` aborted at a flat 120 s per
   attempt and retried three times, `packages.ts` had no deadline and no retries at all.

examples/common.ps1

@@ -1,10 +1,22 @@
 # Shared helpers for quickchr CLI examples (PowerShell - the Windows mirror of common.sh).
 #
 # Dot-source at the top of <name>.ps1, then wrap the body in try/finally:
+#   $ErrorActionPreference = 'Stop'
 #   . "$PSScriptRoot/../common.ps1"
 #   $name = Get-ExampleName 'quickstart'; Register-Cleanup $name
 #   try { Invoke-Qc start $name --channel stable; ... } finally { Invoke-QcCleanup }
 #
+# The caller sets $ErrorActionPreference BEFORE the dot-source even though this
+# file sets it too, and that duplication is load-bearing. If THIS file fails to
+# load, the preference it sets never takes effect - so unless the CALLER already
+# set it, the preference stays at its 'Continue' default, every helper call below
+# is an ordinary non-terminating CommandNotFound, and the example exits 0 having
+# done nothing: the silent false green in #102. The caller's copy is what makes a
+# failed load terminate. Enforced by scripts/validate-examples.ts.
+#
+# Start-Job runs in a separate process and inherits neither preference, so a job
+# script block must set both itself (see version-matrix.ps1).
+#
 # Resolution rule mirrors common.sh: prefer an explicit $env:QUICKCHR override,
 # else the repo source CLI (so CI/local runs exercise THIS checkout), else a
 # globally installed `quickchr`.

examples/device-mode/device-mode.ps1

@@ -1,4 +1,5 @@
 # device-mode (CLI, Windows) - enable a device-mode feature on first boot, read it back.
+$ErrorActionPreference = 'Stop'  # before the dot-source - see common.ps1 (#102)
 . "$PSScriptRoot/../common.ps1"
 
 $name = Get-ExampleName 'device-mode'

examples/dude/dude.ps1

@@ -1,5 +1,6 @@
 # dude (CLI, Windows) - install the dude package on first boot, enable it, read it back.
 # PowerShell mirror of dude.sh.
+$ErrorActionPreference = 'Stop'  # before the dot-source - see common.ps1 (#102)
 . "$PSScriptRoot/../common.ps1"
 
 $name = Get-ExampleName 'dude'

examples/grounding/grounding.ps1

@@ -1,5 +1,6 @@
 # grounding (CLI, Windows) - apply RouterOS config, read it back, prove it took.
 # PowerShell mirror of grounding.sh.
+$ErrorActionPreference = 'Stop'  # before the dot-source - see common.ps1 (#102)
 . "$PSScriptRoot/../common.ps1"
 
 $name = Get-ExampleName 'grounding'

examples/harness/harness.ps1

@@ -1,6 +1,7 @@
 # harness (CLI, Windows) - hand a CHR's connection env to an external tool.
 # PowerShell can't eval the shell-quoted `env` output, so use --json and set
 # $env:* explicitly. This is the natural Windows showcase for env-passing.
+$ErrorActionPreference = 'Stop'  # before the dot-source - see common.ps1 (#102)
 . "$PSScriptRoot/../common.ps1"
 
 $name = Get-ExampleName 'harness'

examples/mndp/mndp.py

@@ -42,7 +42,21 @@
 
 
 def run_quickchr(*args: str, check: bool = True) -> subprocess.CompletedProcess:
-    return subprocess.run([*QUICKCHR, *args], capture_output=True, text=True, check=check)
+    # check=False: this wrapper does its own returncode handling below, because
+    # subprocess.run's own check= raises without the output (see next comment).
+    proc = subprocess.run([*QUICKCHR, *args], capture_output=True, text=True, check=False)
+    if check and proc.returncode != 0:
+        # subprocess.run(check=True) raises a CalledProcessError naming the command
+        # and the exit code and DISCARDING both streams -- and those streams are the
+        # only place quickchr says what actually went wrong. Run 30852139131
+        # (linux/aarch64) failed here on `/system/identity/set` and reported nothing
+        # but "returned non-zero exit status 1", which is not enough to diagnose it.
+        raise SystemExit(
+            f"FAIL: quickchr {' '.join(args)} exited {proc.returncode}\n"
+            f"--- stdout ---\n{proc.stdout}"
+            f"--- stderr ---\n{proc.stderr}"
+        )
+    return proc
 
 
 # ── frame parsing ────────────────────────────────────────────────────────────
@@ -234,8 +248,10 @@ def main() -> None:
             time.sleep(1)
 
         if not got:
-            print(f"FAIL: no MNDP announcement with identity '{IDENTITY}' within {args.timeout}s", file=sys.stderr)
-            return
+            # `raise`, not `return`: a bare return leaves main() through the finally
+            # and never reaches `sys.exit(rc)` below, so the failure would exit 0 —
+            # the same silent false green as #102's PowerShell example.
+            raise SystemExit(f"FAIL: no MNDP announcement with identity '{IDENTITY}' within {args.timeout}s")
         print("\nMNDP received over L2 (socket-connect):")
         for k in ("identity", "version", "platform", "board", "ifname", "ipv4", "uptime", "mac", "softwareId"):
             if k in got:

examples/quickstart/quickstart.ps1

@@ -1,5 +1,6 @@
 # quickstart (CLI, Windows) - boot a CHR, read resource + descriptor, tear down.
 # PowerShell mirror of quickstart.sh. try/finally guarantees teardown.
+$ErrorActionPreference = 'Stop'  # before the dot-source - see common.ps1 (#102)
 . "$PSScriptRoot/../common.ps1"
 
 $name = Get-ExampleName 'quickstart'

examples/rollback/rollback.ps1

@@ -1,4 +1,5 @@
 # rollback (CLI, Windows) - snapshot a CHR, change it, restore the snapshot.
+$ErrorActionPreference = 'Stop'  # before the dot-source - see common.ps1 (#102)
 . "$PSScriptRoot/../common.ps1"
 
 $name = Get-ExampleName 'rollback'

examples/service-forward/service-forward.ps1

@@ -1,4 +1,5 @@
 # service-forward (CLI, Windows) - pin a guest service to a chosen host port.
+$ErrorActionPreference = 'Stop'  # before the dot-source - see common.ps1 (#102)
 . "$PSScriptRoot/../common.ps1"
 
 $name = Get-ExampleName 'service-forward'

examples/trial-license/trial-license.ps1

@@ -1,5 +1,6 @@
 # trial-license (CLI, Windows) - apply a CHR trial license, read it back. MANUAL-ONLY.
 # MikroTik rate-limits trial requests, so this is excluded from CI.
+$ErrorActionPreference = 'Stop'  # before the dot-source - see common.ps1 (#102)
 . "$PSScriptRoot/../common.ps1"
 
 $name = Get-ExampleName 'trial-license'

examples/version-matrix/version-matrix.ps1

@@ -2,6 +2,7 @@
 # PowerShell mirror of version-matrix.sh (parallel start via background jobs).
 param([switch]$Lite)
 
+$ErrorActionPreference = 'Stop'  # before the dot-source - see common.ps1 (#102)
 . "$PSScriptRoot/../common.ps1"
 
 $channels = if ($Lite) { @('long-term', 'stable') } else { @('long-term', 'stable', 'testing', 'development') }
@@ -19,6 +20,14 @@ foreach ($ch in $channels) {
 	# PSUseUsingScopeModifierInNewRunspaces wants -- it doesn't recognize the older
 	# param()+-ArgumentList pattern and flags those as missing the Using: scope.
 	$jobs += Start-Job -ScriptBlock {
+		# Set INSIDE the block: Start-Job runs in a separate process that inherits
+		# no preference variables, so the caller's and common.ps1's copies do not
+		# reach here. Without these, a `quickchr start` that exits non-zero without
+		# writing to stderr is invisible - Receive-Job returns, the script continues,
+		# and the example exits 0 having booted nothing. Measured (pwsh 7.4.6): a
+		# silent `exit 3` in a job gives rc=0 unset, rc=1 with these two set.
+		$ErrorActionPreference = 'Stop'
+		$PSNativeCommandUseErrorActionPreference = $true
 		$parts = $using:qc -split '\s+'
 		& $parts[0] @($parts[1..($parts.Length - 1)]) start $using:name --channel $using:ch --no-secure-login --port-base $using:base --add-package container --mem 256
 	}

scripts/validate-examples.ts

@@ -7,16 +7,20 @@
  *   - no Makefile / stray files (only .ts/.test.ts/.sh/.ps1/.py/.md + known subdirs)
  *   - the primary script is named after its directory
  *   - every relative link and run-command file reference in the README resolves
+ *   - each .ps1 sets $ErrorActionPreference before dot-sourcing common.ps1
  *
  * Wired into `bun run check`. Exits non-zero on any violation.
  */
-import { readdirSync, statSync } from "node:fs";
+import { readFileSync, readdirSync, statSync } from "node:fs";
 import { existsSync } from "node:fs";
 import { join, resolve } from "node:path";
 
 const EXAMPLES = resolve(import.meta.dir, "..", "examples");
 const ALLOWED_EXT = new Set([".ts", ".sh", ".ps1", ".py", ".md"]);
-const SKIP_DIRS = new Set(["_template", "config", "tool", "node_modules"]);
+// `__pycache__` is gitignored, so it never reaches CI and can never be a real
+// violation — but any local `python3 -m py_compile` / import of a .py example
+// creates one, and flagging it turns `bun run check` red for a build artifact.
+const SKIP_DIRS = new Set(["_template", "config", "tool", "node_modules", "__pycache__"]);
 // (Top-level files like README.md / lib.ts / common.sh are skipped implicitly:
 // the dir scan below only descends into directories, never stat's loose files.)
 
@@ -80,6 +84,24 @@ for (const name of dirs) {
 		}
 	}
 
+	// PowerShell examples must arm $ErrorActionPreference themselves, BEFORE
+	// dot-sourcing common.ps1. A common.ps1 that fails to load leaves the
+	// preference at 'Continue', so every undefined helper is a non-terminating
+	// error and the example exits 0 having run no quickchr command at all — the
+	// false green in #102, reproduced locally 2026-08-03 (pwsh 7.4.6: rc=0 without
+	// the guard, rc=1 with it).
+	for (const f of files.filter((x) => x.endsWith(".ps1"))) {
+		const lines = readFileSync(join(dir, f), "utf8").split(/\r?\n/);
+		const dotSource = lines.findIndex((l) => /^\s*\.\s+["']\$PSScriptRoot/.test(l));
+		if (dotSource < 0) continue; // self-contained script, nothing to guard against
+		const armed = lines
+			.slice(0, dotSource)
+			.some((l) => /^\s*\$ErrorActionPreference\s*=\s*'Stop'/.test(l));
+		if (!armed) {
+			err(where, `"${f}" must set $ErrorActionPreference = 'Stop' BEFORE dot-sourcing common.ps1`);
+		}
+	}
+
 	// README references resolve.
 	const readmePath = join(dir, "README.md");
 	if (existsSync(readmePath)) {