Skip to content

CHANGELOG.md

Latest commit

 

History

History
672 lines (501 loc) · 318 KB

CHANGELOG.md

File metadata and controls

672 lines (501 loc) · 318 KB
Raw

WeOwn AI Repository Changelog

This file tracks repository-level (infrastructure, workflows, governance, cross-cutting documentation) changes.

Application-specific changes live in per-directory CHANGELOGs. See the index below.

Format: Keep a Changelog 1.1 Versioning: Per VERSIONING_WEOWNVER.md — vSEASON.MONTH.WEEK.ITERATION

Per-Directory Changelog Index

Component Changelog
AnythingLLM anythingllm/CHANGELOG.md
Gitea runner Docker gitea-runner-docker/template/CHANGELOG.md.jinja
Landing/Purchase landing-purchase/CHANGELOG.md
Matomo matomo/CHANGELOG.md
n8n n8n/CHANGELOG.md
Nextcloud nextcloud/CHANGELOG.md
Sandbox Docker sandbox-docker/template/CHANGELOG.md.jinja
SearXNG Docker searxng-docker/template/CHANGELOG.md.jinja
SigNoz Docker signoz-docker/template/CHANGELOG.md.jinja
Vaultwarden vaultwarden/CHANGELOG.md
WordPress wordpress/CHANGELOG.md
WordPress Dev wordpress-dev/docs/CHANGELOG.md

[Unreleased]

Changes in this section will be promoted to a dated release entry on merge to main.

Security

  • Fixes that missed #320's merge (2026-10-12, #323). anythingllm-docker embed-filter: a HEAD, 204, 205 or 304 with application/json passes through instead of becoming a 502, and an empty 200 still fails closed. scripts/publish_to_buzz.py: hostname redaction is case-insensitive, so example.XN--P1AI no longer leaves <host>--P1AI in a public note. beta-weown-chat gets the embed-filter change on its next deploy; Buzz reports on their next run.

  • anythingllm-docker — a fresh wrap token is never ignored (2026-10-12, weown-fleet#168). The openbao deploy's host unwrap used the token in .bao-wrap.token whenever that file was non-empty. After a failed unwrap the file kept the spent token, so every later BAO_WRAP_TOKEN was ignored and the deploy failed until someone blanked the file by hand. A supplied token now always replaces the file and always rotates the secret-id, even one that still works; a failed unwrap blanks the file, and if the current secret-id works the deploy warns and keeps it. Template and the beta-weown-chat render; applies on that instance's next deploy. Two deploys of one instance are serialized by a lock held from the validity check through the restarts; overlapping rotations could unwrap each other's token and leave one live. Test: anythingllm-docker/tests/deploy-unwrap-check.sh, 40 checks against a real OpenBao dev server; main's task fails the stale-token and rotation checks.

  • Folded fixes from the ai-grok seat (2026-10-12, #320). Six stranded PRs (#286–#289, #299, #300) and the redaction fix on merge/nik-mac (#309), cherry-picked onto current main:

    • billing: an OIDC login links by sub or verified e-mail only, never onto staff
    • anythingllm-docker: MCP children get an env allowlist; embed-filter strips reasoning from JSON chat history and fails closed (502) on a body over 16 MiB or one cut short
    • Node 24 LTS pinned by digest, with a CI check
    • Matomo and Nextcloud securityContext (weown-fleet#102)
    • ((X++)) under set -e (weown-fleet#148)
    • IP addresses and dotted hostnames redacted from Buzz notes (weown-fleet#128)

    Fixed in review: Matomo's init container ran as root with every capability dropped, so its chown and config edits failed silently; it now runs as the pod's non-root user 33. The Nextcloud cron job ran as root, which cron.php refuses; it now runs as 33. Two of the source branches predated weown-fleet#48 and would have reverted the tenant document scope; the fold keeps it. The CI-flow change from the same seat (fca598a4) is not adopted: the current auto-PR behaviour stays. Live renders change on their next deploy (beta-weown-chat, sites/billing); the helm charts change on their next upgrade.

  • gitea-docker — hardening (2026-10-09, weown-fleet#163 group 6). Caddy serves TLS 1.3 only. ssh_source_cidrs, which opens the admin sshd, has no default (it was the whole internet) and every entry must be a /24 or narrower (IPv6 /64), the gitea-runner-docker rule; it is no longer a copier answer, so an admin CIDR never lands in a committed render. ansible/deploy.yml refuses a box that keeps Gitea volumes under another project name, so a mismatched render cannot start Gitea on new, empty volumes. Applies to the live forge on its next deploy (Caddyfile, play) and plan (the variable; the live value already passes).

  • gitea-docker — manual MI rotation refreshes the containers' auth copy (2026-10-08, weown-fleet#163). scripts/rotate-mi-manual.sh swapped v2 into the host's auth file but not the copy the containers read (written by the deploy). Revoking v1, the next step of the runbook, would then leave any container that restarts before the next deploy unable to log in to Infisical. The swap now refreshes that copy too, keeping its owner and mode. Found while rotating the live forge, whose v1 bootstrap secret was confirmed still valid by --verify.

  • gitea-docker — pinned installs (2026-10-09, weown-fleet#163 group 3). Cloud-init installs Docker from its signed apt repository (fingerprint-checked key, pinned and held versions) instead of get.docker.com, and the Infisical CLI from a sha256-pinned release .deb instead of curl | bash, as gitea-runner-docker does. The awscli apt package, which does not exist on Ubuntu 24.04 and failed the whole package step, is removed. Ansible collections are exact pins. Applies on a rebuild (cloud-init) and the next collection install.

  • gitea-docker — bootstrap rotation marks done only after v1 is proven revoked (2026-10-08, weown-fleet#163). The first-boot Machine Identity rotation wrote .rotation-complete even when v1 could not be identified or its revoke was not confirmed, and it took "the oldest active secret that is not v2" for v1, so with another older secret active it revoked the wrong one and left v1, which sits in terraform state and droplet user_data, valid. The gitea-runner-docker rotation (#302) is ported to the template and the live forge render: the marker records the proof (v1-401 <time>) only when a login with the bootstrap client id and secret from user_data answers 401 "Invalid credentials", ansible/deploy.yml refuses to deploy without that proof, and scripts/rotate-mi-manual.sh plus a README section (the failure log pointed at one that did not exist) finish it by hand. Cloud-init changes apply on a rebuild. The deploy gate applies on the forge's next deploy: the empty marker the old script wrote no longer passes, so the operator runs rotate-mi-manual.sh --verify once first, which proves v1 dead there or says it is still alive. Verification: gitea-docker/tests/rotation-check.sh gitea-docker/sites/git and gitea-docker/tests/manual-rotation-check.sh gitea-docker/sites/git.

  • gitea-docker — secrets off command lines and backups owner-only (2026-10-08, weown-fleet#163 group 1). backup.sh passed the database password to docker exec -e, visible in ps for the whole dump: it now reaches pg_dump on stdin, and the backup runs under umask 077 in a 0700 directory (dumps and archives were 0644). The first-boot rotation script put the Machine Identity secret and bearer tokens on curl's argv: logins now send JSON on stdin (--data @-) and tokens go in a curl config on stdin (-K -). The README's OAuth setup now prefers the web UI and feeds the CLI secret on stdin, saying plainly what remains on argv. The Quick Start uses itofu.sh (secrets from Infisical at run time); init.sh, which reads credentials from a local terraform.tfvars, warns that it is deprecated. Also removed one em-dash from the template's cloud-init (DigitalOcean cloud-init must be ASCII-only). Applies to the live forge on its next deploy (backup.sh) or rebuild (cloud-init).

  • gitea-docker — consistent backups, safe and recoverable restores (2026-10-09, weown-fleet#163 group 4). backup.sh stops Gitea while it dumps the database and copies the data volume, so the two match; a trap restarts it. restore.sh accepts only a name backup.sh produces and passes it to the root-run program as an argument (it was spliced into the program text), in a mktemp file instead of a fixed /tmp path. The archive is checked while Gitea still serves; after the stop the current database and volumes are copied, and a restore that fails part-way loads that copy back and restarts Gitea. psql stops on the first SQL error. Retention keeps a backup whose date does not parse (it was deleted as epoch 0); off-box copies get the same policy, report-only until REMOTE_RETENTION=enforce. Applies to the live forge on its next deploy.

  • gitea-docker — AWS CLI pinned (2026-10-09, weown-fleet#163 group 3). The deploy play installs AWS CLI 2.37.12 from AWS's versioned zip, checked against a recorded sha256 (AWS's PGP signature, key FB5D B77F D5C1 18B8 0511 ADA8 A631 0ACC 4672 475C, verified when the pin was taken), instead of whatever the unversioned zip served, unchecked. A box on another version is moved to the pin on its next deploy. The other 18 unpinned AWS CLI installs (anythingllm, billing, keycloak, openclaw, sandbox, supabase) are listed on weown-fleet#164.

  • gitea-docker/sites/git — the live Gitea render is committed (2026-10-08, weown-fleet#159). The forge's own render had only ever existed as an untracked directory in an operator checkout, so its infrastructure (firewall included) ran from code no PR had reviewed. It is now the canonical committed render, with two changes from the copy that built the box: the dormant 10.0.0.0/8 → 5432 firewall rule is removed (as for the templates in #303), and the monitoring alert_email default is the role address alerts@weown.net instead of a personal address from another engagement. The next tofu apply changes both on the live firewall and monitors; the operator runs it.

  • billing-docker — backups and restores keep secrets off command lines and files owner-only (2026-10-12, weown-fleet#166 review). For both backends: backup.sh passed the database password to docker exec -e PGPASSWORD=..., so it was in the docker client's argv for the whole dump. It now reaches pg_dump on stdin, and backups run under umask 077 (archives were 0644). restore.sh remote mode wrote its program to a fixed /tmp/weown-restore-cmds.sh; it now uses a mktemp file removed by an EXIT trap. The AWS S3 remote-storage choice is removed, because backup.sh never uploaded to it: an s3 render made only a local archive while the cron reported success. On an openbao render, a login the store refuses (HTTP 400 or 403) is not retried. It is recorded in /opt/<project>/bao-state/login-refused, and no consumer logs in again until a deploy clears it. Before, provisioning_watch alone would have used up the shared role's lockout budget in five ticks. A BAO_WRAP_TOKEN supplied to a deploy now always rotates the secret-id, even one that still works; if its unwrap fails, the deploy warns and keeps the working one. The old secret-id still logs in until a secrets-admin destroys it (the box's AppRole cannot), so the deploy prints the new accessor and the README gives the destroy step. backup.sh also tightens archives an earlier version left 0644. restore.sh (both backends) accepts only a backup name of the shape backup.sh produces, or an s3:// URL in this site's own location, and checks it again inside the program it runs as root; a quote in the name was spliced into that program. The backup.sh and restore.sh fixes are also applied to the live sites/billing render (re-rendered with its name and domain; the diff is only these lines): backup.sh reaches the box on its next deploy, and restore.sh runs from the operator's checkout. Tests: backup-bao-check.sh 30 checks, deploy-unwrap-check.sh 33, entrypoint-bao-check.sh 36; each new check fails on the head before it.

  • billing-docker — OpenBao backend, stage 5: the runbook (2026-10-12, weown-fleet#166). An openbao render's README covers the steps before a first deploy (mint the AppRole, seed the document with the listed keys, open the store's 8200 allowlist to the droplet, keep a break-glass copy), the deploy with a hidden wrap-token prompt, the backup that must be seen landing afterwards, changing a value with bao kv patch KEY=- (value on stdin; checked against a dev server), and what to do after the store refuses the secret-id. It also says that POSTGRES_* only seed an empty database: rotating the password starts in Postgres. The infisical README is unchanged.

  • billing-docker — OpenBao backend, stage 4: backups and restores read OpenBao (2026-10-12, weown-fleet#166). Billing holds customer and payment data, so an openbao box never runs without backups. A new scripts/bao-run.sh (openbao renders only) runs a command on the host with the backup keys (POSTGRES_*, plus SPACES_* with DO Spaces) loaded through entrypoint-bao.sh, the wrapper the containers use. The daily backup cron, backup.sh's remote mode and restore.sh's remote mode run through it. A missing key or a refused login stops the run before backup.sh starts, after one login attempt and with no cooldown sleep, so the cron fails at once and adds one failed login, not two, to the lockout count of the role db and web share. The provisioning_watch cron also makes one login attempt per tick. The stage-3 refusal is removed: an openbao deploy now installs the backup cron. An infisical render is byte-identical to before (CHANGELOG aside). Test: billing-docker/tests/backup-bao-check.sh, 14 checks against a real OpenBao dev server; removing the one-attempt setting fails it.

  • billing-docker — OpenBao backend, stage 3: the deploy playbook (2026-10-11, weown-fleet#166). An openbao render's ansible/deploy.yml installs a pinned bao CLI, places the store CA, fills the store URL into the entrypoint on the box from BAO_ADDR_INSTANCE, probes the store, and unwraps the AppRole secret-id on the host from BAO_WRAP_TOKEN on stdin, restarting db and web when it is replaced. Infisical-only tasks (MI checks, rotation proof, registry login, container auth file, Infisical backup cron) are skipped; scripts/deploy.sh asks for BAO_ADDR_INSTANCE instead of INFISICAL_PROJECT_ID. The bao CLI step checks /usr/bin/bao (the path compose mounts) at the pinned version, so a pin bump upgrades the box and a bao elsewhere on PATH does not count. Fixed while porting from anythingllm-docker: after a failed unwrap the spent token stayed in .bao-wrap.token and was preferred over every fresh BAO_WRAP_TOKEN. The wrap token now never touches disk: it goes from stdin straight into bao unwrap and is unset whether the unwrap works or not, and a .bao-wrap.token left by the earlier logic is removed. bao_cli_version must be a plain release version, since it is rendered into shell. Until stage 4 the openbao deploy refused before touching the box, because billing never runs without backups. An infisical render is byte-identical to before. Test: billing-docker/tests/deploy-unwrap-check.sh, 24 checks against a real OpenBao dev server (the three no-file checks fail on the earlier logic).

  • billing-docker — OpenBao backend, stage 2: containers, cloud-init, terraform (2026-10-10, weown-fleet#166). An openbao render wires db (--keys) and web (--document) to the stage-1 entrypoint, drops every Infisical step from cloud-init, tags the droplet openbao-consumer and stops injecting a per-site Infisical folder in itofu.sh; compose comments name OpenBao, not Infisical, as the store. Both backends now tag the droplet billing instead of the keycloak and sso tags copied from keycloak-docker, which put billing in every SSO-wide scripts/manage-droplets.sh run; apart from that tag line the infisical render is byte-identical to main. The live sites/billing render is not re-rendered here and keeps the old tags (weown-fleet#167).

  • billing-docker — OpenBao backend, stage 1: container entrypoint (2026-10-09, weown-fleet#166). Wave 2 of the Infisical → OpenBao move starts with billing (the forge is held by the custody gate, WeOwnCloud/openbao#16). The template gains a secret_backend choice (default infisical; the render is unchanged) and an OpenBao entrypoint that works in postgres:16 and the Python web image (AnythingLLM's needs node). Compose, deploy, backup and restore wiring follow in later stages; no live values move until Nik's go. Verification: billing-docker/tests/entrypoint-bao-check.sh runs it against a real OpenBao 2.7.0 dev server (32 checks).

  • billing-docker — bootstrap rotation marks done only after v1 is proven revoked (2026-10-09, weown-fleet#164). The gitea-docker fix (#307, #308) ported to the billing template and the live sites/billing render: v1 is proven dead by a login that answers 401 "Invalid credentials" before .rotation-complete is written (v1-401 <time>); the old script took "the oldest active secret that is not v2" for v1 and wrote the marker regardless; secrets leave curl's argv; runs are serialized. ansible/deploy.yml gates on the proof and scripts/rotate-mi-manual.sh finishes by hand. Cloud-init applies on a rebuild; the gate applies on the next deploy, which stops until rotate-mi-manual.sh --verify proves v1 dead on the live box. Verification: gitea-docker/tests/rotation-check.sh billing-docker/sites/billing and gitea-docker/tests/manual-rotation-check.sh billing-docker/sites/billing.

  • anythingllm-docker — dashboard delete purged nothing, and could purge shared files once fixed (2026-10-08, weown-fleet#157). remove-documents received an empty body (a DELETE without Content-Length), so "deleted" documents stayed in storage while the dashboard reported success. Fixed with explicit body framing, a purge only for files inside the tenant's folder (detach-only for workspace documents stored elsewhere), and a 502 when the purge is refused; replace-on-duplicate follows the same folder rule, and every AnythingLLM call is pinned to ALLM_URL's origin. Three hand-derived cases in test-library-scope.mjs fail on the old code and pass now. Applies on the next redeploy of each dashboard instance (template and beta-weown-chat).

  • gitea-runner-docker — a Gitea Actions runner on its own droplet (2026-10-07, WeOwnCloud/openbao#8). A copier template and the weown-ci-runner site. The runner serves one repository, and every CI job runs in a Docker-in-Docker daemon that is created for it and destroyed after it: containers, volumes, images and daemon settings never carry over to the next job. The droplet has its own VPC and no inbound port but admin SSH, and the SSH CIDRs and the VPC range are required tofu variables, never in git. Containers are refused the metadata service, and the deploy proves that before any job runs. Docker comes from its signed apt repo (key fingerprint checked, versions held), and the Infisical CLI from a sha256-pinned release .deb. The bootstrap Machine Identity secret is marked rotated only once a login with it answers 401. The registration token is the only secret; a one-shot register step alone holds the Infisical credential, and the runner that handles jobs mounts neither. Decision record: .github/ADR-008-gitea-runner-dedicated-droplet.md (Accepted). Cost: one droplet (s-2vcpu-4gb-amd by default). Compliance: NIST CSF 2.0 PR.AA-05, PR.DS-1, PR.PS-01, DE.CM. Verification: gitea-runner-docker/tests/live-check.sh gitea-runner-docker/sites/weown-ci-runner.

  • gitea / keycloak / supabase — dormant Postgres firewall rules removed (2026-10-07, weown-fleet#159). Each firewall admitted tcp/5432 from 10.0.0.0/8 ("only from within the VPC", but that range is every private network). No production compose template publishes Postgres on the host (the apps reach it as db:5432 on the compose network; only the compose.local dev templates publish 5432, on a workstation), so the rule admitted nothing on a deployed box today. It was a trap: the day anyone published 5432, every box in the shared VPC could reach it. Removed from the three templates and the live keycloak-docker/sites/sso and supabase-docker/sites/supabase-dev renders; the retired sso.weown.dev render is untouched. Applying it is a firewall change, so the operator runs it.

  • anythingllm-docker — the dashboard document library listed every file on the instance, not only the tenant's (2026-10-07, weown-fleet#48). The library merged the tenant's two workspaces with AnythingLLM's system document store, which is instance-wide. A signed-in customer saw the titles of operator test files, files attached to no workspace, and another brand's documents on the same instance. Replace-on-duplicate matched uploads against the same list, so it could pick another brand's same-named file to remove. The store pass now reads only the tenant's own folder, named by the new DASHBOARD_DOCUMENT_FOLDER (copier answer dashboard_document_folder), and dashboard uploads are written to that folder. Unset, malformed, or custom-documents (AnythingLLM's shared default) fails closed: the store is not read and the library lists workspace documents only. The handlers that take a docpath from the client (/api/documents/content, /scope, /delete, /lock) checked only that the string was well-formed. A docpath of another brand, which earlier listings had shown, could be read, attached to the tenant's workspace and chatted with, or detached. All four now pass one gate, tenantMayTouch(). A docpath is allowed only if it is embedded in one of the tenant's workspaces (read fresh per request) or exists at exactly that path in the tenant folder. Anything else gets 404, so the answer does not confirm the file exists, and nothing about it is sent to AnythingLLM. /content looked documents up by file name, and AnythingLLM answers from the first folder holding that name. It now reads the stored file at the exact authorized path first and, when that file is mounted, does not ask AnythingLLM at all. Without the mounted store it uses AnythingLLM's answer only when that answer names the same path. New CI job dashboard-tests runs every dashboard test in the template and the committed sites. test-library-scope.mjs fails 32 of its 46 checks on the old server and passes all 46 on the new one.

  • do_backup_stale_alert.yml — findings and delivery failures use different exit codes (2026-10-03). A delivered audit with no stale or unbacked resources exits 0. A delivered audit that found them exits 3, and the workflow stays green. An API error, an incomplete snapshot list, or a Buzz publish that did not succeed exits 1, so the job goes red. Compliance: NIST CSF 2.0 DE.CM, PR.IP-4. Verification: python check_stale_backups.py --self-check.

  • Cost and backup notes — executive summary and optional mentions (2026-10-02). Both reports open with cost or compliance KPIs. The itemized inventory sits inside <details>. NOTIFICATION_PUBKEYS (comma-separated 64-hex public keys) adds a p tag per person. A backup audit with zero violations publishes two lines, "All backups compliant.", and does not add those tags. A private key in that list is refused and not printed. Compliance: NIST CSF 2.0 ID.AM-1, PR.IP-4, PR.DS-1. Verification: both reporters' --self-check and python scripts/publish_to_buzz.py --self-check.

  • scripts/do-cost-reporter — Buzz gets a short infrastructure summary (2026-10-02). The channel note is the spend, droplet, DOKS, volume, and load-balancer counts, plus how many volume-backed droplets have no backup feature. The team table sits in a collapsed <details> block. Resource names stay in the local DO_MONTHLY_COST_REPORT.md. Compliance: NIST CSF 2.0 ID.AM-1, PR.DS-1. Verification: python do_cost_reporter.py --self-check.

  • Late review findings on Shahid's Buzz and DigitalOcean report PRs (#277–#283), fixed (2026-10-02). publish_to_buzz.py refuses a channel id or name that carries the signing key: nostr:nsec1… (the shape guard ran before the prefix was stripped), and the key's own 64-hex form, which is shaped like an event id. Every tag is compared with the key before anything is signed. Its --self-check now covers the nsec load path and publishes through a local NIP-42 relay on 127.0.0.1, so it fails if uniffi_set_event_loop is removed. Both reporters' requirements.txt are hash-locked with --require-hashes, dependencies included. A 401 names the variable that holds the rejected token (DO_EXTRA_TEAM_TOKENS teams included), and the workflows README lists the secrets and refusal semantics.

  • Late review findings on #272/#276, fixed (2026-10-02). sso scripts (create-weown-user, kc-set-temp-password, kc-configure-smtp, kc-set-login-theme) refuse any argument that is not a plain Keycloak name, e-mail or host, before ssh: a quote plus shell syntax in an argument ran as root on the SSO host. The FIFO secret writers (bootstrap-product, store-sendgrid-key, provision-openrouter-key) are killed by the EXIT trap, so a TERM/HUP to the script no longer leaves one holding the secret. Backup exits 1, before retention, when the dashboard could not be quiesced (the archive is still stored); restore prints "restore complete" only on success. dev/up.sh classifies enable-multi-user by status + exact reply, and its .env.dev rewrites leave no temp copy. A committed dashboard test (test-origins.mjs) covers userinfo origins and the position-only 400.

  • scripts/publish_to_buzz.py — channel notes are also Kind 9 (2026-10-02). Kind 1 reached the relay and did not show in the channel view. A set BUZZ_CHANNEL_ID now signs the same markdown twice: Kind 1 with e (root), h, t (Cloud&Infrastructure), and p (sender pubkey), then Kind 9 with h, t, and p so the channel index can show it. Kind 9 has no e tag, because that marks a reply. If the relay answers unknown event kind for Kind 9, that refusal is printed and the Kind 1 note still stands. Markdown line breaks are kept. The key is never printed. Compliance: NIST CSF 2.0 PR.DS-1. Verification: python scripts/publish_to_buzz.py --self-check signs both kinds offline and does not connect.

  • scripts/publish_to_buzz.py — reports are Kind 1 notes (2026-10-02). The relay rejects Kind 42 with restricted: unknown event kind, so the signed note is always Kind 1. When BUZZ_CHANNEL_ID is set the tags are e (root, relay URL of this send), h (that channel id), and t (Cloud&Infrastructure). The key is never printed. Compliance: NIST CSF 2.0 PR.DS-1. Verification: python scripts/publish_to_buzz.py --self-check signs a Kind 1 event and does not connect.

  • scripts/publish_to_buzz.py — relay refusals are printed, not replaced (2026-10-01). NOSTR_PRIVATE_KEY is stripped of spaces, newlines, and wrapping quotes, then Keys.parse accepts hex or nsec (an nsec is decoded to hex before use). NIP-42 still uses SignerAuthenticator. A send_event or authentication refusal is printed as the relay's own message and raised, so the job exits non-zero. A successful send prints [SUCCESS] Published event <id> to channel <id> on <relay>. The key is never printed. Compliance: NIST CSF 2.0 PR.DS-1. Verification: python scripts/publish_to_buzz.py --self-check.

  • Superseded (2026-10-02) by "relay refusals are printed, not replaced" above: a refused login now fails the job. scripts/publish_to_buzz.py — a rejected relay login does not fail the report (2026-10-01). NOSTR_PRIVATE_KEY is stripped, then parsed with Keys.parse as hex or nsec. NIP-42 still uses SignerAuthenticator (nostr-sdk 0.45 has no Client.handle_auth). If the relay answers authentication failed, the script logs [WARNING] Nostr publish failed: authentication failed and returns auth_failed so the cost and backup jobs keep the report they already wrote (exit 0 unless the audit itself is bad). The key is never printed. Compliance: NIST CSF 2.0 PR.DS-1. Verification: python scripts/publish_to_buzz.py --self-check.

  • scripts/publish_to_buzz.py — NIP-42 AUTH callback has an event loop (2026-10-01). After #280 the monthly cost run still exited 1. The report was written, then make_auth_event raised RuntimeError: no running event loop on the Rust thread and the relay dropped the socket (relay not connected). nostr-sdk 0.45 documents this: call uniffi_set_event_loop with the running loop before connect, and clear it afterward. The key is still never printed. Compliance: NIST CSF 2.0 PR.DS-1, PR.AA-05. Verification: python scripts/publish_to_buzz.py --self-check signs a kind 22242 AUTH event from a worker thread against wss://relay.example.com/ and does not open a socket.

  • scripts/publish_to_buzz.py — NIP-42 AUTH when the relay requires it (2026-10-01). The monthly cost run wrote the report, then the relay answered auth-required: not authenticated and the job exited 1. The client now signs that challenge with the same key that signs the note. The key is still never printed. Compliance: NIST CSF 2.0 PR.DS-1, PR.AA-05. Verification: python scripts/publish_to_buzz.py --self-check builds the authenticated client offline and does not connect.

  • scripts/publish_to_buzz.py — channel id is a public identifier (2026-10-01). The monthly cost workflow wrote the report, then exited 1 because BUZZ_CHANNEL_ID was set and was not a 64-character hex event id. A 64-hex id is still accepted, and so is a short public channel id (channel-1, a UUID, note1…). nsec and dop_v1 values are still refused, and the value is never printed. Compliance: NIST CSF 2.0 PR.DS-1. Verification: python scripts/publish_to_buzz.py --self-check.

  • scripts/do-cost-reporter and scripts/do-backup-monitor — webhook host check (2026-10-01). WEBHOOK_URL is treated as Discord only when the URL hostname is discord.com, discordapp.com, or a subdomain of those. A host that merely ends with those names, or a query string that contains them, stays on the generic webhook path. Compliance: NIST CSF 2.0 PR.DS-1. Verification: both scripts' --self-check.

  • Billing deploy helpers pinned and offline (2026-09-30). The state-file purge and the state-volume owner check (#274) ran alpine:3.19 (end of life, floating tag); both now run alpine:3.24 pinned by manifest digest, with --network none since they only touch a local volume. The repo's other alpine:3.19 uses (backup/restore scripts) are unchanged here.

  • billing-docker — the provisioning state file had its own volume in name only (2026-09-30). In the live sites/billing compose, ops_state and static_data both resolved to weown_billing_static_data (the volume name: sat under the wrong key), so provisioning_watch's state file lived inside the web-served static root. The live compose now gives ops_state its own volume (weown_billing_ops_state; static files keep theirs). The deploy removes the stale state files from the static volume before up, makes sure the new state volume is owned by the app user, and fails unless /static/provisioning.json answers 404 from the running app. The template gets the same tasks. Follow-up to #269, which fixed the template but left the live render aliased.

  • Every *-docker deploy play: the Infisical auth-file copy no longer prints its content under --diff (2026-09-26). Create container-readable Infisical auth file (ADR-006) copies .infisical-auth.env, which holds the Machine Identity secret, and had neither no_log nor diff: false. render-deploy.sh --app-check runs ansible --check --diff and is the documented first step on boxes holding customer data, so after any rotation it would print the secret to the operator terminal and any log. Added task-level diff: false (the task still runs and still reports changed) in 10 templates and 5 committed site renders (15 files). Found while running --app-check on the sales box before the weown-fleet#92 deploy; that run printed nothing because the file had not changed. Verification: non-jinja files parse as YAML; ansible-playbook --syntax-check passes on the committed site renders.

  • *-docker templates: a look-alike registry host could receive the Minimus pull token (2026-09-30, Copilot on #265). registry_token_key picked MINIMUS_TOKEN when image_registry merely started with reg.mini.dev, so reg.mini.dev.evil.example got the Minimus credential. It now requires the host to be exactly reg.mini.dev (a namespace path is still fine); verified by render, and main fails the same test. Also: the .env*.example files in billing, gitea, keycloak and supabase are now templates (they already began with {{ project_name }} and rendered it literally), so they follow image_registry and registry_token_key; the wordpress README no longer says "Minimus" for a custom registry.

  • anythingllm-docker — dashboard SVG logo, /healthz and embed-filter hardening (2026-09-30). A customer-uploaded SVG logo could run script on the instance origin when /app/brand/logo was opened directly (entity-encoded and namespace-prefixed payloads passed the regex check). /app/brand/* now sends a sandboxing CSP plus nosniff, and the upload check refuses DTDs, &# references, prefixed elements and non-fragment href/url(). /healthz no longer returns raw socket errors to unauthenticated callers. embed-filter fixes the upstream host from ALLM_URL (a request target could name another host) and runs as uid 1000 on a read-only root filesystem with no capabilities and no-new-privileges. Template and sites/beta-weown-chat kept identical; details in anythingllm-docker/template/CHANGELOG.md.jinja.

  • anythingllm-docker — OpenBao deploy review sweep: the store's private address left the public render; NUL key names, a leaked login token and the wrap token on argv closed (2026-09-30). Fixes open Copilot threads on #198–#262, in the template and the committed sites/beta-weown-chat render. Store address: the render committed the store's VPC address in four places. ansible/deploy.yml now takes bao_addr from the copier answer when given (weown-fleet's ephemeral renders, unchanged), else from BAO_ADDR_INSTANCE at deploy time (or -e bao_addr=), and refuses to run without an https://host[:port] value. entrypoint-bao.sh holds a placeholder that the deploy fills in as it uploads the file, so the address exists only on the box, and --check --diff does not print it (diff: false on that task). Deploy change: every deploy of beta-weown-chat from this repo now needs BAO_ADDR_INSTANCE. History still holds the old address. Host key gate: a key NAME containing NUL passed the #262 name check as two names and shifted the export loop (dummy JSON exported COMPOSE_PROJECT_NAME=evil); NUL is now refused in names as well as values. The container gate already refused it. Tokens: the template's registry-token fetch now revokes its AppRole login token on every exit path; the wrap token reaches the host on stdin under no_log, not via environment: (process list, -vvv). Also: a failed docker restart of a consumer fails the deploy; store keys named like entrypoint helpers are no longer unset; embed-filter runs as uid 1000 with no capabilities; bootstrap-product.sh keeps the admin JWT and API key off curl's argv; devsec.hardening pinned to 10.6.0; copier validators for the bao_* answers; entrypoint-bao.sh rendered only for OpenBao and deploy.sh detects the backend from compose; site.sh smoke-test sets its repo root; README, cloud-init and comment corrections. CI's render check adds a render with no bao_addr (the committed-site shape). Compliance: NIST CSF 2.0 PR.DS-1, PR.AA-05, PR.PS-01; CIS v8 3.3, 4.1, 16.1. Verification (main as the negative control each time): the real playbook tasks run on localhost under bash 5 with stubbed bao/docker, the real entrypoint in a musl node:22-alpine container, and the real curl lines against a local server. NUL key: main ran compose with COMPOSE_PROJECT_NAME=evil, the branch refuses and compose never runs, a benign document still deploys. Login token: main 0 revokes, the branch 1 on success and on a failed read. Wrap token: main put it on the process list and in -vvv, the branch in neither, and the token still unwraps. Failed restart: main exited 0, the branch fails with the message. Address: missing or malformed value refused before any change; the uploaded entrypoint differs from the committed file on exactly the one line. Helper-named keys: main dropped 3 of 4, the branch keeps all. embed-filter: uid 1000, CapEff 0, /healthz 200. Bearer: main 2 tokens on argv, the branch 0, and the server still authenticates. Plus ansible-playbook --syntax-check on all render shapes, bash -n/dash -n, shellcheck (no new warnings), and a YAML/ASCII check of both cloud-init renders.

  • keycloak-docker/sites/sso — admin password, SendGrid key and temp passwords moved off argv (2026-09-30). store-sendgrid-key.sh's --file=/dev/stdin path never worked (the CLI rejects --file together with a positional name), so every run silently fell back to NAME=value on argv. It now writes a YAML --file through a FIFO (jq --rawfile from stdin; the CLI keeps a YAML string exactly and expands @file only for a user login), and there is no argv fallback. kc-configure-smtp.sh, kc-set-login-theme.sh, kc-set-gitea-logout-redirect.sh, kc-set-temp-password.sh and create-weown-user.sh put the Keycloak admin password on docker compose's argv (-e KC_PASS=<value>) and on kcadm's (--password); kcadm now reads KC_CLI_PASSWORD (Keycloak 26+; sso runs 26.7.0), passed into the container by name. The SendGrid key left -s smtpServer.password=<key> for JSON on kcadm's stdin (update -f - --merge), and generated temporary passwords left --new-password for KC_CLI_PASSWORD. Same PR, same class: devbox-docker setup-zed.sh --infisical (template + dev-weown-devbox render) stored the member's OpenRouter key as OPENROUTER_API_KEY=@file/OPENAI_API_KEY=@file, which under INFISICAL_TOKEN saves the literal path and still reports success. It now uses one YAML document with both names through --file on a FIFO, so the key is exact in both auth modes and the transfer adds no copy on argv or disk. The simple-path dotfile that --infisical also writes is unchanged. A signal cleanup means a Ctrl-C mid-store leaves no blocked writer or temp dir. Verification: the real scripts driven through stub ssh/docker/kcadm/infisical (the docker stub copies bare -e VAR from its environment the way compose does), with every argv logged: 30/30, and every secret-on-argv check fails on main. The test also caught an apostrophe in a new comment inside a bash -c '...' block that truncated the remote script; it was fixed before commit.

  • npm Dependabot alerts cleared: 12 alerts, 0 left (2026-09-30). landing-purchase: astro 7.2.0 → 7.3.5 (the critical one) and five transitive packages. n8n/docker/custom: undici 7.30.0. Lockfiles were regenerated with --package-lock-only --ignore-scripts, and both were verified in a throwaway node:22-alpine container, so no package code ran on the operator machine.

  • Operator scripts — secrets moved off argv; a bootstrapped embed was open to every site (2026-09-30). Found in the Copilot review sweep of merged PRs; each fix has a stub-CLI test that logs every argv, run against main as the negative control.

    • anythingllm-docker/.../bootstrap-product.sh (template + beta-weown-chat render): the JWT and the Developer API key were on curl's argv; now a curl config on a pipe (-K <(...)). The Infisical backend put every generated password on infisical secrets set KEY=value; it now passes --file pointing at a FIFO named push.yaml in a private 0700 dir, fed by jq --rawfile v /dev/stdin, so the value is on no argv and never written to disk. Why --file and not KEY=@file: the CLI expands @file only for a user login, and under a machine-identity token it stores the literal path. Why YAML: the CLI parses .yaml/.yml as YAML and keeps the string exactly, while any other name is parsed as dotenv, which trims whitespace, strips quotes and splits on newlines. allowlist_domains was sent as a JSON array, which AnythingLLM stores as NULL (allow every site); it is now a comma-separated string. Each entry is normalised to an origin (https://host[:port]) exactly as the dashboard's normOrigin() does, and the instance's own origin is included. An unusable entry stops the run before anything is sent, including a URL with userinfo (https://trusted.example@evil.example is host evil.example). The dashboard's normOrigin() refuses userinfo too, so a later save can't reintroduce it. Rejected entries are reported by position only, in the script's message and in the dashboard's HTTP error, because an entry may carry a password (https://user:pw@host). A blank answer now means only the instance's own origin: before, it meant no allowlist, which lets every website use the embed and spend the customer's LLM budget. The same list is stored as EMBED_ALLOWLIST_DOMAINS, the dashboard's seed. On a re-run with an existing embed, the list is owned by the dashboard (which keeps its own copy), so the script no longer writes it to AnythingLLM. It prints the exact dashboard action instead, repeated at the end of the run. A failed store write now stops the run (it used to print one line and end in "Done"), and the generated admin password is stored before the admin is created, so a failed write can never lose it.
    • dev/up.sh: the admin password (jq and curl argv) and the Bearer header (curl argv) now go env → stdin and curl -K -.
    • scripts/offboard-account.sh: the Keycloak admin password was on kcadm.sh --password (docker compose and java argv); kcadm now reads KC_CLI_PASSWORD (Keycloak 26+) passed by name with -e. The GITEA_MODE=api admin password no longer touches disk (it was a temp curl config); curl reads it from a pipe per call, with \ and " escaped for curl's config syntax. The one-shot write:admin token (both Gitea paths) now reaches curl on stdin (-K -), not -H on argv.
    • Found in passing, same class: scripts/provision-openrouter-key.sh sent the OpenRouter provisioning key as -H "Authorization: Bearer …", and anythingllm-docker/scripts/allm-prompt-sync.sh did the same with the admin API key, while both headers promised "never on argv". Both now use -K <(...). provision-openrouter-key.sh also stored the key with OPENROUTER_API_KEY=@file, which under INFISICAL_TOKEN saves the literal path, and its read-back only checked for a non-empty value, so it reported success. It now uses a YAML --file on a FIFO, and the read-back compares a sha256 of the stored value with the minted key. The key no longer touches disk at all: the old mode-0600 temp copy is gone, and the private dir and FIFO are set up before minting, so a setup failure stops the run before a key exists.
    • anythingllm-docker/scripts/allm-user-rename.sh and allm-admin-account.sh validate every value they interpolate into a remote root shell (ssh target as user@host, container name, usernames), so a quote can no longer run commands on the droplet. The ssh target must start with a letter or digit: -Fconfig@host matched user@host, but ssh reads it as the option -F. The same check was fixed in allm-set-role.sh, scripts/offboard-account.sh (GITEA_SSH_HOST) and devbox-docker offboard-user.sh. allm-user-rename.sh also refuses an email-shaped new name (FedArc D35).
    • anythingllm-docker embed-filter (template + beta-weown-chat), the one public-facing container, now runs as uid 1000 with cap_drop: [ALL] and no-new-privileges. Verified by starting it from the rendered compose: uid 1000, CapEff 0, NoNewPrivs 1, healthcheck healthy (main: uid 0).
    • .github/workflows/validation.yml: the workflow token is contents: read; only the security job adds security-events: write for the SARIF upload. pull-requests: write is gone (no job used it).

  • anythingllm-docker — an OpenBao key NAME could run commands, in the container and on the droplet host (2026-09-29). Both OpenBao paths turned every key at an instance's path into shell text and ran it, and quoted only the value. A key name containing shell syntax therefore ran as a command: in the container entrypoint (entrypoint-bao.sh, POSIX sh, before the app starts, with every secret already in its environment), and on the host during ansible deploy (the two openbao docker compose up tasks, as the deploy user). Anyone who can write a key at the instance path could do it: operator tiers, store admins and the provisioner. It's the same bug as WeOwnCloud/openbao#90, which fixed the shared seam. Measured on a local OpenBao 2.6.1 dev server against the committed beta-weown-chat render: the old code executed the key name and booted normally both times, exported NODE_OPTIONS into the node app, and passed a store-supplied DOCKER_HOST straight to docker compose, which would send the whole deployment, secrets included, to another daemon. Now: every key name must be a plain identifier and must not be a loader/shell/node variable (LD_*, DYLD_*, BASH_ENV, ENV, BASH_FUNC_*, SHELLOPTS, BASHOPTS, PS4, PROMPT_COMMAND, IFS, PATH, NODE_OPTIONS); on the host, DOCKER_* and COMPOSE_* are refused as well, because compose runs next. One bad name refuses the start or deploy. It is never skipped, and a malformed name is never printed, since it may be a value pasted into the wrong field. Values containing NUL are refused. The host exports through export "$k=$v" and never parses store data as shell. The container is POSIX sh with no read -d '' and no jq, so node checks every name and single-quotes every value before the (now inert) text is run. Also closed, same class: the approle secret-id and the single-use wrap token were on argv at every login and unwrap (4 host sites, 2 container sites), where ps and /proc/*/cmdline show them to any process. Logins now send the JSON body on stdin (jq --rawfile on the host, node from the environment in the container), and unwrap reads the token from BAO_TOKEN. Every login token, including the host's validity-probe tokens, is now revoked with bao token revoke -self instead of being left live until its TTL. Verification (old code as the positive control each time): container and host each run with a benign doc (quotes, $(…) and backticks in values arrive intact and are not executed), a hostile key name (old: executed, rc 0; new: refused, nothing ran), NODE_OPTIONS and DOCKER_HOST (old: exported; new: refused), and a NUL value (new: refused); a bao argv logger shows the secret-id and wrap token on argv in the old code and never in the new; first-boot unwrap, sid_valid verdicts (real/wrong/empty) and revocation checked. Template and render stay identical outside the site's three values; sh -n/dash -n/shellcheck -s sh, ansible-playbook --syntax-check, and a Jinja parse with copier's jinja2 3.1.6 all pass. Deploy consequence: every openbao instance keeps the vulnerable code until it is redeployed. weown-fleet renders tenants from this template at deploy time, so this covers three live instances (beta-weown-chat, chattest2, weownacademy, per tenants.yaml 2026-09-29), not only the committed render. operator.default_secret_backend is openbao, so every new tenant uses this code too. Reported by openbao-claude.

  • anythingllm-docker — the public chat widget resumed the previous visitor's conversation forever (2026-09-26) — the stock AnythingLLM widget stores its session id in localStorage (allm_<embedId>_session_id) and resumes it with no expiry and no setting to change it. On a CPA's public site, the next person on a shared or family computer saw the last person's tax questions. Tyler reported it on the OCPA bot on 2026-09-25: "loaded this website new … the old one was still there". Every instance already serves the widget script itself, so embed-filter now serves /embed/anythingllm-chat-widget.min.js with a small ES5 session-policy shim prepended. A new tab, a reload, arriving from outside the site (a typed URL, a bookmark, a search result, another site's link, even in the same tab), or 30 idle minutes starts a fresh chat. Moving between pages of the same site, and back/forward, keep the conversation. (A tab duplicated from the current one keeps it within the idle window: same person, minutes apart. A site that sends no referrer resets on every page, toward privacy. Known limit: the widget keeps one session id per origin, so two tabs open at once share it; true per-tab isolation needs the widget to support it.) Per embed it can be changed with data-weown-session-ttl-minutes. The shim is fully guarded (storage that throws never blocks the widget load). The shimmed script carries its own ETag and Cache-Control: no-cache, and embed-filter answers revalidation itself. A copy validated against upstream's old ETag gets the new bytes; one holding ours gets a 304. One shimmed copy is cached keyed by upstream's ETag, and upstream is asked conditionally, so AnythingLLM sends the full ~650 KB widget once, not once per page load, and a widget upgrade replaces the copy. No host-page change is needed (ocpa.group, demo.weown.dev). The Caddy route sits above the catch-all in the template and in the re-synced beta-weown-chat render; the three other sites run no embed-filter and are deliberately untouched. Compliance: NIST CSF 2.0 PR.DS-1, PR.DS-5; CIS v8 3.3. Verification: embed-filter/test.js, 27 assertion groups: the shim's policy is run in a sandbox (moving between pages kept; reload dropped; back/forward kept; new tab dropped; arriving from another site, a typed URL, or a look-alike host dropped; duplicated tab kept within TTL; no Performance API still applies the storage rules; idle > TTL dropped; TTL override; garbage TTL falls back to 30; no embed id is a no-op; throwing storage doesn't escape), plus an end-to-end HTTP test through the real server against a chunked upstream. That test caught a Content-Length + Transfer-Encoding conflict before it shipped. The template Caddyfile, rendered both ways, and the beta Caddyfile pass caddy validate/caddy adapt in caddy:2. weown-fleet#92.

  • .gitignore — local DigitalOcean audit exports are ignored (2026-09-24) — droplet inventory and cleanup-audit markdown/PDF files hold live addresses, account emails, and SSH fingerprints. They are local working copies and must not land on this public repo. Compliance: NIST CSF 2.0 PR.DS-1. Verification: git check-ignore on the six export filenames.

  • anythingllm-docker — the customer dashboard's Private/Public toggle was HARD-DELETING library documents (2026-09-03) — turning a document's last scope off computed that it would be left embedded in neither workspace and then called DELETE /api/v1/system/remove-documents to avoid an "invisible orphan". The result was silent customer data loss: operators watched a library shrink 6 to 5, and a customer's uploaded file disappeared when they clicked the toggle. The purge is now removed from the scope path entirely — a scope toggle only adds or removes embeddings, and the explicit POST /api/documents/delete (behind a confirm, a path check and a lock 409) is the sole destructive call site. Documents left in neither scope stay in the system store and remain listed with a Not in use badge, so the state the purge existed to prevent is now visible instead of destroyed. Also hardened: isValidDocpath (rejects leading /, .., NUL, backslash, over-512-char paths) now guards the lock, scope, delete and content endpoints, three of which previously had none. Forward-only — documents already purged by the old code are gone and need a restore from backup. The rendered site under anythingllm-docker/sites/beta-weown-chat/dashboard/ is re-synced in the same change, because ansible ships the site's copy, not the template's, so a template-only fix would have left the deleting code running in production. Compliance: NIST CSF 2.0 PR.DS-1, PR.AC-4; CIS v8 3.11. Verification: every remove-documents call site traced on the merged tree (one comment, one explicit-delete call); node --check clean on template and render; GET /api/v1/documents response shape verified against a live AnythingLLM (viewLocalFiles returns {name: <on-disk filename>, type: "file"}), confirming the library listing's path reconstruction is correct rather than producing phantom duplicate rows. PRs #222, #226.

  • scripts/tag-droplet.sh — commit-tagging can no longer report success while the tag never lands (2026-08-30) — the DO API only attaches existing tag resources, so every fresh commit-<sha> tag failed the droplet-action; a blanket || true (added to tolerate "already tagged") swallowed the failure, and the deploy task reported ok with no tag on the droplet (doctl-verified 2026-08-26 after the #188 deploy: no commit-0cbcdab). The #185 class one level deeper: silent miss → silent false success. Now the helper creates the tag resource first, then verifies presence (add) / absence (remove) via doctl after every mutation and exits non-zero on a miss — which the deploy playbook's existing rc-gated surface task then prints instead of staying quiet. Per governance assert the step's REAL contract, not a proxy: "the action exited 0" was the proxy; "the tag is on the droplet" is the contract. Compliance: NIST CM-8, CIS v8 1.2. Verification: bash -n clean.

  • twilio-bridge/ removed (2026-08-30) — the Twilio Voice dial-pad spike was dormant since 2026-01-06, deployed nowhere, and referenced only by CODEOWNERS, the README index, and copilot-instructions — while carrying all 35 remaining Dependabot alerts (13 high, 11 axios) on main. Deleting it takes the repo to 0 open alerts at zero ongoing cost; git history is the undo if the spike is ever revived. References removed from CODEOWNERS, README, and copilot-instructions.

  • anythingllm-docker — customer data restored to the encrypted volume; deploy now refuses to strand it (2026-07-31) — a template change had replaced the named volume anythingllm_storage:/app/server/storage with a host bind ./storage:/app/server/storage and deleted the volume declaration. Two consequences, both silent: (1) redeploying an existing site starts the app on an EMPTY directory and orphans the old volume — the app comes up healthy with zero workspaces (this is what took chat.weown.dev's workspaces, threads and API-key registry offline on 2026-07-31; 25M of data survived only because nothing had run down -v), and (2) the bind path /opt/... lives on the droplet's local disk, while Docker's data-root sits on the attached DO block volume that is platform-encrypted at rest — so the change also moved customer documents and vectors off encrypted storage, undoing the at-rest control from the July hardening work. Fixed by restoring the named volume for app data while keeping the declarative MCP pack as nested binds (./storage/mcp, ./storage/plugins) inside the volume mount, so the git-managed tool pack stays IaC. Added a pre-up guard in ansible/deploy.yml: if this project's storage volume exists and is non-empty but the compose about to run does not mount it, the deploy FAILS loudly instead of silently starting on empty storage. Compliance: NIST CSF 2.0 PR.DS-1; CIS v8 3.11. Verification: template re-rendered, compose + playbook YAML parse, guard renders with the correct per-project volume name.

  • anythingllm-docker — dashboard chat-flow hardening on top of the #133 redesign (2026-07-29) — review of the redesign confirmed the four security invariants hold (uniform safe-DOM rendering, CSRF header on every non-GET, public/private boundary, non-emptyable allowlist); this follow-up fixes the correctness gaps found: (1) /api/chat upstream failures returned HTTP 200 with the raw upstream error as text, rendering as a normal bot bubble — now a 502 {error}; the UI shows the error strip, rolls the unsent user message back, and restores it into the composer; (2) a reply landing after the user switched threads was pushed into the wrong conversation — a viewSeq guard drops stale async responses (send, thread-create, history-load) for views the user has left; (3) concurrent double-send blocked with an in-flight gate that also protects rewind and the composer from truncating/clearing for a send that would no-op; (4) thread delete and history-load now surface server errors instead of pretending success. Verification: node --check on server + extracted inline script; live smoke against a stub AnythingLLM (login, whoami, rename incl. CSRF/blank/81-char cases, chat happy + 502 paths, draft-to-thread creation with derived title) — all pass in a real browser session.

  • anythingllm-docker — backup.sh gets the same off-box-upload contract as keycloak/gitea (#134), plus the ai.weown.agency bucket mismatch fix (2026-07-29) — same defect family: missing SPACES_* creds with REMOTE_STORAGE=do-spaces produced a WARNING and exit 0, and — worse here — GFS retention then still pruned local copies, so a silently-unuploaded site also aged out its only backups. Now: missing creds abort with exit 1 BEFORE retention; after aws s3 cp the exact object key must exist remotely with the exact local byte size before success (asserted BEFORE the local encrypted .gpg copy is deleted). Applied to the template and all three rendered sites. Also fixed: sites/ai.weown.agency backup.sh AND restore.sh both pointed at the deprecated weown-backups bucket while restore's own header documented weown-prod-backups — both now use the canonical weown-prod-backups (no stranded objects: no backup was ever observed landing in the old bucket). Verification: bash -n clean on all four scripts + the de-jinja'd template.

  • keycloak-docker + gitea-docker — backup.sh can no longer report success while silently skipping the off-box upload (2026-07-29) — root cause of the "live, backups unverified" finding: when REMOTE_STORAGE=do-spaces but SPACES_ACCESS_KEY/SPACES_SECRET_KEY were absent from the injected env, the script printed a WARNING, skipped the aws s3 cp, ran retention, printed === BACKUP FINISHED === and exited 0 — so the daily cron wrote local-only backups for weeks while reporting green. Two contract assertions added (templates + both rendered sites sites/git, sites/sso): (1) missing Spaces credentials with do-spaces configured is now a hard exit 1; (2) after the upload, the script aws s3 lses the exact object key and fails unless it exists with the exact local byte size — cp exiting 0 is not proof of an object. Per governance assert the step's REAL contract, not a proxy: "the script ran" was the proxy; "a non-zero object exists in the bucket" is the contract. The dead-man alert (absence of a fresh object should page) remains open on the board. Compliance: NIST CSF 2.0 PR.DS-1, RC.RP; CIS v8 11.2/11.4. Verification: bash -n clean on both rendered scripts and both de-jinja'd templates.

  • anythingllm-docker — embed widget can no longer be left callable from any website (2026-07-22) — the public chat widget is the customer's brand and their capped LLM budget, and AnythingLLM treats an embed with no allowlist as allow every origin (EmbedConfig.parseAllowedHosts returns null → canRespond skips the origin check). Two changes close that: (1) EMBED_REQUIRE_ALLOWLIST: "true" in compose.prod.yaml flips "no allowlist" to deny-all, so the failure mode is a visibly dead widget instead of a stranger silently spending the customer's budget (upstream #5759; effective on images built after 2026-06-17, ignored harmlessly on older ones); (2) the customer dashboard gains an Authorized websites panel (GET/POST /app/api/embed-domains) so adding a second site never needs an operator — entries are normalised to scheme://host[:port] (lower-cased, path/trailing-slash stripped, wildcards refused, since ALLM compares the browser Origin byte-for-byte), the instance's own origin is always retained so the list can never be emptied, and the list is persisted to a new dashboard_state volume. The allowlist that provisioning seeds arrives as EMBED_ALLOWLIST_DOMAINS. Compliance: NIST CSF 2.0 PR.AC-4, PR.DS-2; CIS v8 4.1. Verification: 16-check live test of the dashboard (public health probe, auth gate, CSRF gate, wildcard rejection, case/path normalisation, self-origin retention, empty-submission floor, state-file persistence, comma-separated-string payload shape, snippet contents) all pass against a running server + stub AnythingLLM; ALLM's own validatedCreationData was executed directly to confirm an array-valued allowlist_domains stores NULL (allow-all) while the comma-separated string stores the array.

  • anythingllm-docker — the embed snippet now carries the AI disclaimer (2026-07-22) — /app/api/snippet emits data-greeting ("I'm an AI assistant… general information only… don't share private information in this chat") plus data-no-sponsor, so the PRD's non-negotiable disclosure is stated the moment the widget opens rather than depending on the model remembering to say it.

  • anythingllm-docker — per-instance Infisical secret paths (infisical_secret_path) (2026-07-19) — supports the WeOwn-Chat initiative secret model: ONE Infisical project per initiative, one folder per instance (/sites/<slug>), each droplet Machine Identity restricted to its own folder via a path-scoped project role — the client-isolation boundary (D-ISO amendment) without a project per customer. New copier var infisical_secret_path (default / — fully backward-compatible; existing dedicated-project sites render identically) threads through every secret consumer: container entrypoint (entrypoint-infisical.sh infisical run --path), ansible/deploy.yml (new infisical_path var from INFISICAL_PATH env; all five infisical run/secrets get calls), backup.sh/restore.sh (local + remote-ssh modes), bootstrap-product.sh, deploy.sh env pass-through, and scripts/provision-openrouter-key.sh (new --path/--operator-path flags — the OpenRouter PROVISIONING key can now live at the project root where instance-scoped MIs cannot read it, closing the POC caveat that a compromised droplet could mint keys). Verification: bash -n clean on the shell scripts; rendered-template validation via the fleet dry run (weown-fleet render-deploy.sh).

  • anythingllm-docker — CIS host hardening play + encrypted-at-rest data volume (2026-07-15) — Phase A of porting the SOP-INFRA-012 baseline to customer instances. Hardening: new template/ansible/harden.yml applies DevSec os_hardening + ssh_hardening (≈ CIS Ubuntu L1 — kernel/sysctl, auth policy, SSH key-only + strong KEX/ciphers) with the Docker-safe ip_forward=1 task-var override (defaults-level overrides silently lose to inner-role defaults — the 2026-07-03 no-egress landmine), then measures with Lynis and verifies Docker survived; ansible/requirements.yml pins the devsec.hardening collection. Phase A deliberately keeps root key-only SSH (all fleet tooling SSHes as root@); Phase B (ops-user model, root-off, per SOP-INFRA-012) is gated on fleet-tooling migration and tracked for pre-first-customer. Data at rest: new digitalocean_volume (default 50 GB, prevent_destroy, data_volume_size_gb=0 to disable) — DO block storage is platform-encrypted at rest, droplet local disk is not; cloud-init setup-data-volume.sh waits out the attach race, formats/mounts, points the Docker data-root (chats/vectors/uploads) and the backups dir at it before Docker installs. Compliance: FTC Safeguards/GLBA at-rest expectation; NIST CSF 2.0 PR.DS-1, PR.PS; CIS v8 4.1/3.11. Verification: rendered harden.yml passes ansible-playbook --syntax-check; setup-data-volume.sh bash -n clean; rendered main.tf parses (tofu fmt).

  • anythingllm-docker — client-side encrypted backups (GPG) + ZDR LLM routing posture (2026-07-15) — closes two compliance gaps for customer instances holding financial/personal records (GLBA-service-provider posture). Backups: backup.sh now encrypts the tarball with OpenPGP/GPG (per-customer ed25519/cv25519 public key, BACKUP_GPG_PUBLIC_KEY injected via infisical run; keyring-free --recipient-file + ephemeral GNUPGHOME) before the Spaces upload — the object at rest is ciphertext to DigitalOcean; the private key lives ONLY in the operator store (BACKUP_GPG_PRIVATE_KEY_<project>, auto-generated by deploy-new-site.sh in an ephemeral keyring). Fail-loud when encryption is configured but gpg is missing; loud warning on unencrypted uploads. restore.sh fetches .tar.gz.gpg, pulls the private key in-process from operator-tools via the logged-in infisical CLI (paste fallback), stages it into droplet tmpfs (0600, ephemeral GNUPGHOME, shredded on exit via trap), decrypts, and proceeds; legacy plaintext objects still restore. GPG chosen over age for auditor familiarity (owner call); gnupg was already in cloud-init packages. LLM routing: per-customer OpenRouter keys operate under the account-level Zero-Data-Retention guardrail (ZDR-only endpoints — no provider retention or training; OpenRouter itself retains nothing); provision-openrouter-key.sh prints the posture requirement, and the compliance facts table in docs/CUSTOMER_INSTANCE_PROVISIONING.md documents both. Compliance: FTC Safeguards (GLBA) encryption-at-rest expectation; NIST CSF 2.0 PR.DS-1/PR.DS-2; CIS v8 3.11; supports IRC §7216 posture for tax data. Verification: 6-check GPG round-trip (encrypt → ciphertext-opaque → decrypt → byte-identical → wrong-key-rejected) passes with the exact script logic; rendered-shell bash -n clean.

  • M2 fleet-wide — Infisical Machine-Identity secrets & DO Spaces creds moved off command-line argv (2026-07-14) — closes the AGENTS.md S2 finding that infisical login --client-secret="$INFISICAL_CLIENT_SECRET" and tofu init -backend-config="secret_key=…" exposed secrets on process argv (readable via ps / /proc/<pid>/cmdline on the droplet). Infisical login now uses env-var auth — export INFISICAL_UNIVERSAL_AUTH_CLIENT_ID/_SECRET before a flagless infisical login --method=universal-auth (the CLI reads them natively), matching the pattern openclaw's entrypoint already used. Tofu/OpenTofu now exports AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY (the S3 backend reads them from env) and drops the two -backend-config access/secret lines; sse_customer_key stays on -backend-config (no env equivalent). Applied across every *-docker template and rendered site (anythingllm, keycloak, owncloud, wordpress, searxng, signoz, sandbox, openclaw, devbox) plus otel-agent/ and the scripts/*otel* helpers, and the doc examples that showed the old form. Only --plain/--silent and non-secret flags remain on argv. Compliance: NIST CSF 2.0 PR.DS-1/PR.AC-4; CIS v8 3.11. Verification: bash -n + shellcheck -S error clean on all changed scripts; rendered ansible YAML parses; 0 remaining --client-secret=/-backend-config="secret_key=" in tracked code. Note: needs a live deploy test (copier render → ansible/tofu path) before promotion — the human pre-merge gate.

Added

  • scripts/registry-rollover.py — move a committed site render off reg.mini.dev (2026-10-02, WeOwnDev/weown-fleet#112). Minimus shuts down on 2026-10-22. The templates take the registry as one copier answer (#265); the committed renders under *-docker/sites/ predate it. The script applies the template's own rule to a render: image refs move to <registry>/<path>, docker login gets the registry host and --username, and MINIMUS_TOKEN becomes REGISTRY_TOKEN. It prints a diff by default (--write applies it). Anything it cannot rewrite safely, such as a token on a command line or a renamed variable, is listed as MANUAL with exit 3. Verification: python3 scripts/test_registry_rollover.py renders all 11 registry-bearing templates both ways and requires the script's output to match the template on all 243 lines it changes; 4 mutations of the script each fail it.

  • scripts/publish_to_buzz.py — Kind 1 notes on a configured Buzz relay (2026-10-01) — the cost report and the stale-backup audit sign their Markdown with nostr-sdk and publish it only when NOSTR_PRIVATE_KEY is set and NOSTR_RELAY_URL is an explicit wss:// URL. There is no default relay. The key accepts nsec or hex and is never printed; when it is unset the scripts still write the local report and exit normally. BUZZ_CHANNEL_ID must be a 64-character hex event id; it posts a NIP-28 kind 42 (e root tag plus h). (Superseded under Fixed: a short public channel id is also accepted, and a channel note is Kind 1 plus Kind 9.) With no channel id the note stays a kind 1 root note. Scheduled workflows: do_monthly_cost_report.yml (01 08:00 UTC) and do_backup_stale_alert.yml (Mondays 08:00 UTC). Actions logs discard report stdout because this repository is public. Compliance: NIST CSF 2.0 PR.DS-1 (key and relay stay in Actions secrets), PR.IP-4 (backup alert). Verification: python scripts/publish_to_buzz.py --self-check plus both reporters' --self-check (offline sign and skip path; no relay connection, no DigitalOcean token).

  • scripts/do-backup-monitor — read-only stale backup audit (2026-10-01) — check_stale_backups.py issues GET requests only (/v2/droplets, /v2/volumes, /v2/snapshots) and writes BACKUP_STALE_REPORT.md as mode 0600. A Droplet or Volume is PASS when features contains backups or a linked snapshot is under 7 days old; otherwise the row is STALE_WARNING or CRITICAL_NO_BACKUP. The address column is a public IPv4 or the resource id. Team labels may contain spaces. A failed snapshot list is AUDIT_INCOMPLETE so a missing response is not reported as no backup. Tokens come from the process environment, not a .env file. The same Markdown can be posted to the configured relay or WEBHOOK_URL. The file is gitignored because it contains live names and addresses. Compliance: NIST CSF 2.0 PR.IP-4 (backups), PR.DS-1 (local file is owner-only; publish targets are explicit secrets). Verification: python -m py_compile; python check_stale_backups.py --self-check (offline rules; no token, no live API).

  • scripts/do-cost-reporter — read-only DigitalOcean monthly cost report (2026-10-01) — do_cost_reporter.py issues GET requests only against api.digitalocean.com, groups list-price run-rate by project and by tag, and writes DO_MONTHLY_COST_REPORT.md as mode 0600. The token stays in DIGITALOCEAN_TOKEN (optional extra teams name other env vars; the script never prints token values and does not read a .env file). Team labels may contain spaces. The same Markdown can be posted to the configured relay or WEBHOOK_URL. The generated report is gitignored because it contains live resource names. Compliance: NIST CSF 2.0 ID.AM-1 (inventory), PR.DS-1 (local file is owner-only; publish targets are explicit secrets). Verification: python -m py_compile; python do_cost_reporter.py --self-check (offline pricing and grouping; no token, no live API).

  • Changelog entries for tools that shipped without one (2026-09-30): dev/ local stack (dev/up.sh, dev/compose.dev.yaml, dev/README.md — AnythingLLM, dashboard and billing on a laptop, credentials generated into a gitignored 0600 file; PR #251); scripts/offboard-account.sh (one script for Keycloak, Gitea, GitHub orgs and the devbox, disable-over-delete, dry run by default; PR #232/#233); anythingllm-docker/scripts/allm-set-role.sh (role change and suspend/unsuspend in place, refuses to demote or suspend the last admin; PR #235).

  • billing-docker — manage.py stalled_signups, a read-only report of where every sign-up stopped (2026-09-26). Since the no-instance paywall (#252), someone can register and never finish, and nobody could see how many. The command places each Customer created on or after --since (default 2026-09-10, the paywall launch) at the first funnel step it hasn't passed:

    • no_agreement
    • signed_no_checkout
    • checkout_unfinished
    • lapsed
    • paid_no_instance
    • provisioning_stuck
    • instance_inactive
    • live

    "Checkout started" is detected from the Instance row that new_instance creates before redirecting to Stripe, not from the Stripe customer id, which only arrives with checkout.session.completed (a Copilot review finding, fixed with a regression test). An abandoned checkout through the legacy /subscribe/ route leaves no row (that door opens Checkout without an Instance), so it lands in signed_no_checkout: the agreement signature both doors require bounds the gap, and the report prints a note saying so. A signature counts only on the ACTIVE customer template, the same test the checkout gate applies, so publishing a new agreement version moves unsigned customers back to no_agreement.

    It prints a count per step and a total. It prints no personal data: no emails or names; --ids adds Customer primary keys for lookup in the admin. It writes nothing. Not covered: Keycloak registrations that never reached billing have no Customer row, and counting them needs the Keycloak admin API. Added to the committed render and the template (identical). Verification: 6 new tests. Every state lands on its own step; a legacy-route abandon is signed_no_checkout; a signature on a retired agreement version is no_agreement; no email or username appears in the output; --since excludes older rows and --ids is opt-in; a malformed date is refused. The full suite passes, 80/80 in both copies, run against a throwaway Postgres 16. weown-fleet#66.

  • anythingllm-docker — allm-prompt-sync.sh, bringing a workspace system prompt under version control for the first time (2026-09-17) — per docs/design/DESIGN-client-intake-and-agent-workflows.md Phase 1: the agent's entire behaviour (persona, disclaimers, refusals) has lived only in the AnythingLLM runtime DB, with no review, no CI, no rollback and no drift detection, while everything around it (Terraform, compose, the MCP pack, Ansible) is strict IaC. New script (discover / pull / diff / push) closes that gap: discover is a read-only GET that confirms the live system-prompt field name before anything is trusted (AnythingLLM's documented contract uses openAiPrompt, but no AnythingLLM source is vendored in this repo to verify that against, and this instance's actual field has not yet been confirmed); pull bootstraps a committed file from a live workspace; diff detects drift and exits non-zero; push shows the live-vs-committed diff and requires a typed YES before applying. The admin API key is read with read -rs in the operator's own terminal and never crosses an agent's context, argv or a log — per AGENTS.md's secrets-hygiene standard, the bootstrap step is explicitly a human-run action, not one this change performs. sites/dev-weown-anythingllm/prompts/README.md starts the convention (prompts/{public,private}.md per site) on the lowest-risk instance only; no prompt content is committed in this change — both files remain to be bootstrapped by a human holding the live key. Deliberately not wired into deploy.yml.jinja yet: the mechanism needs to prove itself against one live instance before it is extended to ai.weown.agency, beta-weown-chat and s004.ccc.bot, or made part of the deploy path. Verification: bash -n clean; the jq extraction/construction logic (shape check, prompt extraction, missing-field failure path, and JSON round-trip through quotes/newlines/backslashes) exercised against synthetic fixtures — no live instance or secret touched in this change. jq chosen over introducing python3 as a dependency, since no other script in anythingllm-docker/scripts/ uses it, and hand-built JSON string escaping was rejected as unsafe for arbitrary prompt text.

  • docs/design — build plan for the agent review thread's client-intake and workflow changes (2026-09-17) — a stakeholder review of a deployed tax/accounting instance produced ten items spanning four disjoint kinds of work (prompt copy, retrieval behaviour, a new product surface, and a compliance decision), which is how a "ten small changes" estimate goes wrong. docs/design/DESIGN-client-intake-and-agent-workflows.md sequences them into six phases with a traceability table mapping every thread item to a plan item. Deliberately vertical-neutral: the thread's examples are tax-shaped, but the same intake workflow serves advisory, lending and legal, so every vertical-specific string, source and document type is per-instance config and a tax-shaped conditional in shared code is defined as a defect (constraint C6). Findings that materially change scope: roughly half the requested workflow already exists (the dashboard is already the AOP's authenticated surface with upload, allowlist, library, previews and a private analysis workspace, and the public/private lane split is already two workspaces, not a persona choice); the system prompt is the one artifact in the product with no version control — no review, no rollback, no drift detection — while everything around it is strict IaC, so a phase is added to close that before an automated knowledge-base writer is pointed at it; and OPENROUTER_MODEL_PREF stops being a quality knob and becomes a compliance control, because OpenRouter routes to upstream providers whose retention policies differ per provider and per model, so under a no-retention requirement model choice determines which third party sees a client's documents. Two specified behaviours are changed rather than implemented as asked, with the reasoning recorded: client documents are notified by authenticated link rather than emailed as attachments (an attachment puts the document in inboxes, auto-forwards, phone syncs and at rest on the relay, contradicting the security claim being sold), and the agent detects intent with the model but executes deterministically via a UI affordance rather than emitting URLs as text (a hallucinated near-miss URL is a link to a document drop-box). Records the highest-severity failure mode for the build — client uploads must never reach WS_PUBLIC_SLUG, which grounds the public widget, or one client's document becomes retrievable context for the next anonymous visitor — and carries an agent-mode probe to run first, whose method section notes that a response's text is not evidence of a tool call and must be confirmed in the container logs. No code changes; design only, nothing approved. Verification: markdown only; no secrets, real customer URLs, or identities (booking and portal URLs are placeholders per the public-repo rule).

  • anythingllm-docker — dashboard stops silent duplicate uploads, and the library text preview actually shows text (2026-09-10) — two customer-visible gaps on top of the #223/#224/#226 dashboard work. Duplicate uploads: re-uploading a file whose display name already exists (case-insensitive, ignoring the internal …-<uuid>.json storage suffix) now opens an in-app Keep both / Replace / Cancel modal that names how many rows match; the client sends X-Upload-On-Duplicate: keep|replace and the server enforces it. Replace is upload → verify → delete, never delete-first — the new file is uploaded and embedded before a single old copy is touched, so a failed upload can never leave the customer with nothing; the only pre-upload work is a locked-match check so a doomed replace fails fast. A failure while retiring the old copy is reported as a warning on the 200, not swallowed (replaceWarnings[]) — the old remove-documents call quietly .catch(() => null)'d. The client no longer pre-deletes (that doubled the destructive ALLM calls — Copilot review). Preview content: Mintplex GET /api/v1/document/:docName strips pageContent, so the preview was always empty and its subtitle leaked the AI-Playbook-MEGA.pdf-<uuid>.json storage name. The dashboard now reads the stored JSON from ALLM_DOCUMENTS_PATH (a read-only mount of ALLM's storage/documents on the shared named volume), path-traversal-guarded under that root and byte-capped at 16 MiB before the read enters the heap — a preview is truncated to 100 k chars anyway, so a larger file can only be an OOM lever for an authenticated caller. The subtitle is now the de-suffixed human title only; when there is still no text the client shows a muted note, not a red error, and distinguishes an unconfigured instance ("ask WeOwn support to enable previews") from a still-indexing document. The rendered site under anythingllm-docker/sites/beta-weown-chat/dashboard/ is re-synced byte-for-byte from the template in the same change (ansible ships the render, not the template — a template-only fix would never reach the box), and the render's dashboard service gains the anythingllm_storage:/allm-storage:ro mount plus ALLM_DOCUMENTS_PATH, mirrored into template/docker/compose.prod.yaml.jinja. Rebased onto current main: the four defects closed in #227 (chat body cap derived from the attachment budget, document-level dragover/drop guard, /api/documents/content no longer asserting an unverified location, location-mismatch log recording folders not filenames) stay fixed — main's version was taken at every overlap. Compliance: NIST CSF 2.0 PR.DS-1 (Replace can no longer lose the only copy), PR.AC-4 (lock still gates the destructive path); CIS v8 3.11. Verification: node --check clean on template and render server.js and on the extracted inline <script>; compose.prod.yaml parses; disk-read cap and traversal guard reasoned through; render confirmed byte-identical to the template after the copy. PR #245.

  • anythingllm-docker — dashboard drag-and-drop, per-message attachments, library text preview and Word .docx (2026-09-03) — files can be dropped onto the chat or the document library, attached to individual messages, and previewed as extracted text; .docx is accepted and legacy .doc is refused with a convert message. The upload allowlist is enforced server-side and is now two-sided: GET /api/upload-policy reports the server's effective list, so an instance pinning UPLOAD_ALLOWED_EXT in Infisical can no longer advertise a type the server will reject — operators pinning that variable must add docx. Four defects found in review were fixed in the same change: the chat body cap is now derived from the attachment budget (CHAT_ATTACH_MAX x CHAT_ATTACH_MAX_B64 plus overhead) rather than a fixed 6 MiB that two ordinary PDFs exceeded — which reached customers as "could not reach the server" with the attachments still queued; a document-level dragover/drop guard stops a file dropped outside a bound zone (the document list being the obvious one) from navigating the browser away and discarding an unsent draft; /api/documents/content no longer reports a location it never verified; and the location-mismatch log records folders instead of customer document filenames, which on hosts running the fleet OTel agent leave the box unredacted. Verification: node --check clean on template and render including the extracted inline script; derived cap asserted to exceed a full attachment set. PRs #223, #224, #226.

  • billing-docker — white-label affiliate branding on the signup funnel (2026-08-26) — affiliates sell under their own brand, and the billing funnel is the only WeOwn-controlled surface an end customer sees before paying, so every pre-purchase page now wears the referring affiliate's identity. Affiliate gains four optional fields (display_name, logo_url, primary_color, support_email; migration 0011); a new core.context_processors.branding resolves the brand from ?ref= → session → the signed-in customer's referred_by. That last fallback is load-bearing: subscribe() pops the session key once it is recorded on the Customer, so without it the page reverts to WeOwn at exactly the success/portal step — after the customer has paid. Only active affiliates may brand a page, so an unsigned or revoked affiliate cannot dress the payment funnel in their name. Blank fields fall back per-field, so existing affiliates are unchanged until an operator fills them in. create_affiliate gains --display-name / --logo-url / --primary-color / --support-email and full_cleans, so a bad value is rejected at the operator's terminal rather than reaching the page. Security: primary_color lands in a CSS custom property, where Django's HTML autoescaping gives no protection, and model validators do not run on a bare .save() — so the context processor re-validates the hex colour and drops any non-https:// logo whatever is in the database; tests assert a CSS-breakout string, an http:// logo, a javascript: URL and an HTML display name are all neutralised. White-label is not anonymity: an affiliate-branded page still carries a "Service delivered on the WeOwn platform." line. 14 new tests; suite 59 passing.

  • anythingllm-docker + weown-fleet — the instance now honours subscription_active; a cancelled customer no longer keeps a working instance (2026-08-16) — billing has written this Keycloak user attribute since the entitlement work landed, and nothing read it. Diagnosing the chain end-to-end rather than patching the obvious end found the actual break: a Keycloak user ATTRIBUTE is not present in userinfo unless a protocol mapper puts it there, and the live chat-chattest client carried exactly one mapper (groups). So the attribute was invisible to the dashboard even when set — a check bolted onto the dashboard alone would have "enforced" a claim that never arrives. Two halves therefore: weown-fleet kc-provision-tenant.sh now creates a subscription_active claim mapper alongside the groups mapper (idempotent; backfilling an existing tenant is just re-running the script), and the dashboard OIDC callback checks entitlement after the group check, returning 402 with a message that says the documents and workspaces are untouched. Deny only on an explicit false — this asymmetry is deliberate and load-bearing: the claim is absent for every user whose client has no mapper yet and for every user billing has never touched, so failing closed on an absent claim would lock out the entire existing fleet the moment it deploys, turning an entitlement feature into an outage. Stated in the code so nobody assumes more than is true: the gate is exactly as good as billing's writes. The predicate accepts every shape Keycloak can produce — bare string, array (multivalued attribute), and real boolean (JSON-typed mapper) — rather than trusting one. Verification: node --check clean; dashboard/test-entitlement.mjs (zero-dependency, run with node test-entitlement.mjs) exercises 12 claim shapes including absent, null, uppercase FALSE, padded " false ", cleared-to-empty and unexpected values — all pass. ⚠️ Not yet applied to the live tenant client (blocked on SSH-to-identity-box access at the time of writing) and existing sessions survive up to 12h on the HMAC cookie, so revocation is bounded by session lifetime rather than immediate. The warn-then-lock grace period and the droplet-stop are separate owners (transactional email, and the fleet side).

  • billing-docker — refund and chargeback now claw back affiliate commission (2026-08-16) — closes the exposure recorded as accepted in OWNER-GUIDE-stripe-connect-and-splits.md §Refunds and raised directly by Tyler on the 2026-08-11 call: Stripe reverses the platform's share of a refund automatically but leaves every Connect transfer alone, so a $1,000 reversal cost WeOwn the $1,000 plus the ~$250 commission already paid out. New charge.refunded and charge.dispute.created / charge.dispute.funds_withdrawn handlers reverse every paid leg proportionally — refund 30% of a charge and each leg returns 30% of its cut — via stripe_svc.reverse_affiliate_splits and a new SplitReversal audit row per (event, leg). The clawback is target-based, not delta-based: each run computes what the charge's CUMULATIVE refunded amount says should be reversed per leg, subtracts what already was, and moves only the difference. That is what makes it safe against Stripe's at-least-once, sometimes-out-of-order delivery — a replayed webhook computes a delta of zero, and a partial refund followed by a full one totals 100% rather than 130%; a delta-based design over-reverses on any replay. It RETURNS failures and never raises, the same load-bearing shape as pay_affiliate_splits and for the same reason — raising inside the caller's @transaction.atomic would roll back audit rows for reversals that already executed at Stripe, and the retry would reverse the same money twice, which is the mirror image of the 2026-08-04 double payout. SplitPayout gains an indexed charge_id (populated at payout time) because a refund arrives as a CHARGE event and previously had no way to find its legs without re-deriving the invoice. charge.dispute.funds_reinstated deliberately does not auto-repay: re-sending money is a new payment decision, not the undo of a reversal, so it logs loudly and leaves it to an operator — over-reversing is recoverable, under-reversing loses money to a party with no obligation to return it. Verification: 12 new tests (38 total, all passing) — full refund reverses the whole cut; partial refund is proportional; both tiers reverse, not just tier 1 (Tyler's explicit requirement); a replayed event moves money once; successive partial→full refunds total exactly the cut and never more; a failing reversal returns a failure and records it rather than raising; one failing leg does not block the other; a leg that never paid out is skipped; an unknown charge is a no-op; and the webhook paths for refund, dispute and reinstatement each do the right thing. makemigrations --check clean, migrate applies, django check clean. Known gap: rows written before migration 0010 have an empty charge_id, so clawback cannot find them — backfill from Stripe before relying on it for any payout predating this change (in practice none exist yet; the first real charge has not happened).

  • buzz-docker — Block Buzz relay wrapper + PRJ-432.2 deploy docs (2026-08-06) — new buzz-docker/ mirror of Block's production deploy/compose stack (relay + Postgres + Redis + MinIO + Caddy) for buzz.weown.tools, with placeholder .env.example, hardened Caddyfile, host-side secret bootstrap script (scripts/bootstrap-host-env.sh), and runbook docs/PRJ-432.2-buzz-deployment.md. Live deploy uses upstream BUZZ_COMPOSE_TLS=true ./run.sh start (image pull from ghcr.io/block/buzz, not on-droplet build). Secrets stay on the host (0600); public repo docs use <DROPLET_PUBLIC_IP> / example.com placeholders only. Follow-up: require immutable BUZZ_IMAGE digest (reject :main/:latest), pin upstream git SHA in docs, Compose cap_drop/no-new-privileges (Caddy + NET_BIND_SERVICE), bootstrap via local venv.

  • billing-docker — 14-day free trial and a customer agreement gate on checkout (2026-08-11) — the two things blocking WeOwn.Chat's first revenue under the single-agency commercial model (Tyler call, 2026-08-11). Trial: STRIPE_TRIAL_DAYS (config, never a constant — the same image has to sell the $5 drill product and the $1,000 live product, so switching is an env change plus a restart) drives subscription_data.trial_period_days on Checkout, with payment_method_collection: always and trial_settings.end_behavior.missing_payment_method: cancel so a trial can never quietly become free-forever when the card is missing at conversion. New trialing subscription status, entitled like active — a trial with a locked instance is not a trial — plus Subscription.trial_end, handlers for customer.subscription.created / trial_will_end, and a full Stripe→local status map (unpaid and incomplete_expired previously fell through unmapped and silently left entitlement stale). The trial must not pay commission: the zero-amount check in pay_affiliate_splits now runs BEFORE the charge lookup, so Stripe's $0 trial invoice returns no splits and does not spend three API round-trips resolving a charge that cannot exist; it is logged at info, not warning, because it is now the normal path. Customer agreement: affiliates had click-wrap evidence (body SHA-256 + IP + UA + timestamp) and customers had nothing — new_instance went straight to Checkout. ContractTemplate gains a kind (affiliate|customer) with the single-active constraint corrected from GLOBAL to per-kind (publishing a customer agreement would otherwise have deactivated the affiliate one), and CustomerContract mirrors the affiliate signature record. Both routes to Checkout are gated, and the gate fails closed: no published agreement means no sale, because taking money without being able to prove what was accepted is the exposure the feature exists to close. Agreement text is a DB row seeded with a placeholder, so it is replaced in admin without a deploy. Review follow-up (2026-08-11) — five findings on ai#169, two of them real bugs. (1) ContractTemplateAdmin.has_change_permission returned False once a template had any signature, which combined with the one-active-per-kind constraint made publishing a new agreement version impossible after the first signature — v1 could not be deactivated, so v2 could not be activated, and the "a new version re-gates everyone" behaviour was unreachable through admin, the only route the business actually uses (the original tests missed it because they build templates via the ORM, bypassing admin permissions). The row is now changeable with kind/version/body_md frozen instead, and activating a new version retires its predecessor in the same save rather than raising IntegrityError. (2) the signature stored the LEFTMOST X-Forwarded-For hop, which is client-supplied — a signer could forge the very IP their click-wrap attests to. Caddy (reverse_proxy web:8000) sits in front, so plain REMOTE_ADDR is the proxy's constant container address and is no fix; the authoritative value is the RIGHTMOST hop (the one Caddy appends), with the full chain now kept verbatim in a new signer_forwarded_for column so a spoof attempt stays visible. Applied to the affiliate signature path too, which carried the identical flaw. Also: an internal ssh alias removed from the public drill doc, a comment corrected to describe the trial settings actually passed, and an unused import dropped. Verification: 26 tests (core/tests.py) all passing against Django 5.2 — trial kwargs sent/not sent, bad STRIPE_TRIAL_DAYS values degrade to "no trial" rather than crashing at import, $0 trial invoice moves no money and creates no payout rows, first real invoice does pay with the arithmetic checked, trial→active clears trial_end, cancellation de-entitles, checkout blocked unsigned via both routes, 409 with no template, signature hash pins the exact bytes shown, a new agreement version re-gates a previously-signed customer, and both agreement kinds can be active at once; makemigrations --check clean, migrate applies. Both review fixes carry regression tests confirmed to FAIL against the pre-fix code (5 failures) and pass after. Writing the original tests caught a third bug in this change: the trial_end is not None guard skipped the explicit None Stripe sends at conversion, so a paying customer would have kept a stale trial date forever — fixed with a sentinel default.

  • New landing-purchase/ project — Astro Landing/Purchase page, plus a weownchat-design skill capturing the real brand tokens (2026-08-06) — first build-out of PRD roadmap item 2 (the top-of-funnel marketing site; the transactional purchase/billing flow itself already lives separately at billing.weown.dev). Astro 7 (static output) + Tailwind CSS v4 via @tailwindcss/vite; started on Next.js, rewritten on Astro per direct direction to match the framework used elsewhere in the WeOwn ecosystem outside this repo (this repo itself has no other Astro usage to match conventions against — confirmed by a repo-wide search — so the setup is this project's own). npm run build (astro check && astro build) is clean and static, 0 npm audit findings. Design tokens (--bg #0e1726, --accent #00A3FF, system-ui font stack, 8/10/16/999px radius scale) are copied verbatim from the real dashboard (anythingllm-docker/template/dashboard/public/index.html's :root) and the Keycloak weown login theme — not invented separately — and codified as .claude/skills/weownchat-design/ (SKILL.md + references/tokens.md) so future frontend work reuses this source of truth instead of drifting from it (an earlier draft of both the skill and the page shipped an independently-invented light/serif palette before this was caught and corrected). Notable: a "Live preview" section embeds the real, unmodified dashboard HTML in an iframe (landing-purchase/public/_demo/dashboard-preview.html), fed realistic fictional content (Harborview CPA, matching the existing weown-cpa-demo.pages.dev placeholder-brand convention) via a window.fetch mock — no edits to the real template, and the Private/Public toggle drives the actual app's own hash router via vanilla JS. The hero's animated chat preview and the FAQ accordion (native <details name="faq-accordion">, zero JS) are likewise framework-free. Pricing intentionally shows no dollar figure (commercial terms are still draft/unsigned per the PRD). .gitignore narrowed from a blanket .claude/ exclusion to .claude/* + !.claude/skills/, so skills are shareable while local session state stays ignored. Scroll-reveal is hardened, not just ported: content is visible by default and only opts into the hidden-then-reveal transition once the script runs, with a 2.5s timeout that force-reveals anything IntersectionObserver never fires for — added after testing surfaced the observer sometimes not firing at all for an element plainly in the viewport, in the specific automated-browser environment used to test this (cause unconfirmed; the fix holds regardless of cause). Open follow-ups, not yet decided: hosting/deployment target (droplet + Docker Compose, matching every sibling service, vs. a static host — Astro's static output keeps either path open — see landing-purchase/README.md); page copy/IA beyond this first pass still needs a check against the Branding & Marketing guide's guardrails before anything ships as final. Verification: astro check + astro build clean (0 errors/warnings/hints, static generation succeeds); manually checked at desktop/1440px/mobile widths in-browser; FAQ exclusive-accordion and mobile nav toggle behavior confirmed via direct DOM inspection, not just visual screenshots.

  • anythingllm-docker dashboard — lock/unlock guard on shared library documents, plus the correctness gaps Copilot found in review (2026-08-01) — the document library is shared across every conversation in a workspace (Main + all threads all query the same private-workspace embeddings), so deleting a document doesn't just affect the chat you're in — it silently pulls grounding out from under every other conversation too. The existing two-click "Confirm?" on the trash icon wasn't enough of a speed bump for that blast radius. Added a per-document lock toggle in the document library modal (public/index.html): locking a document shows a "Locked" badge next to its name and disables its delete button (with a tooltip explaining why); unlocking restores it. New dashboard-only state (server.js, same STATE_DIR read/write pattern as the embed-domain allowlist) — locked-documents.json + POST /api/documents/lock — since AnythingLLM's API has no concept of a locked document. GET /api/documents now includes locked per document. Enforced server-side too: POST /api/documents/delete 409s on a locked docpath regardless of what the client sends, so a stale UI state or a direct API call can't bypass the lock — the same defense-in-depth posture as the CSRF header and upload allowlist elsewhere in this file. Copilot PR review then caught two related gaps in /api/documents/scope, both fixed: (1) a missing/non-boolean on was coerced to falsy and could silently detach a document nobody asked to detach — now requires an explicit boolean; (2) turning off a document's last remaining scope left it embedded in neither workspace but still sitting in ALLM's system document store — invisible to /api/documents (only shows what's embedded somewhere) and never cleaned up. Now purged there too, same as /api/documents/delete — and a locked document can't have its last scope turned off at all (409), closing the one path that could've bypassed the delete lock via scope toggling instead. Also fixed nameOf()'s unguarded JSON.parse(d.metadata), which could throw on a malformed/legacy metadata blob and 500 the whole /api/documents response. Verification: node --check on server.js + the extracted inline script; live smoke against a stub backend — lock persists across reload, delete button disables/re-enables correctly, direct POST /api/documents/delete on a locked docpath returns 409, unlock restores normal delete behavior. Second review pass (2026-08-05) hardened the remaining edges: docsOf() now treats any /api/v1/workspace/<slug> 200 that doesn't yield a documents array as a failure (fail-loud) rather than an empty list — verified against a live ALLM that an empty workspace still returns documents: [], so the destructive orphan-purge can never fire on a mis-parsed response; locked-documents.json writes are now atomic (temp + rename) so a crash can't silently unlock everything, and a corrupt file is logged loudly rather than hidden; every lock read-modify-write and check-then-purge is serialized through an in-process mutex (no lost-update/TOCTOU on the lock set); readBody caps the request buffer (256 KB); purged documents are dropped from the lock set so it stays bounded; and the scope/delete handlers no longer echo raw upstream ALLM error strings to the client (logged server-side instead).

  • billing-docker: new copier template — the central WeOwn billing service (P0: take payment). Django 5 + Postgres + Caddy on the standard droplet pattern (Infisical ADR-006 entrypoints, proven backup/restore contract). Features: Keycloak OIDC login (same identity as the customer's instance), Stripe Checkout + Billing Portal for the single Private-AI-Instance product, 2-tier affiliate splits paid as Stripe Connect transfers (global defaults + per-affiliate overrides, append-only SplitConfig), click-wrap affiliate agreements (versioned immutable templates, sha256-pinned signature records), idempotent Stripe webhook ledger, subscription_active Keycloak attribute flip for elegant suspend/reactivate (no droplet pausing), and a weown-admin break-glass local superuser (own email, password via Infisical).

  • anythingllm-docker dashboard — customer-visible <think> leak closed + session-expiry auto-recovery (2026-07-31) — reasoning models (DeepSeek/Kimi) emit <think>…</think> blocks that AnythingLLM passes through verbatim; they rendered in customer chat replies (seen live in demo transcripts), narrating the model's internal reading of the customer's documents. Now stripped server-side at every text egress (live /api/chat reply + both history endpoints). Also: an expired session previously surfaced as generic errors on every action; the dashboard now detects 401 and reloads straight to the sign-in page — the exact "please log out and log back in" UX the C-Suite asked for, without the user needing to know it. Verification: node --check + extracted-script parse; strip regex exercised against nested/multiline think blocks.

  • supabase-docker — terraform/ansible/scripts/first-site layers added on top of v0.1 scaffold (2026-07-28) — the layers pending from the Jul 14 v0.1 integration (964d5d2) now land. Layer 1 (DO Spaces remote state): template/terraform/backend.tf.jinja — S3-compatible endpoint, SSE-C encryption, private ACL. Layer 2 (bootstrap-secret rotation): rotate-bootstrap-secret.sh embedded in template/terraform/templates/cloud-init.yaml.jinja — logs in with v1 from terraform state, mints v2 via Infisical Universal Auth API, atomically swaps the auth file, revokes v1, fails closed with logs to /var/log/<project>-rotation.log. Path C (thin cloud-init + ansible): cloud-init handles only first-boot bootstrap (Docker + Infisical CLI + Layer 2 rotation); template/ansible/deploy.yml.jinja owns compose + Caddyfile + backup script + cron + Pop schema migration apply + docker compose up. Ansible roles: template/ansible/roles/{common,docker,supabase}/ tasks authored; the supabase role handles Pop schema migration idempotency (skips when pop schema already present). Pop schema migration: template/docker/migrations/001_pop_schema.sql — pop schema + tenants registry + 6 pop tables + 24 RLS policies + set_tenant_from_jwt() PostgREST hook + pop_admin bypassrls role (structural only). Secret generation helper: template/scripts/generate-secrets.sh.jinja mints all 10 Supabase app secrets including HS256-signed ANON_KEY and SERVICE_ROLE_KEY JWTs (bash + openssl only). First site: sites/supabase-dev/ — supabase-dev.weown.tools deployment rendered from the template as the pilot instance under PRJ-429.16 pgvector migration. Copier template compliance: added template/README.md.jinja + template/CHANGELOG.md.jinja matching the keycloak-docker reference pattern. Compose hardening: Caddy service gains healthcheck (wget spider on port 80) matching the multi-service hardening baseline; depends_on block gets jinja whitespace control for clean per-site rendering. Compliance: NIST CSF 2.0 PR.AC-4 (secret rotation), CIS v8 3.11 (encrypt sensitive data at rest via DO Spaces SSE-C), ISO 27001 A.8.24 (HS256 JWT minting).

  • supabase-docker — pre-merge security/correctness fixes on the layer build-out (2026-07-29) — from the integration review: pop.tenants RLS enabled with a service_role-only policy (the blanket pop-schema grant had made every tenant's api_key/rls_key readable AND writable to anon/authenticated via PostgREST); PGRST_DB_PRE_REQUEST: set_tenant_from_jwt wired (without it app.tenant was never set and every RLS query returned empty); pg_dump --single-transaction removed (invalid flag, ai#135 class); backup.sh gains the ai#134/#136 off-box upload contract (exit 1 before retention, object asserted by byte size); DASHBOARD_PASSWORD_BCRYPT minted by generate-secrets (stdin, never argv); restore.sh moves the Infisical client secret off argv + timeout 30 on headless login + stops meta during restore; GoTrue deploy probe fails the play on non-200; rendered site backup bucket → weown-prod-backups; copier MI vars secret: true.

  • anythingllm-docker — pgvector as a vector_db option (2026-07-29) — adds pgvector alongside the default embedded LanceDB. New vector_db=pgvector copier choice + conditional fail-loud PGVECTOR_CONNECTION_STRING and PGVECTOR_TABLE_NAME in both compose.prod.yaml.jinja and compose.local.yaml.jinja for per-instance table isolation on a shared pgvector substrate. Fail-loud rationale mirrors EMBEDDING_ENGINE: a silent fallback to LanceDB would orphan embeddings on a different DB; a silent shared table name would collide instances on shared pgvector. Password lives only in Infisical, never in compose. Existing lancedb sites render byte-identically.

  • anythingllm-docker/sites/dev-weown-anythingllm — INFISICAL_PATH catch-up to the shared-project + folder-per-instance model — optional INFISICAL_PATH --path scoping in the site's entrypoint-infisical.sh and ansible/deploy.yml (default empty = project root, backward-compatible). DEPLOYMENT_GUIDE §6.5 documents site.conf as the primary mechanism and the current container-side scoping caveat.

  • anythingllm-docker template — kimi-verified fleet defaults + MCP tool pack (2026-07-29) — updates the canonical copier template so new AnythingLLM sites ship with OpenRouter deepseek/deepseek-v4-flash, OPENROUTER_TIMEOUT_MS/LLM_STREAM_TIMEOUT 10000, AGENT_SEARCH_PROVIDER=searxng-engine, AGENT_SEARXNG_API_URL=https://searxng.weown.tools (copier var searxng_api_url), bind-mounted ./storage, and a declarative MCP pack (searxng / web-scraping / web-browsing / rag-memory / document-summarizer) with verified uvx/mcp pins and headless Puppeteer flags. No API keys in git — Infisical / Secure Handoff only. Ansible deploy.yml.jinja seeds storage/mcp + storage/plugins on deploy. Also folds in INT-P07 deploy-hardening leftovers (Minimus token soft-fail + Caddy volume ownership) without regressing the env-var Infisical login pattern already on main.

  • anythingllm-docker dashboard — "Sign in with WeOwn" (Keycloak OIDC) (2026-07-24) — implements the customer-identity decision (one weown-chat realm, per-tenant OIDC client + group; weown-fleet kc-bootstrap-realm.sh / kc-provision-tenant.sh). When the tenant's Infisical folder carries OIDC_ISSUER/OIDC_CLIENT_ID/OIDC_CLIENT_SECRET/OIDC_ALLOWED_GROUP (stored by fleet provisioning), the login page offers SSO as the primary path and tucks the password form behind a break-glass link; absent, nothing changes. Zero-dependency authorization-code flow: HMAC-signed 10-minute state (no server-side session store), server-side code exchange, authorization by tenant-group membership in the issuer's userinfo groups claim (asked directly over TLS — no local JWT verification to get wrong), same HMAC session cookie as password logins (SameSite=Lax on the SSO-set cookie so the cross-site redirect landing works; password cookie stays Strict). This finally gives Tyler/Mike/etc. individual named logins with password reset + MFA in Keycloak instead of a shared credential. Verification: node --check; 10-check live test against a stub IdP — authcfg gate, redirect shape, good-code → cookie → authenticated API call, wrong-group 403 without cookie, tampered-state 400, bad-code 401, password break-glass intact.

  • anythingllm-docker — cloudflare_proxied mode: tenants can live behind Cloudflare's proxy (2026-07-23) — the per-brand <brand>.weown.chat hosting model (Jason 2026-07-21) puts customer instances behind Cloudflare Pro (orange cloud), where the template's ACME HTTP-01 cannot complete — exactly the 525 f1visa.weown.chat shows today. New copier bool cloudflare_proxied (default false — existing direct-DNS sites render identically): Caddy serves a Cloudflare Origin CA certificate instead of running ACME (tls /certs/origin.crt /certs/origin.key; zone SSL = Full (strict)). The PEMs arrive as CF_ORIGIN_CERT/CF_ORIGIN_KEY through the same ADR-006 Infisical entrypoint wrapper as every other secret — the caddy container gains the wrapper, and its command materialises them as 0600 files inside the container at start (never on host disk; re-fetched on restart). Origin-CA over DNS-01 was deliberate: no custom Caddy build and no DNS-edit API token sitting on customer droplets — one wildcard cert for *.weown.chat, readable only by operators at the initiative-project root, copied per-tenant-folder at provision time (droplet MIs stay path-scoped). Verification: copier renders clean in both modes and the rendered compose parses; the cert-materialise command proven against a multi-line PEM in env (content byte-identical, 0600 perms); fleet wiring (registry flag → render-deploy/provision) landed in weown-fleet.

  • gitea-docker/ — new copier template: Gitea with Keycloak SSO (2026-07-17) — self-hosted git for WeOwn internal use (Jason-approved Gitea + Keycloak), cloned from keycloak-docker/ per the repo template standard and keeping its full stack: ADR-006 in-container Infisical injection, Path-C thin cloud-init + ansible app layer, Layer-2 bootstrap-secret rotation, skinny backups with DO Spaces offload, DO monitoring alerts. Gitea-specific: all logins via Keycloak OIDC (confidential client, gitea admin auth add-oauth bootstrap documented in the rendered README with a stdin-only secret prompt; local registration disabled, SSO auto-registration on, USERNAME=email, ACCOUNT_LINKING=auto), git-over-SSH on a dedicated firewall port (gitea_ssh_port, default 2222; inline-justified trivy:ignore:AVD-DIG-0001), GITEA__* env-style config with only non-secret values in compose (SECRET_KEY/INTERNAL_TOKEN/JWT_SECRET from Infisical in prod), INSTALL_LOCK=true (env-only config, no web installer), no admin vhost (avoids keycloak-docker's D437 exposure), health via /api/healthz. Verification: template renders clean via copier; rendered compose/ansible YAML parse; tofu validate passes; all scripts bash -n clean; local end-to-end SSO proven — rendered compose.local.yaml + local Keycloak 24 (realm weown, confidential gitea client with email profile offline_access scopes) → browser-equivalent OIDC dance → Keycloak login → callback → user auto-registered active → authenticated session. Reference configs: mcp/infra gitea + keycloak_client roles.

  • anythingllm-docker — customer dashboard (the product surface) + Simple-SSO + two-workspace bootstrap (2026-07-15) — the customer-facing managed-service layer. New template/dashboard/ is a zero-npm-dependency Node server (bind-mounted read-only into a node:20-alpine container, no build/registry image for v0) that is the customer's ONLY surface: they authenticate to the dashboard (HMAC session cookie, sha256 password hash from Infisical, per-IP login throttle, X-Dashboard CSRF header, SameSite=Strict; Secure), and the server proxies scoped actions to the AnythingLLM API with the admin Developer API key held server-side — the customer never receives ALLM credentials. Two sections: Public (upload public docs → ws-public, embed-snippet screen) and Private (upload business docs → ws-private, authenticated chat proxied with userId attribution). Caddy routes /app* → dashboard, everything else → ALLM (embed widget + WeOwn-operated admin UI). SIMPLE_SSO_ENABLED added to compose so the dashboard can mint single-use 1-hour login links (/api/v1/users/{id}/issue-auth-token → /sso/simple) for a UI hop. New scripts/bootstrap-product.sh provisions the two workspaces + domain-allowlisted embed widget + all dashboard secrets in-process (nothing on disk/history). ansible uploads the dashboard app; docs updated. Verification: node --check on the server; live 8-check smoke test (healthz, unauth-reject, wrong-pw-reject, login, CSRF-reject, authed doc-list, bearer-authenticated chat proxy, logout-clears-session) all pass against a running server + stub upstream; compose YAML parses with all three services; rendered ansible parses.

  • Per-customer LLM key provisioning + single-tenant instance lifecycle doc (2026-07-15) — scripts/provision-openrouter-key.sh mints a per-customer, budget-capped OpenRouter key via the OpenRouter Management API (POST /api/v1/keys, limit + limit_reset: monthly) and writes it into that customer's Infisical app project as OPENROUTER_API_KEY — the value never touches the terminal, disk, or shell history; the provisioning key is sourced in-process from the operator Infisical project (OPENROUTER_PROVISIONING_KEY, read -rs fallback). Refuses to overwrite a live key without --force (no silent orphaning); --dry-run for shared hosts. This automates the previously-manual "create a key in the OpenRouter dashboard and paste it" step in the per-site bootstrap-*-infisical.sh scripts, and gives each dedicated instance blast-radius isolation + a hard monthly spend cap. New docs/CUSTOMER_INSTANCE_PROVISIONING.md sequences the existing deploy pieces into a per-customer provision→deploy→operate→deprovision lifecycle (technical only; commercial/sales flow kept out of this public repo). Verification: bash -n + shellcheck clean. Compliance: NIST CSF 2.0 PR.AC-4 (least privilege per tenant), PR.DS-1; CIS v8 3.11.

  • Open-branch consolidation — 10 branches integrated to main, 19 retired (2026-07-14) — a single signed integration of every mergeable open branch, each individually code-reviewed (secrets/public-repo, correctness, template-standard), rebased onto current main so the latest template versions (incl. #78 Minimus fail-loud registry login) are preserved. New: supabase-docker/ copier template (v0.1 — Postgres 15 + pgvector); anythingllm-docker/sites/dev-weown-anythingllm/ (INT-P07/P08 do.weown.tools); devbox-docker/sites/dev-weown-devbox/ (shared Zed remote-dev droplet, SSH-only, key-only sshd); keycloak-docker/sites/sso.weown.dev/ migrated to Path-C serving sso.weown.id; openclaw-docker/sites/claw-weown-dev/ + template hardening. Changed/Fixed: anythingllm template defaults per Jason (DeepSeek V4 Flash, SearXNG agent-search); searxng bound loopback-only; wordpress core-plugin-bundle fetch task; owncloud oCIS template hardening (backup-name injection guard, ignore_changes=[user_data,tags], corrected security-model claims); smoke-test framework across all 7 templates (fails on unhealthy containers, correct auth-file path). Security/public-repo scrubs: real ProtonVPN egress IP → RFC5737; real Infisical project UUIDs → placeholders; offboarded-dev emails → alerts@example.com; openclaw ADR-006 auth file 0644→0600/UID-1000; keycloak image pinned :24.0 + cloud-init {% raw %} rotation-breaker fixed. Retired (skipped + branches deleted): branches already folded into e8496d1, the retired sites/s004, obsolete pre-copier flat layouts, and (owner-decided) the zeroToHundred webapp and the superseded security-hardening-ssh inventory. Deferred: M2 Infisical --client-secret argv pattern (pre-existing fleet-wide) tracked as a separate hardening pass.

  • scripts/deploy-new-site.sh: Added --skip-infisical flag to deploy to existing Infisical projects without creating new ones

  • docs/WEOWN-APP-CLUSTER-RUNBOOK.md §5.5 — W24 Phase 3 operational findings (2026-06-11) — net-new subsection consolidating durable lessons from W24 Phase 3 dry-run execution against the live source cluster (Tue–Thu). Seven findings, each with reproduction commands and the evidence that surfaced them: §5.5.1 Target cluster sizing (combined-workload sizing, not source-match — 2-node s-1vcpu-2gb-amd was insufficient, scaled to 4 then 6, still 92–99 % allocated; recommend ≥7–8 s-1vcpu-2gb-amd OR 3–4 s-2vcpu-4gb-amd); §5.5.2 Velero controller memory (velero install default 256Mi OOM-kills on namespaces >50 items; explicit --velero-pod-mem-limit=768Mi + --node-agent-pod-mem-limit=512Mi now baked into §4.2/§4.3); §5.5.3 Velero 4 h PVR-wait timeout × capacity wall (Velero marks restore PartiallyFailed after 4 h even when no data has moved — triage via Pending pod describe Events); §5.5.4 Restore retry pattern — namespace nuke before re-fire (if pods scheduled on empty PVCs before PVRs ran, a re-fire produces a hollow restore — Completed in <30 s with no PVRs; clean redo requires kubectl delete namespace <workload>-staging first); §5.5.5 Post-restore CronJob cleanup (matomo-archive / n8n-backup CronJobs restore into staging and fire jobs against unreachable legacy destinations); §5.5.6 Interpreting benign warnings (table of 5 warning patterns that are skip-correct: cert-manager + cilium CRDs already exist, kube-root-ca.crt auto-managed, Helm release annotation drift, CiliumEndpoint cosmetic, 4 h PVR timeout); §5.5.7 AnythingLLM auth drift (post-backup admin reconfig invalidates restored password hash; informational, not a failure; soft-smoke via DB file presence). Companion edits to §3.1 (sizing callout) and §4.2 / §4.3 (--velero-pod-mem-limit=768Mi + --node-agent-pod-mem-limit=512Mi flags). Real cluster identifiers redacted per .github/copilot-instructions.md §3.0. Source: W24 SOW Day-1 through Day-4 execution; Gate 1 evidence captured in PR comment for @ncimino sign-off.

  • scripts/create-signoz-oom-alert.sh — fleet-wide memcg-OOM pager (2026-06-10, #WeOwnVer v4.1.2.2) — creates one SigNoz log-based alert on the kernel's Memory cgroup out of memory: Killed process line (already shipped to SigNoz by the fleet otel-agent's syslog receiver), closing the blind spot exposed by the INT-S004 incident: DO droplet monitor alerts watch host metrics and never fired while the AnythingLLM container was OOM-killed against its own compose memory limit (host RAM was fine). Env-only auth (SIGNOZ_BASE_URL + SIGNOZ_API_KEY via read -rs, :? guards, nothing on argv/disk), idempotent (skips if the rule name exists), with a UI fallback recipe in the header for SigNoz schema drift. Companion: anythingllm-docker/sites/s004.ccc.bot/RESIZE_RUNBOOK.md Phase 7. Compliance: NIST CSF 2.0 DE.CM-1; CIS v8 8.11.

  • AGENTS.md canonical agent rules + cross-tool pointer stubs (2026-06-09, #WeOwnVer v4.1.2.1) — the secrets-hygiene standard now auto-loads into every agentic IDE/CLI, not just Zed/Copilot/Claude. New root AGENTS.md (read natively by Codex, Cursor, Windsurf, Cline, Aider, Jules, Copilot agent mode) carries the portable standard: the agent context-leak ban (never read secrets into chat; Secure-Handoff read -rs pattern), credential-sourcing rules (Infisical runtime / fail-loud env vars with fetch-from-where errors / interactive prompts — never argv), the two-layer gitleaks setup (pre-commit + CI pinned to the same version, bumped together), the 8-step mint-before-revoke rotation flow with multi-orphan cleanup, and the shell traps that have bitten production (SSH+heredoc expansion, unicode-after-$VAR, templatefile() $$). Byte-identical pointer stubs at .cursorrules / .windsurfrules / .clinerules / GEMINI.md route the remaining loaders to it; CLAUDE.md gains the context-leak ban section + a first-read pointer; .rules now points at the canonical file and its templatefile() $$ line is corrected to match CLAUDE.md (the 2026-06-03 text still instructed the $$VAR PID-expansion landmine that the 2026-06-02 fleet-wide fix removed).

  • docs/WEOWN-APP-CLUSTER-RUNBOOK.md (2026-06-05, v4.1.1.1) — migration runbook for the W23 cluster team-move (<source_team> → <target_team>). Companion to docs/WEOWN-APP-CLUSTER-INVENTORY.md. Adopts the parallel-build + DNS-cutover pattern from ADR-005. Ten phases: target cluster provisioning at the highest available 1.33.X-do.Y slug to match source's minor (per inventory §5.1; current operational slug is 1.33.12-do.0 since DOKS retired the source's exact 1.33.1-do.2 slug post-inventory — within-minor patch drift is API-compatible and Velero cross-restore is unaffected), addon install (ingress-nginx, cert-manager ≥v1.16 fresh, metrics-server), Velero install on both clusters pointed at shared <SPACES_BUCKET>, dry-run restore into staging namespaces with per-workload smoke tests (Gate 1), real restore on temp staging hostnames with dual-hostname cert-manager TLS pre-issuance, OTel collector to SigNoz Cloud, DNS TTL lowering 48 h ahead of cutover, non-matomo cutover (anything-llm → n8n → vaultwarden Pattern B-equivalent), matomo Pattern B with explicit freeze window + mysqldump --single-transaction + DNS swap (Gate 2 with @ncimino sign-off), 7-day soak with source stopped-but-intact, decommission. Includes §13 per-phase rollback table, §14 compliance mapping (NIST CSF 2.0 PR.IP-3 / RC.RP-1 / DE.CM-1; CIS Controls v8 11.1 / 12.4; ISO 27001:2022 A.5.30 / A.8.13). Real identifiers redacted per .github/copilot-instructions.md §3.0; operator-internal companion holds resolved values. Source: W23 SOW Day-3 deliverable.

  • docs/WEOWN-APP-CLUSTER-INVENTORY.md (2026-06-03, v4.1.1.1) — pre-migration inventory of the W23 source cluster (<source_team> → <target_team>), captured via read-only kubectl + helm against the source kubeconfig context. Documents 5 active Helm-managed workloads (matomo + MariaDB, anything-llm, n8n, vaultwarden), flags 2 anomalous namespaces (orphaned nextcloud PVC, endpoint-less searxng), and catalogs 10 PVCs totalling 123 Gi plus the ingress / DNS map. §5 records 7 scope decisions resolved with @ncimino during 2026-06-03 review (k8s version pin at 1.33.1-do.2 to match source per Velero cross-restore constraint; vaultwarden Pattern B treatment; nextcloud + searxng + *-backup PVC decommission; Cilium + Hubble continuity; cert-manager fresh install ≥v1.16; ingress-nginx upstream image source). Real cluster identifiers, LB IP, node hostnames, kubeconfig context, Spaces bucket, and PV UUIDs redacted per .github/copilot-instructions.md §3.0; operator-internal runbook holds resolved values. Companion to forthcoming docs/WEOWN-APP-CLUSTER-RUNBOOK.md (W23 follow-up PR). Compliance: NIST CSF 2.0 ID.AM-1, ID.AM-2; CIS Controls v8 1.1, 12.4. Source: W23 SOW Day-2 deliverable.

  • .rules + docs/onboarding/ — Zed dev-governance & setup runbook (2026-06-03, #WeOwnVer v4.1.1.1) — net-new dev-governance layer so WeOwn standards load into each developer's AI editor automatically (Zed loads only the first rules file it finds at a repo root, so the standards must be in that slot). Adds a root .rules (the first-match file: public-repo §3.0 redaction, Infisical secrets, branch naming, #OnlyHumanApproves, PR hygiene, with pointers to .github/copilot-instructions.md + CLAUDE.md + the #FedArch PinnedDocs), docs/onboarding/zed-global-AGENTS.md (the cross-repo global-rules template devs copy to ~/.config/zed/AGENTS.md), and docs/onboarding/ZED_DEV_SETUP.md (step-by-step runbook: clone → global rules → project .rules → #FedArch RAG-sync per GUIDE-006 → Infisical → must-reads → verify). Companion business-governance doc proposed for CCCbotNet/fedarch as GUIDE-016.

  • anythingllm-docker/DEPLOYMENT_GUIDE.md — operator onboarding guide (2026-06-03) — the single step-by-step source of truth for deploying a WeOwn AnythingLLM instance, written so other devs can build on the template. Documents the two-project Infisical secrets model (operator-only weown-tofu per-dev/shared infra tokens via each dev's own login, vs the per-site app project read by the droplet's Machine Identity), DO-Spaces remote tofu state (weown-prod-state, SSE-C) via itofu.sh, the Path C + Layer 2 bootstrap, skinny GFS backups to weown-prod-backups, the reserved/static IP, Caddy auto-HTTPS, OTel→SigNoz observability (separate fleet agent), the fail-loud JWT_SECRET/OPENROUTER_API_KEY/ANYTHINGLLM_IMAGE guards, runtime env injection, the Infisical-injected ANYTHINGLLM_IMAGE versioning (in testing on AnythingLLM; Caddy still in-repo), the Minimus→DOCR mirror strategy (with an open licensing question), and the ssh_source_cidrs → Proton-VPN perimeter plan (currently open). Reference deployment: INT-S004. Linked from the template README.

  • anythingllm-docker/sites/s004.ccc.bot/ — INT-S004 recovery rebuild (2026-06-02, #WeOwnVer v4.1.1.1) — a fresh AnythingLLM droplet replacing the locked-out s004.ccc.bot. Re-rendered from the anythingllm-docker copier template and validated by diff against the hardened sites/ai.weown.agency/ (INT-P01) — at render time its deployable files matched ai.weown.agency modulo names (it has since been hardened further — see the Changed entries) (project_name=int-s004-anythingllm, domain=s004.ccc.bot, image injected from Infisical via ANYTHINGLLM_IMAGE — deployed v1.12.1; the old box actually ran :latest (digest 7a2f7157), so the restore forward-migrated the schema cleanly on first boot). Single-host Caddyfile (s004.ccc.bot, same hostname as the box it replaces), a dedicated s004 Infisical project (least privilege, distinct from INT-P01's weown-anythingllm), a ${JWT_SECRET:?...} fail-loud compose guard, and scripts/bootstrap-s004-infisical.sh (read -rs → push to Infisical, no disk). Ships a recovery MIGRATION_RUNBOOK.md (restore the off-box s004_storage_<TS>.tar.gz export). Closes the two old failures: JWT_SECRET unset on a container restart, and the never-installed daily backup cron.

  • devbox-docker/ — shared multi-user developer machine copier template (2026-05-30) — new *-docker template for a single DigitalOcean droplet the team SSHes into for Zed (zed.dev) remote development (per Jason's 2026-05-20 "Zed is mandatory for the entire team" mandate, ♾️ WeOwn.Dev). Forked from the canonical anythingllm-docker template and adopts the same Layer 1 (DO Spaces remote tofu state) + Layer 2 (first-boot Infisical Machine-Identity bootstrap-secret rotation) + Path C (thin cloud-init + ansible/deploy.yml app layer) pattern from docs/INFRA_BOOTSTRAP_PATTERN.md. Diverges from the web-app templates: no Caddy/compose, no inbound 80/443 — the only inbound port is SSH (22). Security model: no shared team root — a single break-glass admin key (var.ssh_key_fingerprint) is the only root access (PermitRootLogin prohibit-password); each member gets their own non-root account (login = lowercased CCC Short ID, UID ≥ 1000) in a devs group gated by AllowGroups devs root, with authorized_key exclusive: true so a key opens only its owner's account; members get no sudo and docker-group membership is opt-in per member (root-equivalent — documented). Per-user OpenRouter keys for Zed's AI agent are handled by scripts/setup-zed.sh (read -rs prompt, stored only in a 0600 ~/.config/<slug>/openrouter.env, never in settings.json/argv/logs; optional --infisical path keeps the key in the member's own Infisical account behind an infisical run -- zed launcher). The roster lives in ansible/members.yml (gitignored; members.example.yml shipped) so onboarding (scripts/add-user.sh) and offboarding (scripts/offboard-user.sh — archives the member's /home off the box, removes the account, then prints an Infisical/secret-rotation revocation checklist) are an edit + re-run, never a tofu taint. Also ships team-standard Zed settings, a dev toolchain (git, node, opentofu, doctl, copier, ansible-lint, YAML/Python LSPs), daily skinny backups of /home + /etc (secrets hard-excluded), DO monitoring alerts, and operator + end-user docs (README.md, docs/CONNECTING-WITH-ZED.md). Validated: copier render + tofu validate clean, bash -n on all scripts, YAML parse of cloud-init/ansible/roster.

  • .github/ADR-005-int-p01-doks-retirement.md — Proposed ADR (2026-05-25) documenting the INT-P01 retirement from DOKS, the parallel-build + DNS-cutover migration pattern, two-gate validation (Jason/Yonks staging soak; CTO production-cutover approval), and compliance mappings across NIST CSF 2.0, SOC 2, ISO/IEC 27001:2022, ISO/IEC 42001:2023, CIS Controls v8. Status moves Proposed → Accepted at close of the 7-day post-cutover soak.

  • .github/ADR-006-in-container-infisical-injection.md — Proposed ADR (2026-05-30, #WeOwnVer v3.4.5.1) establishing in-container Infisical injection as the runtime secret-delivery standard for *-docker services. Documents the move from the host-side infisical run -- docker compose up wrap (secrets baked at container-create time → refresh only on --force-recreate/redeploy) to infisical run as the container entrypoint (re-fetch on docker restart → bounce-to-refresh), reusing the Layer-2 Machine Identity + CLI already on the droplet (no rebuilt image, no new credential). Covers blast-radius scoping (per-workload, per-project identity), auto-reload (--watch/Infisical Agent), automatic rotation (consumer-side trigger), single-use/wrapping tokens (Machine Identity Token Auth, Max-Uses=1), and the K8s/K3s migration (Infisical Secrets Operator + Kubernetes-native auth + auto-reload annotation). Pilots on anythingllm-docker/sites/s004.ccc.bot/; other services adopt in follow-up PRs. Priority sponsor: Jason (Signal ♾️ WeOwn.Dev 2026-05-21; D383 gate on s004 working).

  • docs/INFRA_BOOTSTRAP_PATTERN.md — new "Runtime secret injection" section (2026-05-30) added as the companion to Layers 1–2 + Path C: documents current host-side-wrap semantics, the in-container target, auto-reload/rotation/one-time-token, and the K8s mapping; updates the single-use-token bullet under "future hardening" to the Infisical-native Token-Auth form; refreshes the stale sites/s004/ proving-ground reference to the live s004.ccc.bot; last-updated 2026-06-04.

  • Per-*-docker README "Secret injection pattern" pointers (2026-05-30) — anythingllm, keycloak, openclaw, sandbox, searxng, signoz, wordpress READMEs now link the canonical pattern (INFRA_BOOTSTRAP_PATTERN.md + ADR-006).

  • anythingllm-docker/sites/ai.weown.agency/ — INT-P01 DOKS → Docker migration site (2026-05-25) — generated from the anythingllm-docker copier template with project_name=int-p01-anythingllm, domain=ai.weown.agency, anythingllm_image=reg.mini.dev/anythingllm:1.7.2 (WeOwnLLM hardened, per D381; pinned to the version verified on s004.ccc.bot). Adopts the Path C + Layer 2 bootstrap pattern from docs/INFRA_BOOTSTRAP_PATTERN.md: slim cloud-init that hands off to ansible/deploy.yml, Layer 2 bootstrap-secret rotation, Layer 1 DO-Spaces remote tofu state, dual-hostname Caddyfile (ai-stage.weown.agency, ai.weown.agency) so the production cutover is a DNS A-record swap with no re-deploy. Includes phased migration runbook (MIGRATION_RUNBOOK.md) with an optional local-laptop dry-run phase, and a one-shot bridge script scripts/migrate-from-doks.sh that streams /app/server/storage out of the DOKS pod via kubectl exec and wraps it in the skinny-backup tarball layout the existing scripts/restore.sh already understands. Source plan: D383 / Tuleap A174 ([#1238]) / Signal ♾️ WeOwn.Dev 2026-05-21 ask from Jason.

  • anythingllm-docker/sites/README.md — new directory-level README documenting the sites/ pattern (matching the existing keycloak-docker/sites/README.md convention), with a current-deployments table and a "creating a new site" walkthrough.

  • sandbox-docker/ adopts canonical Layer 1 + Layer 2 + Path C (2026-05-26) — migrated the AIO Sandbox copier template to the WeOwn bootstrap pattern defined in docs/INFRA_BOOTSTRAP_PATTERN.md. Adds DO Spaces remote state (backend.tf.jinja + init.sh.jinja), embedded rotate-bootstrap-secret.sh in cloud-init (mints v2 Infisical Machine Identity client secret via Universal Auth API on first boot, atomically swaps .infisical-auth.env, revokes v1), thin cloud-init + ansible/deploy.yml.jinja app layer with community.docker.docker_image_pull (CLI-based; no python3-docker SDK dependency), auto-tagging via scripts/tag-droplet.sh (commit-<sha>, feature tags), ssh_source_cidrs firewall pinning, Caddy file logging with /var/log/caddy bind mount for otel-agent ingestion, and pinned sandbox_image default (ghcr.io/agent-infra/sandbox:0.4.0, no more :latest). Switches Infisical CLI install from legacy install-cli.sh (capped at v0.38) to the artifacts-cli.infisical.com apt repo.

  • scripts/bootstrap-otel-agent.sh (2026-05-23) — one-time per-droplet bootstrap for the OTel fleet agent. Installs the Infisical CLI if missing and writes /opt/otel-agent/.infisical-auth.env (0600 root) with the Machine Identity credentials for the Infisical otel project. Supports --droplet <name>, --tag <tag>, and --host <user@ip> selectors. Validates the Machine Identity can log in before exiting. Compliance: NIST PR.DS, CIS 3.11, ISO A.5.16.

  • otel-agent/README.md (2026-05-23) — full operator runbook for the SigNoz Cloud observability path: architecture diagram, secrets model (Infisical runtime injection, never on disk), safety notes for existing droplets (all mounts read-only, loopback health port, 256MB cap), per-droplet bootstrap + deploy steps, verification checklist, key rotation procedure, troubleshooting table, compliance mappings.

Changed

  • *-docker templates — the image registry is one copier answer (2026-09-30) — reg.mini.dev (Minimus) shuts down 2026-10-22 (WeOwnDev/weown-fleet#112), and the host was hardcoded across 11 templates (anythingllm, billing, gitea, keycloak, openclaw, owncloud, sandbox, searxng, signoz, supabase, wordpress). Each now asks image_registry (default reg.mini.dev) and registry_username (default = the username that template logs in with today: minimus for anythingllm/openclaw, token for the rest); image defaults, ansible inventories and every docker login read them. The registry password's secret-store key is chosen BY REGISTRY, not by presence: a hidden computed answer registry_token_key is MINIMUS_TOKEN while image_registry is reg.mini.dev and REGISTRY_TOKEN for any other registry, and every deploy playbook (Infisical and OpenBao paths) reads exactly that one key — no fallback. So the new registry's REGISTRY_TOKEN can be stored in every tenant's store before a site switches without a still-on-Minimus site logging in to reg.mini.dev with the wrong key. With default answers a render is byte-identical to before, except anythingllm's login task below. Committed sites/ renders are untouched; each rolls over per site under weown-fleet#112. The Terraform minimus_token variable (really the DO API token) is not renamed here. anythingllm's registry login now passes the token on ansible's stdin: (command + argv) instead of writing it into a shell script's text, which put it on bash's argv on the droplet.

  • WeOwnChat onboarding (billing + landing) — Keycloak register vs sign-in + no-instance paywall (2026-09-15) — see billing-docker/sites/billing/CHANGELOG.md, billing-docker/template/CHANGELOG.md.jinja, landing-purchase/CHANGELOG.md, and .github/ADR-007-billing-keycloak-registration-onboarding.md. PR #252.

  • scripts/provision-openrouter-key.sh — --bao-path: the minted per-customer key can land in OpenBao instead of Infisical (2026-09-04) — new weown-fleet tenants integrate with the WeOwn OpenBao platform store (Nik, 2026-09-04: "credentials should be integrated with openbao, not infisical"), so the fleet provisioner needs the customer key written to weown/platform/<slug>. The PROVISIONING key is still read from the operator Infisical project (operator tier); with --bao-path the minted key travels env → jq → stdin → bao kv patch (never argv) and is read back from the store before success is reported. Infisical behaviour without the flag is unchanged. Consumer: weown-fleet scripts/provision-instance.sh Phase 1 via scripts/lib-secrets.sh. Verification: bash -n clean; dry-run path unchanged.

  • anythingllm-docker customer dashboard — full UI/UX redesign to a modern chat-app standard (2026-07-29) — public/index.html and public/login.html rebuilt against the same real endpoints, unchanged constraints (zero-dependency, single self-contained file each): a collapsible sidebar with a horizontal elevated-chip tab switcher (private/public selection now reads through elevation/form rather than color — color is reserved for the semantic public/private cue, carried by a pill badge in the main header plus a warning banner on the public pane instead of the tab selector itself; a deliberate mid-review pivot from the original ask, made because two different signals — "which tab is selected" and "is this tab public" — were competing for the same color channel); a real chat surface for the private assistant (message bubbles with an accessibility fix — the original blue-fill/white-text combination measured ~2.7:1 contrast, below WCAG AA's 4.5:1 floor for body text, now a dark navy bubble at ~11.5:1; copy on every message, edit-and-resend on the ones you sent; a landing state that centers the greeting and composer together, homepage-style like modern AI chat products, instead of docking the composer at the bottom before any conversation exists); deferred conversation creation ("New conversation" opens a blank composer immediately with no upfront naming prompt — the thread is created server-side with a title derived from the first message actually sent, once it's sent, so the common case never needs a rename call); a kebab menu (rename/delete) replacing the old flat picker; confirm-before-remove (click once to arm, click again to confirm) added to document and allowed-website removal for parity with conversation delete; the public tab restyled onto card-based sections (Documents / Allowed websites / Embed snippet) with the snippet in a bordered code block that wraps instead of overflowing on narrow screens — the exact bug named in the original brief. All dynamic rendering rebuilt on safe DOM construction (createElement/textContent) instead of the previous innerHTML-based rendering — which, checked against the pre-redesign source, mixed a shared esc() helper (thread names, allowed-website domains) with a separate, weaker inline escape for chat text (stripped < only, not &/>/quotes) and, for document names specifically, no escaping at all (${x.name} interpolated straight into innerHTML). The rewrite applies the same safe-construction treatment everywhere uniformly, closing that document-name gap rather than just preserving the original's (inconsistent) behavior. The composer carries a visibly disabled file-attach affordance (paperclip icon, "coming soon" tooltip) — no upload wiring behind it, since neither AnythingLLM's nor the underlying model's chat-attachment support is confirmed yet.

    Two small additions to server.js (the four named security invariants — X-Dashboard CSRF header, HTML-escaping discipline, upload type/size limits, non-emptyable allowed-websites list — otherwise untouched): POST /api/threads/:slug/rename (proxies to AnythingLLM's thread-update API; path unverified against a live instance — no ALLM access in the environment this was built in, so this is the first thing to check if renaming 502s in practice) and GET /api/whoami (returns the already-existing DASHBOARD_CUSTOMER_EMAIL — previously computed server-side but never sent to the client — now shown in the sidebar; gracefully hidden on instances that don't set that optional env var).

    Known limitation: edit-and-resend ("rewind") truncates the conversation client-side and resends for real, but does not delete the original trailing messages from AnythingLLM's own thread history — no confirmed partial-deletion API exists (only whole-thread clearing, which would be more destructive than intended). Reloading that thread will show the old messages reappear ahead of the resend; nothing is lost, it just isn't a clean server-side history rewrite yet.

    Verification: no live AnythingLLM instance was available while building this, so verification is static + simulated rather than live end-to-end: node --check on all three files; HTML tag-balance parsing on both pages; a scripted jsdom session against a fetch stub matching every real endpoint's request/response contract, driving the full interaction surface (login success/failure, tab switch, thread switch/create/rename/delete, send/rewind including the thread-creation-failure rollback path, document/domain add/remove, snippet copy) with zero runtime errors; server.js booted locally against a throwaway dev password hash and curled directly to confirm the auth gate, the CSRF gate, and the new rename endpoint's input validation all behave correctly against the real server, not just the mock. Layout fixes (composer-width parity between the landing and docked states, mobile-width snippet overflow) verified with real browser screenshots at 1280×800 and 390×844. Before merge: needs a pass against a real AnythingLLM instance to confirm the rename endpoint and general live behavior — flagged to the reviewer in the PR.

  • anythingllm-docker dashboard — document upload redesigned into one shared library with per-surface toggles (2026-07-30) — the private and public tabs each had their own separate "upload a document" box, so a document useful for both (a service menu, say) had to be uploaded twice, and there was no single place to see everything a customer had ever uploaded. public/index.html replaces both with a Documents button (header, either tab) opening one shared library modal: every uploaded document gets two independent toggle pills — Private chat and Public site — reusing the exact green/red the app already uses for the private/public distinction elsewhere, so "this is visible on the public website" reads as the same warning color everywhere, not a new one to learn. Each tab's header strip now shows a compact, live summary ("documents your private assistant uses" / "documents your website chatbot uses") instead of a duplicate upload UI. Uploading from a tab defaults the new document on for that tab's scope, so the common one-click case still works; widening it to the other surface is a separate toggle click.

    server.js replaces the three scoped routes (GET /api/documents/:scope, POST /api/upload/:scope, POST /api/remove/:scope) with four unscoped ones: GET /api/documents (the library — the union of what's embedded in the private and public AnythingLLM workspaces, merged by docpath), POST /api/upload (now scoped by an X-Upload-Scope header instead of the URL — a document is always embedded into at least one workspace at upload time, so nothing is ever orphaned in the system document store with no scope pointing at it), POST /api/documents/scope (toggles one workspace's embedding on/off for an existing document), and POST /api/documents/delete (detaches from both workspaces and purges the underlying file). All four are built entirely from ALLM calls the pre-redesign code already made and had proven working (workspace GET, update-embeddings adds/deletes, system/remove-documents) — no new/unverified ALLM API surface for this feature, unlike the earlier thread-rename endpoint. The four security invariants (X-Dashboard CSRF header, HTML-escaping, upload type/size limits, non-emptyable allowed-websites list) are untouched.

    Verification: node --check on server.js; HTML tag-balance check on index.html; a stateful local stub of the four ALLM endpoints involved, with server.js booted for real against it (throwaway dev password hash) — curled directly to confirm the merge-by-docpath logic (a document embedded in both workspaces, one in only private, one in only public, all reported correctly), the scope-toggle and delete round-trips, the upload-scope default, and that the CSRF/auth/upload-type guard rails are all still enforced. Then driven through a real Chromium session against the same running server (not a static mock): logged in, opened the library, toggled a document's public scope on, searched, ran the two-click delete confirm, and confirmed both tabs' header summaries and counts updated live after each action.

  • anythingllm-docker — migrated itofu.sh to the A405 folder model (shared + per-site), fixing stale deploy-specific TF_VARs (2026-07-15) — anythingllm was the last template still reading weown-tofu flat (infisical run --projectId=<weown-tofu> with no --path), so every deploy inherited whatever TF_VAR_infisical_project_id was last set at the project root — surfaced live as a droplet pointed at a stale app project (8420b42e) while its secrets lived elsewhere. Migrated to the openclaw/claw reference model: infisical run --path=/infra/shared --path=/infra/sites/$SITE (SITE = SITE_NAME or the rendered project_name), so shared infra vars (do_token, spaces_*, ssh_key_fingerprints, alert_email) come from /infra/shared and per-site vars (infisical_client_id/_secret/_project_id — the app-project MI the droplet reads) come from /infra/sites/<site>. No deploy-specific TF_VAR at the root anymore. Consequently reverts the earlier singular ssh_key_fingerprint change (correct for the flat model) back to the plural ssh_key_fingerprints / list(string) that /infra/shared provides. Requires a /infra/sites/<site> folder per instance in weown-tofu. Verification: rendered itofu.sh bash -n + shellcheck -S error clean; terraform parses (tofu fmt).

  • anythingllm-docker/template/ansible/deploy.yml.jinja — Minimus registry login + rotation marker (2026-06-12) — adds docker login reg.mini.dev via CLI (--password-stdin, no Docker SDK dependency) using MINIMUS_TOKEN from Infisical, so fresh droplets can pull caddy:2 and the AnythingLLM image without manual auth. Adds fail-loud .rotation-failed marker check in pre_tasks (gated on .rotation-complete not existing, so the warning stops after manual rotation succeeds). Cloud-init fail() now writes .rotation-failed with chmod 0600 to prevent API response leakage to non-root users.

  • anythingllm-docker template: bounce-proof config + right-sized memory after the INT-S004 OOM (2026-06-10, #WeOwnVer v4.1.2.2) — incident-driven hardening of the copier template (and the s004.ccc.bot site — full incident narrative in anythingllm-docker/sites/s004.ccc.bot/CHANGELOG.md). The 2026-06-10 01:00 MT memcg OOM kill of INT-S004's AnythingLLM container revealed two template-level gaps: (1) EMBEDDING_ENGINE had a silent :-native compose fallback — after the auto-restart the app came back on a different embedder than the one that built the LanceDB vectors and every RAG query failed on a dimension mismatch; it is now a fail-loud :? guard like JWT_SECRET/OPENROUTER_API_KEY/ANYTHINGLLM_IMAGE, and the compose header + template README now state the rule that Infisical is the only source of truth for runtime config (UI changes do not survive a bounce). (2) The AnythingLLM memory limit was hardcoded at 2G regardless of droplet size; it is now the anythingllm_memory_limit / anythingllm_memory_reservation copier vars (defaults unchanged: 2G/1G on the 4 GB droplet) with sizing guidance, since agent-RAG chats with pinned docs + the in-process native reranker legitimately spike RSS. Also adds explicit resize_disk = false to the droplet resource (template + s004): droplet size changes stay CPU/RAM-only, reversible, and leave the disk + Docker volumes untouched. Compliance: NIST CSF 2.0 PR.IP-3, RC.RP-1; CIS v8 4.1/4.2; ISO 27001 A.8.9.

  • Refreshed all anythingllm-docker docs to the current pattern (2026-06-03) — swept the template README + top-level README + sites/README.md + both live sites' READMEs/runbooks/changelogs to remove stale "old way" content now that INT-S004 is live. Replaced the copier copy … --data-file answers.yaml render with the real sites/<domain> + --data form; documented ANYTHINGLLM_IMAGE (Infisical-injected image, no longer a render-time / variables.tf value) in every secrets table + Updating row; corrected the wrong "docker compose restart triggers a fresh infisical run" claim everywhere to "re-run deploy.sh (recreates the container); restart reuses the old env"; switched provisioning prose from terraform.tfvars / tofu apply to itofu.sh / weown-tofu (TF_VAR_*, no disk); added OTel→SigNoz observability + the OpenRouter-key-from-restored-DB caveat (401 Missing Authentication header → re-enter in the UI). Slimmed template/README.md.jinja to point at DEPLOYMENT_GUIDE.md instead of duplicating the flow, fixed the broken site-relative ../docs/ links (→ ../../../docs/), marked the s004.ccc.bot MIGRATION_RUNBOOK.md COMPLETED, and trimmed the retired sites/s004/ tombstone to a banner + pointers. The top-level "Migration status" section (which still called the retired sites/s004/ the "first deployment" and said other templates "should migrate to match") is now "Pattern status" pointing at the live s004.ccc.bot reference.

  • Infisical-native OpenTofu provisioning (2026-06-02) — added terraform/itofu.sh (+ template) that runs tofu under infisical run against a separate operator-only weown-tofu Infisical project, so the TF_VAR_* infra secrets (DO API token, Spaces keys, droplet Machine Identity) never touch a committed file, a local terraform.tfvars, or the s004 app project (whose Machine Identity the droplet can read). init forwards the Spaces creds to the S3 backend. terraform.tfvars + init.sh remain a gitignored fallback. Runbook Phase 1 documents the weown-tofu setup + the isolation rationale.

  • Team SSH access standard for AnythingLLM ops boxes (2026-06-02) — ansible/deploy.yml (+ the template) writes the team's public keys from the Infisical OPS_AUTHORIZED_KEYS var to root's authorized_keys.weown-ops (managed exclusively, alongside the untouched DO break-glass key) and adds it to sshd's AuthorizedKeysFile via a drop-in. Grant/revoke = edit OPS_AUTHORIZED_KEYS + re-run deploy.sh (revoke-on-termination supported); sshd -t gates the reload so a bad config can't lock the box. scripts/bootstrap-s004-infisical.sh prompts for the key list. Piloted on s004.ccc.bot.

  • Standardized DO Spaces bucket names (2026-06-02) — weown-terraform-state → weown-prod-state (state) and weown-backups → weown-prod-backups (backups), in the template + s004.ccc.bot. Other sites/services keep the old names (separate coordinated migration to avoid orphaning live state).

  • Renamed terraform var minimus_token → do_token in anythingllm-docker (2026-06-02) — the variable feeds provider "digitalocean" { token = ... }, so it is the DigitalOcean API token (droplet host), not a Minimus credential; the Minimus image-registry token is the separate optional MINIMUS_TOKEN Infisical secret (reg.mini.dev pulls). Renamed in the template .jinja + the s004.ccc.bot site (+ its runbook TF_VAR_do_token), with the distinction documented in the tfvars example. ai.weown.agency and the retired s004/ keep minimus_token for now (renaming a live site's var breaks its local terraform.tfvars — separate coordinated change).

  • Fail-loud JWT_SECRET guard + /api/ping caveat across anythingllm-docker (2026-06-02) — added a ${JWT_SECRET:?...} guard to the template (anythingllm-docker/template/docker/compose.prod.yaml.jinja) and both existing sites (ai.weown.agency and the retired s004), so a docker compose up that bypasses infisical run refuses to start instead of serving broken auth (the INT-S004 lockout mode). Each README now notes that /api/ping returns 200 even when auth is broken, so health checks must not rely on it alone. The retired s004/ (s004.ccc.bot) row in anythingllm-docker/sites/README.md is marked superseded by s004.ccc.bot/.

  • k8s/docs/CLUSTER_MIGRATION.md — forward note on secrets (2026-05-30) — added an Important-Notes callout that the K8s/K3s target is Infisical Secrets Operator (InfisicalSecret CRD) + Kubernetes-native auth + auto-reload annotation, the same runtime-injection pattern as the Docker fleet (ADR-006), rather than the native kubectl Secret backups this node-pool runbook currently uses.

  • docs/WEOWN-APP-CLUSTER-RUNBOOK.md (2026-06-05, v4.1.1.1) — Phase 2 pivoted from chart-based to velero install CLI after Day-4 execution surfaced the cluster-backup/ Helm chart as broken across multiple resource types and flag formats (pattern matches code helm template-validated but never helm install-tested against modern Velero). Bugs found include: CRDs not packaged in helm/crds/; BSL helper uses pre-Velero-1.0 schema (spec.bucket instead of spec.objectStorage.bucket); Schedule helper wraps BackupSpec in Pod-style template.metadata + template.spec instead of flat template; VSL template emits invalid default: true field; --default-volume-snapshot-locations flag passes location name without required provider: prefix; missing AWS plugin init container (mandatory since Velero 1.10); deprecated --restic-timeout and --restic-restore-timeout flags; unknown --metrics-enabled flag; Velero appVersion pinned at retired 1.12.2. §4 (Phase 2) rewritten to use velero install CLI with --provider aws --plugins velero/velero-plugin-for-aws:v1.11.0 --bucket <SPACES_BUCKET> --backup-location-config region=atl1,s3Url=...,s3ForcePathStyle=true --secret-file <mktemp_creds> --use-volume-snapshots=false --use-node-agent --uploader-type=kopia (Velero v1.18 removed Restic uploader, Kopia is canonical), with --velero-pod-cpu-request=50m / --node-agent-pod-cpu-request=50m overrides tuned for the W23 cluster's s-1vcpu-2gb-amd nodes (defaults at 500m/200m don't schedule on this size). §4.2 / §4.3 reordered so target install runs first (zero risk on empty cluster, validates bucket connectivity). §13 rollback Phase 2 entry updated from helm uninstall to velero uninstall --force. Chart's preset Pattern A Schedules and ServiceMonitor are NOT installed by velero install — re-added as a thin "WeOwn add-ons" layer post-cutover (separate follow-up, out of W23 scope). §3.1 (cluster create) version pin updated to 1.33.12-do.0 — DOKS retired source's exact 1.33.1-do.2 slug for new cluster creation post-inventory; within-minor patch drift is API-compatible and Velero cross-restore is unaffected, but the explicit pin needed to reflect operational reality. Added --auto-upgrade=false to the create flags to prevent DOKS from bumping the cluster's K8s version mid-migration. §1 (Approach) bullet 1 reframed from exact slug pin to "highest available 1.33.X-do.Y slug to match source's minor". Source: W23 SOW Day-4 execution. Full chart-bug list tracked in operator-internal notes for the next @ncimino sync; possible follow-ups are either a focused fix(cluster-backup) PR or chart deprecation in favour of velero install CLI + a thin Helm wrapper for WeOwn-specific Schedules / ServiceMonitor (out of W23 scope).

  • docs/WEOWN-APP-CLUSTER-RUNBOOK.md (2026-06-05, v4.1.1.1) — earlier same-day amend (now superseded by the chart-pivot entry above): Phase 2 + Phase 5 patterns amended after Day-4 validation of the secret-pull flow end-to-end. Phase 2 (Velero install): replaced interactive deploy.sh prompts with an infisical run --projectId=<W23_INFISICAL_PROJECT_ID> --env=prod -- bash -c '...' wrapper that maps SPACES_ACCESS_KEY / SPACES_SECRET_KEY (per @ncimino 2026-06-05 directive on Infisical key naming) to the chart's expected S3_ACCESS_KEY / S3_SECRET_KEY env-var contract (see cluster-backup/deploy.sh lines 111-122). Non-secret values (bucket, region, endpoint, tenant, cluster, environment) inlined as explicit export lines. Bucket strategy switched from "shared bucket + prefix" to a dedicated per-cluster bucket (<SPACES_BUCKET>) — the cluster-backup chart's BackupStorageLocation helper does not emit a prefix field, so per-cluster bucket matches the existing per-tenant-bucket precedent in the WeOwn DO Spaces fleet without requiring a chart change. Phase 5 (OTel): dropped the Machine Identity universal-auth pattern in favour of user-level infisical login + infisical secrets get OTEL_URL / OTEL_KEY --env=prod (the otel project's dev env values are placeholders; only prod is operationally configured). Operator's CLI session is the auth boundary; secret values pass to kubectl create secret generic via the existing $(mktemp) + trap + --from-env-file pattern, preserving the --from-literal ban from .github/copilot-instructions.md §3.0 PR.DS. Infisical K8s Operator + InfisicalSecret CRD path tracked as a post-W23-soak follow-up so the migration itself stays scope-tight. §2.1 (Access) rewritten to surface the user-level Infisical access pattern (Viewer on otel prod env; owner on <W23_INFISICAL_PROJECT> prod env) and drop the MI bullets. §13 (Rollback) Phase 5 entry updated to point at Infisical prod env, not Pass vault. Date stamp bumped to 2026-06-05. Source: W23 SOW Day-4 execution + @ncimino Signal 2026-06-05 directive.

  • Observability pivot to SigNoz Cloud (2026-05-23) — per CTO direction, WeOwn AI now sends telemetry to SigNoz Cloud (Yonks' managed account), not a self-hosted SigNoz droplet. The signoz-docker/ copier template is preserved as an optional/fallback path only — see signoz-docker/README.md for the status banner. OTEL_URL + OTEL_KEY live in the Infisical otel project and are injected at every docker compose up via infisical run (never stored on disk; container restarts pick up rotated values automatically).

  • otel-agent/config.yaml rewritten for SigNoz Cloud (2026-05-23) — exporter switched from in-VPC otlp (insecure) to otlphttp/signoz with TLS + signoz-access-token header reading ${env:OTEL_KEY}; endpoint reads ${env:OTEL_URL}. OTLP/HTTP chosen over gRPC so the Infisical-stored OTEL_URL (a full https://ingest.us2.signoz.cloud URL) works as-is — the gRPC exporter requires bare host:port form. SigNoz Cloud destination region is us2 (not us1 or us). Health endpoint moved from 0.0.0.0:13133 → 127.0.0.1:13133 (loopback only) so network_mode: host does not expose it. Added filter/skip_self processor to prevent the otel-agent's own logs creating a feedback loop. Added sending_queue for buffering during transient SigNoz Cloud unavailability. Stricter filesystem exclude list (avoids container/snap pseudo-filesystems). Compliance: NIST DE.CM, CIS 8.5, ISO A.8.15.

  • otel-agent/compose.yaml rewritten for Infisical runtime injection (2026-05-23) — OTEL_URL and OTEL_KEY are now read from the environment with :? shell-fail-if-missing guards, so a misconfigured infisical run fails fast with a clear message instead of silently shipping no data. Health probe target updated to 127.0.0.1:13133. JSON-file log driver pinned with rotation caps (10MB × 3 files). All host mounts remain read-only.

  • otel-agent/deploy.yml rewritten (2026-05-23) — removed broken Jinja escaping pattern ({{ '{{' }} … {{ '}}' }}) that Gemini had introduced; these are plain Ansible playbooks, not copier templates. Now requires <otel_agent_dir>/.infisical-auth.env to exist (created by bootstrap-otel-agent.sh) and starts the container under infisical run using the bootstrapped Machine Identity. Path is configurable via -e otel_agent_dir=... (default /opt/otel-agent; burnedout-xyz uses /root/observability/otel-agent). Infisical env slug is configurable via -e infisical_env=... (default dev to match the existing "Development" environment in the otel project). Added pre-flight checks (docker present, infisical CLI present, auth file present). Added wait-for-health task.

  • scripts/bootstrap-otel-agent.sh adds --dir and --env-slug flags (2026-05-23) — --dir <path> overrides the install directory (default /opt/otel-agent; burnedout-xyz uses /root/observability/otel-agent). --env-slug <slug> records the Infisical env to use at runtime in the auth file (default dev); the deploy script reads this back so the env-slug only needs to be specified once during bootstrap.

  • scripts/deploy-otel-fleet.sh rewritten + --dir / --env-slug flags (2026-05-23) — dropped the <signoz-private-ip> positional argument (no longer applicable with SigNoz Cloud). Now supports --droplet <name>, --tag <tag>, and --host <user@ip> selectors, plus --dir, --env-slug, --environment, --namespace, --dry-run options. --env-slug is optional and overrides the value recorded by bootstrap. Refuses to deploy to a host that has not been bootstrapped at the specified --dir. Wraps docker compose up in infisical run for runtime secret injection. Includes inline health wait (up to 60s).

Deprecated (kept for fallback only)

  • signoz-docker/ (self-hosted SigNoz copier template) — superseded by SigNoz Cloud + otel-agent/. Files retained, deployment paused. Reactivation requires an ADR; see signoz-docker/README.md for the procedure.

Fixed

  • anythingllm-docker — the deploy refuses an image on a registry host the box does not log in to (2026-10-02, WeOwnDev/weown-fleet#112). The box logs in to one registry host, the one from image_registry. If ANYTHINGLLM_IMAGE (read from the store) names another registry host, its pull would run unauthenticated and fail at compose up. Each compose-up task now checks it first and stops with both hosts named. A Docker Hub ref (no dot or colon in its first part) needs no login and still passes. Verification: scripts/test-image-host-check.sh cuts the check out of both compose-up tasks of four renders (infisical/openbao × default/mirror) and runs 28 cases. Without the host comparison, 8 fail.

  • anythingllm-docker (openbao) — the deploy stops with the reason when the platform store is unreachable (2026-10-02, WeOwnDev/weown-fleet#36). A new droplet's private /32 is not in the store firewall's 8200 allowlist until an operator adds it. The host unwrap then failed with an opaque non-zero rc (a first openbao deploy, three runs, 2026-09-07). Every new tenant has defaulted to openbao since 09-04. A pre-check now curls <bao_addr>/v1/sys/health with the instance CA before the unwrap and fails naming the /32 allowlist (platform_api_source_cidrs). The firewall change itself stays with the operator. Verification: scripts/test-bao-reach-check.sh runs the two tasks from the rendered playbook against a refused port, a reachable HTTPS host and an unusable CA (3/3); with the stop condition disabled, 2 of the 3 fail.

  • Tombstoned two dead renders (2026-10-02, WeOwnDev/weown-fleet#112). wordpress-docker/sites/burnedout-xyz (droplet off) and anythingllm-docker/sites/dev-weown-anythingllm (no droplet) get RETIRED-DO-NOT-DEPLOY.md, and their scripts/deploy.sh exits 1. They are not rolled over to the post-Minimus registry.

  • wordpress-docker — the template renders again (2026-10-02). copier copy failed on main for two reasons. scripts/backup.sh.jinja had Go-template braces (docker inspect --format '{{range ...}}') outside {% raw %}. The domain_style and droplet_size choices were written value→label, where copier expects label→value, so the defaults failed validation. Stored answers keep the same values (apex, s-2vcpu-2gb-amd). Committed site renders are unaffected.

  • anythingllm-docker — dashboard review fixes (2026-09-30): booking button ink now picks the higher-contrast of dark/white, or black for mid-tones where neither reaches 4.5:1 (white on Harbor Gold was 2.42:1; now 7.38:1), and the preview matches; the Appearance and Booking cards track unsaved changes separately (including edits made while a save is in flight) so Copy stays blocked until both are saved; locationVerified is true only for a folder-qualified exact match; an oversize request body now gets a 413 with the limit instead of a dropped connection or a misleading 400. Tests: dashboard/test-embed-appearance.mjs, new dashboard/test-http.mjs, embed-filter/test.js.

  • Operator scripts and CI — failures that were reported as success (2026-09-30). From the same review sweep; each has a test run against main as the negative control.

    • dev/up.sh: only a 2xx counts as ready (a dashboard answering 503 was announced as up); an unexpected enable-multi-user reply now stops the run; --destroy also clears the saved ALLM_ADMIN_API_KEY, whose database it just deleted, so the next run mints a new one; an existing credentials file is re-tightened to 0600 before any mode runs, --down and --destroy included.
    • scripts/offboard-account.sh: a failed Keycloak disable printed KC_DONE and was reported as done; a failed session logout is now a manual step. Both Gitea paths refuse to PATCH unless the user GET returned that user, so an error body can no longer reset an SSO account's login_name/source_id to local.
    • anythingllm-docker/scripts/allm-prompt-sync.sh: no mode ever ran (${1:?…{discover|pull|diff|push}…} ended at the first }, so every mode string had the usage text appended); a failed pull truncated the committed prompt to empty; it now writes a temp file and moves it into place on success.
    • anythingllm-docker/scripts/allm-admin-account.sh, allm-set-role.sh, allm-user-rename.sh: mktemp -t name fails on GNU mktemp ("too few X's"); now an explicit XXXXXX template. allm-admin-account.sh stages its helper at a per-run mktemp path on the droplet instead of a fixed /tmp/allm-admin.js.
    • scripts/provision-openrouter-key.sh: the store-failure message printed OpenBao weown/platform/xplatform/x; it now names the real destination.
    • .github/workflows/validation.yml: the pre-commit cache key now includes the Python version (setup-python, pinned) and a pinned pre-commit version, in both the key and the restore prefix. (The Django-comment guard fix from this sweep was superseded on merge by #269's stricter version, recorded in its billing-docker entry.)
    • anythingllm-docker/sites/{ai.weown.agency,dev-weown-anythingllm,s004.ccc.bot}: added the .python-version pin (3.12.12) that only beta-weown-chat had, so ansible-playbook resolves under pyenv with a system global.

  • Docs (2026-09-30): runbook receipt no longer prints the operator's Infisical token; affiliate runbook says create_affiliate never sets active; gitea-docker README and help text say admin SSH is ssh_source_cidrs on admin_ssh_port (default open to all); anythingllm README uses an <openbao-checkout> placeholder; dev/README.md gives the dashboard login email; the client-intake design doc states the embedding routes and provider-routing control for R1/R2, the event hand-off, token transport, scan order, quarantine key model and client binding, and CUSTOMER_INSTANCE_PROVISIONING.md marks its ZDR claim pending verification.

  • billing-docker — sweep of review findings on merged PRs #236, #239, #240, #242 and #252 (2026-09-30). prune_demo_data --restore no longer re-activates a code that has referred customers, and its "(none)" line now prints. The template gives provisioning_watch state its own volume (<project>_ops_state) instead of the staticfiles one, and the app image creates /app/state owned by appuser, so a fresh volume is writable. The provisioning-watch cron log, whose ALERT lines name customer e-mails, gets a logrotate rule (daily, 14 kept), is created or corrected as root:root 0640 on every deploy, and is written under umask 027. .github/workflows/validation.yml: the Django template-comment check now tests every comment opener on a line (PCRE lookahead), and a failed scan (grep exit >1) fails the job instead of reading as clean. ADR-007 gets its #WeOwnVer version line. The volume split is not applied to the live sites/billing render yet: there both keys resolve to the same volume, so renaming needs a planned migration (see billing-docker/sites/billing/CHANGELOG.md).

  • anythingllm-docker — smoke test check 3.2 now tests AnythingLLM's real health endpoint (2026-09-30): /api/ping inside the container, PASS only on "online":true, SSH failure reported as NOT CHECKED. It used to pass on any body from a SigNoz-only path. In-container probes use docker exec by compose label (a bare docker compose exec can't parse the compose file outside infisical run).

  • anythingllm-docker — backups skipped the dashboard's state volume (2026-09-29). Embed allowlist domains, document delete-locks and embed appearance/logos live in <project>_dashboard_state, which backup.sh/restore.sh never archived. Both now include it (existence-guarded on backup, archive-guarded on restore); template + sites/beta-weown-chat re-synced. Follow-up 2026-09-30: restore never touches the dashboard volume it can't stop (and still attempts to restart it), always restarts AnythingLLM, and exits 1 on any dashboard error; the backup quiesces the dashboard around tar so the volume is one consistent snapshot. A failed stop no longer aborts the backup before the dashboard restart.

  • billing-docker/sites/billing — the committed render was 3 migrations behind the template, and PRODUCTION was running it (2026-08-26) — verified against the live service: billing.weown.dev/agreement/ returned 404 while /instances/new/ returned 302, so the deployed app was the stale render. Absent from production as a result: the customer-agreement gate on checkout (the template had closed the second door with "a gate on only one door is not a gate"), the free-trial lifecycle (TRIALING, trial_end), transactional lifecycle email (core/mail.py + templates), customer_agreement.html, tests.py, and migrations 0008–0010. The stale render also still hardcoded $1,000/month on the checkout button — which the template had deliberately removed so one image can serve both the $5 real-money drill product and the $1,000 live product via STRIPE_PRICE_ID — meaning the $5 pilot could not have worked on what was deployed. sites/billing/app/ is now byte-identical to template/app/. docker/, site.conf and terraform/ were deliberately not touched: they carry the render's app_dir and volume-name identity, and a mismatch there brings the app up on fresh empty volumes. Verified against the site copy: 59 tests pass, makemigrations --check reports no changes, and a from-scratch migrate applies 0001→0011 cleanly. ⚠️ Code change only — the droplet keeps running the old image until a deploy.

  • buzz-docker — Windows Desktop DM/@mention toasts (2026-08-18) — Buzz on Windows false-denies notification permission at every start (block/buzz#2445), then auto-disables Desktop alerts so real DMs and channel tags never toast. A prior local CDP “repair” fired 3–4 fake startup toasts and still left the mention pipeline off. Standard per-machine workaround: buzz-docker/scripts/windows-fix-desktop-notifications.ps1 (silent Notification interceptor, one reload, no test toasts, rewires Start Menu/Desktop/Startup shortcuts). Runbook: docs/PRJ-432.2-buzz-deployment.md.

  • buzz-docker — Desktop Mobile pairing 404 (NIP-AB pairing relay) (2026-08-12) — membership relays advertise NIP-43 but, without pairing_relay_url, Desktop v0.5.9 falls back to wss://<host>/pair on the main relay (no handler → HTTP 404). Mirror now ships a pairing-relay sidecar (buzz-pair-relay), Caddy handle_path /pair*, required BUZZ_PAIRING_RELAY_URL, plus scripts/enable-pairing-relay.sh for live hosts (recreates only pairing/caddy/relay; leaves Postgres/Redis/MinIO up). Runbook: docs/PRJ-432.2-buzz-deployment.md.

  • searxng-docker — image pin + Valkey cap_drop (2026-08-11) — bump static docker-compose.yaml / copier searxng_image default to searxng/searxng:2026.8.10-0a118066d (prior 2026.5.13-eb12c9b 404s on Docker Hub). Drop cap_drop: ALL on the Valkey service — its entrypoint setpriv/setresuid exits 127 under a full capability drop.

  • auto-pr-to-main.yml — present-but-dead WEOWN_BOT_PAT no longer reds every push (2026-08-11) — empty-PAT skip was already on main, but a present expired token still failed actions/checkout with could not read Username. Gate now probes GET /user and skips (exit 0) unless HTTP 200. Rotate WEOWN_BOT_PAT to restore auto-PR as weown-bot.

  • auto-pr-to-main.yml — invalid job-level secrets gate (2026-08-11) — if: ${{ secrets.WEOWN_BOT_PAT != '' }} is not legal on jobs.*.if (secrets context unavailable there), so every feature-branch push failed at workflow setup with "Invalid workflow file …#L1" / "No jobs were run". Gate moved into a step (env: + GITHUB_OUTPUT), remaining steps skipped, job exits 0 when the PAT is absent. Rotate WEOWN_BOT_PAT to restore auto-PR authorship as weown-bot.

  • buzz-docker — Copilot review: pin image + harden compose (2026-08-11) — require immutable BUZZ_IMAGE (digest/tag; reject :main/:latest) in compose, .env.example, bootstrap, and runbook; pin upstream git SHA in docs; add cap_drop: ALL + no-new-privileges (Caddy gets NET_BIND_SERVICE); bootstrap uses a local venv instead of pip --break-system-packages.

  • anythingllm-docker dashboard — assistant replies rendered raw markdown syntax instead of formatted text, and new conversations could silently overwrite an existing thread instead of stacking (2026-08-01) — two customer-reported dashboard bugs. (1) buildMsg()'s message bubble used textContent for every reply, so assistant answers containing **bold**, lists, or code fences showed the literal markdown source in the chat instead of formatted text. Added a markdown-lite renderer (mdBlocks/mdInline/renderMarkdown in public/index.html) covering bold/italic, inline code, fenced code blocks, headings, bullet/numbered lists, blockquotes, tables, links and rules — built entirely with createElement/textContent (never innerHTML), preserving the redesign's safe-DOM invariant; user-sent messages are unchanged (still plain text). Mobile type was also bumped and given real breakpoints (860px/480px/360px) instead of one fixed size, and every text input's font-size is pinned to 16px under 860px so iOS Safari no longer auto-zooms the page on input focus. (2) POST /api/threads (server.js) let AnythingLLM derive the new thread's slug from name, which is just the first message's text via deriveTitle(); two conversations opening with the same or a similarly-worded first message could then generate the same slug, so "New conversation" would resolve to the SAME existing thread instead of creating a new one — silently merging one conversation's history into another rather than stacking a new sidebar entry. Fixed by generating an explicit unique slug (t-<timestamp36>-<8 hex chars>) server-side for every new thread, independent of the message text. Verification: node --check on server.js + the extracted inline script; the markdown renderer was exercised in a real browser against bold/list/table/blockquote/code-block/link input at 800px, 375px and 320px viewports, confirming correct parsing and no page-body horizontal scroll at any width. No live AnythingLLM instance was available to reproduce the thread-slug collision directly, so item (2) is a code-level hardening fix for the exact mechanism that produces the reported symptom, not a live-confirmed root-cause fix — worth a live re-test before closing this out.

  • Template consolidation from the Keycloak/Gitea production rebuild (2026-07-24) — every defect found while driving keycloak-docker and gitea-docker to production ported back to template level so future renders deploy first-try: Caddyfile empty tls {} (does not parse — Caddy never listened on 80/443) + dead admin vhost removed; host CA bundle mounted for the bind-mounted Infisical CLI (both app containers crash-looped without it) with a neutral mount path so image truststores are never shadowed; Caddy log-dir bind mount; Keycloak 26 migration (image 26.7.0, KC_PROXY_HEADERS, health on the :9000 management port, tmpfs resource cache, container healthcheck removed); Gitea command: s6 path (/usr/bin/s6-svscan); headless infisical login --plain </dev/null everywhere a droplet logs in (bare login hangs on keyring); backup/restore POSTGRES_* names + canonical weown-prod-backups bucket; health-probe retries 20→40 for first-boot schema migration; entrypoint wrapper 0755 for non-root hardened images; gitea smoke test probes git-over-SSH via hostname to catch proxied DNS. Fleet-wide: executable bit restored on 44 *.sh.jinja/init.sh.jinja across 10 further templates (copier renders the source file's mode — a 644 template renders an unrunnable script); anythingllm-docker deliberately untouched (owned by its own workstream). Verification: both templates re-rendered; keycloak render is byte-identical (excluding comments) to the live sso.weown.id site files; compose YAML parses; bash -n clean.

  • anythingllm-docker dashboard — /app/healthz answered 401, so every deploy reported a scary false warning (2026-07-22) — the dashboard's public health route was checked before the base path was stripped, so it only ever matched the container-internal /healthz; Caddy routes only /app* to the dashboard, which meant the public probe fell through to the auth gate. Added the health check after base-path stripping too, leaving the container healthcheck's /healthz working unchanged.

  • anythingllm-docker terraform — ssh_key_fingerprint(s) name/type regression broke every fresh provision (2026-07-15) — the 12-PR consolidation (e8496d1) pluralized the template var to ssh_key_fingerprints / list(string), but weown-tofu provides the singular TF_VAR_ssh_key_fingerprint (one shared DO bootstrap key; team access is post-boot via OPS_AUTHORIZED_KEYS), and the live sites (s004) use the singular string. Result: tofu plan failed with No value for required variable ssh_key_fingerprints. Reverted the template (variables.tf, main.tf → ssh_keys = [var.ssh_key_fingerprint], terraform.tfvars.example) to the singular convention that matches live + weown-tofu. openclaw-docker intentionally keeps the plural (different folder-based weown-tofu structure; its var is designed as a dev-key list). Another consolidation regression never live-tested until the first weown-fleet provisioning run. Verification: rendered terraform parses (tofu fmt); no ssh_key_fingerprints refs remain in the template.

  • itofu.sh — M2 regression: plan/apply/output failed on the S3 backend (2026-07-15) — closes handoff H1 (the M2 argv→env change was static-verified only; copier wasn't installed in the agent env, so the tofu path was never live-tested). The M2 change added export AWS_ACCESS_KEY_ID/SECRET (from TF_VAR_spaces_access_key/secret) to the init case so the DO Spaces S3 backend could authenticate — but not to plan, apply, or the passthrough (output/destroy) cases. Those creds are deliberately not persisted in .terraform, so every state op needs them in env; tofu plan therefore failed with No valid credential sources found ... no EC2 IMDS role. Fixed by routing all non-init tofu calls through a trun() helper that exports the AWS_* creds (sse_customer_key stays init-only — it is persisted). Applied to both templates carrying this wrapper: anythingllm-docker and openclaw-docker. Surfaced by the first live provisioning run of the fleet orchestrator (weown-fleet). Verification: rendered bash -n + shellcheck -S error clean on both.

  • deploy-new-site.sh — automated AnythingLLM deploys now bootable (2026-07-15) — the script pushed JWT_SECRET/ADMIN_EMAIL/Spaces creds but none of the other fail-loud-required AnythingLLM values, so an automated deploy produced a container that refused to boot (the manual DEPLOYMENT_GUIDE.md flow was unaffected). New anythingllm-docker branch in Phase 2 pushes ANYTHINGLLM_IMAGE (from env — the private registry namespace is never committed), EMBEDDING_ENGINE (default openrouter) + optional EMBEDDING_MODEL_PREF, and mints the per-customer capped OPENROUTER_API_KEY via scripts/provision-openrouter-key.sh (cap override OPENROUTER_LIMIT_USD, default $50); loud warnings when a value can't be set automatically. Also adds a "Compliance & data handling" facts table to docs/CUSTOMER_INSTANCE_PROVISIONING.md (tenancy, locality, encryption, retention, subprocessors, framework mapping) backing the customer-facing terms. Verification: bash -n + shellcheck clean.

  • otel-agent/config.yaml — syslog parser matched zero Ubuntu 24.04 lines; agent flooded SigNoz with its own error spam (2026-06-10, #WeOwnVer v4.1.2.2) — the filelog/syslog receiver's regex_parser assumed RFC3164 timestamps (Jun 10 07:00:33), but Ubuntu 24.04 rsyslog writes RFC3339 (2026-06-10T07:00:33.179378+00:00 host unit[pid]: msg), so every syslog/auth.log line failed to parse and the collector logged 2 ERROR lines + Go stack traces per input line (84–504/min on int-s004; ~24% of that host's log rows in the 2026-06-10 incident export were the agent's own stanza v0.114.0 stack traces — transformer.go:100 / reader.go:214). Reproduced and fixed against otelcol-contrib:0.114.0 (the deployed version) using real incident log lines: new regex parses the three real shapes (unit[pid]:, kernel: no-pid, (udev-worker)[pid]:) with gotime RFC3339 timestamp parsing (microsecond-accurate; repro: 9/9 lines parsed + 2 garbage lines passed through, 0 error lines, was 18), plus on_error: send_quiet so any future unmatched line passes through as raw body logged at DEBUG — the agent can never ERROR-spam per line again. Also hardened the previously-ineffective filter/skip_self (its container.name/log.file.name conditions never matched docker's <container-id>-json.log paths, which is how the self-spam reached SigNoz): added a body-signature condition matching the collector's own tab-separated zap console format, starving the feedback loop even during future agent error bursts (e.g. exporter outages). otelcol validate passes on 0.114.0. Rollout: scripts/deploy-otel-fleet.sh (canary int-s004-anythingllm → --tag weown-ai). Compliance: NIST CSF 2.0 DE.CM-1 (signal fidelity); CIS v8 8.5, 8.11.

  • Propagated the INT-S004 / anythingllm-docker deploy-bug battery across all *-docker copier templates (2026-06-03) — the AnythingLLM rebuild surfaced a set of latent first-boot/deploy bugs (the entries below) that the other templates inherited from the same authoring patterns. Scanned every template (and rendered site) and applied only the bug classes actually present in each. cloud-init + deploy + scripts were re-validated per template: grep -cP '[^\x00-\x7F]' = 0 on every cloud-init, copier render (5/6; wordpress-docker has a pre-existing copier-choices render incompatibility, and searxng-docker needed a one-line {% raw %} wrap on a docker ps --format '{{.Names}}' Go-template that previously blocked rendering — same pattern signoz-docker already used), simulated-templatefile() YAML parse, bash -n on all extracted/rendered shell, and pre-commit (gitleaks/yamllint/shellcheck/markdownlint) all green.

    • keycloak-docker — B1 non-ASCII scrub of cloud-init; B3 kebab-case infisical login --client-id/--client-secret (cloud-init infisical-auth.sh + cron + the inline-SSH remote restore in scripts/restore.sh); B4 export INFISICAL_TOKEN="$(infisical login … --plain --silent)" headless capture (auth helper now sourced into the compose-up shell; the redundant standalone auth runcmd dropped); B5 dropped awscli from apt packages: + added unzip and the official AWS CLI v2 installer in runcmd; B6 pre-create /var/log/caddy (0777) in runcmd before compose-up. (B2 N/A — its embedded backup.sh already used single-$; the only $$ is the legitimate Compose $$? healthcheck escape. B7/B8 N/A.)
    • sandbox-docker — B1 ASCII scrub; B2 $$VAR/$$(…)→$VAR/$(…) de-escape in the embedded rotate-bootstrap-secret.sh (kept $${…}); B3+B4 in ansible/deploy.yml (×3 login sites) + scripts/backup.sh + scripts/restore.sh; B5 awscli→AWS CLI v2; B6 /var/log/caddy ansible file: task before compose-up; B8 published 127.0.0.1:8080:8080 (loopback-only) on the sandbox service so the deploy's host-side uri: health probe (http://127.0.0.1:8080/v1/docs) can connect — caddy still proxies the public over the internal network. (B7 N/A — sandbox_image is the public ghcr.io/agent-infra/sandbox.)
    • searxng-docker — B1 ASCII scrub; B2 de-escape in the embedded backup.sh (kept $${…} for Compose/volume refs); B3+B4 in cloud-init infisical-auth.sh + cron + scripts/restore.sh; B5 awscli→AWS CLI v2; B6 /var/log/caddy (0777) in runcmd before compose-up. (B7/B8 N/A — public searxng/valkey images; in-container healthcheck only.)
    • signoz-docker — B1 ASCII scrub; B2 de-escape in the embedded backup.sh; B3+B4 in cloud-init infisical-auth.sh + cron; B5 awscli→AWS CLI v2; B6 /var/log/caddy (0777) in runcmd before compose-up. (B7/B8 N/A — public signoz/* images.)
    • wordpress-docker — B1 ASCII scrub (comments + the two MYSQL_*:?… fail-loud messages); B3+B4 in the cloud-init infisical-auth.sh heredoc (already sourced by the boot + cron infisical run). (B2 N/A — the only $$ are the legitimate Compose escapes $$_SERVER/$${WP_IMAGE} etc. B5/B6 N/A — local-only backups, no aws; caddy does no file logging. B7 N/A — the reg.mini.dev/1923/wordpress-fluentsmtp ref is the already-accepted, repo-wide pattern, not introduced here. B8 N/A.)
    • openclaw-docker (+ sites/claw-weown-tools) — B1 ASCII scrub; B2 de-escape in the embedded rotate-bootstrap-secret.sh; B3+B4 in ansible/deploy.yml (×3) + scripts/backup.sh + scripts/restore.sh; B5 awscli→AWS CLI v2; B6 /var/log/caddy ansible file: task before compose-up. The same fixes were applied to the rendered claw-weown-tools site so the committed artifact matches the hardened template; this is a repo edit only — the live droplet would need a redeploy (ansible-playbook deploy.yml) and, for the cloud-init changes, a tofu apply -replace to pick them up. No droplet was recreated, replaced, or SSH'd. (B7 N/A — openclaw_image default is top-level reg.mini.dev/openclaw, same category as the retained reg.mini.dev/caddy:2 copier-var pattern; B8 N/A — the host probe targets caddy's published :80.)
    • Bug 7 (private-registry image hardcoded) applied to NO target template — none of the six hardcode a private-namespaced app image with an invalid pin the way anythingllm-docker did; app images are either public (ghcr.io, docker.io, signoz/*) or top-level reg.mini.dev/<image> (the same shape as caddy_image, which the canonical fix deliberately kept as a copier var). wordpress-docker's namespaced reg.mini.dev/1923/… ref is already committed repo-wide and out of scope for this hardening pass. devbox-docker is not present in this repo (no such template or sites/weown-devbox), so it could not be assessed.

  • Post-deploy follow-ups: caddy log-dir perms + ANYTHINGLLM_IMAGE bootstrap (2026-06-02) — (1) the hardened (non-root) caddy:2 crash-looped with permission denied writing /var/log/caddy/<site>.log because nothing created that host bind-mount dir writable; ansible/deploy.yml now ensures /var/log/caddy (mode 0777) before docker compose up (template + s004.ccc.bot + ai.weown.agency). (2) scripts/bootstrap-s004-infisical.sh now prompts for the now-required ANYTHINGLLM_IMAGE, and the s004.ccc.bot README documents it, so a rebuild repopulates the image ref in Infisical.

  • AnythingLLM published no host port → deploy health probe + runbook tunnel could never connect (2026-06-02) — the anythingllm service had no ports: (caddy proxies it over the internal Docker network), yet ansible/deploy.yml's health task polls http://127.0.0.1:3001/api/ping on the host and the runbook's Phase-4 validation uses ssh -L 3001:127.0.0.1:3001 — both assume 3001 is published on the droplet loopback. The container was healthy the whole time (docker inspect health=healthy, internal ping {"online":true}); the probe was checking an unpublished port and failed with Connection refused. Added ports: ["127.0.0.1:3001:3001"] (loopback-only — not internet-exposed; caddy still serves the public via the internal network) to the template + s004.ccc.bot + ai.weown.agency.

  • AnythingLLM image pin 1.7.2 was invalid + hardcoded a private registry path (2026-06-02) — the compose pinned reg.mini.dev/anythingllm:1.7.2, but that tag was never in the Minimus registry (the source box actually ran :latest, and Minimus rotates tags + had moved the image to a private namespaced path), so the deploy 404'd on image pull. The image ref is now injected from Infisical (ANYTHINGLLM_IMAGE, e.g. reg.mini.dev/<ns>/anythingllm:v1.12.1): compose reads ${ANYTHINGLLM_IMAGE:?…} (fail-loud) and docker compose up pulls it under infisical run, so the private namespace never enters this public repo and each site pins its own version. The deploy's image-pull task now pulls only the static caddy ref. Applied to the template + s004.ccc.bot + ai.weown.agency; render parity + compose interpolation verified.

  • Infisical machine-identity auth hung on headless boxes — infisical run fell back to an interactive login prompt (2026-06-02) — ansible/deploy.yml, scripts/backup.sh, and scripts/restore.sh ran infisical login --method=universal-auth … followed by a bare infisical run …, but a universal-auth (machine identity) login does not leave a session that infisical run reuses. run therefore saw no auth and dropped to the interactive Select hosting option / Email / Password prompt, which hangs forever on a no-TTY ansible/cron run (Login via browser failed. inappropriate ioctl for device). Fixed by capturing the token — export INFISICAL_TOKEN="$(infisical login … --plain --silent)" — which infisical run reads directly (the documented headless/CI pattern). Applied to the template + s004.ccc.bot + ai.weown.agency; render parity and bash -n verified.

  • Infisical CLI login used camelCase flags the current CLI rejects (2026-06-02) — ansible/deploy.yml, scripts/backup.sh, and scripts/restore.sh (template + s004.ccc.bot + ai.weown.agency) called infisical login --method=universal-auth --clientId … --clientSecret …, but the CLI installed from artifacts-cli.infisical.com requires kebab-case and fails with unknown flag: --clientId — breaking the deploy's docker compose up task, the daily backup cron, and remote restore. Renamed to --client-id/--client-secret. (infisical run --projectId/--env and the cloud-init rotation's API JSON clientId/clientSecret keys are correct as-is and unchanged.)

  • cloud-init $$ over-escaping broke Layer-2 rotation; awscli apt-install errored the bootstrap (2026-06-02) — two more anythingllm-docker cloud-init bugs found bootstrapping INT-S004. (1) The embedded rotate-bootstrap-secret.sh used $$VAR/$$(…), assuming Terraform templatefile() collapses $$→$; it does not (only $${→${), so the literal $$ became the bash PID and the rotation silently failed on every deploy (it is best-effort/exit-0, so it went unnoticed) — the v1 bootstrap secret was never rotated. Fixed $$VAR→$VAR / $$(…)→$(…), keeping the legitimate $${…} escapes. (2) awscli in the apt packages: list fails to install on current Ubuntu, which set cloud-init status: error and skipped the rest of runcmd; replaced with the official AWS CLI v2 installer in runcmd (best-effort, non-fatal) and added unzip. Applied to the template + both sites (s004.ccc.bot, ai.weown.agency); rendered YAML validated and the rotation script passes bash -n after render. Also corrected the now-known-wrong $$ guidance in CLAUDE.md.

  • cloud-init silently skipped its entire bootstrap on non-ASCII characters (2026-06-02) — the anythingllm-docker cloud-init template and both rendered sites (s004.ccc.bot, ai.weown.agency) had em-dashes (—), arrows (→), and a section sign (§) in comments and log strings. The em-dash's UTF-8 middle byte is 0x80, a C1 control char that DigitalOcean's ConfigDrive → cloud-init YAML 1.1 parser rejects (unacceptable character #x0080), so the whole cloud-config was discarded as "empty cloud config" — Docker, the Infisical CLI, .infisical-auth.env, and the Layer-2 secret rotation never ran (cloud-init status = degraded, /opt/<project>/ absent). Scrubbed all three cloud-init files to pure ASCII (—→-, →→->, §→section); the template re-renders byte-identical to the site. cloud-init user-data must be ASCII-only. Caught bootstrapping the INT-S004 rebuild.

  • itofu.sh + restore.sh header inconsistencies (2026-06-02) — itofu.sh (template + s004.ccc.bot) now lists TF_VAR_alert_email (a DO-verified email) in its "Required Infisical secrets" header, with the TF_VAR_enable_monitoring=false opt-out, matching the runbook/tfvars and closing the email is not verified apply footgun. restore.sh (template + site) no longer advertises an s3://… path argument it would actually reject — the header now documents that the input is the backup name only (validated ^[A-Za-z0-9._-]+$) and that the tarball is auto-fetched from DO Spaces when absent locally. Both scripts were also scrubbed to pure ASCII. Caught by Copilot review on PR #39.

  • OPS_AUTHORIZED_KEYS sync could revoke all team SSH access on a blank value (2026-06-02) — the ansible SSH-key sync task guarded only against an unset/empty OPS_AUTHORIZED_KEYS ([ -z ]), but a whitespace-only value passed that check and then had every line stripped by the blank-line filter, producing an empty temp file that install would write over authorized_keys.weown-ops — wiping all team keys and locking ops out (the DO break-glass key, managed separately, would survive). Reordered to filter first and treat "no non-blank lines" the same as unset (warn + exit without touching the destination). Fixed in the template (ansible/deploy.yml.jinja) and the s004.ccc.bot site; verified with a harness (unset/empty/whitespace-only all preserve the existing file; real keys still sync). Caught by Copilot review on PR #39.

  • TF_VAR_alert_email missing from the weown-tofu setup caused a tofu apply failure (2026-06-02) — the s004.ccc.bot runbook's weown-tofu TF_VAR_* table omitted alert_email, so var.alert_email fell back to its placeholder default alerts@example.com and DigitalOcean rejected the CPU/mem/disk monitor-alert creation with 400 … email is not verified (the rest of the apply — droplet, reserved IP, firewall — succeeded). Added TF_VAR_alert_email to the runbook table with a note that it must be a DO-verified email (plus a TF_VAR_enable_monitoring=false opt-out), and added the same ⚠️ warning to terraform.tfvars.example in both the s004.ccc.bot site and the template .jinja.

  • OPENROUTER_API_KEY never injected into the AnythingLLM container across anythingllm-docker (2026-06-02) — the compose files documented OPENROUTER_API_KEY as a required Infisical secret, but the environment: block only injected JWT_SECRET and ADMIN_EMAIL (the original Helm chart mounted all three). A redeploy/restore therefore came up with no OpenRouter credential in the container env — and for the INT-S004 restore specifically, AnythingLLM would fall back to the expired key persisted in the restored storage/.env, breaking inference with an opaque OpenRouter 401. Added a fail-loud ${OPENROUTER_API_KEY:?...} injection (same pattern as the JWT_SECRET guard) to the template (compose.prod.yaml.jinja) and both live sites (s004.ccc.bot, ai.weown.agency); a value injected via infisical run now correctly overrides the stale persisted key. Also gave OPENROUTER_MODEL_PREF a default (${OPENROUTER_MODEL_PREF:-anthropic/claude-opus-4.5} in the sites, {{ openrouter_model_pref }} in the template) to silence the Compose "variable is not set" warning. Caught by Copilot review on PR #39.

  • otel-agent/deploy.yml broken Jinja escaping (2026-05-23) — the file is a plain Ansible playbook (not a copier template), but it had been written with {{ '{{' }} otel_agent_dir {{ '}}' }} patterns that would render as the literal string {{ otel_agent_dir }} and prevent Ansible variable substitution. Replaced with plain {{ otel_agent_dir }} throughout.

  • signoz-docker/ copier template (2026-05-14, Added) — new copier template for self-hosted SigNoz observability platform (logs + metrics + traces via ClickHouse). 19 files: Docker Compose (6-service stack: ZooKeeper, ClickHouse, SigNoz, OTel Gateway, Caddy), OpenTofu IaC (droplet, reserved IP, firewall with OTLP ports VPC-only), cloud-init bootstrap, Ansible deploy playbook, skinny backup/restore scripts, monitoring alerts. Compliance: NIST DE.CM, CIS 8.2, ISO A.8.15. Status as of 2026-05-23: see Deprecated section above — kept as optional fallback only.

  • searxng-docker/ copier template (2026-05-14, Added) — restructured SearXNG deployment (previously plain Ansible on feature/searxng-deployment branch) into full copier template matching keycloak-docker patterns. 20 files: Docker Compose (SearXNG + Valkey + Caddy), OpenTofu IaC, Infisical secrets management, skinny backups, monitoring alerts, browser-search integration playbook (Chrome/Edge/Firefox on Linux + Windows). Resolves PR #20 alignment gaps.

  • otel-agent/ fleet deployment (2026-05-14, Added) — OpenTelemetry Collector agent deployed per-host to /opt/otel-agent/. Collects: container metrics (docker_stats), host metrics (CPU, memory, disk I/O, filesystem, load), container logs, Caddy access logs, syslog/auth.log. Exports via OTLP gRPC. Original 2026-05-14 design targeted an in-VPC SigNoz gateway; superseded 2026-05-23 by SigNoz Cloud destination — see Changed section above.

  • scripts/deploy-otel-fleet.sh (2026-05-14, Added) — fleet deployment script. Original 2026-05-14 design took <signoz-private-ip> arg; superseded 2026-05-23 by Infisical-driven flagged interface — see Changed section above.

  • scripts/enable-do-agent.sh (2026-05-14, Added) — idempotent script to install free DigitalOcean extended metrics agent (do-agent) on all tagged droplets. Provides memory, disk, and load metrics in DO dashboard at zero cost.

  • anythingllm-docker/ansible/configure-allm.yml (2026-05-14, Added; moved 2026-05-25) — idempotent Ansible playbook for Docker-based AnythingLLM instances. Configures MCP servers (SearXNG at searxng.weown.app) via docker exec + jq deep merge. Originally landed under anythingllm/ansible/ but moved alongside the other Docker-stack assets in anythingllm-docker/ since anythingllm/ is reserved for the helm/k8s deployment path. Gemini-style broken Jinja escapes were fixed in PR #26 cleanup.

  • CLAUDE.md (2026-05-14, Added) — project-level Claude Code guide. References .github/copilot-instructions.md for compliance standards, documents copier template patterns, branch naming convention, Infisical secrets model, and fleet management scripts.

  • .github/CODEOWNERS contributor handoff (2026-05-13, Changed) — @romandidomizio removed from all path assignments (left 2026-05-15); @dhruvmalik007 references removed (no longer with team). @ncimino is now the sole assigned reviewer on every path. @iamwaseem18 (Mohammed) and @mshahid538 (Shahid) noted as available at @ncimino's discretion. All TODO(2026-05-15) handoff comments removed. Header updated to reflect 1-approval requirement and Roman as previous steward.

  • Required reviewers changed from 2 → 1 (2026-05-13) — updated across .github/CODEOWNERS header, CONTRIBUTING.md §6.4 merge requirements, README.md Code Review Model, .github/ADR-004-copilot-auto-review-ruleset.md CIS Controls v8 18.3 compliance row + Layer 1 pruning criteria reviewer requirement, .github/workflows/auto-pr-to-main.yml PR body "Required Reviewers" text, .github/workflows/README.md §11 troubleshooting row.

  • CONTRIBUTING.md §1 prerequisites + §10 Questions updated (2026-05-13) — removed @romandidomizio and @dhruvmalik007 references; org-access contact updated to @ncimino; process/governance contact updated to @ncimino + CODEOWNERS link.

  • .github/workflows/auto-pr-to-main.yml reviewer assignment updated (2026-05-13) — --add-reviewer ncimino,romandidomizio → --add-reviewer ncimino (both existing PR update + new PR create paths). TODO(2026-05-15) comments removed.

  • .github/ADR-004-copilot-auto-review-ruleset.md ruleset structure amended (2026-05-01) — deletion rule removed from ~ALL branches in both Layer 1 (repo-level) and Layer 2 (enterprise-level) rulesets. Retained 2 rules: non_fast_forward + copilot_code_review. Rationale: post-merge branch deletion was permanently blocked by deletion on ~ALL with empty bypass_actors — no one (including org admins) could delete any branch. main branch deletion protection remains via ADR-003 rule #11. Decision Log appended. Compliance mappings adjusted (SOC 2 CC6.1, ISO 27001 A.8.13).

  • CONTRIBUTING.md §8 troubleshooting expanded (2026-05-01) — added "My PR is blocked — commits are unsigned" scenario. Removed Option 1 (rebase + force-push) because non_fast_forward ruleset on ~ALL branches blocks force-push. Retained Option 2: close PR, cherry-pick to fresh branch with git cherry-pick -S, open new PR. Also added "Copilot review didn't start on my auto-created PR" troubleshooting entry documenting the PR-creation-time caching behavior: Copilot evaluates auto-review at PR-creation time, not push time, so the first commit on a new branch won't trigger review until a subsequent push. All subsequent pushes on an open PR are reviewed automatically. Replaces the older single "I need to sign commits I already pushed unsigned" subsection.

  • Repo Settings → General → Pull Requests (2026-05-01): enabled "Automatically delete head branches" (merged branches auto-delete, replacing the retired deletion rule on ~ALL); enabled "Always suggest updating pull request branches" (keeps PRs current with main for cleaner merges).

  • runcmds: → runcmd: typo in all cloud-init templates (2026-05-14, Fixed) — cloud-init only recognizes runcmd: (singular). The typo runcmds: silently caused boot commands to never execute. Fixed in: keycloak-docker/template/terraform/templates/cloud-init.yaml.jinja, anythingllm-docker/template/terraform/templates/cloud-init.yaml.jinja, keycloak-docker/sites/sso.weown.dev/terraform/templates/cloud-init.yaml. New templates (signoz-docker, searxng-docker) use the correct form.

  • scripts/manage-droplets.sh SSH key rotation parsing bug (2026-05-14, Fixed) — rotate-authorized-keys used tr ' ' '\t' which mangled SSH key names containing spaces. Replaced with per-key-ID loop that fetches Name and PublicKey individually via doctl compute ssh-key get.

  • Copilot PR #17 round-2 comment fixes (2026-05-13, Fixed) — addressed remaining Copilot comments from the 2026-05-13 push:

    • workflows/README.md stale "2 reviewer" references (comment 3236477797): updated 5 remaining "2 reviewers/approvals" → "1 reviewer/approval" at §6 PAT rotation step 13, §8.1 rule #1 table row, §8.1 "Interaction with workflows", §9 step 4, §10 item #7.
    • workflows/README.md §9 Reviewer Rotation: updated step 1 to reflect handoff complete (2026-05-15, PR #17); removed @dhruvmalik007 from the done-placeholder list; --add-reviewer ncimino only.
    • workflows/README.md §10 Transition Checklist: marked items 1 (PAT stewardship), 4 (CODEOWNERS), 5 (workflow reviewer) as ✅ done; updated owners to reflect @ncimino and @YonksTEAM post-transition; removed @romandidomizio from action-owner column.
    • workflows/README.md §11 line 587 duplicate link (comment 3236447687): removed redundant second [ADR-004] link in the "No Copilot review on first commit" row; kept single [ADR-004 § Empirical Validation Results] link with correct anchor.
    • CODEOWNERS risk-acceptance note (comment 3236477912): added explicit RISK ACCEPTANCE block in the CODEOWNERS header documenting the deliberate 2-reviewer → 1-reviewer posture change, compensating controls (Copilot AI review + CodeQL), and trigger for revisiting.

  • Copilot PR #17 round-1 comment fixes (2026-05-13, Fixed) — addressed all open Copilot review comments on branch feature/roman-ruleset-cleanup-and-docs:

    • CONTRIBUTING.md §8 Cause B force-push (comments 3175275840 + 3175470086): removed git push --force-with-lease step; replaced with note that force-push is blocked by non_fast_forward and directs users to the close+recreate path already documented in the same §8 section.
    • ADR-004 title — removed "Deletion Protection" from title since the deletion rule was removed from ~ALL; title now accurately reflects the 2 active rules.
    • ADR-004 Verification Procedure expected output (comment 3175470115): updated Layer 1 expected rules array from ["deletion", "non_fast_forward", "copilot_code_review"] → ["non_fast_forward", "copilot_code_review"]; updated Layer 2 expected count from "identical 3 rules" → "identical 2 rules".
    • ADR-004 Decision Log (comment 3175275917): removed reference to gitignored PR7_HANDOFF_CHECKLIST.md in the round-7 close-out entry.
    • auto-pr-to-main.yml NOTE (comment 3175470133): rewrote NOTE to clarify the push-before-PR pattern specific to this workflow — commits are pushed before the PR exists, so review_on_push: true has no new push delta at PR-creation time; first Copilot review fires on the next push to the open PR. Reconciled with ADR-004 § Empirical Validation Results.
    • workflows/README.md §11 Copilot row (comment 3175470144): updated "No Copilot review on first commit of auto-created PR" row to clarify this is specific to the auto-pr-to-main.yml push-before-PR pattern (not all PRs), and that manually-created PRs do get Copilot at PR-creation time.

[v3.3.5.1] — 2026-04-27 to 2026-04-28

Continuation iteration of the in-flight PR #13 (originally tagged [v3.3.4.2] below). Bumped to v3.3.5.1 because this section captures the portion of the PR that moved into the 5th week of April under the active #WeOwnVer interpretation (vSEASON.MONTH.WEEK.ITERATION → Season 3, April, Week 5, Iteration 1). The original 2026-04-23 work + Copilot rounds 1–3 (2026-04-24) remain under [v3.3.4.2] below; the later continuation rounds documented here span 2026-04-27 (rounds 4–11) through 2026-04-28 (rounds 12–13). The PR itself remains a single squash-merge unit; the CHANGELOG split is purely about #WeOwnVer fidelity per the active versioning calendar. Version stays at v3.3.5.1 across the day boundary because the PR is a single semantic unit (no fresh iteration started post-merge).

Added

  • .github/ADR-004-copilot-auto-review-ruleset.md — new ADR (added 2026-04-27 in round-6 close-out, refined in round-7) documenting the two-layer defense-in-depth ruleset that targets ~ALL branches with three rules: deletion, non_fast_forward, and copilot_code_review. Layer 1: repo-level "Copilot auto-review" ruleset (id 12131972, configured 2026-04-23). Layer 2: enterprise-level ruleset configured 2026-04-27. Both layers enforce identical rules; the union-of-rules semantics provides defense-in-depth that survives enterprise migrations / SKU changes / individual misconfigurations and mirrors WeOwn's branch-name regex defense-in-depth pattern. Compliance mappings: SOC 2 CC6.1, CC7.1, CC7.2, CC8.1; ISO/IEC 27001:2022 A.8.32, A.8.13; ISO/IEC 42001:2023 A.6.2.7, A.6.2.8; NIST CSF 2.0 PR.IP-3, DE.CM-1; CIS Controls v8 18.3. Includes explicit pruning criteria for Layer 1 (do not delete unless ALL of: 5+ consecutive bot-authored PRs auto-trigger Copilot via Layer 2 alone + no enterprise migration planned + Layer 2 deletion+non_fast_forward verified by deliberate test + reviewer sign-off). Includes end-to-end auto-trigger validation procedure pointing at the ADR's own Decision Log table for in-repo logging (replaces a transient reference to a gitignored operational checklist; resolves Copilot R7 #5). Includes an Empirical Validation Results section documenting the round-7 controlled experiment finding: Layer 2 + Copilot Business entitlement DO work for newly-created PRs but Copilot auto-trigger is evaluated at PR-creation time, not push time — pre-existing PRs (created before the entitlement was provisioned) will never auto-trigger and the only remediation is close+re-open (usually not worth the cost).
  • .github/workflows/README.md §2A "What auto-pr-to-main.yml Does (Step-by-Step)" — new authoritative narrative walkthrough of the auto-PR workflow, with three subsections: Trigger matrix (push + workflow_dispatch + concurrency); Step-by-step walkthrough (banners 1–9 with what-it-does + failure-modes columns); Why the team benefits (SOC 2 CC8.1 evidence narrative, zero-friction onboarding, Copilot AI review quality, defense-in-depth pattern). Inserted between §2 and §3 to give workflow-level context before the attribution model details. TOC updated.
  • .github/workflows/README.md §11 "Troubleshooting — Symptom → Cause → Verification" — consolidated symptom/cause/verification table covering all three workflows (auto-pr-to-main.yml, branch-name-check.yml, pat-health-check.yml) plus Copilot auto-review, branch protection / rulesets, and Infisical sync drift. Old §11 ("Related Documents") renumbered to §12. TOC updated. Closes the long-standing v3.3.4.3 queue item to add troubleshooting documentation; accelerated into this PR per user direction.

Changed

  • .github/workflows/auto-pr-to-main.yml step 7 contributor aggregation — replaced the while read -r count handle pipeline with awk '{count=$1; $1=""; sub(/^[[:space:]]+/, "", $0); ...}'. The read-based approach split on whitespace and silently truncated multi-word names in the fallback path (e.g., Jane Doe rendered as - Jane (3 commits) instead of - Jane Doe (3 commits)). The awk-based approach preserves the entire remainder of the line after the count. Includes an inline comment explaining the rationale so future maintainers don't regress. Resolves Copilot R7 comment #8. (The fallback path triggers when gh api /commits/{sha} returns a null login — typical for unlinked external commits.)
  • auto-pr-to-main.yml step 6 attribution comment — softened the immutability claim from "first commit is immutable under the non_fast_forward ruleset" (under-specified) to "stable across pushes because the 'Copilot auto-review' ruleset (id 12131972, see ADR-004) enforces non_fast_forward on ~ALL branches in this repo, blocking the rebase / force-push that would change the first-commit identity". Resolves Copilot R6 comment A by making the enforcement chain auditable. Identical clarifications applied to workflows/README.md §3 Opened-by table row (Copilot R6 D) and CONTRIBUTING.md §4 Opened-by bullet (Copilot R6 C).
  • auto-pr-to-main.yml step 7 contributors fallback PII strip — changed git log -1 --format='%an <%ae>' to git log -1 --format='%an'. Names are already public on GitHub commit pages; emails are PII per GDPR/CCPA + WeOwn's existing PII-minimization posture (already applied to CONTRIBUTING.md handle table + CODEOWNERS). Resolves Copilot R6 comment E.
  • auto-pr-to-main.yml step 8 Recent Commits PII strip — changed git log --format='### %h %s%n%n**Author:** %an <%ae>%n**Date:** %ad...' to '**Author:** %an'. Same principle as step 7 fallback. Reviewers can still cross-reference via the commit SHA + GitHub web UI. Closes the only remaining email exposure path in the PR body. User-directed extension of round-6 PII work.
  • auto-pr-to-main.yml step 8 PR body parenthetical descriptions removed (user-directed UX polish): the Opened by:, Last pushed by:, and Contributors on this branch: lines previously included italic explanatory parentheticals that duplicated information already visible in CONTRIBUTING.md §4 + workflows/README.md §3. Trims visual noise without information loss.
  • auto-pr-to-main.yml workflow_dispatch: trigger added — enables manual re-run for debugging / PR-body refresh without needing an empty commit. The defense-in-depth branch-name regex (step 1) makes dispatch on main or unconventional branches a safe no-op (exit 0). Resolves the 2026-04-23 PAT-rotation scenario that required commit ba6d7ee (empty re-trigger commit) — going forward this is one click in the Actions tab.
  • auto-pr-to-main.yml concurrency: block added — group: auto-pr-${{ github.ref }}, cancel-in-progress: true. Prevents race conditions when developers rapid-push: the older in-flight run is cancelled and only the latest run creates / updates the PR. Per-branch isolation; pushes on different branches don't interfere with each other.
  • PR-body attribution upgrade — three-tier model: replaced the prior single **Triggered by:** @<user> line (semantically ambiguous — implied "opener" but actually showed most recent pusher) with three distinct, semantically-precise fields:
    • Opened by: — GitHub @handle of the FIRST commit's author on the branch. Resolved via git rev-list --reverse "${GIT_RANGE[@]}" | head -n 1 → gh api /repos/$GITHUB_REPOSITORY/commits/$FIRST_SHA --jq '.author.login'. Stable across pushes (see ADR-004 for the enforcement chain). Two fallbacks: (1) .committer.login if author.login is null, (2) LAST_PUSHED_BY if both fail.
    • Last pushed by: — ${{ github.triggering_actor || github.actor }}. Updates every push.
    • Contributors on this branch: — list now uses GitHub @handles with commit counts (- @romandidomizio (4 commits)). Per-commit gh api /repos/.../commits/{sha} --jq '.author.login // .committer.login // ""' resolution; fallback to commit-author NAME ONLY (no email) for unlinked external commits.
  • workflows/README.md §3 rewritten — (a) intro sentence corrected to clarify attribution comes from GitHub event context + commits API, not branch-name parsing (resolves Copilot R5 #1); (b) Parsing Rules retitled "Parsing Rules (branch name → regex validation only)" with 3 simplified steps; (c) new "PR body attribution — three-tier model (by design)" subsection with Field/Value/Source/Updates table; (d) "Branch name vs. PR body" table expanded from 2 rows to 4 (branch <dev> + Opened by: + Last pushed by: + Contributors on this branch:).
  • CONTRIBUTING.md §4 rewritten — (a) <dev> paragraph updated to clarify attribution uses commits API + event context, not branch parsing; (b) blockquote callout table expanded to 4 rows (matches workflows/README.md); (c) new explanatory bullets for each of the three PR-body fields with their resolution mechanism; (d) feature/add-thing trailing note updated; (e) round-5 Copilot comment #2 fix: attribution paragraph rewritten to drop misleading phrasing; (f) round-7 alignment fix: Contributors: label updated to Contributors on this branch: in both the table cell and the explanatory bullet to match the exact PR-body wording (resolves Copilot R7 #6).
  • workflows/README.md header version + Last updated: **Version**: v3.3.4.1 → v3.3.5.1, **Last updated**: 2026-04-23 → 2026-04-27. The original 2026-04-23 bump was to v3.3.4.2 (documented in [v3.3.4.2] below); this iteration's continuation work bumps it again to v3.3.5.1 per the user's #WeOwnVer week-5 interpretation. Resolves Copilot R7 comments #1 + #2 (workflows/README.md header inconsistency) and #7 (CONTRIBUTING.md header inconsistency); same bump applied to CONTRIBUTING.md (Last updated 2026-04-23 → 2026-04-27), .github/ADR-003-main-branch-ruleset.md (Date "2026-04-23" → "2026-04-23 (ruleset configured) / 2026-04-27 (ADR last revised)" — resolves Copilot R7 #3), and .github/ADR-004-copilot-auto-review-ruleset.md Decision Log + Empirical Validation Results section (resolves Copilot R7 #4).
  • .github/ADR-003-main-branch-ruleset.md Related section — added cross-reference to ADR-004 (under Related), making the two-ADR structure discoverable.
  • .github/ADR-002-infisical-github-sync.md naming convention revised (2026-04-28) — the original convention used WEOWN_BOT_PAT__<ORG>_<REPO> as the Infisical secret name with the assumption that the GitHub Sync UI exposed a per-secret rename feature at sync-config time. Empirical finding while configuring the sync on 2026-04-28: Infisical's "Key Schema" can ADD prefixes/suffixes around the {{secretKey}} template but cannot STRIP them, so the original convention is unworkable at sync-creation time. Revised convention: Infisical secret name = WEOWN_BOT_PAT (identity-mapped, no suffix) + namespace across repos via separate Infisical projects (weown-bot/<org>-<repo>), not secret-name suffixing. Updates: (a) Architecture diagram in ADR-002 redrawn to show project-per-scope; (b) Naming Convention section rewritten with explanation of why the original convention fails; (c) Implementation Notes "Initial setup" steps revised; (d) NEW Decision Log section appended with 2026-04-23 + 2026-04-28 rows documenting the convention evolution; (e) ADR-002 header Version bumped from v3.3.4.1 to v3.3.5.1, Date updated to 2026-04-23 (initial) / 2026-04-28 (naming convention revised — see Decision Log). Status remains "Accepted" — this is an implementation-detail revision, not a decision reversal. Cross-doc updates: workflows/README.md §2.4 Usage Table + §5.1 Onboarding steps 2 + 4 + §6 Rotation step 6 + new §6.1 Sync Options Configuration sub-section. The Sync Options table documents the four configuration parameters (Initial Sync Behavior = Overwrite [forced]; Key Schema = {{secretKey}} identity; Disable Secret Deletion = Yes; Auto-Sync Enabled = Yes) with rationale per row.
  • .github/workflows/README.md §6.1 NEW: Sync Options Configuration — step-by-step guidance for configuring an Infisical → GitHub Secret Sync. Covers the four UI options (Initial Sync Behavior, Key Schema, Disable Secret Deletion, Auto-Sync Enabled) plus a Migration Steps sub-section for the 2026-04-28 transition from the legacy WEOWN_BOT_PAT__WEOWNNETWORK_AI secret to the revised WEOWN_BOT_PAT identity-mapped secret in the new weown-bot/weownnetwork-ai Infisical project. Cross-references ADR-002 Decision Log.
  • .github/workflows/README.md §2.4 Usage Table refreshed — active row updated to reflect the 2026-04-28 PAT rotation: Expiration: 2026-07-22 → 2026-07-27 (90 calendar days from 2026-04-28); Last Rotated: 2026-04-23 → 2026-04-28; PAT Secret (Infisical) column: WEOWN_BOT_PAT__WEOWNNETWORK_AI → WEOWN_BOT_PAT (Infisical project: weown-bot/weownnetwork-ai). Placeholder rows for future repos updated to match the new project-per-scope naming.
  • .github/workflows/pat-health-check.yml example date refreshed — line 133 expected-format example string updated from '2026-07-22 17:13:00 UTC' to '2026-07-27 17:13:00 UTC' for currency. Behavior unchanged (the value is just an example in the diagnostic message; parsing logic is generic).

Fixed

  • .github/ADR-004-copilot-auto-review-ruleset.md validation step — the auto-trigger validation procedure previously instructed "record the run number + timestamp in PR7_HANDOFF_CHECKLIST.md" but that file is gitignored (operational notes; not in repo). Updated to point at this ADR's own Decision Log table for in-repo logging. Resolves Copilot R7 comment #5.

  • CHANGELOG drift on three-tier attribution wording — the v3.3.4.2 "Added" bullet for platform-sourced attribution (originally written round-1) still described the workflow as producing **Triggered by:** @<real-github-username>. Round-6 rewrote it to describe the three independent fields (Opened by:, Last pushed by:, Contributors on this branch:) currently emitted, with a parenthetical noting the round-5 evolution. Resolves Copilot R6 comment B.

  • Round-5 auto-trigger claim REVERTED (carried forward from round-6, refined in round-7): prior session incorrectly inferred from the 2026-04-27T21:59:30Z Copilot review timestamp on commit 34f28bf that auto-trigger had succeeded. User clarified the round-5 review was MANUALLY triggered. Memory + checklist + this changelog corrected. Round-7 controlled-experiment finding: a fresh PR opened on 2026-04-27 from an old branch DID auto-trigger Copilot review (Copilot AI review requested due to automatic review settings), validating the Layer 2 ruleset + Copilot Business entitlement chain, but PR #13 (created 2026-04-23) does NOT auto-trigger because Copilot's auto-review decision is made at PR-creation time. See ADR-004 § Empirical Validation Results.

  • Round-20 Copilot review fixes (2026-04-28, 2 Copilot R20 fixes — 1 correctness fix + 1 doc-vs-UI display canonicalization with cross-doc cascade):

    • R20 #1 / .github/workflows/README.md §11 Troubleshooting "Merge blocked with 'requires signed commits'" row — guidance was incorrect. Old text: "Configure commit signing per CONTRIBUTING.md §3; retroactive signing is impossible (would force-push) — add a new signed commit to the branch." This was wrong: adding a new signed commit does NOT fix earlier unsigned commits — the merge will still be blocked because the main branch protection requires ALL commits in the PR be signed (not just the most recent). Copilot R20 #1 caught the correctness bug. Fix (per Copilot's literal suggested change): rewrote the row to explain that adding a new signed commit does not fix earlier unsigned commits, and that because retroactive signing would rewrite history and non-fast-forward is blocked on ~ALL branches by Layer 1 + Layer 2 rulesets, the contributor must recreate the branch/PR with all commits signed (or otherwise ensure every commit in the PR is signed). Compliance basis: SOC 2 CC8.1 (incident-response runbooks must give correct remediation; misleading guidance during a merge-blocked incident wastes contributor time and may cause repeated failed remediation attempts); ISO/IEC 27001:2022 A.5.24 (runbook accuracy). Pattern: this is a runbook-correctness bug that survived 19 prior rounds of Copilot review because the row's narrative sounded plausible (the "force-push is blocked" framing was correct; the conclusion "add a new signed commit" was wrong). Lesson: troubleshooting tables describing remediation steps need explicit logic-validation against the underlying enforcement chain (here: branch-protection + ruleset rules), not just internal narrative consistency.
    • R20 #2 / Folder-path canonical display form across all docs — Copilot R20 #2 flagged that Infisical folder paths were shown with a trailing slash in several places (e.g., /WeOwnNetwork-ai/) but the Sync "Source secret path" was documented without it (/WeOwnNetwork-ai). This creates real-world copy/paste configuration errors during incident response and new-repo onboarding when stewards transcribe paths from docs into the Infisical UI. Copilot suggested standardizing — gave a literal "add trailing slashes everywhere" suggestion. User explicit preference: NO trailing slash (opposite of Copilot's literal suggestion). Rationale: the Infisical UI's "Source secret path" field accepts the path without trailing slash and mirrors the path verbatim, so the no-slash form is the form that survives the round-trip from docs → UI → docs without transformation. Applied trailing-slash → no-trailing-slash canonicalization across:
      • .github/workflows/README.md — §2 Scope item 3, §2 Scope item 4 (×2 path examples), §2.4 Usage Table active row + 2 placeholder rows, §4.4 Step B step 3 + step 4, §5.1 Common steps step 2 + step 4, §6 Rotation step 6, §6.1 Sync Options Configuration explanatory paragraph + Migration Steps step 1 + step 4. Total: ~14 path normalizations within a single doc.
      • .github/INCIDENT_RESPONSE.md — Scenario 6 step 6 single instance (steward-handoff Infisical access verification).
      • .github/ADR-002-infisical-github-sync.md — Architecture diagram lines 40 + 42 (with right-edge whitespace adjustment to maintain box-border alignment); Naming Convention bullet (line 64) with explicit "canonical form: NO trailing slash, per R20 close-out 2026-04-28" parenthetical added for future authors; Implementation Notes step 2 (line 138).
    • .github/ADR-002-infisical-github-sync.md Decision Log appended with R20 row (in correct chronological position after R17 + R18, NOT as a replacement). The R20 row documents: (a) the canonical form decision; (b) the explicit PRESERVATION of R17 + R18 historical rows with their original trailing-slash forms per the R19-codified "Historical-narrative preservation in audit trails" rule (modifying the historical text would corrupt the audit trail of the substantive R17 + R18 convention decisions); (c) Status remains "Accepted" — this is a display canonicalization, NOT a convention change (both forms refer to the same Infisical folder); (d) NEW operational rule codified (R20 close-out): when documenting a path-like identifier that users will copy into a vendor UI text field, pick the canonical form that the vendor UI itself uses (or the simpler form when the UI accepts both) and use it consistently across all docs.
    • Header date bumps for traceability:
      • .github/ADR-002-infisical-github-sync.md Date header: 2026-04-23 (initial) / 2026-04-28 (naming convention revised twice — see Decision Log) → 2026-04-23 (initial) / 2026-04-28 (naming convention revised twice + canonical no-trailing-slash form for folder paths — see Decision Log).
      • .github/INCIDENT_RESPONSE.md Date header: appended R20 trailing-slash canonicalization on Scenario 6 step 6 to the existing R13 + R18 list.
    • workflows/README.md Last updated header (line 6) was already simply 2026-04-28 — no parenthetical detail to extend, no bump needed.
    • CHANGELOG [v3.3.5.1] § Changed historical bullets PRESERVED — entries documenting the R17 (project-per-repo weown-bot/weownnetwork-ai reference) + R18 (folder-per-repo with trailing slashes) close-outs in their historical narrative form; modifying the original-state references would corrupt the close-out audit trail per the R19 rule.
    • Files touched (4): .github/workflows/README.md, .github/ADR-002-infisical-github-sync.md, .github/INCIDENT_RESPONSE.md, CHANGELOG.md (this file). PR7_HANDOFF_CHECKLIST.md (gitignored) updated separately.
    • Pattern continuation: R20 #1 surfaced a runbook-correctness bug (different drift class from R19's label-citation drift); R20 #2 surfaced doc-vs-UI display drift (also distinct from prior rounds). Two new sub-classes of accuracy drift identified within a single round, both caught by Copilot late in the v3.3.5.1 review cycle. Cumulative pattern across rounds 7-20: late-round Copilot reviews continue surfacing new sub-classes of accuracy drift even after 19 prior rounds. Going-forward implication: the v3.3.5.1 review cycle is establishing a stable taxonomy of doc-accuracy-drift classes that future PRs can pre-validate against.

  • Round-19 Copilot review fixes (2026-04-28, 1 Copilot R19 label-canonicalization fix + 3 same-doc + cross-doc cascades per the R18 #1 "Same-Document Consistency Sweep" rule):

    • R19 #1 / .github/workflows/README.md §3 "Branch name vs. PR body" table row label — the table row at line 170 was labeled **PR body \Contributors:` list**while the workflow itself emitsContributors on this branch:(auto-pr-to-main.ymlstep 8 line 320), and the SAME doc's earlier §3 three-tier table at line 161 already used the canonicalContributors on this branchlabel. Copilot R19 #1 flagged the inconsistency: readers cross-referencing the table against the actual PR body would see different labels and might think the doc is describing a different field than the one rendered. **Fix**: renamed the row label toPR body `Contributors on this branch:` listper Copilot's literal suggested change. **This fix is the canonical example of why the R18 #1 "Same-Document Consistency Sweep" rule matters** — the table row at line 170 saidContributors:while line 161 in the same doc already usedContributors on this branch`; R18's new rule mandated sweeping all earlier sections for the same convention; R19 #1 demonstrates the rule's value by catching exactly the kind of intra-doc drift it was designed to surface. Compliance basis: SOC 2 CC8.1 (documented operating procedures must be internally consistent across the same document); ISO/IEC 27001:2022 A.5.37 (no contradictory label citations within an authoritative reference doc); ISO/IEC 42001:2023 A.6.2.7 (AI system control descriptions must accurately match what the system emits — here, the actual workflow output line).
    • R19 same-doc cascade / .github/workflows/README.md §11 Troubleshooting table row — applied the same fix to the §11 row "PR body shows wrong attribution" which used Contributors: in the resolution column when listing the three field-label semantics. Updated to canonical Contributors on this branch: for cross-section consistency. (Not Copilot-flagged; caught proactively per the R18 #1 same-doc-sweep rule.)
    • R19 cross-doc cascade #1 / CONTRIBUTING.md §4 explanatory bullet — line 334 bullet **\Contributors:`→`Contributors on this branch:`**` to match the canonical workflow output. The same doc's earlier inline-quoted table at lines 289 + 294 already used the canonical form; the bullet at line 334 was the only inconsistent label-citation in the doc.
    • R19 cross-doc cascade #2 / CONTRIBUTING.md §4 parenthetical at line 328 — within the "regex-valid but convention-violating" example paragraph, the parenthetical "(The PR body's Opened by:, Last pushed by:, and Contributors: fields will still attribute correctly regardless — see next paragraph.)" listed the three field names but used the shortened form for the third one while the other two used the canonical form. Updated to canonical Contributors on this branch: for full intra-paragraph consistency.
    • CONTRIBUTING.md Last updated header bump — "2026-04-28 (R12 §4 attribution-fallback fix + R13 header date sync)" → "2026-04-28 (R12 §4 attribution-fallback fix + R13 header date sync + R19 §4 Contributors on this branch: label canonicalization)" for traceability of the R19 cross-doc cascade.
    • CHANGELOG [v3.3.5.1] line 56 PRESERVED unchanged — the v3.3.5.1 § Changed entry documenting the R7 fix that renamed Contributors: → Contributors on this branch: quotes the OLD label as the "before" value in a historical close-out narrative ("Contributors: label updated to Contributors on this branch:"). Modifying that quoted historical value would corrupt the audit trail of when + why the rename happened. The R10 PII-recursive-quote lesson applies in reverse here: when a CHANGELOG entry documents a label rename, the original-value reference must remain verbatim for the audit-trail entry to retain its meaning. Compliance basis: SOC 2 CC7.2 (historical audit trail integrity); ISO/IEC 27001:2022 A.5.28 (records of revision history must accurately reflect the original state being revised).
    • Files touched (4): .github/workflows/README.md, CONTRIBUTING.md, CHANGELOG.md, PR7_HANDOFF_CHECKLIST.md (gitignored). Single commit; squash-merge target preserved.

  • Round-18 Copilot review fixes + Infisical naming convention 2nd revision (folder-per-repo) (2026-04-28, 4 Copilot R18 doc-consistency fixes + user-driven operational simplification of the R17 naming convention):

    • R18 #1 / .github/workflows/README.md §2 Scope & Principles — convention contradiction with §2.4 Usage Table — Copilot R18 #1 flagged that §2.4 Usage Table reflected the revised Infisical convention (WEOWN_BOT_PAT in per-repo containers) but §2 "Scope & Principles" earlier in the doc still stated the old convention (single project + WEOWN_BOT_PAT__<ORG>_<REPO> naming), creating contradictory onboarding guidance. Fix: rewrote §2 item 3 ("Centralized storage") to describe the single shared project weown-bot GitHub PATs with one folder per target repo (e.g., /WeOwnNetwork-ai/), and rewrote item 4 ("Consistent naming") to match the revised convention — Infisical secret per folder = WEOWN_BOT_PAT (identity-mapped), namespace via folder path, GitHub Actions secret = WEOWN_BOT_PAT. Item 4 also notes the 2026-04-28 ADR-002 Decision Log revision and points at §6.1 Sync Options for the full rationale. Compliance basis: SOC 2 CC8.1 (documented operating procedures must be internally consistent across the same document); ISO/IEC 27001:2022 A.5.37 (no contradictory onboarding guidance — onboarding contributors must read one canonical convention).
    • R18 #2 / .github/INCIDENT_RESPONSE.md Scenario 6 step 6 — legacy project-name reference — Copilot R18 #2 flagged that Scenario 6 (Stewardship Gap) step 6 still referenced the legacy Infisical project name (weown-bot GitHub PATs only, no folder qualifier) but this PR's revised convention namespaces per-repo via folders inside that project. The runbook needed to match the new structure so a new steward verifying access during a SEV-4 hand-off would know which folder to check. Fix: per Copilot's suggested change, updated step 6 to "Verify new steward has Infisical access (project weown-bot GitHub PATs, folder /WeOwnNetwork-ai/)". Also bumped the file's Date header to include the R18 update for traceability. Compliance basis: NIST CSF 2.0 RS.MA-1 + ISO/IEC 27001:2022 A.5.24 (incident-response runbooks must reference current resource identifiers under stress; legacy references slow steward onboarding during an active incident).
    • R18 #3 / .github/ADR-004-copilot-auto-review-ruleset.md Layer 1 bullet #3 — copilot_code_review claim qualification — Copilot R18 #3 flagged that the Layer 1 rules bullet #3 stated "auto-request Copilot review on every PR (draft + ready) and on every push to an open PR", which contradicts the same ADR's later § Empirical Validation Results section documenting PR-creation-time eligibility caching (pre-existing PRs do not auto-request on subsequent push). Fix: rewritten per Copilot's suggested change to "auto-request Copilot review for newly-created, eligible PRs (draft + ready), and re-request review on later pushes only for open PRs where Copilot was auto-requested at creation"; appended a Note that explicitly cross-references § Empirical Validation Results and documents the close+reopen / merge+open-fresh remediation paths for pre-existing PRs. Also bumped the file's Date header to include R18 for traceability. Compliance basis: SOC 2 CC7.2 (control-evidence must be internally consistent across rule descriptions and validation results); ISO/IEC 42001:2023 A.6.2.7 (AI system enforcement claims must accurately describe the population they apply to — here, newly-created PRs only).
    • R18 #4 / .github/workflows/README.md header Last updated — stale date — Copilot R18 #4 flagged that the file's header Last updated: 2026-04-27 was stale because this PR's R17 close-out added 2026-04-28 changes (PAT rotation dates + new §6.1 Sync Options Configuration sub-section). Fix: per Copilot's suggested change, bumped to Last updated: 2026-04-28. Compliance basis: SOC 2 CC8.1 + ISO/IEC 27001:2022 A.5.37 (versioned-doc headers must reflect the most recent edit so auditors can locate the audit-trail entry covering the change).
    • Folder-per-repo namespacing — second revision of the 2026-04-28 naming convention (user-directed operational simplification): the R17 first-iteration revision used project-per-repo (one Infisical project per target repo, e.g., weown-bot/weownnetwork-ai). User feedback during the actual sync configuration on 2026-04-28: a single shared Infisical project with one folder per target repo is operationally cleaner — (a) one project-level RBAC boundary instead of N; (b) one expiration-reminder convention; (c) sibling folders discoverable from the same project landing page; (d) the existing weown-bot GitHub PATs project absorbs new repos without new project creation. Cascade: ADR-002 architecture diagram redrawn to show project-with-folders; Naming Convention section rewritten with the folder-per-repo model + a "why folder-per-repo, not project-per-repo" comparison; Implementation Notes initial setup steps revised (5 steps → 7 steps to add folder creation); Decision Log appended with a SECOND 2026-04-28 row tagged (R18) documenting the project→folder shift while preserving the R17 row for audit trail. .github/workflows/README.md updates: §2 Scope items 3+4 rewritten (R18 #1 above); §2.4 Usage Table rows updated to show Infisical project: weown-bot GitHub PATs, folder: /WeOwnNetwork-ai/; §4.4 Step B + §4.5 Step C rewritten for folder-based initial setup with explicit Source Path = /WeOwnNetwork-ai; §5.1 replication steps 2 + 4 rewritten for folder-based; §6 Rotation step 6 references folder; §6.1 Sync Options Configuration updated with new "Why folder-per-repo, not project-per-repo" sub-section + revised Migration Steps. .github/INCIDENT_RESPONSE.md Scenario 6 step 6 updated (R18 #2 above). Compliance basis: same as ADR-002 baseline (Infisical-primary-with-GitHub-Sync; Status remains "Accepted") — this is a second implementation-detail revision documenting the operationally-cleaner namespacing axis. NEW operational rule (R18 close-out): when a vendor-feature-driven convention is documented in an ADR, validate the convention against actual UI/API configuration before broad cascade — first-cut conventions may need a second-cut revision once the namespacing primitives are exercised in practice (R17 → R18 here is the canonical example; the R17 project-per-repo convention was correct in principle but operationally suboptimal; R18 folder-per-repo arrived from actual sync-creation feedback within the same day).

  • Round-17 Copilot review fixes + PAT rotation + Infisical sync configuration revision (2026-04-28, 1 Copilot R17 doc-accuracy fix + ecosystem-shaping naming-convention revision + audited PAT rotation event):

    • R17 #1 / auto-pr-to-main.yml step 4 trap — unconditional rm -f "$CONTRIB_RAW" is NOT a silent no-op — the R14 #1 trap consolidation comment block claimed "rm -f \"\"" was a "silent no-op until step 7's mktemp populates the variable", but on GNU coreutils (the Ubuntu-based GitHub-hosted runner) rm -f "" emits rm: cannot remove '': No such file or directory and exits 1. If the script aborts before step 7's mktemp populates CONTRIB_RAW, the trap's rm -f would alter the trap's own exit status and could mask the real script-failure exit code. Copilot R17 #1 flagged the empty-operand failure mode. Fix: split the trap into an unconditional cleanup for the always-populated paths + a conditional cleanup branch for the optional placeholder — trap 'rm -f "$PR_BODY" "$PR_TITLE" "$CONTRIBUTORS_FILE"; if [ -n "$CONTRIB_RAW" ]; then rm -f "$CONTRIB_RAW"; fi' EXIT. Comment block above the trap rewritten to (a) describe the actual GNU coreutils empty-operand behavior, correcting the R14 #1 "silent no-op" claim that was incorrect, (b) explicitly direct future contributors to add new optional temp files to the conditional branch (not the unconditional list) when their mktemp assignment happens after the trap is set. Compliance basis: SOC 2 CC7.2 (cleanup procedures must not alter the exit status of monitored job steps); ISO/IEC 27001:2022 A.5.37 (documented operating procedures must reflect actual runtime behavior, not assumed behavior); ISO/IEC 42001:2023 A.6.2.7 (correctness of code-comment claims is part of the AI-system documentation surface).

  • Round-16 Copilot review fixes (2026-04-28, 4 items — 2 workflow improvements on auto-pr-to-main.yml + 1 ADR-004 heading rename for stable anchor + 1 CHANGELOG cross-doc link fix):

    • R16 #1 / auto-pr-to-main.yml step 4 — RUNNER_TEMP fail-fast (policy-code alignment) — the comment block above the temp-dir setup explicitly stated "$RUNNER_TEMP — the GitHub-runner-scoped temp directory that is isolated from the shared /tmp", but the code did TEMP_DIR="${RUNNER_TEMP:-/tmp}" — silently falling back to /tmp if RUNNER_TEMPwas unset, which contradicts the documented isolation policy. Copilot R16 #1 flagged the policy-code drift. Fix: replaced the silent fallback with an explicit fail-fast check (if [ -z "${RUNNER_TEMP:-}" ]; then echo "::error::..." >&2; exit 1; fi) that aborts the workflow with a clear error message ifRUNNER_TEMP is unset. The error message explains: GitHub-hosted runners always set RUNNER_TEMP; an unset value means the workflow is being executed in an unsupported environment (e.g.,act, local emulation without env shimming) where the isolation guarantee cannot be honored. Comments above the check were preserved unchanged because they now accurately describe the runtime behavior. Compliance basis: SOC 2 CC7.2 + CC8.1 (controls must enforce the documented policy, not silently degrade to a weaker stance); ISO/IEC 27001:2022 A.5.37 (documented operating procedures must be enforced by the system, not just described); ISO/IEC 42001:2023 A.6.2.7-A.6.2.8 (AI system controls must fail-closed, not fail-open).
    • R16 #2 / auto-pr-to-main.yml step 7 — email-keyed cache prevents per-commit API call storm on large branches — the contributors-list builder did one gh api repos/.../commits/$sha call per commit in the branch range. On a long-running branch with 50+ commits this could trigger GitHub API rate-limits, degrading attribution to the name-only fallback for the rest of the run. Copilot R16 #2 flagged the rate-limit risk. Fix: added a bash associative array EMAIL_LOGIN_CACHE keyed by author email. Before each API call, check if the email already has a memoized login (cache hit → reuse). On miss, do the API call and memoize the result (including empty-string "unresolved" outcomes so we don't retry the same email twice within one run). On a typical PR (1–5 unique authors over 5–50 commits), this reduces API calls by 50–95%. Cache is correct because GitHub's commits API resolves the same email to the same login deterministically. Cache scope is the workflow run (rebuilt fresh per auto-pr-to-main execution); no persistence needed. Compliance basis: NIST CSF 2.0 PR.IP-1 (efficient resource use); NIST CSF 2.0 PR.AC-4 (rate-limit resilience prevents control degradation under load); SOC 2 A1.2 (system processing integrity — attribution remains accurate even on large PRs).
    • R16 #3 / ADR-004 § Empirical Validation Results — heading rename for stable anchor — the section heading was ## Empirical Validation Results (round-7, 2026-04-27). GitHub auto-generates Markdown anchors via lowercase + space-to-hyphen + non-alphanumeric drop, producing #empirical-validation-results-round-7-2026-04-27 (with the parenthetical date suffix baked into the anchor). Internal links written as [Empirical Validation Results](#empirical-validation-results) (the human-intuition-friendly form) DO NOT resolve. Copilot R16 #3 flagged the broken intra-doc anchor. Fix: renamed the heading to ## Empirical Validation Results (clean anchor #empirical-validation-results) and moved the round/date provenance to a *Source: round-7 controlled experiment, 2026-04-27 (sharpened in round-13 #1, 2026-04-28).* italic sub-line right under the heading. Audit trail of when the section was added + revised is preserved without polluting the anchor. Lesson: section headings that need to be cross-referenced should NEVER include parenthetical dates / round-numbers / version qualifiers — those go in a sub-line. Codified as a NEW operational rule (see Meta below).
    • R16 #4 / CHANGELOG.md line 66 — broken intra-CHANGELOG anchor for cross-doc reference — the R15 #1 narrative said Cross-references [Empirical Validation Results](#empirical-validation-results) from the procedure intro, but #empirical-validation-results resolves WITHIN THE CHANGELOG (where there is no such heading), not in ADR-004 where the section actually lives. Copilot R16 #4 flagged the implicit-anchor failure mode (Markdown (#anchor) always resolves within the current doc, not the intended target doc). Fix: rewritten as an explicit cross-doc link (.github/ADR-004-copilot-auto-review-ruleset.md#empirical-validation-results) per Copilot's suggestion. Combined with the R16 #3 heading rename, the link now resolves correctly from the CHANGELOG to the renamed clean-anchor section in ADR-004. Lesson: when a CHANGELOG narrative paraphrases or summarizes a cross-doc link, ALWAYS render the link as an explicit relative path ((./relative/path.md#anchor)), never as a bare (#anchor). The latter form silently resolves to a current-doc anchor that doesn't exist. Codified as a NEW operational rule (see Meta below).

  • Round-15 Copilot review fixes (2026-04-28, 3 items — 1 ADR-004 verification-procedure alignment + 2 workflow improvements on auto-pr-to-main.yml):

    • R15 #1 / ADR-004 § End-to-end auto-trigger validation — procedure conflicted with Empirical Validation Results — the validation steps instructed "the next bot-authored push to any open PR is the live test" + "push a commit authored via weown-bot (i.e., the auto-PR workflow runs and updates / creates a PR)", but the same ADR's § Empirical Validation Results (added in R7, sharpened in R13 #1) documents that Copilot auto-review eligibility is evaluated at PR-creation / reopen time, NOT at push time. Running the procedure against a pre-existing PR (like PR #13 itself) produces a false negative because the PR-creation-time cache was set BEFORE ruleset enablement. Copilot R15 #1 flagged the internal contradiction. Fix: rewritten to explicitly require ONE of two trigger paths: (a) New PR path — push weown-bot-authored commits to a fresh branch so auto-pr-to-main.yml opens a brand-new PR, OR (b) Close+reopen path — gh pr close <N> then gh pr reopen <N> on an existing PR (Copilot re-evaluates eligibility on the reopen event). Added bold Important callout at top of section + explicit warning in step 4 ("Do not retry against a pre-existing PR via plain push — that path is known-unreliable per § Empirical Validation Results"). Cross-references Empirical Validation Results from the procedure intro. Compliance basis: SOC 2 CC7.2 (validation procedures must be executable end-to-end without producing false negatives); ISO/IEC 27001:2022 A.5.37 (documented operating procedures must correctly describe the system behavior, including timing-sensitive enforcement semantics); ISO/IEC 42001:2023 A.6.2.7-A.6.2.8 (AI validation procedures must account for the population they apply to — here, newly-created PRs only).
    • R15 #2 / auto-pr-to-main.yml step 6 — two API calls collapsed into one with jq fallback chain — the OPENED_BY resolution did two gh api repos/$REPO/commits/$FIRST_SHA calls on the same SHA: first with --jq '.author.login // ""', then on empty fall through, with --jq '.committer.login // ""'. Both calls hit the same endpoint with the same SHA, wasting one API request per PR build + doubling rate-limit exposure. Copilot R15 #2 suggested consolidating via jq's // operator. Fix: rewritten to a single call gh api "repos/$GITHUB_REPOSITORY/commits/$FIRST_SHA" --jq '.author.login // .committer.login // ""' — jq evaluates the fallback chain in-process, returning the first truthy value (or empty string on total miss). Halves per-PR API requests, reducing rate-limit exposure especially under burst conditions (e.g., multiple pushes within the concurrency: cancel-in-progress window). Added comment block explaining the consolidation rationale + rate-limit justification. Compliance basis: operational cost reduction + defense-in-depth against API rate-limits (NIST CSF 2.0 PR.IP-1 — configuration baselines include efficient resource use).
    • R15 #3 / auto-pr-to-main.yml step 8 — @unknown sentinel leakage in PR body — lines echo "**Opened by:** @${OPENED_BY}" + echo "**Last pushed by:** @${LAST_PUSHED_BY_RESOLVED}" always prefixed @ even when the resolved value was the fallback sentinel unknown (e.g., when LAST_PUSHED_BY env var was empty AND the API fallback chain exhausted). Output would read @unknown which is misleading — @unknown is not a valid GitHub login and looks like a broken mention. Copilot R15 #3 flagged the UX degradation. Fix: added render_handle() helper function before the step 8 { block that uses a bash case to conditionally prefix @ only when the value is non-empty AND not the literal string unknown; otherwise renders plain unknown. Call sites changed from @${OPENED_BY} / @${LAST_PUSHED_BY_RESOLVED} to $(render_handle "$OPENED_BY") / $(render_handle "$LAST_PUSHED_BY_RESOLVED"). Output under degraded resolution now reads **Opened by:** unknown (no broken mention) instead of **Opened by:** @unknown. Helper is reusable for any future handle-rendering needs in the same workflow. Compliance basis: SOC 2 CC8.1 (user-facing output must accurately reflect system state); ISO/IEC 42001:2023 A.6.2.7 (AI-adjacent outputs must not produce misleading artifacts).

  • Round-14 Copilot review fixes (2026-04-28, 6 items — 1 workflow refactor + 1 ADR-004 control-evidence wording + 4 anchor-link sweeps after R13 § header date-range change):

    • R14 #1 / auto-pr-to-main.yml step 4 + step 7 trap consolidation — the workflow set trap '...' EXIT twice in the same shell (line 133 covering 3 temp files, line 213 overwriting it to cover 4). Functionally correct (each trap covered exactly the files extant at that point) but Copilot flagged the override pattern as regression-risky: any future temp file added between steps would silently leak if the dev forgot to update both trap calls. Fix: declared CONTRIB_RAW="" placeholder in step 4 alongside the other mktemp calls; the single trap on line 141 now references all 4 paths upfront. The trap body single-quotes $CONTRIB_RAW, so variable expansion happens at fire time — empty string expands to a silent rm -f "" no-op until step 7 mktemp populates the variable. Step 7 retains only the CONTRIB_RAW="$(mktemp ...)" assignment; the duplicate trap call is removed. Added in-line comment block explaining the deliberate single-trap-with-deferred-mktemp pattern + regression-safety rationale (any future temp file just adds itself to the trap body once, with empty placeholder upfront). Compliance basis: SOC 2 CC8.1 (controls must be unambiguous to engineers) + ISO/IEC 27001:2022 A.5.37 (documented operating procedures must accurately describe enforcement mechanism) + WeOwn mktemp + trap defense-in-depth pattern from § 6 of .github/workflows/README.md.
    • R14 #2 / ADR-004 § "Defense-in-depth" bullet — inaccurate control-evidence claim — the bullet read "Ruleset non_fast_forward AND auto-pr-to-main.yml's commit-signing requirement" but auto-pr-to-main.yml does NOT enforce commit signing — that's the main ruleset (per ADR-003 § 8.1 row 5: required_signatures: true). The auto-PR workflow signs ITS OWN commits (the bot-authored one creating the PR) but doesn't gate human commits on the branch. Copilot R14 #2 flagged the example as inaccurate control-evidence. Fix: rewritten to "Ruleset non_fast_forward AND ADR-003 / main ruleset signed-commit enforcement" — attributes the signing-enforcement to the actual control mechanism (the main branch ruleset) instead of the workflow. The defense-in-depth point still holds: non_fast_forward (Layer 1, ~ALL branches) + required_signatures (main ruleset only) are independent enforcement layers. Compliance basis: SOC 2 CC7.2 (audit-trail accuracy: control-evidence statements must point to the actual enforcement mechanism, not an implementer that doesn't enforce); ISO/IEC 42001:2023 A.6.2.7-A.6.2.8 (AI control attribution must be precise to support compliance audit).
    • R14 #3-#6 / CHANGELOG.md 4 broken intra-doc anchor links — R13 #5 changed the [v3.3.5.1] section header from — 2026-04-27 to — 2026-04-27 to 2026-04-28. GitHub auto-generates Markdown anchors by lowercasing + replacing spaces with hyphens + dropping non-alphanumerics; the new anchor is #v3351--2026-04-27-to-2026-04-28, breaking the previous #v3351--2026-04-27 references. Copilot R14 #3-#6 flagged 4 occurrences across the [v3.3.4.2] section: (3) Changed bullet about workflows/README.md header version bump cross-reference; (4) Changed bullet about Continuation rounds 4–7 cross-reference; (5) Added bullet about Platform-sourced developer attribution three-tier upgrade cross-reference; (6) Meta bullet about Version cadence cross-reference. Fix: single sed-equivalent sweep — all 4 occurrences updated from (#v3351--2026-04-27) to (#v3351--2026-04-27-to-2026-04-28) via replace_all on the exact pattern. Verified zero false positives (no other #v3351--2026-04-27 substring exists). Lesson: when a ## [vN.N.N.N] section header is modified mid-PR (e.g., date-range expansion), sweep ALL intra-doc links in the same commit — GitHub-generated anchors regenerate automatically and silently break references. Codified as a NEW operational rule in this round's close-out.

  • Round-13 Copilot review fixes (2026-04-28, 5 items — 4 doc edits committed in this round + 1 deferred manual PR title alignment):

    • R13 #1 / ADR-004 line 24 — copilot_code_review claim qualified — the rule description claimed "every PR (regardless of base branch) gets Copilot AI review automatically" but the same ADR's § Empirical Validation Results section documents that auto-trigger is evaluated at PR-creation time and does NOT apply to pre-existing PRs (e.g., PR #13). Copilot flagged the internal contradiction. Fix: rewritten to "every newly-created PR (regardless of base branch) gets Copilot AI review automatically after ruleset enablement" + added explicit Note paragraph: "Copilot evaluates auto-review eligibility at PR-creation time, so PRs that already existed before the ruleset was applied (e.g., PR #13) do not retroactively gain auto-review — they must be triggered manually for the duration of their open lifecycle. See § Empirical Validation Results below for the controlled experiment confirming this PR-creation-time caching behavior." Also bumped Date header to add 2026-04-28 (R13 clarification on auto-trigger timing) for audit traceability. Resolves the contradiction without softening the forward-looking claim (all NEW PRs auto-trigger; only PR #13 is grandfathered to manual-only for its remaining lifecycle).
    • R13 #2 / INCIDENT_RESPONSE.md Scenario 6 step 2 — awkward "per ... per ..." phrasing rephrased — the line read "Assign new primary steward ... per CODEOWNERS .github/CODEOWNERS per-path TODO comments (decision pending 2026-05-15 handoff)" which had repeated "per" and ambiguous parsing during a live incident. Fix: rewritten per Copilot's suggestion to "based on .github/CODEOWNERS; use the handoff TODO comments there to resolve the pending 2026-05-15 decision" — single canonical reference to .github/CODEOWNERS, semicolon-separated decision-procedure clarification, no duplicate "per" keywords. Also bumped header Version from v3.3.4.1 to v3.3.5.1 and Date to 2026-04-23 (initial) / 2026-04-28 (R13 phrasing fix on Scenario 6 step 2) so the header reflects the actual last-edit state and matches the rest of the v3.3.5.1 PR scope. Compliance basis: incident-response runbooks must be unambiguous under stress (NIST CSF 2.0 RS.MA-1, ISO/IEC 27001:2022 A.5.24); awkward phrasing is a documented incident-response failure mode.
    • R13 #3 / PR title vs. CHANGELOG version mismatch — deferred to manual gh pr edit after this commit lands — PR #13 title was set at PR-creation time (2026-04-23) to Auto-PR: docs(v3.3.4.2): sync main ruleset docs + ADR-003 + confirmed contributor handles. The auto-PR workflow intentionally preserves PR titles across pushes (see R11 fix in workflows/README.md §2A step 9 + ADR-001), so the title naturally drifts from the effective in-flight version (v3.3.5.1). Copilot R13 #3 flagged the divergence between the PR title and the [v3.3.5.1] CHANGELOG section. Fix: cannot be applied via a regular commit (workflow ignores --title on existing PRs by design). Action plan: after this commit lands, manually run gh pr edit 13 --title "docs(v3.3.5.1): main ruleset docs + ADR-003/004 + auto-PR workflow attribution + Copilot review hardening" to align the PR title with the current effective version. This is a one-time manual override of the workflow's title-preservation contract, justified by audit-trail clarity at squash-merge time. The workflow's title-preservation rule remains correct for normal use; this is the documented exception when the in-flight version bumps mid-PR.
    • R13 #4 / CONTRIBUTING.md header Last updated: 2026-04-27 — stale date refreshed — the file was modified in R12 (2026-04-28) but the header date wasn't updated. Copilot R13 #4 suggested bumping both Version (v3.3.5.1 → v3.3.5.2) and Last updated (2026-04-27 → 2026-04-28). Fix applied: Last updated: 2026-04-28 (R12 §4 attribution-fallback fix + R13 header date sync). Fix REJECTED: keeping Version: v3.3.5.1 rather than bumping to v3.3.5.2. Rationale for rejection: per the explicit decision in this [v3.3.5.1] section's intro paragraph ("Version stays at v3.3.5.1 across the day boundary because the PR is a single semantic unit (no fresh iteration started post-merge)"), the in-flight PR keeps the same #WeOwnVer iteration number throughout. Bumping to v3.3.5.2 mid-PR would create cascade churn (CONTRIBUTING.md, workflows/README.md, ADR-003, ADR-004, CHANGELOG section header all need to match) without changing the PR's semantic scope. Documenting the rejection in the Last updated parenthetical communicates intent to future auditors. (A future v3.3.5.2 will be the FIRST PR opened on 2026-04-28 or later — a fresh iteration boundary.)
    • R13 #5 / CHANGELOG.md [v3.3.5.1] header date + intro paragraph — stale single-day date refreshed to span — the section header read ## [v3.3.5.1] — 2026-04-27 and the intro claimed "all changes in this section were made on 2026-04-27", but R12 added a 2026-04-28-dated entry. Copilot R13 #5 flagged the inconsistency. Fix: header rewritten to ## [v3.3.5.1] — 2026-04-27 to 2026-04-28 per Copilot's suggested rewording; intro paragraph rewritten to "this section captures the portion of the PR that moved into the 5th week of April under the active #WeOwnVer interpretation ... the later continuation rounds documented here span 2026-04-27 (rounds 4–11) through 2026-04-28 (rounds 12–13)" + added explicit "Version stays at v3.3.5.1 across the day boundary because the PR is a single semantic unit (no fresh iteration started post-merge)" sentence so the v3.3.5.1-not-v3.3.5.2 decision is auditable from the section header alone (cross-references R13 #4 above).

  • Round-12 Copilot review fixes (2026-04-28, 1 item — in CONTRIBUTING.md §4 attribution paragraph):

    • R12 #1 / §4 post-attribution closing sentence — the trailing sentence claimed "The PR body always shows real GitHub usernames (@ncimino, @romandidomizio, etc.), regardless of what <dev> segment was chosen" which contradicted the documented step-7 aggregation fallback behavior (per workflows/README.md §2A step 7 + auto-pr-to-main.yml step 7: gh api returns .author.login // .committer.login // ""; on empty, falls through to git log -1 --format=%an producing a plain-text author name, no @ prefix, no GitHub link). Copilot flagged "always" as overly absolute. Fix: rewritten to "shows real GitHub usernames (@ncimino, @romandidomizio, etc.) when available and otherwise falls back to commit-author names (for commits where the commits API doesn't return a linked GitHub login — e.g., unlinked email addresses), regardless of what <dev> segment was chosen for the branch name." Matches Copilot's suggested rewording and adds a parenthetical explaining the specific failure mode (unlinked email) so maintainers understand why the fallback exists. Compliance alignment: SOC 2 CC8.1 + ISO/IEC 27001:2022 A.5.37 (documented operating procedures must be accurate) — the "always" claim would fail an audit where the auditor traces the code path and finds the fallback branch.

  • Round-11 Copilot review fixes (2026-04-27, 1 item — in workflows/README.md §2A walkthrough table):

    • R11 #1 / §2A step 9 row — the "What it does" column claimed the existing-PR idempotent-update path runs gh pr edit $N --title --body --add-reviewer ncimino,romandidomizio (single combined call, updates title). In reality, auto-pr-to-main.yml lines 353–357 intentionally preserve the existing PR title and make two separate gh pr edit calls: gh pr edit $N --body-file $PR_BODY (body only — no --title flag passed), followed by gh pr edit $N --add-reviewer ncimino,romandidomizio (reviewers only). Title is set exactly once at PR creation time from $PR_TITLE (step 5). Row rewritten to accurately reflect the implementation: "If found → gh pr edit $N --body-file $PR_BODY (preserves the existing title — PR titles are set once at creation, not refreshed on subsequent pushes), followed by a separate gh pr edit $N --add-reviewer ncimino,romandidomizio. If not → gh pr create --base main --head $BRANCH_NAME --title $(cat $PR_TITLE) --body-file $PR_BODY followed by the same --add-reviewer call." Rationale for title preservation (now explicit in doc): PR titles tend to carry human semantic context (feature scope, PR tracking nicknames); refreshing them on every push based on the latest commit subject would create churn in PR lists + notifications. Body is always refreshed to reflect latest commits + attribution. This matches GitHub best-practice for long-lived PRs with iterative review rounds (exactly the scenario PR #13 is in — title stable for 11 rounds, body refreshed each push). Compliance alignment: SOC 2 CC8.1 change-management — accurate developer documentation is part of the audit-trail evidence chain (the walkthrough table IS the onboarding doc for new contributors); inaccurate documentation of a security-sensitive workflow step would be flagged in an audit.

  • Round-10 Copilot review fixes (2026-04-27, 2 items — both in this CHANGELOG.md):

    • R10 #1 / ### Fixed round-8 close-out entry + R10 #2 / ### Security / Compliance PII audit bullet — the round-8 entry quoted the literal personal email 2x ('Replace temp <actual-email> with permanent bot email' + 'stale <actual-email> references in §2 + §7') as part of describing the round-8 fix, and the PII-audit bullet quoted it 1x ('all 3 <actual-email> references in workflows/README.md rewritten...'). All 3 occurrences flagged by Copilot R10 as still exposing PII despite the fix being correctly described. Fix: all 3 occurrences replaced with <redacted-personal-email> placeholder — same convention used elsewhere in the codebase for placeholder identifiers (e.g., <dev>, <real-github-username> in v3.3.4.2 entry). Rationale and round-8 audit history fully preserved; only the literal email string is gone. Anticipated in round-8 close-out commit message: that commit explicitly noted "the email appears 3x in the round-8 commit body documenting Before: state of the fix" and identified squash-merge as the long-term sanitization mechanism. Round-10 brings forward that sanitization to the CHANGELOG narrative now (rather than waiting for squash-merge) so the in-flight CHANGELOG no longer carries the email. Same redaction approach should be applied to the round-10 close-out commit message itself — i.e., this entry must NOT quote the literal email it just redacted.

  • Round-9 Copilot review fixes (2026-04-27, 3 items + 1 plain-text-handle defense-in-depth extension — all in workflows/README.md):

    • R9 #1 / §2 Account Security Requirements line 64 — "✅ 2FA mandatory (TOTP + recovery codes held by Yonks as enterprise admin)" disclosed specific 2FA credential custody on a public repo. Round-3 had already removed this pattern from §10 Transition Checklist row 2 ("Transfer 2FA (TOTP seed) + recovery codes..." → "Transfer 2FA administration per internal runbook...") but missed the §2 line. Fix: rewritten to "✅ 2FA mandatory (administration, recovery, and custody tracked per internal runbook — no credential details in this public repo)". Aligns with the round-3/7/8 internal-runbook posture. Reduces social-engineering surface area (no specific credential types named, no custodian named in this section). Compliance basis: ISO/IEC 27001:2022 A.5.34 (PII protection) + A.8.1 (user endpoint devices) + NIST CSF 2.0 PR.AC-3 (access management without revealing custody chain).
    • R9 #2 / §11 Troubleshooting Copilot auto-review row — manual-trigger snippet used reviewers[]=Copilot but ADR-004 § Validation (line 169) references the canonical login copilot-pull-request-reviewer. Inconsistency could cause API call to fail (Copilot is not a valid GitHub login) or confuse maintainers cross-referencing the two docs. Fix: snippet rewritten to use reviewers[]=copilot-pull-request-reviewer with explicit "canonical GitHub Copilot reviewer login — same value referenced by ADR-004 § Validation" cross-reference. Single canonical identifier across the docs reduces divergence risk.
    • R9 #3 / §10 Transition Checklist rows 3 + 6 — Owner column had plain-text "Yonks" (row 3 sole owner; row 6 second owner) while adjacent rows used @YonksTEAM handle. Inconsistency makes unambiguous paging during transition impossible (audit trail can't @-mention plain-text names). Fix: both rows updated to @YonksTEAM handle. All 6 rows in the Transition Checklist now use GitHub handles consistently for the Owner column.
    • PII / handle-consistency defense-in-depth note: a separate sweep found that SECURITY_ASSESSMENT.md line 14 ("Yonks-held recovery codes") and INCIDENT_RESPONSE.md lines 61, 149, 152 (similar credential-custody disclosures + plain-text "Yonks" handles) carry the same anti-patterns. These files are OUT OF SCOPE for PR #13 (which is the main-branch-ruleset + auto-PR + Copilot-review docs sweep) and intentionally NOT modified to avoid scope creep. Tracked: a follow-up PR (v3.3.5.2 or v3.3.6.1) will apply the same internal-runbook + handle-consistency rewrites across SECURITY_ASSESSMENT.md + INCIDENT_RESPONSE.md. Documented in PR7_HANDOFF_CHECKLIST.md queue.

  • Round-8 Copilot review fixes (2026-04-27, 3 items — all in workflows/README.md):

    • R8 #1 / §10 Transition Checklist row 3 — "Replace temp <redacted-personal-email> with permanent bot email" exposed a real personal email on a public repo. Fix: rewritten to "Update the service account's email to the permanent bot email (details tracked per internal runbook)". Defense-in-depth extension (same principle): stale <redacted-personal-email> references in §2 Account Security Requirements (line 65) and §7 Layer 1 email alert recipient (line 390) also rewritten to generic internal-runbook language to prevent Copilot from flagging them in a future round and to complete the PII audit. Aligns with the round-6/7 PII-minimization posture already applied to CONTRIBUTING.md handle table + CODEOWNERS + step 7 contributors fallback + step 8 Recent Commits. Personal emails are PII per GDPR Art. 5(1)(c) data minimization + ISO/IEC 27001:2022 A.5.34 even when the individual is publicly associated with the repo.
    • R8 #2 / §11 Troubleshooting 403 row — the suggested PAT scopes (contents:write, pull_requests:write, issues:write, metadata:read) contradicted the least-privilege guidance at §2.4 + ADR-001 (Contents: Read, Pull requests: Read/Write, Metadata: Read; Issues: Write intentionally NOT on the PAT). Fix: row rewritten to specify the 3 correct least-privilege scopes, explicit "Do NOT add Issues: Write to the PAT" warning, cross-reference to §2.4 + ADR-001, and explanation that permissions: issues: write on the workflow job (using the ephemeral GITHUB_TOKEN) is the correct mechanism for issue actions — not a broader PAT scope. Prevents a future maintainer following the troubleshooting table from over-provisioning the PAT against the repo's own security posture.
    • R8 #3 / §2A step 6 failure mode — column said gh api rate-limited "uses $GITHUB_TOKEN", but auto-pr-to-main.yml actually runs every gh call under GH_TOKEN=${{ secrets.WEOWN_BOT_PAT }} (the PAT is what makes PRs show weown-bot as author, which is what makes Copilot auto-review trigger — see inline comment in the workflow). Fix: failure mode now reads "check WEOWN_BOT_PAT rate-limit status / scopes / rotation state (the workflow runs all gh calls under GH_TOKEN=${{ secrets.WEOWN_BOT_PAT }}, not the ephemeral GITHUB_TOKEN)". Correct diagnostic reference for on-call debugging.

Security / Compliance

  • Copilot Business entitlement validated end-to-end on new PRs (2026-04-27) — controlled experiment confirmed the Layer 2 enterprise-level ruleset triggers Copilot AI review requested due to automatic review settings for newly-created weown-bot-authored PRs without manual intervention. PR #13 stays manual-review-only (PR-creation-time caching). All future bot-authored PRs (v3.3.5.2+ work) will auto-trigger. Compliance impact: ISO/IEC 42001:2023 A.6.2.7 (AI-aware technical controls) + A.6.2.8 (review of AI-generated output) are now actively enforced for all NEW bot-authored PRs in addition to human-authored ones. SOC 2 CC8.1 change-management evidence narrative now has a verified AI-assisted review layer. CIS Controls v8 18.3 (development security — code review) coverage upgraded from "manual on bot PRs" to "automatic for new PRs / manual for legacy".
  • PII minimization audit (round-6 + round-7 + round-8 + round-9): zero email addresses anywhere in PR body output paths. Step 7 contributors fallback (%an only), step 8 Recent Commits header (%an only). Round-8 extension: all 3 <redacted-personal-email> references in workflows/README.md rewritten to generic internal-runbook language. Round-9 extension: §2 Account Security Requirements line 64 rewritten to remove "TOTP + recovery codes held by Yonks as enterprise admin" credential-custody disclosure (last social-engineering-surface item in workflows/README.md §2 — the §10 row 2 equivalent was already fixed in round-3). Aligns with prior PII minimization on CONTRIBUTING.md handle table + CODEOWNERS. Per GDPR Art. 5(1)(c) data minimization, ISO/IEC 27001:2022 A.5.34 (privacy + protection of PII), and ISO/IEC 27001:2022 A.5.7 (threat intelligence — reduces public attacker recon surface). (Out-of-scope files NOT modified: vaultwarden/README.md, vaultwarden/CHANGELOG.md, anythingllm/helm/Chart.yaml for emails; SECURITY_ASSESSMENT.md line 14 + INCIDENT_RESPONSE.md lines 61/149/152 for credential-custody + plain-text handles — follow-up PR queued.)
  • Defense-in-depth ruleset architecture documented in ADR-004 — see "Added" entry above. SOC 2 auditor narrative: WeOwnNetwork/ai now has THREE rulesets (main rules id 12119304 strict on ~DEFAULT_BRANCH, Copilot auto-review id 12131972 baseline on ~ALL repo-scoped, plus enterprise-level layer baseline on all orgs/repos/branches). Bypass list empty on all three.

Meta

  • Version cadence (revised under user's #WeOwnVer week-5 interpretation): 2026-04-23 work was Season 3, April Week 4 → v3.3.4.x (original PR + rounds 1–3). 2026-04-27 work is Season 3, April Week 5 → v3.3.5.x (rounds 4–7). The exact ISO-week-to-WEEK mapping methodology is currently under leadership review per docs/VERSIONING_WEOWNVER.md; this CHANGELOG follows the user's directive at PR #13 round-7 close-out time. Future iterations on 2026-04-27 itself would be v3.3.5.2, v3.3.5.3, etc.
  • Round 7 close-out scope = 8 Copilot R7 doc/PII/bug fixes + 2 user-directed README additions (§2A + §11) + new ADR-004 Empirical Validation Results section + version bump cascade across 4 files + CHANGELOG split. Single commit; squash-merge target preserved.
  • Round 8 close-out scope = 3 Copilot R8 doc-accuracy fixes on workflows/README.md (PII, least-privilege PAT scopes, correct GH_TOKEN diagnostic reference) + 2 PII defense-in-depth extensions (same stale email in §2 + §7 rewritten before Copilot flags them). Single-file change; single commit; squash-merge target preserved. Running total: 38 Copilot comments resolved across 8 rounds.
  • Round 9 close-out scope = 3 Copilot R9 fixes on workflows/README.md (credential-custody PII removal in §2, canonical Copilot reviewer login copilot-pull-request-reviewer aligned across docs in §11, plain-text "Yonks" → @YonksTEAM handle in §10 rows 3 + 6 for unambiguous paging). Single-file change; single commit; squash-merge target preserved. Running total: 41 Copilot comments resolved across 9 rounds. Out-of-scope follow-ups queued: cross-doc PII sweep on SECURITY_ASSESSMENT.md + INCIDENT_RESPONSE.md for the same credential-custody anti-pattern.
  • Round 10 close-out scope = 2 Copilot R10 fixes on CHANGELOG.md itself (PII self-redaction — the round-8 close-out narrative + PII audit bullet quoted the literal personal email 3x while documenting the round-8 fix; replaced with <redacted-personal-email> placeholder, full audit rationale preserved). Single-file change; single commit; squash-merge target preserved. Running total: 43 Copilot comments resolved across 10 rounds. Lesson learned: when documenting a PII fix, the rationale entry itself must use a placeholder — quoting the original 'Before:' value defeats the purpose of the fix. This applies recursively to the round-10 commit message + this CHANGELOG entry.
  • Round 11 close-out scope = 1 Copilot R11 doc-accuracy fix on workflows/README.md §2A walkthrough table step 9 (documented --title --body --add-reviewer single combined call → rewritten to match actual implementation of two separate gh pr edit calls with title preservation). Single-file change; single commit; squash-merge target preserved. Running total: 44 Copilot comments resolved across 11 rounds. Pattern: late-round reviews are surfacing walkthrough-table accuracy drift that earlier rounds missed — §2A is now the most heavily-reviewed section of the doc.
  • Round 12 close-out scope = 1 Copilot R12 doc-accuracy fix on CONTRIBUTING.md §4 attribution paragraph (removed overly-absolute "always" claim; added fallback-to-name path that matches step-7 implementation). Single-file change; single commit; squash-merge target preserved. Running total: 45 Copilot comments resolved across 12 rounds. Pattern continuation: accuracy-drift fixes now extending beyond workflows/README.md §2A into adjacent docs (CONTRIBUTING.md §4) that describe the same workflow behavior. Going-forward rule: when the attribution / step-7 code path changes, sweep BOTH workflows/README.md §2A step 7 + CONTRIBUTING.md §4 for consistency in the same commit.
  • Round 13 close-out scope = 5 Copilot R13 fixes touching 4 files (ADR-004 auto-trigger qualification + Date bump; INCIDENT_RESPONSE.md Scenario 6 phrasing + version/date header sync; CONTRIBUTING.md Last-updated date sync; CHANGELOG.md [v3.3.5.1] section header + intro paragraph spanning 2026-04-27→-28) + 1 deferred manual gh pr edit 13 --title override (R13 #3). Multi-file change; single commit; squash-merge target preserved. Running total: 50 Copilot comments resolved across 13 rounds. Pattern continuation: accuracy-drift fixes are now sweeping ALL versioned-doc headers in the PR scope simultaneously (ADR-004, INCIDENT_RESPONSE.md, CONTRIBUTING.md, CHANGELOG section header) rather than waiting for Copilot to flag each one. Going-forward rule: when crossing the day boundary mid-PR, refresh Last updated on every modified file in the same commit AND update the [vN.N.N.N] CHANGELOG section header to span the date range. R13 § Empirical Validation Results consistency rule: when a forward-looking compliance claim (e.g., "every PR gets X automatically") is documented in an ADR, ensure ALL such claims explicitly reference the validation section that confirms or qualifies the claim — prevents internal contradictions like the one R13 #1 caught.
  • Round 14 close-out scope = 6 Copilot R14 fixes touching 3 files (auto-pr-to-main.yml trap consolidation; ADR-004 defense-in-depth bullet control-evidence wording; CHANGELOG.md 4 anchor-link sweeps). Multi-file change; single commit; squash-merge target preserved. Running total: 56 Copilot comments resolved across 14 rounds. Pattern continuation: R14 surfaced a NEW class of accuracy drift (intra-doc anchor breakage caused by R13 § header date-range change), confirming that consistency sweeps must extend to all link-target derivations, not just text content. Three NEW operational rules codified: (a) Same-shell trap discipline — within a single run: block (one bash shell), use a SINGLE trap '...' EXIT set ONCE upfront, with all temp-file paths declared as variables (empty placeholder OK; bash expands at fire time). NEVER re-trap in the same shell to add a new file; just initialize the variable to "" upfront and assign mktemp later. Override patterns are regression-risky and Copilot-flag-prone. (b) Post-anchor-change link sweep — when modifying a ## [vN.N.N.N] section header in CHANGELOG (or any Markdown header that other parts of the same doc cross-reference), sweep ALL intra-doc (#anchor) references in the same commit. GitHub auto-generates anchors from header text via lowercasing + space-to-hyphen + non-alphanumeric drop; date-range expansions, version bumps, and any other header edit silently break links. (c) ADR control-evidence accuracy — when an ADR cites a control-evidence example (e.g., "X is enforced by mechanism Y"), verify Y is the ACTUAL enforcement mechanism (not just an implementer that interacts with Y). Cross-reference the relevant ADR's § "Rules enabled" or § "Configuration" section to confirm the control-evidence path is end-to-end accurate.
  • Round 15 close-out scope = 3 Copilot R15 fixes touching 2 files (ADR-004 validation procedure alignment with Empirical Validation Results; auto-pr-to-main.yml two code-quality improvements — API call consolidation via jq fallback chain + render_handle helper preventing @unknown sentinel leakage). Multi-file change; single commit; squash-merge target preserved. Running total: 59 Copilot comments resolved across 15 rounds. Pattern continuation: R15 surfaced a NEW class of accuracy drift (validation-procedure semantic conflict with the same ADR's own empirical findings), confirming that consistency sweeps must extend to verification / troubleshooting / runbook sections of an ADR, not just the main body. ONE NEW operational rule codified: (a) Sentinel-value output discipline — when emitting user-visible strings that may include fallback sentinel values (e.g., unknown, n/a, unresolved), NEVER prefix them with formatting that implies a valid reference (e.g., @unknown implies a GitHub mention, #unknown implies an issue/PR). Use a small render helper function with a case to conditionally apply the prefix only for real values, and render a plain sentinel otherwise. This pattern applies to GitHub handles, issue/PR numbers, commit SHAs, email addresses, and any other bracket/sigil-prefixed identifier. R15 also sharpened the ADR validation cross-reference rule (R13): whenever a validation / debug / runbook procedure is written IN an ADR, cross-reference the ADR's own § Empirical Validation Results from the procedure intro — prevents the procedure from drifting out of sync with empirical findings when the latter are updated. This is an extension of the R13 rule (which covered main-body compliance claims) to cover procedural / instructional content as well.
  • Round 16 close-out scope = 4 Copilot R16 fixes touching 3 files (auto-pr-to-main.yml two improvements: RUNNER_TEMP fail-fast for policy-code alignment + email-keyed cache for step 7 contributors-list builder; ADR-004 heading rename for stable anchor; CHANGELOG.md cross-doc link fix). Multi-file change; single commit; squash-merge target preserved. Running total: 63 Copilot comments resolved across 16 rounds.
  • Round 17 close-out scope = 1 Copilot R17 doc-accuracy fix (auto-pr-to-main.yml trap conditional cleanup, correcting the R14 #1 "silent no-op" claim that was incorrect on GNU coreutils) + ecosystem-shaping ADR-002 naming-convention revision (Infisical Sync Key Schema is identity-only, not strip-capable; revised to project-per-scope + identity-mapped secret name) + audited PAT rotation event (regenerated 2026-04-28; 90-day expiration 2026-07-27; stored in Infisical under the new convention). Multi-file change touching 5 files (auto-pr-to-main.yml, workflows/README.md, pat-health-check.yml, ADR-002, CHANGELOG.md). Single commit; squash-merge target preserved. Running total: 64 Copilot comments resolved across 17 rounds.
  • Round 18 close-out scope = 4 Copilot R18 doc-consistency fixes (R18 #1 workflows/README.md §2 Scope & Principles items 3+4 rewritten to match §2.4 Usage Table; R18 #2 INCIDENT_RESPONSE.md Scenario 6 step 6 updated to reference per-repo folder; R18 #3 ADR-004 Layer 1 bullet #3 qualified to align with § Empirical Validation Results PR-creation-time caching semantics + close+reopen remediation note; R18 #4 workflows/README.md header Last updated 2026-04-27 → 2026-04-28) + user-driven 2nd-iteration revision of the 2026-04-28 Infisical naming convention (project-per-repo → folder-per-repo inside the single shared weown-bot GitHub PATs project; namespacing axis is now folder path scoped via Sync Source Path; cleaner operational footprint — one project-level RBAC boundary instead of N). Multi-file change touching 5 files (workflows/README.md, ADR-002, ADR-004, INCIDENT_RESPONSE.md, CHANGELOG.md). Single commit; squash-merge target preserved. Running total: 68 Copilot comments resolved across 18 rounds. Pattern continuation: R18 surfaced a NEW class of drift — same-day convention iteration drift (R17 introduced a vendor-feature-corrected naming convention that itself needed a second-cut revision once the operational namespacing primitives were exercised in practice; the R17 → R18 evolution happened within the same 24-hour window). TWO NEW operational rules codified: (a) Same-document consistency sweep on convention changes — when revising a convention in section X of a doc (e.g., §2.4 Usage Table), sweep ALL earlier sections of the same doc that introduce or summarize the convention (e.g., §2 Scope & Principles, §1 inventory) in the same commit. Documents that introduce a convention before describing the table that uses it must keep the introductory text in lockstep with the table; otherwise onboarding contributors read contradictory guidance. (b) Convention iteration discipline within the same close-out window — when a vendor-feature-driven convention is revised under empirical sync-configuration findings (R17), expect one to two same-day operational-simplification iterations (R18) before the convention stabilizes. Document each iteration as its own Decision Log row; preserve earlier rows for audit trail. The R17 first-iteration row (project-per-repo) was preserved verbatim in ADR-002 Decision Log even though the R18 second-iteration row supersedes it operationally — this preserves the empirical narrative for future ADR review cadences. R18 also extends the R17 rule ("vendor-feature verification before convention design"): the verification step must include exercising the namespacing primitive against the actual UI before publication, not just confirming the primitive exists. Pattern continuation: R17 surfaced a NEW class of drift — vendor-feature assumption drift (an ADR encoded a convention based on an assumed vendor capability that does not exist; the assumption was unchallenged for 5 days until empirical sync-configuration on 2026-04-28). ONE NEW operational rule codified: (a) Vendor-feature verification before convention design — when an architecture or convention relies on a specific vendor feature (e.g., "Infisical Sync supports per-secret rename", "GitHub Actions exposes triggering_actor in pull_request events"), validate the feature exists in the actual UI/API before encoding it in ADR conventions. Document the verification step (UI screenshot, API response capture, or documentation excerpt) in the ADR's Implementation Notes section. Future ADR review cadences should include a "verify cited vendor features still exist" step. R14 #1 "silent no-op" claim correction: the R14 #1 close-out narrative claimed "rm -f \"\"" was a silent no-op; this was incorrect on GNU coreutils. The R17 fix replaces the unconditional cleanup with a conditional branch + corrects the comment block. The R14 #1 CHANGELOG entry is preserved as historical record (the trap consolidation itself was correct; only the empty-operand sub-claim was wrong). PAT rotation 2026-04-28 audit trail: PAT regenerated by @romandidomizio from weown-bot account; old PAT invalidated; new PAT 90-day fine-grained, single-repo scope (WeOwnNetwork/ai), permissions Contents: R / PRs: R/W / Metadata; expiration 2026-07-27. Health-check verification scheduled post-Infisical-sync-completion. Resolves the 2026-04-23 sync drift permanently via the revised naming convention. Pattern continuation: R16 surfaced TWO NEW classes of drift: (i) policy-code drift (comments declared a stricter policy than the code enforced — R16 #1) and (ii) anchor-derivation drift (heading text containing parenthetical date suffixes silently breaks intra-doc references; cross-doc links rendered as bare (#anchor) silently resolve to non-existent current-doc anchors — R16 #3 + #4). TWO NEW operational rules codified: (a) Stable-anchor heading discipline — section headings that need to be cross-referenced should NEVER include parenthetical dates / round-numbers / version qualifiers / any non-alphanumeric noise; provenance metadata goes in an italic sub-line right under the heading (*Source: ...*) instead. Keeps the auto-generated anchor stable across edits. (b) Cross-doc link explicitness — when a CHANGELOG narrative (or any document) paraphrases / summarizes / references content in another file, ALWAYS render the link as an explicit relative path ((./relative/path.md#anchor)), never as a bare (#anchor). The bare form silently resolves to a current-doc anchor that may not exist. R16 also sharpened the policy-code alignment rule: when a comment block declares a security / isolation / privacy policy (e.g., "isolated from /tmp", "emails are PII and not surfaced"), verify the code actually enforces it without silent fallbacks. Fail-fast on policy violations, never fall back to a weaker stance. This is the natural code-side counterpart of the R14 #2 ADR control-evidence accuracy rule (which covered ADR text claims about control mechanisms).
  • Round 19 close-out scope = 1 Copilot R19 label-canonicalization fix (workflows/README.md §3 "Branch name vs. PR body" table row label Contributors: → canonical Contributors on this branch: to match the actual workflow output at auto-pr-to-main.yml step 8 line 320 + the same doc's earlier §3 three-tier table at line 161) + 3 same-doc + cross-doc cascades caught proactively per the R18 #1 "Same-Document Consistency Sweep" rule codified in the previous round (workflows/README.md §11 Troubleshooting row; CONTRIBUTING.md §4 explanatory bullet at line 334; CONTRIBUTING.md §4 parenthetical at line 328) + CONTRIBUTING.md Last updated header bump for R19 traceability. CHANGELOG [v3.3.5.1] line 56 historical narrative was explicitly PRESERVED to retain the "before/after" audit trail of the original R7 rename event (modifying the historical "before" value would corrupt the rename narrative). Multi-file change touching 3 in-repo files (workflows/README.md, CONTRIBUTING.md, CHANGELOG.md) + gitignored PR7_HANDOFF_CHECKLIST.md. Single commit; squash-merge target preserved. Running total: 69 Copilot comments resolved across 19 rounds. Pattern continuation: R19 #1 is the canonical demonstration of the R18 #1 "Same-Document Consistency Sweep" rule's value — Copilot caught a label inconsistency at line 170 that the SAME doc had already fixed at line 161 (same §3 section, just 9 lines earlier); the R18 rule mandates exactly this kind of intra-section sweep, and R19 #1 is the kind of drift the rule was designed to surface within 1 round of the rule being codified. TWO additional sub-rules added under the R18 rule's scope: (a) Workflow-output vs. doc-citation parity — when a doc cites the label of a workflow-emitted line (e.g., a PR body field name), the citation MUST match the actual emitted string verbatim, including any modifier phrases (e.g., "on this branch"). Search-grep before publication: grep '<label>' $WORKFLOW_FILE to confirm the canonical form. (b) Historical-narrative preservation in audit trails — when a CHANGELOG entry documents a rename or label change, the original-value reference in the close-out narrative MUST be preserved verbatim even if subsequent rounds touch the same area. Modifying the quoted historical value corrupts the audit trail (R10 PII-recursive-quote lesson applies in reverse). For R19, this meant explicitly NOT touching CHANGELOG line 56 even though it contains a Contributors: reference, because that reference is part of the historical "before" narrative of the R7 rename event.
  • Round 20 close-out scope = 2 Copilot R20 fixes — (1) R20 #1 runbook-correctness fix on workflows/README.md §11 Troubleshooting "Merge blocked with 'requires signed commits'" row: previous guidance said "add a new signed commit to the branch" which is incorrect (adding a new signed commit does NOT fix earlier unsigned commits — the merge will still be blocked because branch protection requires ALL commits be signed); rewritten per Copilot's literal suggested change to direct contributors to recreate the branch/PR with all commits signed since non-fast-forward blocks history rewriting. (2) R20 #2 doc-vs-UI display canonicalization: Infisical folder paths were inconsistently shown with vs. without trailing slash; user explicitly chose the NO-trailing-slash form (opposite of Copilot's literal suggestion to add slashes everywhere) because the Infisical UI's "Source secret path" field accepts the path without trailing slash and mirrors it verbatim, so the no-slash form survives the docs → UI → docs round-trip without transformation. Cascade across 3 docs: workflows/README.md (~14 path normalizations across §2, §2.4, §4.4, §5.1, §6, §6.1); INCIDENT_RESPONSE.md (Scenario 6 step 6); ADR-002 (Architecture diagram + Naming Convention + Implementation Notes). ADR-002 Decision Log appended with R20 row in correct chronological position after R17 + R18 (NOT a replacement); R17 + R18 historical rows PRESERVED with their original trailing-slash forms per the R19-codified historical-narrative-preservation rule. CHANGELOG [v3.3.5.1] § Changed historical bullets (lines describing R17 + R18 work in their original narrative form) also PRESERVED for the same reason. ADR-002 + INCIDENT_RESPONSE.md Date headers bumped to include R20 references; workflows/README.md Last updated header was already simply 2026-04-28 so no change needed. Multi-file change touching 4 in-repo files (workflows/README.md, ADR-002-infisical-github-sync.md, INCIDENT_RESPONSE.md, CHANGELOG.md) + gitignored PR7_HANDOFF_CHECKLIST.md. Single commit; squash-merge target preserved. Running total: 71 Copilot comments resolved across 20 rounds. Pattern continuation: R20 surfaced TWO new drift classes — (a) runbook-correctness drift (R20 #1) where the narrative was internally consistent but the conclusion was wrong; survived 19 prior rounds because Copilot reviews tend to flag inconsistency rather than logical-correctness bugs in remediation steps. (b) doc-vs-UI display drift (R20 #2) where docs displayed paths in a form that differs from the form the vendor UI accepts/displays, creating real-world copy/paste configuration errors during incident response or new-repo onboarding. TWO NEW operational rules codified under R20: (1) Runbook remediation steps must validate against the underlying enforcement chain — when a troubleshooting row gives a remediation step, trace the step against the enforcement mechanism (here: branch-protection ruleset rule + signed-commits requirement + non_fast_forward enforcement) to verify the step actually unblocks the user. Internal narrative consistency is necessary but not sufficient. (2) Vendor-UI parity for path-like identifiers — when documenting a path-like identifier that users will copy into a vendor UI text field, pick the canonical form that the vendor UI itself uses (or the simpler form when the UI accepts both) and use it consistently across all docs. The cost of doc-vs-UI display mismatch is real-world copy/paste configuration errors during incident response or new-repo onboarding. Going-forward implication: the v3.3.5.1 review cycle (rounds 7-20, 14 rounds total) is establishing a stable taxonomy of doc-accuracy-drift classes that future PRs can pre-validate against — currently 12+ distinct sub-classes documented across all R-rounds.

[v3.3.4.2] — 2026-04-23

Second #WeOwnVer iteration in the same week, following PR #7 (v3.3.4.1) merge. Ships documentation sync for the main branch ruleset (configured on the same day as the merge), a new ADR capturing ruleset decisions + compliance mappings, an inline contributors table + enforcement-posture matrix in CONTRIBUTING.md §4, and a small doc correction surfaced during workflow testing.

Added

  • .github/ADR-003-main-branch-ruleset.md — new ADR documenting the 12 rules enabled on the main ruleset (configured 2026-04-23), mapped to SOC 2 (CC6.1, CC6.3, CC7.1, CC7.2, CC8.1), ISO/IEC 27001:2022 (A.5.15, A.5.28, A.5.37, A.8.13, A.8.24, A.8.29, A.8.32), ISO/IEC 42001:2023 (A.6.2.7, A.6.2.8, A.7.2, A.8.3, A.9.3, A.9.4), NIST CSF 2.0 (PR.AC-4, PR.DS-6, PR.IP-1, PR.IP-3, DE.CM-1, DE.CM-7, DE.CM-8), and CIS Controls v8 (3.11, 11.2, 11.3, 16.1, 16.9, 16.11, 16.12, 18.3). Also documents rules intentionally NOT enabled (linear history, code scanning results, signed tags — latter two tracked as future items), rationale for empty bypass list under SOC 2 CC6.3, and a "Dev Attribution Enforcement Posture" section comparing three options (A=strict regex allowlist, B=reviewer-enforced convention [chosen], C=hybrid warning layer) with explicit numeric upgrade triggers (e.g., "escalate to Option C if team >15 or >10% of merged PRs use non-table handles in a quarter").
  • Platform-sourced developer attribution in auto-pr-to-main.yml step 6 — the workflow reads ${{ github.triggering_actor || github.actor }} to attribute PRs, producing **Triggered by:** @<real-github-username> in the PR body. Zero maintenance: no case-statement mapping to keep in sync, no onboarding / offboarding workflow edits required, no risk of stale mappings drifting from CONTRIBUTING.md. The <dev> segment in branch names is preserved for human-readable naming only. Resolves 4 of 6 Copilot round-1 review comments on PR #13 that flagged @<short-handle> as potential misattribution. (An intermediate iteration added a case-statement mapping; subsequent iteration replaced it with ${{ github.triggering_actor || github.actor }} for correctness-by-default. The single Triggered by: line was later split into a three-tier model on 2026-04-27 — see the [v3.3.5.1] section above for the canonical record of that semantic upgrade.)

Changed

  • .github/workflows/README.md §8.1 rewritten to reflect the as-configured ruleset (12 enabled rules with compliance column) instead of an aspirational recommendation list. Removes pat-health-check.yml from "required status checks" (it is schedule: triggered, cannot be a PR-time gate). Clarifies that "Require code quality results at warning and higher" is satisfied by CodeQL Default Setup (distinct from "Require code scanning results" which requires SARIF via a different API). Adds explicit note on why linear history is intentionally NOT enabled and how "Restrict who can push" is effectively covered by "Require a PR" + empty bypass list.
  • CONTRIBUTING.md §4 Branch Naming — (a) removed the false-invalid example feature/add-thing (which actually DOES pass the regex: add = 2+ char <dev>, thing = 3+ char description) and replaced with genuinely invalid examples (feature/ab-a, feature/roman--double-hyphen, random/roman-test); (b) added a "Convention beyond the regex" note clarifying that <dev> is a human-readable short handle (typically first name / alias), not used for PR attribution — attribution is sourced from ${{ github.triggering_actor || github.actor }} directly; (c) added a "Known contributor handles" table with GitHub handles + branch <dev> segments only (extended context such as legal names, tenure, and roles lives in internal onboarding docs, not this public repo); Nik's branch <dev> segment changed from ncimino to nik (shorter, matches first-name convention); placeholder TODO handles replaced with real ones (@iamwaseem18, @mshahid538); @YonksTEAM added as executive stakeholder reference; (d) added an "Enforcement posture" mini-matrix summarizing the three options documented in detail in ADR-003 so contributors understand why convention-not-mechanical is the current choice and what would trigger a change. Self-contained — no separate file.
  • .github/workflows/README.md header — version bumped from v3.3.4.1 to v3.3.4.2 per #WeOwnVer pre-merge iteration policy (this is the second merge-event iteration in the same ISO week). A subsequent bump to v3.3.5.1 was applied in the same PR on 2026-04-27 once the work crossed into ISO week 5 of April — see [v3.3.5.1] above.
  • .github/CODEOWNERS — replaced placeholder handles @mohammed-TODO / @shahid-TODO / @dhruv-TODO with confirmed real GitHub handles @iamwaseem18 / @mshahid538 / @dhruvmalik007 (active per-path rules unchanged — still @ncimino @romandidomizio until 2026-05-15 handoff decisions are finalized). Added @YonksTEAM as an executive stakeholder reference in the header (not a path reviewer — avoids notification noise). Header PII minimization: legal names, tenure descriptors, and recovery-credential custody details removed from the public header; extended operational context lives in internal onboarding / security docs.
  • Branch naming example sync — updated example branch fix/ncimino-... → fix/nik-... across .github/workflows/branch-name-check.yml (error-message output), .github/workflows/README.md §3, and root README.md §"Branch Strategy" to match Nik's confirmed branch <dev> segment in CONTRIBUTING.md §4. auto-pr-to-main.yml reviewer assignment remains --add-reviewer ncimino,romandidomizio — Nik's GitHub handle @ncimino is unchanged; only his <dev> branch segment changed from ncimino to nik.
  • .github/workflows/README.md §9 + §10 — Reviewer Rotation Procedure step 2 and Transition Checklist row 4 updated to reflect that placeholder-handle replacement is complete (~~strikethrough~~ + ✅), leaving only the per-path specialist-assignment decision as pending. Keeps the transition checklist an accurate audit artifact of what's done vs. still open.
  • .github/INCIDENT_RESPONSE.md Scenario 6 — Stewardship Gap response steps 2 and 3 updated to reference real GitHub handles (@iamwaseem18 / @mshahid538 / @dhruvmalik007) instead of @<name>-TODO placeholders. Incident playbook accuracy matters for SEV-4 on-call response — no ambiguity about who to page.
  • CONTRIBUTING.md §4 now explicitly documents the branch-name-vs-PR-body identifier split: <dev> is a human-readable short handle; PR **Triggered by:** attribution uses ${{ github.triggering_actor || github.actor }} automatically. Added a blockquote callout table for quick reference. Resolves a user-facing ambiguity flagged in Copilot round-1 review of PR #13.
  • .github/workflows/README.md §3 Parsing Rules simplified from 5 steps (including a handle-to-username mapping step) down to 3 steps (regex validation + direct ${{ github.triggering_actor || github.actor }} attribution). Added a "Branch name vs. PR body—two different identifiers (by design)" subsection with a clarity table.
  • .github/workflows/README.md §9 Reviewer Rotation Procedure — stripped contributor legal name in the "placeholder-handle replacement complete" note; kept only the GitHub handles (PII minimization in public runbook, addressing Copilot round-2 review feedback).
  • .github/CODEOWNERS PII minimization — header re-written to remove contributor legal names, tenure descriptors, intern status, executive titles, and recovery-credential custody details. Kept only GitHub handles + minimal functional-area descriptors (IaC, Docker, Agentic AI). Reduces social-engineering surface area on a public repo, addressing Copilot round-2 review feedback.
  • Round-3 consistency + PII follow-ups (Copilot review, 2026-04-24):
    • .github/ADR-003-main-branch-ruleset.md Option A regex example updated from (roman\|ncimino\|mohammed\|...) to (roman\|nik\|mohammed\|...) to match the current <dev> segment convention in CONTRIBUTING.md §4.
    • .github/ADR-003-main-branch-ruleset.md Option B rationale bullet rewritten: previously described a now-retired inline <dev> → GitHub-username mapping + git-author-email fallback; now correctly describes the ${{ github.triggering_actor || github.actor }} platform-sourced attribution. Keeps the ADR a truthful control-evidence artifact.
    • CONTRIBUTING.md §4 — all three references (<dev> intro paragraph, blockquote callout table cell, blockquote explanatory paragraph, Known-contributor-handles onboarding paragraph) updated from shortform github.actor to the full ${{ github.triggering_actor || github.actor }} expression to match the workflow env variable exactly.
    • .github/workflows/README.md §3 "Branch name vs. PR body" table cell updated to use the full ${{ github.triggering_actor || github.actor }} expression to match the Parsing Rules step 3 above it.
    • .github/workflows/README.md §10 Transition Checklist row 2 softened: Transfer 2FA (TOTP seed) + recovery codes to enterprise admin (Yonks) + rotation lead → Transfer 2FA administration per internal runbook to enterprise admin + rotation lead. Removes specific credential-type details from a public runbook (social-engineering surface reduction).
  • Continuation rounds 4–7 (all 2026-04-27) — see [v3.3.5.1] above for the canonical record. Includes the round-5 three-tier PR-body attribution refactor + workflow_dispatch + concurrency, the round-6 PII strip + ADR-004 creation + auto-trigger claim revert, and the round-7 Copilot 8-comment close-out + §2A + §11.

Fixed

  • .github/ADR-003-main-branch-ruleset.md stale path reference — "Not enabled → Require signed tags" bullet previously pointed to .pr7-handoff-checklist (a local-only, gitignored file that does not exist in the repo tree). Replaced with /CHANGELOG.md + docs/VERSIONING_WEOWNVER.md, both of which are in-repo tracking locations. Resolves 1 of 6 Copilot review comments on PR #13.
  • .github/ADR-003-main-branch-ruleset.md date arithmetic — "Next scheduled review" incorrectly read 2026-07-23. 2026-04-23 + 90 calendar days = 2026-07-22 (April 30-23=7 remaining days in April, May=31, June=30 → 7+31+30=68 days, 90-68=22 → July 22). Corrected to 2026-07-22 and clarified "90 calendar days" to remove math ambiguity. Resolves 1 of 6 Copilot review comments on PR #13 about date inconsistency.

Security / Compliance

  • SOC 2 CC6.3 evidence-ready: §8.1 table maps each of the 12 enabled rules to specific SOC 2 Trust Services Criteria. Auditors can cross-reference the GitHub ruleset config with this table for one-shot coverage proof.
  • ISO 27001 A.5.15 + A.5.37: path-reviewer binding (CODEOWNERS #3) + change-of-state reset (stale approval dismissal #2) documented and enforced.
  • ISO 42001 AI governance: rule #10 (auto-request Copilot review) now mapped to Annex A.6.2.7 (AI-aware technical controls) and A.6.2.8 (review + approval of AI-generated output).
  • WEOWN_BOT_PAT rotation audit note — the fine-grained PAT scoped to WeOwnNetwork/ai was regenerated on 2026-04-23 during this PR's cycle to restore the auto-pr-to-main.yml workflow (prior PAT was invalidated; PAT Health Check workflow run 24870414632 confirmed the post-rotation PAT authenticated as weown-bot with 89 days to next expiration on 2026-07-22). New PAT is 89-day fine-grained, single-repo scope, stored via Infisical → GitHub Secret sync. The rotation event itself is an auditable control (no sensitive credential material recorded here).

Meta

  • Testing evidence: all three workflows validated via a 42-case solo regression matrix (branch-name regex coverage, defense-in-depth parity, developer attribution extraction, jq fallback behavior, YAML parse) — see .pr7-test-workspace/solo-tests.sh (local-only, not committed).
  • Version cadence: this section captures the Season 3, April Week 4 work (original PR + Copilot rounds 1–3). Subsequent same-PR work on 2026-04-27 crossed into Week 5 of April and is documented under [v3.3.5.1] above. See docs/VERSIONING_WEOWNVER.md for the full cadence rules.

[v3.3.4.1] — 2026-04-23

First repository-level CHANGELOG entry (#WeOwnVer vSEASON.MONTH.WEEK.ITERATION — Season 3, April, Week 4 of April, Iteration 1). Establishes auto-PR workflow hardening, ecosystem-wide service account, Infisical GitHub Sync, branch naming enforcement, and the initial compliance roadmap.

Added

  • weown-bot ecosystem-wide GitHub service account with per-repo fine-grained PATs, centralized in Infisical project weown-bot GitHub PATs (see ADR-001)
  • Infisical → GitHub Sync integration for WEOWN_BOT_PAT__WEOWNNETWORK_AI (see ADR-002)
  • New workflows:
  • New documents:
    • .github/ADR-001-service-account-pat.md
    • .github/ADR-002-infisical-github-sync.md
    • .github/SECURITY_ASSESSMENT.md — threat model, risk register, compliance mapping
    • .github/INCIDENT_RESPONSE.md — SEV-1..4 runbooks for PAT/account/Infisical/stewardship scenarios
    • .github/CODEOWNERS — path-based reviewer assignment with post-2026-05-15 handoff TODOs
    • .github/workflows/README.md — authoritative ops reference (usage table, rotation procedure, alert stack, transition checklist, branch protection, replication steps)
    • docs/COMPLIANCE_ROADMAP.md — detailed 5-phase compliance roadmap (NIST CSF → CIS → CSA CCM → ISO 27001 → SOC 2 → ISO 42001) with CI/CD integration per phase, success metrics, and forward-looking guardrails
    • CONTRIBUTING.md — first-time developer onboarding. Covers mandatory SSH-based commit signing (NIST PR.DS-6, SOC 2 CC7.1, ISO 27001 A.8.28, CIS 16.12) with 7-step setup including gpg.ssh.allowedSignersFile for local verification and verified-email alignment with GitHub account; the GitHub Flow branching model (explicitly named + lifecycle diagram + what we do NOT use + rationale table + branch lifetime expectations); branch naming convention; commit message conventions; full PR workflow with post-merge cleanup; review expectations; expanded troubleshooting (signature-present-vs-absent pre-check, 6-cause diagnostic for %G?=N, 2-cause diagnostic for GitHub "Unverified" including committer-email mismatch, retro-sign-and-force-push recipe, ssh-agent/keychain fixes); and compliance cross-references
    • CHANGELOG.md — this file
  • PR body enhancements (in auto-pr-to-main.yml):
    • NIST CSF Function-aligned human review checklist (Govern, Identify, Protect, Detect, Respond, Recover)
    • Full commit bodies visible to Copilot for context (%b not just %s)
    • Author identity (%an <%ae>) and date (%ad) shown per commit
    • Developer attribution parsed from branch naming convention (<type>/<dev>-<description>)
    • Deduplicated contributor list
  • Auto-assignment of 2 reviewers (@ncimino + @romandidomizio) via gh pr edit --add-reviewer

Changed

  • auto-pr-to-main.yml: token reference ROMAN_PAT → WEOWN_BOT_PAT
  • .github/copilot-instructions.md: full rewrite — removed command/test directives (Copilot is static-only), added phase-aware compliance, expanded §3 checklist to cover all six frameworks (NIST CSF 2.0, CIS v8 IG1, CSA CCM v4, ISO/IEC 27001:2022, SOC 2 TSC, ISO/IEC 42001:2023) plus seven ecosystem best-practice blocks (Kubernetes, Docker/Compose, IaC [OpenTofu for infrastructure + Ansible for software/config], Infisical, Observability, GitOps, Security/Supply Chain), added §3.0 explicit PUBLIC-repo precautions (never-commit list, placeholder patterns, git history hazards), moved checklist to top of document, added forward-looking guardrails, ecosystem awareness, anti-pattern reference, and cross-framework review output guidelines. Ansible integration documented throughout: Infisical secrets via community.hashi_vault / infisical run -- ansible-playbook; idempotency / handlers / vault / inventory best practices; anti-patterns for plaintext secrets, non-idempotent shell, implicit root
  • .github/CI_CD_WORKFLOWS.md: cross-referenced the new workflows README, ADRs, and compliance roadmap; documented pat-health-check.yml
  • docs/VERSIONING_WEOWNVER.md: rewritten to the corrected calendar-driven methodology (L-094 REVISED): vSEASON.MONTH.WEEK.ITERATION, L-115 ISO-week-offset rule, finalized Season Calendar, Helm/OCI mapping, and calculation cheat sheet
  • README.md (top-level): added "Compliance & Governance" section linking all new docs

Removed

  • .github/PAT_MIGRATION_GUIDE.md deleted — content was superseded by ADR-001 (rationale), ADR-002 (Infisical sync), CHANGELOG (history), and .github/workflows/README.md (authoritative rotation procedure). Retaining the old guide created duplicate-source risk.
  • maintenance branch removed from auto-pr-to-main.yml triggers and all documentation. The repository now standardizes on GitHub Flow: short-lived feature/*, fix/*, docs/*, hotfix/* branches off main, merged back via reviewed PRs.

Security

  • Fine-grained PAT replaces broad-scoped tokens: minimally scoped to Contents: Read + Pull requests: R/W + metadata (auto) on WeOwnNetwork/ai. Issue creation in pat-health-check.yml intentionally uses the ephemeral per-run GITHUB_TOKEN (with workflow-level issues: write) rather than expanding the PAT — principle of least privilege (NIST PR.AC-3 / CIS 5.4)
  • Secret management centralized in Infisical with 90-day audit logs (SOC 2 evidence)
  • Branch naming enforced by branch-name-check.yml (blocks non-conforming branches via required status check). Description segment requires 3+ alphanumeric chars before any hyphen suffix (e.g., feature/ab-a now rejected). Regex kept in sync with the defense-in-depth guard in auto-pr-to-main.yml
  • Branch protection to be configured: require 2 approvals + review from Code Owners + signed commits + no bypass (see .github/workflows/README.md §8)
  • Workflow hardening (Copilot review rounds 3–5):
    • pat-health-check.yml now fails closed (exit 1) when the github-authentication-token-expiration header is missing — previously silently exited 0, which defeated the workflow's safety-net purpose. A missing header indicates token-type misconfiguration (e.g., classic PAT instead of fine-grained) and must surface as a red-X in Actions
    • pat-health-check.yml now fails closed (exit 1) when the header IS present but the timestamp cannot be parsed into an epoch — previously emitted a ::warning:: and silently exited 0, creating a second bypass path that would activate exactly when GitHub changed the header format. Error message includes the offending raw value for forensics (round 4)
    • pat-health-check.yml separates gh api /user exit-code check from header-grep so a transient API/network failure (previously swallowed by 2>/dev/null | grep ... || true) is not misclassified as a PAT-configuration issue. Three distinct red-X paths now exist with targeted error messages: (1) API call itself failed, (2) call succeeded but header absent, (3) header present but unparseable. Stderr of the failing gh api is surfaced in the Actions log (round 5)
    • branch-name-check.yml sets permissions: {} — the workflow makes no API calls and does not check out the repo (only reads github.head_ref / github.ref context and runs a local grep), so the ephemeral GITHUB_TOKEN is stripped of every permission (round 5)
    • pat-health-check.yml removed unused ISSUE_LABELS variable — labels are passed directly to gh issue create via three explicit --label flags; the unused variable falsely implied a single source of truth (round 4)
    • Temp files in both workflows now route through $RUNNER_TEMP (GitHub-runner-scoped, auto-cleaned at job end) instead of the shared /tmp — defense in depth beyond the existing mktemp + trap cleanup pattern
    • pat-health-check.yml issue-body links use ${{ github.server_url }} instead of hardcoded https://github.com — portable to GitHub Enterprise Server (matches the BLOB_BASE pattern already in auto-pr-to-main.yml)
    • auto-pr-to-main.yml PR-existence check uses jq .[0].number // empty — avoids jq's literal "null" string on empty arrays, which would previously cause the script to attempt gh pr edit null
    • docs/VERSIONING_WEOWNVER.md Helm chart mapping corrected for SemVer precedence: every iteration gets a -N prerelease suffix (3.3.4-1 < 3.3.4-2 < 3.3.4-3 < 3.3.4), preventing the SemVer-downgrade pitfall where 3.3.4-2 would sort BELOW 3.3.4 in Helm/OCI tooling

Compliance

  • NIST CSF 2.0 — Govern, Protect (Access Control, Data Security), Detect, Respond functions addressed for auto-PR workflow
  • CIS Controls v8 IG1 — Controls 3, 5, 6, 7, 8, 13, 16, 17 in scope
  • ISO/IEC 27001:2022 — A.5.15 (access control), A.5.37 (documented operating procedures), A.8.2, A.8.24 (cryptographic/secret mgmt), A.8.32 (change mgmt)
  • SOC 2 TSC — CC6.1–CC6.3 (logical access), CC7.1–CC7.2 (system operations), CC8.1 (change mgmt)

Transition Note (2026-05-15)

@romandidomizio departs 2026-05-15. PAT rotation responsibility transitions to one of Mohammed / Shahid / Dhruv. Full handoff checklist in .github/workflows/README.md §10.

Links