Skip to content

fix(infra,scripts): resilient terraform imports and morning-start bring-up - #30

Merged
Bihan-Banerjee merged 1 commit into
mainfrom
fix/morning-start-and-infra-robustness
Jun 4, 2026
Merged

Bihan-Banerjee merged 1 commit into
mainfrom
fix/morning-start-and-infra-robustness

Conversation

@Bihan-Banerjee

Copy link
Copy Markdown
Owner

What & why

Four lifecycle-robustness fixes found while bringing the ephemeral prod cluster up
end-to-end (morning-start.ps1) and recovering Terraform state. No application
behavior changes
— these harden the AWS bring-up so a fresh run doesn't wedge on
benign warnings or pre-existing resources.

1. ECR lifecycle policy → static for_each keys (modules/ecr/main.tf)

aws_ecr_lifecycle_policy.repos keyed its for_each off aws_ecr_repository.repos.
When the repo isn't in state, those keys are "known only after apply", so Terraform
refuses to evaluate the config — which silently rolled back every terraform import (S3 reports bucket, GitHub OIDC provider, ECR repo) during state-drift
recovery, all failing with …already exists. Switched to
for_each = toset(var.image_names) (same source the repo resource uses) + explicit
depends_on for create-ordering. Imports of pre-existing long-lived resources now
persist.

2. Guard ArgoCD kubectl apply against the PowerShell 5.1 stderr trap

Under $ErrorActionPreference = 'Stop', a native command writing to stderr (e.g.
kubectl's harmless non-domain-qualified finalizer warning) is promoted to a
terminating error even though it exited 0 — aborting the whole run after the
objects were already created. Wrapped the applies in a 'Continue' scope (the same
pattern the script's terraform helper already uses) with a finally restore.

3. ArgoCD login via port-forward, not the public host

argocd login argocd.<domain> can't resolve on a fresh bring-up — the ALB and its
Route53 record are seconds old. Now logs in over kubectl port-forward svc/argocd-server on localhost:8088 (cleaned up in finally) and degrades to a
warning + manual instructions
instead of halting. Manual-setup hint updated to the
port-forward method too.

4. Read-TfVar strips inline comments / quotes

kyverno_policy_action = "Audit" # observe first; flip later parsed as
Audit" # observe first; flip later, which splatted as extra args into
guardops admission → Usage: guardops admission [OPTIONS]. Now parses the quoted
value (or drops a trailing # comment for unquoted values) before returning.

Reviewer notes

  • All four were observed live in a real cluster bring-up; each fix was verified
    (terraform validate passes after Phase 3 #1; the script parses clean and Read-TfVar
    now returns Audit for the commented tfvars line).
  • test-project/security/metadata/guardops.db (a local scan DB that mutates every
    run) was intentionally left out of this PR.

Follow-ups (not in this PR)

  • ArgoCD login: add a readiness-wait + retry around the port-forward (login still
    failed once on a 6s fixed sleep).
  • Staging Helm deploy timed out on a single t3.large (resource pressure) — bump
    timeout or make staging optional on one-node clusters.
  • Helm chart guardops-app emits a duplicate port definition warning — drop the
    duplicate container port.
  • Add an Operational Recovery runbook (stale kube-context on local deploys,
    terraform import cheat-sheet, the 2>$null stderr trap, ALB/DNS warm-up).

🤖 Generated with Claude Code

Four fixes found while bringing the ephemeral prod cluster up end-to-end. None
change app behavior; all harden the lifecycle so a fresh bring-up doesn't wedge.

1. ECR lifecycle policy: static for_each keys.
   aws_ecr_lifecycle_policy.repos keyed its for_each off aws_ecr_repository.repos,
   so when the repo was not in state the keys were "known only after apply" - which
   made terraform refuse to evaluate the config and silently rolled back EVERY
   terraform import (S3 bucket, OIDC provider, ECR repo). Switch to
   for_each = toset(var.image_names) (the same source the repo resource uses) with
   an explicit depends_on for ordering. Unblocks importing pre-existing long-lived
   resources back into state.

2. morning-start: guard ArgoCD kubectl apply against the PowerShell 5.1 stderr trap.
   Under ErrorActionPreference=Stop, a native command writing to stderr (e.g.
   kubectl's harmless non-domain-qualified finalizer warning) is promoted to a
   terminating error even on exit 0, aborting the whole run after the objects were
   already created. Wrap the applies in a Continue scope (same pattern the
   terraform helper already uses) with a finally-restore.

3. morning-start: log into ArgoCD via port-forward, not the public host.
   argocd login argocd.<domain> cannot resolve on a fresh bring-up (the ALB +
   Route53 record are seconds old). Log in over kubectl port-forward
   svc/argocd-server on localhost instead, cleaned up in a finally, and degrade to
   a warning + manual instructions instead of halting. Manual-setup hint updated to
   the port-forward method too.

4. morning-start: Read-TfVar strips inline comments / quotes.
   kyverno_policy_action = "Audit"  # comment was parsed as Audit" # comment, which
   splatted as extra args to guardops admission ("Usage: guardops admission
   [OPTIONS]"). Parse the quoted value (or drop a trailing # comment for unquoted
   values) before returning.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@vercel

vercel Bot commented Jun 4, 2026

Copy link
Copy Markdown

Someone is attempting to deploy this pull request to the Bihan Banerjee's projects Team on Vercel.

No GitHub account was found matching the commit author email address.

To deploy this pull request, the commit author's email address needs to be associated with a GitHub account.

Learn more about how to change the commit author information.

@Bihan-Banerjee
Bihan-Banerjee merged commit 68aa4a0 into main Jun 4, 2026
2 of 6 checks passed

This branch had an error being deployed

1 failed deployment
Preview — 65f8f763 Deployed Jun 4, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant