Repository navigation
fix(infra,scripts): resilient terraform imports and morning-start bring-up - #30
Merged
Merged
Conversation
Four fixes found while bringing the ephemeral prod cluster up end-to-end. None
change app behavior; all harden the lifecycle so a fresh bring-up doesn't wedge.
1. ECR lifecycle policy: static for_each keys.
aws_ecr_lifecycle_policy.repos keyed its for_each off aws_ecr_repository.repos,
so when the repo was not in state the keys were "known only after apply" - which
made terraform refuse to evaluate the config and silently rolled back EVERY
terraform import (S3 bucket, OIDC provider, ECR repo). Switch to
for_each = toset(var.image_names) (the same source the repo resource uses) with
an explicit depends_on for ordering. Unblocks importing pre-existing long-lived
resources back into state.
2. morning-start: guard ArgoCD kubectl apply against the PowerShell 5.1 stderr trap.
Under ErrorActionPreference=Stop, a native command writing to stderr (e.g.
kubectl's harmless non-domain-qualified finalizer warning) is promoted to a
terminating error even on exit 0, aborting the whole run after the objects were
already created. Wrap the applies in a Continue scope (same pattern the
terraform helper already uses) with a finally-restore.
3. morning-start: log into ArgoCD via port-forward, not the public host.
argocd login argocd.<domain> cannot resolve on a fresh bring-up (the ALB +
Route53 record are seconds old). Log in over kubectl port-forward
svc/argocd-server on localhost instead, cleaned up in a finally, and degrade to
a warning + manual instructions instead of halting. Manual-setup hint updated to
the port-forward method too.
4. morning-start: Read-TfVar strips inline comments / quotes.
kyverno_policy_action = "Audit" # comment was parsed as Audit" # comment, which
splatted as extra args to guardops admission ("Usage: guardops admission
[OPTIONS]"). Parse the quoted value (or drop a trailing # comment for unquoted
values) before returning.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
Someone is attempting to deploy this pull request to the Bihan Banerjee's projects Team on Vercel. No GitHub account was found matching the commit author email address. To deploy this pull request, the commit author's email address needs to be associated with a GitHub account. Learn more about how to change the commit author information. |
This branch had an error being deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
Four lifecycle-robustness fixes found while bringing the ephemeral prod cluster up
end-to-end (
morning-start.ps1) and recovering Terraform state. No applicationbehavior changes — these harden the AWS bring-up so a fresh run doesn't wedge on
benign warnings or pre-existing resources.
1. ECR lifecycle policy → static
for_eachkeys (modules/ecr/main.tf)aws_ecr_lifecycle_policy.reposkeyed itsfor_eachoffaws_ecr_repository.repos.When the repo isn't in state, those keys are "known only after apply", so Terraform
refuses to evaluate the config — which silently rolled back every
terraform import(S3 reports bucket, GitHub OIDC provider, ECR repo) during state-driftrecovery, all failing with
…already exists. Switched tofor_each = toset(var.image_names)(same source the repo resource uses) + explicitdepends_onfor create-ordering. Imports of pre-existing long-lived resources nowpersist.
2. Guard ArgoCD
kubectl applyagainst the PowerShell 5.1 stderr trapUnder
$ErrorActionPreference = 'Stop', a native command writing to stderr (e.g.kubectl's harmless non-domain-qualified finalizer warning) is promoted to a
terminating error even though it exited
0— aborting the whole run after theobjects were already created. Wrapped the applies in a
'Continue'scope (the samepattern the script's terraform helper already uses) with a
finallyrestore.3. ArgoCD login via port-forward, not the public host
argocd login argocd.<domain>can't resolve on a fresh bring-up — the ALB and itsRoute53 record are seconds old. Now logs in over
kubectl port-forward svc/argocd-serveronlocalhost:8088(cleaned up infinally) and degrades to awarning + manual instructions instead of halting. Manual-setup hint updated to the
port-forward method too.
4.
Read-TfVarstrips inline comments / quoteskyverno_policy_action = "Audit" # observe first; flip laterparsed asAudit" # observe first; flip later, which splatted as extra args intoguardops admission→Usage: guardops admission [OPTIONS]. Now parses the quotedvalue (or drops a trailing
# commentfor unquoted values) before returning.Reviewer notes
(terraform
validatepasses after Phase 3 #1; the script parses clean andRead-TfVarnow returns
Auditfor the commented tfvars line).test-project/security/metadata/guardops.db(a local scan DB that mutates everyrun) was intentionally left out of this PR.
Follow-ups (not in this PR)
failed once on a 6s fixed sleep).
t3.large(resource pressure) — bumptimeout or make staging optional on one-node clusters.
guardops-appemits aduplicate port definitionwarning — drop theduplicate container port.
terraform import cheat-sheet, the
2>$nullstderr trap, ALB/DNS warm-up).🤖 Generated with Claude Code