Skip to content

[199] Define each MMGIS environment in Terraform#211

Draft
CarsonDavis wants to merge 8 commits into
feature/193-dev-deploy-cifrom
feature/199-terraform-environments
Draft

[199] Define each MMGIS environment in Terraform#211
CarsonDavis wants to merge 8 commits into
feature/193-dev-deploy-cifrom
feature/199-terraform-environments

Conversation

@CarsonDavis

Copy link
Copy Markdown

Closes #199

Stacked on #210 (targets feature/193-dev-deploy-ci; retarget to development after #210 merges — the Closes fires then).

Defines a complete MMGIS environment in Terraform, ending the hand-built/runbook era. One module, instantiated per environment (development / production — full-word naming throughout), nothing shared between environments, each with its own state bucket.

What's here:

  • infrastructure/terraform/modules/mmgis-environment/ — cluster, Express Mode admin service (aws_ecs_express_gateway_service, AWS provider ≥ 6.22 with committed lock files), per-env task-definition families, RDS (managed master password, deletion protection, postgres username enforced), security groups, log groups, secret shells (values never touch Terraform or its state), per-env ECR, CloudFront + VPC origin + OAC, task/exec/infra roles, and a branch-scoped OIDC deploy role carrying the four empirically-learned IAM/CLI corrections from Deploy the lean dev environment officially from development via CI #193's first live runs
  • environments/development/ + environments/production/ — thin roots with per-env S3 backends (production's bootstrap belongs to Add a protected production branch that deploys its own production environment #195)
  • Workflow: task-def family names move from hard-coded env values to Actions variables (per-env families prevent cross-environment publish contamination)
  • infrastructure/README.md rewritten as apply-docs: two-phase apply (the provider doesn't export the ALB ARN / on.aws endpoint — phase 2 takes them as inputs, including the ALB security-group rule, so no hand-executed mutation remains), first-image-deploy ordering, secret CLI examples with no real values
  • Fidelity: every attribute traced to the production-tested June recipes (independent review verified line-by-line); no account IDs, network IDs, URLs, or secret material committed

Not in this PR: applying it. The dev rebuild under mmgis-development-* names happens after #210 merges and the e2e verification passes; production instantiation is #195.

Translate the June recipes and the live environment's production-tested
values into infrastructure/terraform/modules/mmgis-environment: ECS cluster,
the admin Express Mode gateway service, per-environment admin/publish task
definitions, RDS postgres (master user postgres, manage_master_user_password,
deletion protection), two task security groups, two log groups, five Secrets
Manager secret shells, a per-environment ECR repo, the shared asset bucket +
OAC, the CloudFront distribution + VPC origin, task/exec/infra IAM roles, and
the branch-scoped GitHub OIDC deploy role.

The Express service does not expose its internal ALB ARN or on.aws endpoint as
Terraform attributes, so the CloudFront front door is created in a documented
second apply gated on two phase-2 inputs. No secret values or random_password
anywhere: secret shells only.
Each root is a module call with commit-safe values, an S3 backend configured
via -backend-config (per-environment state bucket, S3 native locking, no
account-specific value committed), a terraform.tfvars.example and
backend.hcl.example carrying only placeholders, and pass-through outputs
(including the workflow variable values). Network ids, the RDS CA bundle, and
the two-phase CloudFront inputs are uncommitted tfvars. Production exists as
config; #195 bootstraps its state bucket and instantiates it.
…cs as apply-docs

deploy-lean.yml now reads ADMIN_TASK_FAMILY/PUBLISH_TASK_FAMILY from repo
variables, falling back to the legacy mmgis-admin/mmgis-publish names. Families
are region-global, so per-environment names stop a production deploy from
registering a revision development's publish-by-family flow would pick up.

Rewrite infrastructure/README.md as apply-docs: prerequisites (existing
VPC/subnets as inputs, per-environment state-bucket bootstrap, secret values
set out-of-band with CLI examples referencing no real values), the two-phase
plan/apply flow, and the workflow variables each environment needs. The runbook
prose is gone; the hard-won operational notes (RDS postgres username, regional
CA bundle, Express Mode not task-def-driven, dual DB secrets, CloudFront origin
details, CI deploy-role facts) are kept and updated to the Terraform world. The
recipe JSON files stay in place as provenance.
….11 floor, phase-2 ALB rule, secret recovery windows
@CarsonDavis

Copy link
Copy Markdown
Author

Reviewer's guide to this diff

A diff explainer for anyone reviewing cold. The PR is large (~2,400 lines) but highly structured: one Terraform module translated from the already-production-tested June recipes (infrastructure/{ecs,iam}/*.json, cloudfront-admin.json, s3-asset-bucket.json), two thin per-environment roots, a README rewrite, and a two-line workflow change. Nothing is applied by this PR — it's all definition.

How to read this diff (2–3 minutes)

  1. Start with infrastructure/README.md — it's the narrative: layout, two-phase apply flow, secret handling, workflow variables. Once you've read it, every .tf file is confirmation rather than discovery.
  2. Then the module, in this order: main.tf (naming/locals — everything else derives from it), ecs.tf, iam.tf + iam_deploy.tf (the security-relevant core), then rds.tf, secrets.tf, network.tf, s3.tf, cloudfront.tf. ecr.tf, logs.tf, outputs.tf, variables.tf, versions.tf are quick reads.
  3. The two environments/{development,production}/ roots are nearly identical thin wrappers — read development, then just diff production's main.tf against it (RDS sizing, branch, recovery window are the only real differences). Lock files, .gitignore, and the tfvars/backend .example files are mechanical.
  4. .github/workflows/deploy-lean.yml is a two-line change plus comments.

File group walkthrough

Workflow (deploy-lean.yml, .gitignore). The task-definition family names move from hard-coded env: values to Actions variables with the legacy names as fallbacks (vars.ADMIN_TASK_FAMILY || 'mmgis-admin'). This is the one required workflow change: ECS task-def families are region-global, so without per-environment families (mmgis-development-admin etc.) a future production deploy would register a revision that development's publish-by-family-name RunTask flow silently picks up — cross-environment contamination. The fallback keeps an unconfigured repo behaving exactly as before. The root .gitignore stops ignoring .terraform.lock.hcl because lock files are now deliberately committed (below).

README rewrite. The old README was the hand-run runbook; the new one is apply-docs: state-bucket bootstrap, the two-phase apply, out-of-band secret values, the workflow-variables table (fed by terraform output workflow_variables), and the operational gotchas carried forward from the recipe era. Key point: the recipe JSON files are kept as provenance — every module attribute traces to a production-tested value in them — but nothing is applied by hand anymore.

Module naming and shape (main.tf, variables.tf, versions.tf). Everything derives from name_prefix = "mmgis-<environment>" — full-word environment names matching the branch names, which also delivers the staging-name retirement from #193/#199. versions.tf floors Terraform at 1.11 and the AWS provider at 6.22: 6.22 is where aws_ecs_express_gateway_service landed, and 1.11 is the first release with S3-native state locking (use_lockfile in the backends — no DynamoDB table). The committed .terraform.lock.hcl files pin the provider (6.55.0) so every operator applies with the same bits.

ECS (ecs.tf). Three resources: the two task definitions and the Express Mode gateway service. Things to understand rather than skim: (a) the placeholder_image trap, documented in a big comment — task defs point at a :latest tag that never exists in ECR (CI pushes SHA tags only), so any Terraform change touching a task def leaves Publish silently broken until the deploy workflow re-registers the families with a real image; (b) the service ignores drift on primary_container[0].image (lifecycle.ignore_changes) because the deploy workflow owns the image; (c) the admin task def is registered as a human-auditable mirror while the service actually runs from its inline primary container — Express Mode is not task-definition-driven; the publish task def is the genuinely load-bearing one; (d) the depends_on on the service prevents the provider's documented delete-time race (service stuck DRAINING if its IAM policies are destroyed first).

Two-phase CloudFront (cloudfront.tf, network.tf bottom, s3.tf bucket policy, the three express_* variables). The provider does not export the Express service's internal ALB ARN or on.aws endpoint (only service_arn and ingress_paths), and the CloudFront VPC origin needs the ALB ARN. So the front door is gated behind local.enable_cloudfront: phase 1 creates everything else, the operator reads three values from one describe-express-gateway-service call (ALB ARN, on.aws host, the ECS-managed ALB's SG id), sets them in tfvars, and re-applies. Phase 2 also covers the :443-from-VPC-CIDR ingress rule on the ECS-owned ALB SG (aws_vpc_security_group_ingress_rule — a rule on a foreign SG, not ownership of it) and the asset-bucket policy (needs the distribution ARN). Net effect claimed by the PR: no hand-executed mutation remains anywhere in the flow. The load-bearing CloudFront details from the recipes carry over verbatim: origin DomainName must be the on.aws endpoint (SNI + host-header rule), AllViewerExceptHostHeader + CachingDisabled on the default behavior, CachingOptimized on /assets/*. MinimumProtocolVersion is intentionally dropped — the provider forbids it alongside the default certificate.

Secret-shell discipline (secrets.tf, rds.tf). Five aws_secretsmanager_secret resources with no aws_secretsmanager_secret_version and no random_password anywhere — either would land a secret value in Terraform state. Values are set out-of-band via CLI. RDS squares the "no values in state" rule with aws_db_instance's create-time password requirement via manage_master_user_password = true (RDS generates/rotates the master password in its own managed secret). That managed secret's {username,password} shape doesn't match what the app reads, so the separate app-shaped mmgis/<env>/db secret exists on purpose — the operator copies the password across. Also encoded from the June pain: master username hard-coded to postgres (the app's init defaults the maintenance DB to the username), db_name deliberately unset, and dev's secret_recovery_window_days = 0 so destroy/re-apply doesn't collide with names held 30 days (production keeps 30).

IAM (iam.tf, iam_deploy.tf). iam.tf is a faithful translation of the recipe roles — exec/task split per task, mmgis-dashboard-* pinning, the PassRole-both-publish-roles requirement, and Resource: "*" appearing only where AWS defines the action as unscopable (ecr:GetAuthorizationToken here; ecs:RegisterTaskDefinition in the deploy role) — with literal ARNs replaced by resource references. iam_deploy.tf is the newest thinking: the per-environment GitHub OIDC deploy role, carrying the corrections learned in #193's live runs (DescribeServices on the service ARN; the Express actions authorizing against the service ARN so Resource lists both ARN shapes; Update+Describe only — no Create, so a compromised branch token can't stand up new services). Its trust is branch-scoped (repo:NASA-IMPACT/MMGIS:ref:refs/heads/<branch>), not Environment-scoped — deliberately: the workflow declares no environment: yet, and a job without one presents a branch-ref OIDC subject, so Environment-scoped trust would fail authentication today. #195 tightens it when it wires environment: in.

Environment roots. Thin by design: only values that must not be committed (VPC/subnet ids, CA bundle, phase-2 inputs) are variables via git-ignored tfvars; everything safe to commit is inline in main.tf. Development: db.t3.micro, single-AZ, skip_final_snapshot, recovery window 0 — disposable. Production: db.t3.medium, Multi-AZ, final snapshot, 30-day window — exists as config only; #195 applies it. Separate state buckets per environment, keyed at init time via -backend-config so no account id is committed.

What to scrutinize

  1. iam_deploy.tf statements — this is the role a GitHub branch token assumes. Check each statement's Resource pinning and the PassRole condition; confirm you agree Update-without-Create is the right ceiling for CI.
  2. rds.tf settingsdeletion_protection = true unconditionally (even dev; teardown requires a two-step), apply_immediately = false, engine "17" floating minor versions, gp3/20GiB dev sizing. These are judgment calls, not recipe translations.
  3. The placeholder-image trap in ecs.tf — the failure mode is silent (Publish breaks until the next CI deploy). Is a comment + README note enough mitigation, or should something enforce it?
  4. Two-phase gating consistencyenable_cloudfront requires ALB ARN + endpoint, but the ALB SG rule gates independently on its own variable. Trace the count conditions and make sure a partial phase-2 tfvars can't produce a broken in-between state.
  5. Recipe fidelity spot-check — pick one role (say publish_task) and diff it against infrastructure/iam/publish-task-role.json; the PR claims line-by-line fidelity, and one sample verifies the method.

What's deliberately NOT here

…nvironments

Carries the ADMIN_URL manually-maintained-by-design note (966b531) up the
stack. Resolution: the workflow header keeps this branch's extended var list
plus the note; 193's README sentence has no counterpart in this branch's
rewritten README, so the note lands in the Workflow variables table row for
ADMIN_URL instead.
…nvironments

Carries the ADMIN_URL removal (d0e8306) up the stack: the variables-table
row is gone, ADMIN_URL leaves the workflow_variables output map (it is no
longer an Actions variable), and the terraform admin_url output stays as
the authoritative — and only — place to read the admin URL.
The admin is internal tooling — the shipped product is the published
dashboards, which serve independently of this stack — so production gets
no availability/sizing headroom development doesn't have. The
environments now differ only in deletion policy (final snapshot on
destroy, secret recovery window) and the deploy branch.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant