hardening(deploy): runtime override, robust env vars, smoke test by karthikeyangs9 · Pull Request #8 · last9/aws-cost-reporter

karthikeyangs9 · 2026-04-30T05:42:55Z

Stacked on top of #7. Merge #7 first; this PR's diff will then auto-rebase against main.

Summary

Three follow-up improvements to deploy.sh after the idempotency fixes in #7. Each addresses a real (or anticipated) lower-severity failure mode.

#	Issue	Risk
1	Lambda runtime hardcoded as `python3.13`	When AWS deprecates the runtime, every redeploy errors. No path to upgrade existing functions without delete + recreate.
2	Env vars serialized as `Variables={k=v,k=v}` shorthand	Breaks for any value containing `=` or `,` — possible for custom JWTs and unusual `OTEL_SERVICE_NAME` values.
3	No post-deploy verification	A customer's IAM-policy-drift bug (#7 fix #2) only surfaced on manual `lambda invoke` after deploy "succeeded" — would have been caught at deploy time with a smoke test.

Fixes

1. Configurable runtime

RUNTIME="${RUNTIME:-python3.13}"

Used in both create-function and update-function-configuration paths. Existing customers can migrate runtimes via:

RUNTIME=python3.14 ./deploy.sh

2. JSON-encoded environment via temp file

Replaces:

ENV_VARS="Variables={OTEL_EXPORTER_OTLP_ENDPOINT=...,OTEL_EXPORTER_OTLP_HEADERS=...}"
aws lambda update-function-configuration --environment "${ENV_VARS}"

With a python json.dumps-built JSON file passed via file://:

ENV_FILE=$(mktemp ...)
python3 -c 'import json,os; print(json.dumps({"Variables": {...}}))' > "${ENV_FILE}"
aws lambda update-function-configuration --environment "file://${ENV_FILE}"

Python is already a hard dependency of this Lambda (the runtime itself), so no new tooling. JSON escapes every character class correctly. Temp file is registered with the existing trap.

3. Smoke test (default on, `SMOKE_TEST=1`)

FUNCTION_ERROR=$(aws lambda invoke ... --query 'FunctionError' --output text)
if [[ "${FUNCTION_ERROR}" != "None" ]]; then
  echo "✗ Smoke test FAILED" >&2
  exit 1
fi

Catches IAM permission gaps, OTLP endpoint typos, and network egress blocks at deploy time instead of silently shipping a broken Lambda. Uses --output text (returns literal "None" for null) for reliable comparison instead of JSON grep.

SMOKE_TEST=0 ./deploy.sh opts out — useful for CI roles without lambda:InvokeFunction permission, or to avoid extra Cost Explorer API charges outside the daily cron budget.

Test plan

bash -n deploy.sh — syntax OK
Diff self-review — fixed brittle grep '"FunctionError"' (matches null too) → reliable [[ "$VAR" != "None" ]]
Sandbox: RUNTIME=python3.14 ./deploy.sh against existing function — confirm runtime upgrade applies
Sandbox: deploy with OTEL_EXPORTER_OTLP_HEADERS='Authorization=Bearer token=with=equals,and,commas' — confirm Lambda env vars are correct in console
Sandbox: deploy with intentionally-broken IAM policy — confirm smoke test exits 1 with FunctionError: Unhandled message
Sandbox: SMOKE_TEST=0 ./deploy.sh — confirm script ends without invoking

Three quality-of-life improvements that build on the idempotency fixes in the parent commit. Each addresses a real (though lower-severity) failure mode observed or anticipated. 1. Configurable Lambda runtime (RUNTIME env var, default python3.13). Runtime was hardcoded — when AWS deprecates the version, every redeploy starts erroring. The update path now also passes --runtime so existing functions migrate (e.g. RUNTIME=python3.14 ./deploy.sh) instead of needing delete + recreate. 2. Robust environment variable serialization via python json.dumps. The shorthand "Variables={k=v,k=v}" syntax breaks if any value contains '=' or ',' — possible for custom JWT tokens and comma-bearing service names. Switching to "--environment file://path.json" with a properly JSON-encoded file handles every character class. Temp file is registered with the existing trap. 3. Post-deploy smoke test (SMOKE_TEST=1, default on). The previous deploy printed "✓ Deployed" even when runtime errors existed (a customer's IAM-policy-drift bug only surfaced on manual invoke). The smoke test invokes the function once, captures the FunctionError field via --query/--output text (returns "None" for null — reliable string comparison, avoids brittle JSON grep), and exits 1 with a hint message if the Lambda returned an error. Set SMOKE_TEST=0 to skip — useful for CI deploys that don't have permission to invoke or where the smoke run would charge Cost Explorer API fees outside the daily cron budget.

…test Integrates hardening from #7 + #8 into the idempotent deploy work: - trap-based temp file cleanup (BUILD_DIR, ZIP_PATH, ENV_FILE) - wait function-updated before update-function-code (handles in-flight state from a prior crashed/concurrent run) - wait function-active after create-function (newly created Lambdas start in Pending; add-permission requires Active) - selective ResourceConflictException handling on add-permission so unrelated failures (regional, IAM, throttling) surface instead of being swallowed by `2>/dev/null || true` - RUNTIME override (default python3.13) on both create-function and update-function-configuration - JSON env vars via python json.dumps — fixes shorthand breakage when any value contains '=' or ',' (Last9 OTLP tokens, comma-bearing service names) - SMOKE_TEST=1 opt-in post-deploy lambda invoke that fails the script on FunctionError. Default 0 to avoid spurious data points on every deploy during initial setup. Co-Authored-By: karthikeyangs9 <174431822+karthikeyangs9@users.noreply.github.com> Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(deploy): always reapply IAM policy so existing roles pick up new perms Previously put-role-policy only ran on initial role creation. Customers re-running deploy.sh after a new code release (e.g., PR #5 added ce:GetDimensionValues) hit AccessDeniedException at runtime because the existing role wasn't patched. put-role-policy and attach-role-policy are idempotent — always reapply. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(deploy): wait for in-flight Lambda updates; document troubleshooting deploy.sh: Add `aws lambda wait function-updated` before and between update-function-code / update-function-configuration calls. Without these waits, back-to-back deploys hit ResourceConflictException ("An update is in progress for resource: ..."). README: Add Troubleshooting section covering the four errors seen in customer rollouts: missing CE IAM permissions, in-flight Lambda update conflict, stale GroupBy 3-dim build, and the old mktemp zip failure. Also clarify that AWS_REGION controls Lambda placement only — Cost Explorer is global — so users override it via env var rather than editing the script. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * simplify: drop redundant Lambda waits and clean up role flag - Drop pre-update wait (defensive, no prior async mutation in this path) - Drop trailing wait after update-function-configuration (EventBridge put-rule/put-targets don't read Lambda update state) - Keep the wait between update-function-code and update-function-configuration (the actual race that caused ResourceConflictException) - ROLE_CREATED flag → simpler NEW_ROLE empty/non-empty test - Trim narrating comments Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(deploy): harden Lambda update flow, JSON env vars, opt-in smoke test Integrates hardening from #7 + #8 into the idempotent deploy work: - trap-based temp file cleanup (BUILD_DIR, ZIP_PATH, ENV_FILE) - wait function-updated before update-function-code (handles in-flight state from a prior crashed/concurrent run) - wait function-active after create-function (newly created Lambdas start in Pending; add-permission requires Active) - selective ResourceConflictException handling on add-permission so unrelated failures (regional, IAM, throttling) surface instead of being swallowed by `2>/dev/null || true` - RUNTIME override (default python3.13) on both create-function and update-function-configuration - JSON env vars via python json.dumps — fixes shorthand breakage when any value contains '=' or ',' (Last9 OTLP tokens, comma-bearing service names) - SMOKE_TEST=1 opt-in post-deploy lambda invoke that fails the script on FunctionError. Default 0 to avoid spurious data points on every deploy during initial setup. Co-Authored-By: karthikeyangs9 <174431822+karthikeyangs9@users.noreply.github.com> Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: karthikeyangs9 <174431822+karthikeyangs9@users.noreply.github.com>

prathamesh-sonpatki · 2026-04-30T09:32:23Z

Integrated into #9 (squash-merged as 65b6452). Thanks @karthikeyangs9!

prathamesh-sonpatki mentioned this pull request Apr 30, 2026

fix(deploy): make deploy.sh idempotent and document troubleshooting #9

Merged

5 tasks

prathamesh-sonpatki closed this in #9 Apr 30, 2026

Navigation Menu

Repository navigation

hardening(deploy): runtime override, robust env vars, smoke test - #8

hardening(deploy): runtime override, robust env vars, smoke test#8
karthikeyangs9 wants to merge 1 commit into
fix/deploy-lambda-update-racefrom
hardening/deploy-script-polish

karthikeyangs9 commented Apr 30, 2026

Uh oh!

prathamesh-sonpatki commented Apr 30, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

karthikeyangs9 commented Apr 30, 2026

Summary

Fixes

1. Configurable runtime

2. JSON-encoded environment via temp file

3. Smoke test (default on, SMOKE_TEST=1)

Test plan

Uh oh!

prathamesh-sonpatki commented Apr 30, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

3. Smoke test (default on, `SMOKE_TEST=1`)