Repository navigation
fix(deploy): make deploy.sh idempotent and document troubleshooting - #9
Merged
prathamesh-sonpatki merged 4 commits intoApr 30, 2026
Merged
Conversation
…perms Previously put-role-policy only ran on initial role creation. Customers re-running deploy.sh after a new code release (e.g., PR #5 added ce:GetDimensionValues) hit AccessDeniedException at runtime because the existing role wasn't patched. put-role-policy and attach-role-policy are idempotent — always reapply. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
deploy.sh:
Add `aws lambda wait function-updated` before and between
update-function-code / update-function-configuration calls. Without
these waits, back-to-back deploys hit ResourceConflictException
("An update is in progress for resource: ...").
README:
Add Troubleshooting section covering the four errors seen in customer
rollouts: missing CE IAM permissions, in-flight Lambda update conflict,
stale GroupBy 3-dim build, and the old mktemp zip failure. Also
clarify that AWS_REGION controls Lambda placement only — Cost
Explorer is global — so users override it via env var rather than
editing the script.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Drop pre-update wait (defensive, no prior async mutation in this path) - Drop trailing wait after update-function-configuration (EventBridge put-rule/put-targets don't read Lambda update state) - Keep the wait between update-function-code and update-function-configuration (the actual race that caused ResourceConflictException) - ROLE_CREATED flag → simpler NEW_ROLE empty/non-empty test - Trim narrating comments Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…test Integrates hardening from #7 + #8 into the idempotent deploy work: - trap-based temp file cleanup (BUILD_DIR, ZIP_PATH, ENV_FILE) - wait function-updated before update-function-code (handles in-flight state from a prior crashed/concurrent run) - wait function-active after create-function (newly created Lambdas start in Pending; add-permission requires Active) - selective ResourceConflictException handling on add-permission so unrelated failures (regional, IAM, throttling) surface instead of being swallowed by `2>/dev/null || true` - RUNTIME override (default python3.13) on both create-function and update-function-configuration - JSON env vars via python json.dumps — fixes shorthand breakage when any value contains '=' or ',' (Last9 OTLP tokens, comma-bearing service names) - SMOKE_TEST=1 opt-in post-deploy lambda invoke that fails the script on FunctionError. Default 0 to avoid spurious data points on every deploy during initial setup. Co-Authored-By: karthikeyangs9 <174431822+karthikeyangs9@users.noreply.github.com> Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
prathamesh-sonpatki
deleted the
fix-deploy-idempotent-and-troubleshooting
branch
April 30, 2026 09:32
This was referenced Apr 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Customer rollouts surfaced two latent bugs in
deploy.shand a docs gap.1. IAM policy not reapplied on existing roles
put-role-policywas inside theif ! get-rolebranch, so it only ran on initial role creation. When PR #5 addedce:GetDimensionValuesto the policy JSON, customers re-running./deploy.shagainst an existing role didn't get the new permission. Lambda then hit:put-role-policyandattach-role-policyare idempotent — moved outside the branch so deploy.sh self-heals after permission changes.2.
ResourceConflictExceptionon back-to-back updatesupdate-function-codeis async —update-function-configurationimmediately after races with the in-flight code update:Added
aws lambda wait function-updatedbefore each update call. Also waits beforeupdate-function-codeitself (handles rerun-after-crash) and aftercreate-function(newly created Lambdas start in Pending;add-permissionrequires Active).3. Troubleshooting docs
Added a Troubleshooting section to
aws-cost-explorer/README.mdcovering all four errors hit during rollouts: missing CE IAM perms, in-flight Lambda update, stale GroupBy 3-dim build, the old mktemp zip failure. Also clarified thatAWS_REGIONis an env var override (some users edited the script directly).4. Hardening folded in from #7 + #8 (Closes #7, Closes #8)
trap-based temp file cleanup (zip + env file)ResourceConflictExceptionhandling onadd-permissionso unrelated failures surfaceRUNTIMEoverride env varpython json.dumps— fixes shorthand breakage when any value contains=or,(OTLP tokens, comma-bearing service names)SMOKE_TEST=1opt-in post-deploy invoke that fails on FunctionErrorCloses #7
Closes #8
Test plan
./deploy.shon a fresh account creates role + policy + Lambda./deploy.shre-run against existing role re-applies the inline policy (verify withaws iam get-role-policy --role-name aws-cost-reporter-role --policy-name cost-explorer-read)./deploy.shruns no longer hitResourceConflictExceptionOTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic <token-with-=>"survives round-trip into Lambda env (no shorthand breakage)SMOKE_TEST=1 ./deploy.shfails the script when IAM/OTLP misconfigured