|
| 1 | +# Staged rollout of the framework-mode build |
| 2 | + |
| 3 | +This is the plan for moving a deployment that uses the [`cloudfront_intercode`](../terraform/modules/cloudfront_intercode) |
| 4 | +module from the Rails-rendered shell (the data-mode build) to the static framework-mode shell served from S3, one |
| 5 | +convention domain at a time. It is part of the migration tracked in |
| 6 | +[#12219](https://github.com/neinteractiveliterature/intercode/issues/12219) and |
| 7 | +[#12260](https://github.com/neinteractiveliterature/intercode/issues/12260). |
| 8 | + |
| 9 | +Nothing here removes the data-mode build. Every release still builds and publishes both, so each step can be undone by |
| 10 | +changing one Terraform variable. The code cutover (v9.0.0), which makes the framework build the only one and gives the |
| 11 | +Rails image its own copy of the shell, comes after this rollout is finished. Domains that Rails serves directly, with no |
| 12 | +CloudFront in front, move with that release. |
| 13 | + |
| 14 | +## Before the first domain |
| 15 | + |
| 16 | +- The repository variables `BUILD_FRAMEWORK`, `FRAMEWORK_SHELL_BUCKET` and `FRAMEWORK_SHELL_UPLOAD_ROLE_ARN` are set, so |
| 17 | + every release builds the framework image and uploads it to the shell bucket. Leave them set for the whole rollout: |
| 18 | + unsetting `BUILD_FRAMEWORK` while the bucket is in use means the bucket stops getting new releases. |
| 19 | +- The shell bucket comes from [`assets_bucket_intercode`](../terraform/modules/assets_bucket_intercode) (a separate |
| 20 | + bucket from the one for `/packs/*`), and its `cloudfront_distribution_arns` includes the distribution you are about to |
| 21 | + change. |
| 22 | +- At least one release has uploaded to it. Check the bucket has `index.html` and an `assets/` directory. |
| 23 | +- The distribution has access logs on (`access_logs_enabled`, from |
| 24 | + [`cloudfront_intercode_logging`](../terraform/modules/cloudfront_intercode_logging)), and the logging module's 5xx |
| 25 | + alarm has an `alarm_actions` topic someone reads. Both are what you will be watching. |
| 26 | + |
| 27 | +## The order |
| 28 | + |
| 29 | +1. **P** (for example `p.interconlarp.org`), the test bed. |
| 30 | +2. **I**, once P has been quiet through at least one release. |
| 31 | +3. The rest of the CloudFront domains, together or in small groups. |
| 32 | +4. The cutover release (v9.0.0), which moves everything that Rails serves directly. |
| 33 | + |
| 34 | +Each step is the same change on one distribution: |
| 35 | + |
| 36 | +```hcl |
| 37 | +static_shell_enabled = true |
| 38 | +static_shell_bucket_domain_name = module.shell_bucket.bucket_regional_domain_name |
| 39 | +static_shell_origin_access_control_id = module.shell_bucket.origin_access_control_id |
| 40 | +``` |
| 41 | + |
| 42 | +After the apply, the distribution takes a few minutes to deploy. Let it finish before testing. |
| 43 | + |
| 44 | +## Checking a domain |
| 45 | + |
| 46 | +Do this right after the apply, and again after the first release that lands while the domain is on the static shell. |
| 47 | + |
| 48 | +- **The shell comes from the bucket.** `curl -sI https://<host>/` returns the shell with `Cache-Control: max-age=60` |
| 49 | + (the same value as `static_shell_ttl`), and a deep link such as `/events` returns the same document. A request for a |
| 50 | + file under `/assets/` that doesn't exist is a 404, not the shell. |
| 51 | +- **Crawler previews.** A link-preview user agent gets per-resource metadata from Rails, not the shell: |
| 52 | + |
| 53 | + ```sh |
| 54 | + curl -s -A 'Slackbot-LinkExpanding 1.0' https://<host>/events/<an event id> | grep -E 'og:(title|url)' |
| 55 | + ``` |
| 56 | + |
| 57 | + `og:title` should be the event's title and `og:url` the page's own address. Paste a link into Slack as well, since |
| 58 | + that is the real consumer. |
| 59 | + |
| 60 | +- **Sign-in.** In a private window: sign in, reload (still signed in), open another page, sign out, reload (still |
| 61 | + signed out). Sign-in goes through the root site's sign-in page, so check that the round trip ends signed in on the |
| 62 | + domain you started from. |
| 63 | +- **A page that needs the API.** Open a page with a loader, such as an event, and an admin page if you have the access. |
| 64 | + These fail loudly if `/graphql` or `/client_configuration` isn't reaching Rails. |
| 65 | +- **A stale tab survives a release.** Leave a tab open on the site across a release, then click through to a page you |
| 66 | + haven't opened yet. It should keep working, because the bucket keeps the earlier release's files, and if a file is |
| 67 | + missing the app reloads once and then shows a message with a reload button. |
| 68 | + |
| 69 | +## What to watch |
| 70 | + |
| 71 | +- **The 5xx alarm** from `cloudfront_intercode_logging`. It counts responses, not a rate, because these sites see tens |
| 72 | + to hundreds of requests a day. Requests to a raw `*.cloudfront.net` name are excluded. |
| 73 | +- **Logs Insights** on the `/cloudfront/<name>` log group, for anything that is failing but not 5xx: |
| 74 | + |
| 75 | + ``` |
| 76 | + filter `sc-status` like /^5/ |
| 77 | + | stats count(*) by `x-host-header`, `cs-uri-stem` |
| 78 | + ``` |
| 79 | + |
| 80 | + Also look at 404s under `/assets/`, which mean a page is asking for a file the bucket doesn't have: |
| 81 | + |
| 82 | + ``` |
| 83 | + filter `sc-status` like /^404$/ and `cs-uri-stem` like /^\/assets\// |
| 84 | + | stats count(*) by `x-host-header`, `cs-uri-stem` |
| 85 | + ``` |
| 86 | + |
| 87 | + 4xx responses overall are not alarmed on: anonymous visitors get an expected 401 from `/oauth_session/refresh` on |
| 88 | + every page load. |
| 89 | + |
| 90 | +- **Sentry and Rollbar** for new frontend errors from the domain after the change. |
| 91 | +- **Your release workflow.** The `framework-shell-release-*` jobs have to succeed, or the bucket is a release behind. |
| 92 | + |
| 93 | +Leave each domain on the static shell for at least one release and a few days of normal traffic before moving the next. |
| 94 | + |
| 95 | +## Rolling back |
| 96 | + |
| 97 | +Remove the three variables above (or set `static_shell_enabled = false`) and apply. The default behavior goes back to the |
| 98 | +Rails origin, and Rails renders the shell and loads the data-mode build from `/packs/`. The distribution takes a few |
| 99 | +minutes to deploy, and a browser that already has the static shell keeps it until it reloads the page. Nothing else |
| 100 | +changes with it: the API, the sign-in cookies and the database are the same in both modes, so a tab open on either build |
| 101 | +keeps talking to the same backend. |
| 102 | + |
| 103 | +Don't delete the bucket or unset the repository variables to roll back; leaving them in place means going forward again |
| 104 | +is the same one-line change. |
| 105 | + |
| 106 | +This only works while releases still build the data-mode build. Once v9.0.0 is out it no longer does, and a rollback is |
| 107 | +redeploying the last 8.x image to Fly and removing the static shell variables. |
0 commit comments