Provision the Graviton 5 rustc-perf collector - #1173
Conversation
Allocate an M9g Dedicated Host in us-east-2 and place an m9g.12xlarge partition on its isolated no-ingress network. Use Session Manager for administration, require an approved host quota, and smoke-test cycles, instructions, branches, and branch-miss PMU counters during boot.
|
|
||
| filter { | ||
| name = "name" | ||
| values = ["ubuntu/images/hvm-ssd-gp3/ubuntu-noble-24.04-arm64-server-*"] |
There was a problem hiding this comment.
Can we switch to 26.04 from the start?
| // `aws ssm start-session`; it is not a startup script for the instance. | ||
| // Managing it here makes the no-SSH access path available without a manual | ||
| // console step, runs operators as Ubuntu's normal sudo-capable user, and puts | ||
| // finite idle and total limits on forgotten sessions. |
There was a problem hiding this comment.
I think we can try SSM as the access mechanism to start, but we should think about whether just exposing ssh on port 22 to the bastion is reasonable. We do that for other instances we manage and it feels fairly reasonable to me.
| sysctl --system | ||
|
|
||
| # Canonical's AWS images ship the agent as a snap. Keep this fallback for an | ||
| # image where it is not preinstalled yet. |
There was a problem hiding this comment.
I assume agents are generating this? I'd prefer to avoid random "fallbacks" that we don't actually use.
| fi | ||
| systemctl enable --now snap.amazon-ssm-agent.amazon-ssm-agent.service | ||
|
|
||
| install -d -m 0755 /var/lib/rustc-perf |
There was a problem hiding this comment.
What is this doing? I don't think we normally install perf into /var/lib...
At a high level, I think I would keep the user data minimal (empty?) unless our intent is for perf to provision new instances on demand. That might make sense, but then perf should probably be building an AMI with this pre-provisioned or the script and run-instances should live in rustc-perf (as it does in bors for r-l/r CI).
| encrypted = true | ||
| iops = 12000 | ||
| throughput = 500 | ||
| volume_size = 500 |
There was a problem hiding this comment.
12000 IOPS seems very much excessive to me. I would keep this at defaults personally unless we encounter issues. Most of the perf workload should be able to fit in memory.
500 gb is probably fine (I doubt we need that much though...).
| #!/usr/bin/env bash | ||
| set -euxo pipefail | ||
|
|
||
| export DEBIAN_FRONTEND=noninteractive |
There was a problem hiding this comment.
We'll want to mask the tmp.mount systemd unit and unmount /tmp most likely, we probably can't fit everything into memory unless we get an even larger instance allocated.
|
In the pr description I wrote explicitly that this was generated with AI and I didn't want anyone to review it because I didn't review it yet. Please don't review PRs from me that are in draft. I don't want to waste anyone time. Anyway I will have a look at your feedback, thank you. |
2 participants
Allocates an M9g Dedicated Host and places a no-ingress
m9g.12xlargeGraviton 5 collector on it, with SSM access, host recovery, termination protection, encrypted storage, and a boot-time hardware performance-counter smoke test.Stack: 3/4. Depends on the account bootstrap PR and an approved host quota increase.
AI disclosure
I used GPT5.6-Sol with the codex harness to generate this change. I reviewed its output and changed it where necessary.