Skip to content

Feat/self host public ip - #9824

Closed
vicky040 wants to merge 25 commits into
multica-ai:mainfrom
opex-labs-ai:feat/self-host-public-ip
Closed

vicky040 wants to merge 25 commits into
multica-ai:mainfrom
opex-labs-ai:feat/self-host-public-ip

Conversation

@vicky040

@vicky040 vicky040 commented Nov 2, 2026

Copy link
Copy Markdown

What does this PR do?

Related Issue

Closes #

Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality)
  • Refactor / code improvement (no behavior change)
  • Documentation update
  • Tests (adding or improving test coverage)
  • CI / infrastructure

Changes Made

How to Test

Checklist

  • I have included a thinking path that traces from project context to this change
  • I have run tests locally and they pass
  • I have added or updated tests where applicable
  • If this change affects the UI, I have included before/after screenshots
  • I have updated relevant documentation to reflect my changes
  • If I added a new runtime / coding tool / UI tab, I synced the change to landing copy (apps/web/features/landing/i18n/) and relevant docs (apps/docs/content/docs/)
  • If this PR touches Chinese product copy, I checked it against apps/docs/content/docs/developers/conventions.zh.mdx (terminology, mixed-rule for task / issue / skill)
  • I have considered and documented any risks above
  • I will address all reviewer comments before requesting merge

AI Disclosure

AI tool used:

Prompt / approach:

Screenshots (optional)

vicky040 and others added 24 commits October 4, 2026 19:17
This commit adds comprehensive AWS CDK infrastructure code for deploying
the Multica platform to AWS. The infrastructure is production-ready,
secure, and follows AWS best practices.

Infrastructure Components:
- Network Stack: VPC with public/private/isolated subnets, NAT gateway,
  security groups, and VPC Flow Logs
- Database Stack: RDS PostgreSQL 16 with Multi-AZ support, automated
  backups, encryption, and CloudWatch monitoring
- Backend Stack: EC2 Auto Scaling Group behind Application Load Balancer
  with health checks and auto-scaling policies
- Frontend Stack: S3 + CloudFront CDN for Next.js static hosting with
  custom cache policies and HTTPS enforcement

Key Features:
- Multi-environment support (dev, staging, prod) with environment-specific
  configurations
- Security: encryption at rest and in transit, AWS Secrets Manager for
  credentials, least-privilege IAM roles, isolated database subnets
- High availability: Multi-AZ database, Auto Scaling, load balancing
- Monitoring: CloudWatch Logs, metrics, and alarms for all resources
- Cost optimization: Environment-specific instance sizing, lifecycle policies

Deployment Tools:
- Bootstrap script for initial AWS CDK setup
- Deployment scripts with confirmation prompts
- Comprehensive documentation with architecture diagrams
- Rollback and disaster recovery procedures

Documentation:
- Complete README with prerequisites, deployment steps, and troubleshooting
- Architecture diagram in Mermaid format
- Cost estimation for dev and prod environments
- Security best practices and compliance features

The infrastructure code is ready for deployment but does NOT include
automatic deployment to avoid incurring AWS costs. Users should review
the configuration in lib/config.ts and deploy using the provided scripts
after adding their AWS credentials.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Updated the AWS CDK infrastructure to focus exclusively on the
development environment, simplifying configuration and reducing
complexity as requested.

Changes:
- Simplified config.ts to only support dev environment with cost-
  optimized settings (db.t4g.micro, t3.small, single NAT gateway)
- Updated package.json scripts to default to dev (deploy, synth, diff)
- Simplified deployment scripts to remove multi-environment logic
- Rewrote README.md to focus on development deployment with:
  - Quick start guide for dev environment
  - Simplified architecture diagram
  - Cost estimate: ~$95/month for development
  - Streamlined deployment instructions
  - Development-focused troubleshooting

The infrastructure maintains all security best practices and AWS
recommendations while being optimized for development use and cost
efficiency.

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Removed NAT gateway from the infrastructure to achieve significant
cost savings in the development environment, reducing monthly costs
from ~$95 to ~$60.

Changes:
- network-stack.ts: Set natGateways to 0, removed PRIVATE_WITH_EGRESS
  subnet configuration, now only public and isolated (database) subnets
- backend-stack.ts: Moved Auto Scaling Group to public subnets instead
  of private subnets
- config.ts: Added comment about NAT gateway removal and cost savings
- README.md: Updated architecture diagram to remove NAT gateway, moved
  backend to public subnets, updated cost estimate from $95 to $60/month

Architecture change:
- Backend EC2 instances now run in public subnets with direct internet
  access instead of using NAT gateway for outbound connections
- Database remains in isolated subnets with no internet access
- Security groups still enforce least-privilege access patterns

Cost savings: ~$35/month (NAT gateway + data processing charges)
New monthly estimate: ~$60/month

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Multica runs as a hosted service, so the four stacks this branch started with
(network, database, backend, frontend) deployed infrastructure the team does
not need and could not run: apps/web is a standalone Next.js server rather than
a static site, and the backend's systemd unit was a placeholder that never
started anything.

What the team needs from AWS is somewhere for agents to execute. This is one
stack: a one-AZ VPC, one egress-only security group with no ingress rules, one
Graviton instance running the Multica daemon under systemd as a non-root user,
a Bedrock-and-own-parameters IAM role, and a log group. No key pair and no SSH;
access is Session Manager. Roughly $30/month.

Claude Code is installed once and exposed through two wrappers on PATH,
claude-max (subscription OAuth token) and claude-bedrock (instance role), each
unsetting the other's credentials so the credential a run used is unambiguous.
Both are registered as custom runtime profiles in the workspace.

Credentials are SecureString parameters created outside CloudFormation, so no
secret value enters a template or a change set. On the machine they are pulled
into tmpfs at 0400 on every service start, the Multica PAT reaches
`multica login` on stdin rather than argv, and git reads its token through a
credential helper instead of ~/.git-credentials.

Also addressed from review: one environment instead of three, a single security
group instead of four (three of them unused), Graviton claims backed by an
arm64 AMI and a synth-time rejection of x86 instance types, a CloudWatch agent
that is actually configured, outputs without clashing export names, and
deploy.sh no longer prompting for confirmation and then passing
--require-approval never.

Adds 20 assertions over the synthesized template covering the claims that are
easy to break by hand, a workflow that bash -n's the bootstrap script before it
can ship as user data, and brings the package into the pnpm workspace as
@multica/infrastructure.

No AWS deployment has been performed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three pieces of the infrastructure workflow cost time without producing a
signal anyone reads:

- the lockfile and workspace manifest in the pull_request path filter ran the
  whole job on every dependency PR, none of which change the stack
- the synthesized template was uploaded as an artifact that nothing downloads;
  synth already fails the job when it cannot produce one
- the deploy job diffed and then deployed, and `cdk deploy` prints the same
  change set itself, so the separate diff step now runs only for a diff run

bash -n, typecheck, test and synth are unchanged.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
frontend-test failed on this branch with "Test timed out in 5000ms" on
whichever assertion synthesized the stack first. Two causes, both fixed here.

The test file synthesized the stack once per case, twenty times. The first one
costs several seconds on a cold runner because it stages the asset behind the
default-security-group custom resource, which is over vitest's 5s default and
has nothing to do with what that test asserts. Each of the four configurations
is now built once and shared, and vitest.config.mts raises the timeout so a
slow runner cannot decide the result. Test time drops from 11.2s to 2.5s.

The suite should not have been in that job at all. infrastructure/ has its own
path-filtered workflow, imports nothing from the other packages, and nothing
imports it, so no frontend change can alter its result — it ran there only
because joining the pnpm workspace put it in turbo's default scope. It is now
excluded from the root turbo scripts and from ci.yml's filters, the same way
apps/mobile is. knip still covers it, which is where the workspace-membership
rules are enforced.

AGENTS.md records the new boundary and the commands that verify it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Add AWS CDK infrastructure for Multica deployment
systemd opens the StandardOutput=append: file before running the root
("+") ExecStartPre, and LogsDirectory= is not set up for that step, so
/var/log/multica did not exist and multica-daemon failed with
209/STDOUT on every start. Create the directory in the bootstrap script.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Without a terminal the CLI routes its logger to ~/.multica/daemon.log,
so /var/log/multica/daemon.log only ever held the credential fetch and
login output. Ship the daemon's log as the <id>/daemon stream and keep
the unit output as <id>/service.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
fix(infra): start the agent daemon and ship its log
Multica keys runtimes by daemon id, and the daemon mints a random one on
first boot. Every instance replacement (any bootstrap.sh change) would
register new, private runtimes and leave the public ones agents are bound
to offline. Pin MULTICA_DAEMON_ID through a daemonId context key, set in
cdk.json to the id the current instance registered with.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
fix(infra): pin the daemon id so a replacement keeps its runtimes
Records the deployed state (runtimes, ownership, pinned daemon id), how to
reach and operate the machine, and the remaining setup steps. connect.sh
now points at the daemon's own log, which #2 moved.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
docs(infra): add a runbook for the deployed agent runtime
Updates the runbook after credential rotation and operator access, and
replaces the short next-steps list with the workspace setup a new admin
follows: repositories and projects, planning with a Sprint property,
agents, skills, squads and a sprint-planning autopilot.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
docs(infra): record the rotated state and the workspace setup plan
The agent runtime stack assumes Multica is used as a hosted service. This adds
the other option: Multica running in our own account, with every issue,
comment, agent and run in a PostgreSQL volume we control and nothing sent to
multica.ai. The backend's first-party deployment telemetry is off unless
-c telemetry=true turns it back on.

One instance runs the repository's own docker-compose.selfhost.yml — Postgres,
the Go backend and the Next.js web app — behind Caddy, configured from the
single-domain example in SELF_HOSTING_ADVANCED.md. The compose file ships as a
CDK asset rather than being fetched from a branch at boot, so the deployment
runs the reviewed file and cdk diff shows when it changes.

A bare IP cannot hold a trusted certificate, so the Elastic IP is reached
through sslip.io, which resolves <ip>.sslip.io to <ip> and gives Caddy a real
hostname for Let's Encrypt. -c domainName= switches to a name we own.

Postgres lives on a separate encrypted EBS volume mounted at /var/lib/multica,
retained on stack delete and snapshotted daily. That volume is also why this
stack, unlike the agent runtime, does not replace its instance on a user data
change: replacing an instance out from under an attached volume is how a
database gets lost. cdk deploy updates the launch template and
scripts/selfhost-apply.sh runs the new script on the live machine.

443 is open to the internet, so synth refuses to build the stack without an
ALLOWED_EMAILS signup allowlist, and the guide's fixed
MULTICA_DEV_VERIFICATION_CODE shortcut is never set.

19 template assertions cover the parts that are easy to get wrong by hand:
only 80 and 443 reachable and never 5432, 3000, 8080 or 22; the data volume
encrypted, separate and retained; snapshots scheduled; the WebSocket route
ahead of the catch-all with flush_interval -1; and the allowlist reaching the
backend. The retention helper and the script renderer are now shared between
the two stacks.

Rendering a script is now renderScript(fileName, substitutions), so the two
bootstraps share one renderer. That moves the substitution map out to each
stack, and the agent runtime's map has to keep carrying DAEMON_ID from
853cff9 — without it the deployed runtime machine would mint a new daemon id
on replacement and lose its runtime rows. The renderer's unknown-placeholder
guard catches that, and a test covers it.

The self-host stack has not been deployed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Resolves the two conflicts the PR reported against upstream main.

AGENTS.md: upstream dropped the "Root frontend commands and `make check` do
not verify mobile" sentence, and this branch had rewritten the same line to
name infrastructure as well. Took upstream's shorter wording — the exclusion
is already stated by the two table rows this branch adds, and AGENTS.md says
not to duplicate rules.

pnpm-lock.yaml: a generated file, so it was reset to upstream's and
regenerated with `pnpm install --lockfile-only` rather than merged by hand.
The result differs from upstream only by the @multica/infrastructure importer.

Verified after the merge: the ci.yml turbo filters, the root package.json
scripts, the knip workspace entry and .github/workflows/infrastructure.yml all
survived; bash -n on all eight scripts; typecheck clean; 43/43 tests; both
stacks synthesize.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…story

Both describe one particular deployment rather than the stack, so neither
belongs in a repository other people read.

cdk.json pinned a real daemon id. It names that installation's runtime rows in
Multica, so committing it is both meaningless to anyone else and a fact about
our workspace. The knob stays: pass -c daemonId=<uuid> at deploy. Left empty,
the daemon mints its own on first boot, which is the right default for a fresh
installation.

RUNBOOK.md recorded the deployed state, who has access and how credentials
were rotated. That is operational record for one team, not documentation of the
stack.

The regression test from 853cff9 stays, with its own fixture id instead of
reading the committed one, plus a second case covering the empty default.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A first deploy required two SecureString parameters created by hand before the
instance booted, or the bootstrap exited on its first read. That is the step
most likely to be skipped, and it puts both values through a shell history on
the way in.

Neither is ever read or typed by a person, so CloudFormation generates them
instead. Two stores, one rule: what the stack generates lives in Secrets
Manager, what a human supplies lives in Parameter Store. The optional Resend
API key is the only thing left in the second category.

Punctuation is excluded from the generated password because it is interpolated
into a postgres:// URL, where an unescaped @ or / would point the backend at
the wrong host rather than fail. The password is retained when the stack is
deleted, alongside the volume it opens.

Costs $0.80/month for the two secrets.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…value

The job ran a bare `cdk deploy`, which fails now that the app holds two
stacks, and it had no way to pass the self-host context, so Multica-SelfHost
could not be deployed through it at all.

It now takes the stack as a dispatch input and resolves the one value that
stack needs but the repository deliberately does not hold:

- Multica-SelfHost: the signup allowlist, as a dispatch input. Missing, the
  job fails rather than deploying an internet-facing server anyone can sign
  up to.
- Multica-AgentRuntime: the daemon id, from the MULTICA_DAEMON_ID repository
  variable. It names this installation's runtime rows, so it was removed from
  cdk.json; missing, the job fails rather than silently minting a new id and
  orphaning the runtimes it already has.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@vercel

vercel Bot commented Nov 2, 2026

Copy link
Copy Markdown

@vicky040 is attempting to deploy a commit to the IndexLabs Team on Vercel.

A member of the Team first needs to authorize it.

@vicky040 vicky040 closed this Nov 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants