diff --git a/plans/sandboxed-e2e-test-runtime.md b/plans/sandboxed-e2e-test-runtime.md new file mode 100644 index 0000000000..e752da5bd1 --- /dev/null +++ b/plans/sandboxed-e2e-test-runtime.md @@ -0,0 +1,489 @@ +# Sandboxed app runtime for user-triggered E2E tests + +## Status + +Implemented for host-runtime test execution. The Tests panel and agent test tool +now use a disposable workspace, run-scoped server, sandbox-only Neon env, and +retained artifact directory. Interactive test recording remains on its existing +preview-oriented lifecycle and was deliberately not migrated. + +## Problem + +Neon test execution currently isolates the data but not the app runtime. Dyad +creates a temporary Neon branch, rewrites the real app's `.env.local`, and +restarts the user's existing preview against that branch. Playwright then uses +the existing preview proxy. While the run is active, any user interaction with +the preview therefore reaches the throwaway branch; setup and teardown also +interrupt the user's dev server. + +The test runner also uses the real app directory as its working directory, so +reports and any test-time filesystem writes land in the user's project. + +## Goal + +Run user-triggered E2E tests against a disposable copy of the app with its own +environment, process, and port, while the normal preview continues running from +the real app directory and against the user's normal database configuration. + +The sandbox must represent the app as it exists when Run is pressed, including +tracked modifications and relevant untracked/ignored runtime files. It must not +be limited to the last Git commit. + +## Non-goals + +- Do not change the recorder. Recording continues to drive the normal preview + and use the existing database-isolation lifecycle. +- Do not replace the app preview iframe or introduce an Electron session + partition. +- Do not copy test-generated source changes back into the real app. +- Do not use a Git worktree as the required implementation. Git may be absent, + unhealthy, or behind the app's current on-disk state. +- Do not support concurrent test runs for the same app in the first version. +- Do not install dependencies independently inside every sandbox. +- Do not silently fall back to running a Neon test against the user's real + database or normal preview when sandbox setup fails. + +## Product behavior + +1. The user presses Run while their normal preview is running. +2. Setup progress reports distinct steps: preparing Playwright if necessary, + copying the current app, creating isolated data, and starting the test + server. +3. The normal preview remains available and is neither stopped nor restarted. +4. Playwright targets the sandbox server URL only. +5. Stop terminates Playwright, then the sandbox server, then remote test-data + resources, and finally deletes the sandbox directory. +6. Test results remain visible through the existing result model. Artifacts + needed by the result UI are promoted to a Dyad-owned artifact directory + before the sandbox is deleted. +7. A setup failure says which stage failed and confirms that the normal preview + was not changed. + +The first-ever Playwright bootstrap remains a transparent, one-time project +setup operation because it intentionally adds `@playwright/test`, Dyad's config, +and ignore entries to the user's app. It runs before the snapshot. Ordinary +runs after bootstrap do not mutate the real app. + +## Proposed architecture + +### 1. `E2eTestWorkspace` + +Add an E2E-only workspace service, separate from database isolation: + +```ts +interface E2eTestWorkspace { + realAppPath: string; + workspacePath: string; + artifactPath: string; + dispose(): Promise; +} + +createE2eTestWorkspace({ + appId, + appPath, + signal, + onProgress, +}): Promise +``` + +Place workspaces beneath a Dyad-owned user-data directory such as +`/test-sandboxes//`, rather than beside the user's repo. +This gives startup reconciliation a bounded, recognizable cleanup root and +prevents temp-directory policy differences from leaking into the feature. + +#### Copy strategy + +Use the existing repository pattern of recursive `fs.cp` with +`COPYFILE_FICLONE` and a filter on macOS/Linux. A reflink-capable filesystem +makes the initial clone copy-on-write. Fall back to an ordinary filtered copy +when cloning is unsupported. Use a filtered ordinary copy on Windows initially +and measure it before adding platform-native block cloning. + +Never hard-link application files: a write through a hard link would mutate the +real file. + +Exclude heavyweight or disposable roots, initially: + +- `.git` +- `node_modules` +- framework output/caches: `dist`, `build`, `out`, `.vite`, `.next`, `.nuxt`, + `.svelte-kit`, `.turbo`, `.cache` +- test output: `test-results`, `playwright-report`, `coverage` +- OS metadata such as `.DS_Store` + +Do not use `.gitignore` wholesale. Ignored files can be runtime inputs, including +environment files and generated assets. Preserve all other files as they exist +on disk, including uncommitted and untracked files. + +After copying the source tree, clone `node_modules` into the workspace with the +same copy-on-write strategy. Do not link the dependency root back to the real +app: Vite resolves that link to the real directory, after which Nitro runtime +entries fall outside the sandbox filesystem boundary and fail with +`ERR_LOAD_URL`. Preserve pnpm's relative links within the cloned dependency tree. +If dependencies are absent after the pre-snapshot bootstrap, fail setup with an +actionable error. + +Treat writable dependency caches as a measured risk. Verify Vite, Next, and the +supported package managers in tests; add sandbox-local cache environment +overrides or exclusions for any tool proven to write beneath `node_modules`. + +#### Snapshot consistency + +Take the copy while holding the app path and repository-worktree coordination +claims so Dyad-driven edits, restores, imports, and path changes cannot +interleave. Release those claims once the copy is complete so the user can keep +editing while Playwright runs against the captured state. External IDE writes +cannot be made globally atomic; document the snapshot boundary as the moment +Run is pressed and make copy failures fail closed. + +### 2. `E2eTestDataIsolation` + +Do not make `prepareIsolatedTestDatabase` serve two incompatible lifecycles. +Keep it unchanged for recording, including its real-preview restart and env +restoration behavior. Extract shared provider primitives where useful, then add +an E2E-only data-isolation service whose contract targets the workspace: + +```ts +interface PreparedE2eTestData { + isolation: TestIsolation; + testCredentials?: Record; + applyToWorkspace(workspacePath: string): Promise; + dispose(): Promise; +} +``` + +#### Neon + +1. Create and durably track the temporary branch using the existing Neon branch + primitives. +2. Read the already-copied workspace environment. +3. Call the existing framework-aware Neon env updater with `workspacePath`, + never `realAppPath`. +4. Provision the optional Better Auth test account as today. +5. On cleanup, delete/mark the branch using the existing durable ordering. + +There is no real `.env.local` restoration step because it was never changed. +Accordingly, remove E2E-only `envRestored` failure handling after migration; +retain it in the recorder path. Continue tracking the branch ID durably so a +crash can clean up a leaked remote branch even though the user's env is safe. + +#### Supabase + +Continue creating an RLS-scoped test user and cleaning up its data/user. The +sandbox runtime still provides filesystem/process isolation even though the +database endpoint does not change. + +#### No database + +Still use the workspace and test runtime. Return `mode: "none"` only for data +isolation; it must not mean runtime isolation was skipped. + +#### Runtime modes + +Implement the first version for host runtime. For Docker/cloud runtime, retain +the current behavior only where it is already safe and make the lack of runtime +sandboxing explicit in the UI. For Neon specifically, never degrade to the +normal preview/real database on sandbox failure. A later phase may provide +Docker-specific volumes or cloud sandbox clones behind the same contract. + +### 3. `E2eTestRuntime` + +Add a short-lived runtime owned by the test run: + +```ts +interface E2eTestRuntime { + baseUrl: string; + processId?: number; + stop(): Promise; +} + +startE2eTestRuntime({ + app, + workspacePath, + signal, + onOutput, +}): Promise +``` + +Extract and reuse app-runtime command detection, managed Node/pnpm selection, +environment construction, and readiness parsing where possible. Do not call the +normal `executeApp` path unchanged: it is keyed by `appId`, publishes into +`runningApps`, drives the app-run actor, owns the normal deterministic ports, +and could replace or stop the user's preview. + +The test runtime must instead: + +- have a run-scoped identity rather than using `appId` as a singleton key; +- run with `cwd: workspacePath`; +- receive a unique app port; +- remain outside `runningApps` and normal preview/app-run state; +- stream setup output into the existing test output channel with a clear + `[test server]` prefix; +- resolve only after an HTTP readiness probe succeeds; +- stop only the exact child/process tree it started; +- never call a cleanup helper that kills an unknown process merely because it + owns the desired port. + +Prefer Playwright targeting the test dev server directly. It does not need the +normal preview proxy because there is no iframe, stable browser origin, or +preview navigation UI to preserve. When the disposable Neon branch has Auth +enabled, register the test server's exact run-scoped origin as a trusted domain +after its random port is known and before Playwright starts. Registration must +preserve the HTTP scheme rather than applying the HTTPS normalization used for +deployment domains, and must target only the disposable branch; teardown +deletes it with that branch. If authorization fails, stop before Playwright +rather than producing misleading per-test `INVALID_ORIGIN` failures. Do not +disable Better Auth's origin checks, use a loopback wildcard, or reuse the +normal app proxy. + +Allocate an OS-available port and pass it through the same framework-specific +start-command machinery used by the normal runtime. Because selecting a free +port has a bind race, detect address-in-use startup failure and retry with a new +port a bounded number of times. Do not reserve a port by killing its listener. + +### 4. Make the test core accept explicit execution inputs + +Change `runAppTestsCore` so it no longer looks up the real app path and normal +preview URL internally: + +```ts +runAppTestsCore({ + appId, + appPath: workspace.workspacePath, + baseUrl: testRuntime.baseUrl, + artifactPath: workspace.artifactPath, + // existing selector, output, timeout, and credential options +}); +``` + +Keep path/URL resolution in the orchestration layer. Run `npx playwright` with +the sandbox as `cwd`, write the JSON report inside it, and parse paths relative +to the sandbox. Before disposal, copy retained traces/screenshots/report data to +the run's artifact directory and rewrite result attachment paths if the UI +stores them. + +Bootstrap Playwright against the real app before workspace creation. This +preserves the current user-visible project setup semantics and ensures the +generated Dyad config and dependency are present in the snapshot. Split +`ensurePlaywrightBootstrap` from the pure "verify ready" step so code inside the +sandbox cannot accidentally invoke an install through the shared +`node_modules` clone. + +### 5. Lifecycle orchestration and coordination + +Refactor the E2E handler into explicit stages with one cleanup stack: + +```text +register run/cancellation owner + -> bootstrap Playwright in real app (when required) + -> snapshot workspace + -> prepare isolated test data in workspace + -> start test runtime + -> run Playwright against explicit baseUrl + -> retain artifacts +finally + -> stop Playwright/process tree + -> stop test runtime + -> dispose isolated data + -> delete workspace + -> publish terminal result +``` + +Register every acquired cleanup immediately after acquisition. Run cleanup in +reverse order and continue after individual cleanup failures, reporting all +failures without hiding the original run error. Cancellation must be checked +between every awaited setup stage. Teardown is not abortable once a remote +branch/user exists. + +Do not hold the current broad coordinator claim for the entire Playwright run. +Use staged claims: + +1. **Bootstrap/snapshot:** read `app-path` and `repository-ref`; claim + `repository-worktree` and `test-files` while bootstrap files or the snapshot + may change/read them. +2. **Provider lifecycle:** hold `provider` from temporary branch/user creation + through deletion, preventing unlink/delete races. +3. **Sandbox runtime:** use a new run-scoped owner/registry rather than claiming + the normal app `runtime` or `runtime-config`, because those claims would + unnecessarily block the preview the feature is meant to preserve. + +Audit app deletion, app relocation, provider unlink, Stop, replacement Run, and +application shutdown. These must cancel and await a run where required; they +must not delete provider state or the sandbox root while cleanup is active. +Preserve the existing same-app single-run controller ordering. Runs for +different apps may proceed concurrently once port and resource ownership are +run-scoped. + +### 6. Crash recovery + +On Dyad startup: + +- remove abandoned directories only beneath the recognized + `test-sandboxes` root; +- never follow directory links during recursive cleanup; +- rely on the existing durable Neon branch marker/reconciliation to remove + leaked remote branches; +- ensure no real-env restoration gate is applied to E2E-only leaked branches, + because the real env was never modified; +- keep recorder recovery behavior unchanged. + +The durable branch state may need to distinguish `recorder-env-swapped` from +`e2e-cleanup-only`; add that distinction before sharing startup reconciliation. +Do not infer it solely from whether a sandbox directory still exists. + +## Implementation phases + +### Phase 1: Workspace primitive and benchmarks + +- Add the filtered reflink/copy workspace service and safe disposal. +- Add copy-on-write dependency-tree cloning with pnpm-link preservation. +- Add timing/size telemetry without recording absolute user paths. +- Benchmark representative Vite and Next apps on macOS, Windows, and Linux. +- Set a soft progress threshold (show ongoing file count/bytes after 500 ms), + not a hard correctness timeout. + +Exit criterion: a workspace contains current tracked/untracked source and env +files, excludes known heavy outputs, cannot mutate real source, and starts with +resolvable dependencies. + +### Phase 2: Unregistered host test runtime + +- Extract reusable command/env/readiness pieces from `app_runtime_service`. +- Implement the run-scoped process registry, port retry, output, readiness, and + process-tree teardown. +- Prove that starting/stopping it does not change `runningApps`, the app-run + actor, normal proxy URL, or normal preview process. + +Exit criterion: normal and test servers run simultaneously from different +directories and ports, and stopping either leaves the other alive. + +### Phase 3: E2E-only data isolation and runner wiring + +- Add workspace-targeted Neon env rewriting and shared provider primitives. +- Keep the recorder on `prepareIsolatedTestDatabase` unchanged. +- Pass explicit `appPath`/`baseUrl` into `runAppTestsCore`. +- Promote artifacts, then dispose the sandbox. +- Replace E2E env-restoration messaging with sandbox-cleanup messaging. + +Exit criterion: during a Neon test, the real `.env.local` is byte-identical, +the normal preview keeps its PID/URL and real branch, and Playwright reaches the +throwaway branch through the test server. + +### Phase 4: Coordination, recovery, and rollout + +- Narrow broad whole-run coordinator claims into staged ownership. +- Complete delete/relocate/unlink/shutdown/rapid-rerun audits. +- Add abandoned-workspace startup cleanup and distinguish durable Neon cleanup + states. +- Gate the new path behind a temporary feature flag for soak testing, but fail + closed rather than silently use the legacy runtime when the flag is enabled. +- Remove the legacy E2E env-swap path after cross-platform validation. Do not + remove or redirect the recorder path. + +## Test plan + +### Unit tests + +- Workspace filter includes modified/untracked/ignored runtime inputs and + excludes every declared heavyweight root. +- LF/CRLF environment files remain valid and only sandbox Neon values change. +- Reflink failure falls back to copy; cancellation removes a partial workspace. +- Pnpm dependency realpaths remain inside the sandbox and missing-dependency + errors are platform-correct. +- Sandbox disposal rejects paths outside its owned root and does not follow + malicious links. +- Port collision retries without killing the existing listener. +- Cleanup stack runs in reverse order, attempts every cleanup, and preserves the + primary error plus cleanup diagnostics. +- Explicit runner paths parse reports and attachments relative to the sandbox. + +### Vitest integration tests + +- Start a real fixture dev server normally, create a sandbox server, and assert + distinct CWDs, PIDs, ports, and environment values. +- Modify and add fixture files before Run; assert the sandbox sees them while a + post-snapshot edit affects only the normal workspace. +- Assert Playwright bootstrap happens before snapshot and no install command is + invoked from the sandbox. +- Cancel during copy, branch creation, server readiness, Playwright, and cleanup; + assert no live child/workspace remains and remote cleanup is attempted. +- Start Run twice rapidly; assert the second waits for the first cleanup and + cannot delete the successor's workspace or branch. +- Delete/unlink/relocate/shutdown coordination tests prove there is no orphaned + process or provider race. +- Recorder regression tests assert it still uses the existing preview-oriented + isolation service and restart behavior. + +### Playwright E2E coverage + +Add one broad test using a fixture app whose page displays a server-read env +marker: + +1. Start the normal preview with marker `real` and capture its PID/URL. +2. Run a test with sandbox marker `isolated`. +3. While it is running, verify the normal preview still displays `real` and is + interactive. +4. Verify the generated test observes `isolated`. +5. Stop or finish the run and assert the original PID/URL and `.env.local` + content never changed. +6. Assert the sandbox process/directory is gone and retained results remain + readable. + +Add a second targeted cancellation case only if the integration harness cannot +exercise real process-tree teardown reliably. + +## Observability and acceptance criteria + +Log structured timings for bootstrap, copy, data setup, server readiness, test +execution, artifact promotion, and cleanup. Include app ID/run ID but never env +contents, credentials, or absolute project paths. + +The feature is complete when: + +- the real app environment is byte-identical before, during, and after E2E; +- the normal preview process and URL do not change during E2E; +- Playwright can only reach the sandbox server URL supplied to its config; +- current uncommitted and relevant untracked files are tested; +- test writes and reports do not pollute the real app; +- Stop and app shutdown leave no child process, sandbox, test user, or Neon + branch after cleanup/reconciliation; +- filtered workspace creation is normally faster than test-server startup; +- Windows fallback performance is measured and acceptable; +- recorder behavior and its tests are unchanged. + +## Key risks and mitigations + +- **Shared `node_modules` is mutated:** prohibit sandbox installs, split + bootstrap verification from installation, and isolate proven writable caches. +- **Framework command logic diverges:** extract command construction/readiness + from the normal runtime rather than duplicating it. +- **Artifacts disappear with the workspace:** promote them before disposal and + test every attachment path consumed by the UI. +- **External editor writes during copy:** coordinate all Dyad writers and define + a best-effort on-disk snapshot boundary; fail on inconsistent filesystem + errors rather than silently mixing in the real directory later. +- **Port race:** bounded retry on bind failure; never kill an unknown listener. +- **Crash leaks remote data:** preserve durable branch markers and distinguish + E2E cleanup-only state from recorder env-restoration state. +- **Sandbox cleanup escapes its root:** validate canonical containment and do + not traverse links. +- **Normal preview is accidentally registered/restarted:** keep the test runtime + out of `runningApps` and add assertions around the app-run actor and proxy. + +## Likely code areas + +- `src/ipc/handlers/tests_handlers.ts`: orchestration, explicit runner inputs, + cancellation, artifacts, and staged coordination. +- `src/ipc/services/isolated_test_db.ts`: recorder-compatible service remains; + extract only provider primitives shared with the new E2E service. +- `src/ipc/services/app_runtime_service.ts`: extract reusable command/env and + readiness helpers without reusing singleton runtime registration. +- New `src/ipc/services/e2e_test_workspace.ts`. +- New `src/ipc/services/e2e_test_runtime.ts`. +- New `src/ipc/services/e2e_test_data_isolation.ts`. +- `src/ipc/utils/playwright_bootstrap.ts`: split mutating bootstrap from sandbox + readiness verification. +- `src/ipc/utils/neon_test_branch.ts`: durable cleanup-state distinction. +- Startup reconciliation and app deletion/shutdown paths that currently recover + or stop tests. +- Unit/integration tests beside each service plus one broad packaged E2E spec. diff --git a/shared/ports.ts b/shared/ports.ts index 7123fa6937..123ed5ccde 100644 --- a/shared/ports.ts +++ b/shared/ports.ts @@ -53,6 +53,41 @@ export const PROXY_FALLBACK_PORT_START = PROXY_PORT_BASE + PROXY_PORT_RANGE; /** How many consecutive fallback ports to scan before giving up. */ export const PROXY_FALLBACK_MAX_ATTEMPTS = 50; +/** + * Start of the band reserved for run-scoped E2E test servers. It sits directly + * above the proxy fallback band so a sandbox server never squats on the + * deterministic app or proxy port of some *other* app that merely happens to be + * stopped right now — which would make that app fail to start later, with + * nothing to point at as the cause. An OS-assigned ephemeral port cannot give + * that guarantee: on Linux the ephemeral range (32768–60999) covers almost all + * of the reserved bands below. + */ +export const E2E_TEST_SERVER_PORT_START = + PROXY_FALLBACK_PORT_START + PROXY_FALLBACK_MAX_ATTEMPTS; +/** Width of the E2E test-server band, so it spans 52150..52349. */ +export const E2E_TEST_SERVER_PORT_RANGE = 200; + +/** + * Whether a port falls in a band Dyad hands out deterministically, and so must + * not be taken by anything that only needs *some* free port. + */ +export function isReservedDyadPort(port: number): boolean { + const e2ePortBlockBase = getE2ePortBlockBase(); + if ( + e2ePortBlockBase != null && + port >= e2ePortBlockBase && + port < e2ePortBlockBase + E2E_PORT_BLOCK_SIZE + ) { + return true; + } + return ( + (port >= APP_PORT_BASE && port < APP_PORT_BASE + APP_PORT_RANGE) || + (port >= PROXY_PORT_BASE && port < PROXY_PORT_BASE + PROXY_PORT_RANGE) || + (port >= PROXY_FALLBACK_PORT_START && + port < PROXY_FALLBACK_PORT_START + PROXY_FALLBACK_MAX_ATTEMPTS) + ); +} + export function getProxyFallbackPortStart(): number { const e2ePortBlockBase = getE2ePortBlockBase(); if (e2ePortBlockBase != null) { diff --git a/src/atoms/testRuntimeAtoms.ts b/src/atoms/testRuntimeAtoms.ts index 176e089ab2..3500a39182 100644 --- a/src/atoms/testRuntimeAtoms.ts +++ b/src/atoms/testRuntimeAtoms.ts @@ -44,7 +44,7 @@ export type TestRunPhase = | "setup" // first-run Playwright bootstrap streaming | "running" // playwright test executing | "stopping" // Stop pressed; killing the Playwright process tree - | "cleaning-up"; // tests gone; isolation teardown still restoring the app + | "cleaning-up"; // tests gone; isolated provider data is still being removed export interface TestRunState { phase: TestRunPhase; @@ -72,6 +72,12 @@ export interface TestRunState { * a run completes. */ isolation?: TestIsolation; + /** + * Whether the run executed in an isolated sandbox. Drives the cleanup copy: + * the fallback path (Docker/cloud runtime, or the user's opt-out) creates no + * workspace, so there is no sandbox to claim Dyad is deleting. + */ + sandboxed?: boolean; startedAt?: number; } @@ -275,6 +281,8 @@ export const applyTestRunStartedAtom = atom( ), runError: undefined, isolation: undefined, + // Reported by the main process once the run has picked its path. + sandboxed: undefined, startedAt: startedAt ?? Date.now(), }), }); diff --git a/src/components/SandboxedE2eTestsSwitch.tsx b/src/components/SandboxedE2eTestsSwitch.tsx new file mode 100644 index 0000000000..b8c2097bd6 --- /dev/null +++ b/src/components/SandboxedE2eTestsSwitch.tsx @@ -0,0 +1,29 @@ +import { useSettings } from "@/hooks/useSettings"; +import { Label } from "@/components/ui/label"; +import { Switch } from "@/components/ui/switch"; + +/** + * Escape hatch for the sandboxed E2E runtime. Stored inverted + * (`disableSandboxedE2eTests`) so the sandbox stays the default for everyone + * who never opens this, and only an explicit opt-out falls back to running + * against the normal preview. + */ +export function SandboxedE2eTestsSwitch() { + const { settings, updateSettings } = useSettings(); + const enabled = !settings?.disableSandboxedE2eTests; + return ( +
+ { + updateSettings({ disableSandboxedE2eTests: !checked }); + }} + /> + +
+ ); +} diff --git a/src/components/chat/CancellationBanner.test.tsx b/src/components/chat/CancellationBanner.test.tsx index 0ec8d60def..364fdac1c6 100644 --- a/src/components/chat/CancellationBanner.test.tsx +++ b/src/components/chat/CancellationBanner.test.tsx @@ -18,10 +18,12 @@ vi.mock("react-i18next", () => ({ ({ stoppingGeneration: "Stopping…", cancellationEndingTestRun: "Ending the test run.", - cancellationRestoringTestApp: - "Restoring your app's database and preview. This can take a while.", + cancellationRemovingTestDatabase: + "Removing the temporary test database. This can take a while.", cancellationCleaningTestData: "Cleaning up the test data from this run.", + cancellationCleaningTestSandbox: + "Cleaning up this run's test sandbox and data.", })[key] ?? key, }), })); @@ -30,6 +32,7 @@ function renderBanner(runState?: { phase: TestRunPhase; isolationMode?: TestIsolation["mode"]; source?: "panel" | "agent"; + sandboxed?: boolean; }) { const store = createStore(); if (runState) { @@ -45,6 +48,7 @@ function renderBanner(runState?: { isolation: runState.isolationMode ? { mode: runState.isolationMode } : undefined, + sandboxed: runState.sandboxed, }, ], ]), @@ -87,24 +91,46 @@ describe("CancellationBanner", () => { }); expect( - screen.getByText(/Restoring your app's database and preview/), + screen.getByText(/Removing the temporary test database/), ).toBeTruthy(); expect(screen.getByText(/can take a while/)).toBeTruthy(); + // The run had its own sandbox and its own server; the user's preview and + // `.env.local` were never touched, so nothing is being restored. + expect(screen.queryByText(/Restoring/i)).toBeNull(); }); - it("does not claim a restore on the Supabase path", () => { - // That teardown only deletes the temporary test user — no env swap, no - // dev-server restart, nothing the user sees. + it("names the sandbox on the Supabase path when the run took one", () => { + // That teardown only deletes the temporary test user and the run's sandbox + // copy — no env swap, no dev-server restart, nothing the user sees. renderBanner({ phase: "cleaning-up", isolationMode: "supabase-test-user", source: "agent", + sandboxed: true, }); - expect(screen.getByText(/Cleaning up the test data/)).toBeTruthy(); + expect( + screen.getByText(/Cleaning up this run's test sandbox/), + ).toBeTruthy(); expect(screen.queryByText(/Restoring/)).toBeNull(); }); + it("claims no sandbox for a run that never took one", () => { + // The fallback path (Docker/cloud runtime, or the opt-out) creates no + // workspace, and neither does a run whose setup failed before the copy. + renderBanner({ + phase: "cleaning-up", + isolationMode: "supabase-test-user", + source: "agent", + sandboxed: false, + }); + + expect( + screen.getByText("Cleaning up the test data from this run."), + ).toBeTruthy(); + expect(screen.queryByText(/sandbox/i)).toBeNull(); + }); + it("reports the kill before the teardown starts", () => { renderBanner({ phase: "stopping", source: "agent" }); diff --git a/src/components/chat/CancellationBanner.tsx b/src/components/chat/CancellationBanner.tsx index ac699c92db..ff20323dd5 100644 --- a/src/components/chat/CancellationBanner.tsx +++ b/src/components/chat/CancellationBanner.tsx @@ -11,11 +11,11 @@ import { * Pinned above the composer while a stopped turn settles. * * Stopping is not instant. The agent awaits its in-flight tool, and a - * `run_tests` call first kills the Playwright process tree and then runs an - * isolation teardown that accepts no AbortSignal — restoring `.env.local`, - * restarting the dev server and deleting the temporary Neon branch, whose - * delete retries with backoff. That wait can pass a minute, and the composer - * stays locked for all of it. + * `run_tests` call first kills the Playwright process tree and then runs a + * teardown that accepts no AbortSignal — deleting the temporary Neon branch, + * whose delete retries with backoff, and removing the run's sandbox copy of the + * app. That wait can pass a minute, and the composer stays locked for all of + * it. The user's own `.env.local` and preview are never touched. * * The transcript's inline status card scrolls out of view; this stays fused to * the composer the user just clicked Stop in, so the wait is never unexplained. @@ -38,8 +38,10 @@ export function CancellationBanner({ appId }: { appId?: number | null }) { ? null : runState.phase === "cleaning-up" ? runState.isolation?.mode === "neon-branch" - ? t("cancellationRestoringTestApp") - : t("cancellationCleaningTestData") + ? t("cancellationRemovingTestDatabase") + : runState.sandboxed + ? t("cancellationCleaningTestSandbox") + : t("cancellationCleaningTestData") : runState.phase === "stopping" ? t("cancellationEndingTestRun") : null; diff --git a/src/components/preview_panel/TestsPanel.test.tsx b/src/components/preview_panel/TestsPanel.test.tsx index 03c21a9bf2..cd912afafe 100644 --- a/src/components/preview_panel/TestsPanel.test.tsx +++ b/src/components/preview_panel/TestsPanel.test.tsx @@ -31,6 +31,8 @@ const mocks = vi.hoisted(() => ({ appUrl: "http://localhost:32100" as string | null, previewUrl: "http://localhost:32100/" as string | null, previewUrlSource: "dyad" as "none" | "dyad" | "app", + app: { id: 1, testingEnabled: true } as Record, + settings: {} as Record, })); vi.mock("@/ipc/types", () => ({ @@ -52,11 +54,11 @@ vi.mock("@/lib/toast", () => ({ })); vi.mock("@/hooks/useLoadApp", () => ({ - useLoadApp: () => ({ app: { id: 1, testingEnabled: true } }), + useLoadApp: () => ({ app: mocks.app }), })); vi.mock("@/hooks/useSettings", () => ({ - useSettings: () => ({ settings: {}, updateSettings: vi.fn() }), + useSettings: () => ({ settings: mocks.settings, updateSettings: vi.fn() }), })); vi.mock("@/hooks/useRunApp", () => ({ @@ -131,6 +133,8 @@ describe("TestsPanel", () => { mocks.appUrl = "http://localhost:32100"; mocks.previewUrl = "http://localhost:32100/"; mocks.previewUrlSource = "dyad"; + mocks.app = { id: 1, testingEnabled: true }; + mocks.settings = {}; mocks.listAppTests.mockResolvedValue({ specs: [ { @@ -314,6 +318,62 @@ describe("TestsPanel", () => { ).toBe(true); }); + it("runs sandboxed tests without the preview being up", async () => { + // A sandboxed run serves its own copy of the app on its own port. Requiring + // the user's preview would block the whole point of the feature — and would + // contradict the panel's own "your preview keeps running" disclosure. + mocks.appUrl = null; + + renderPanel(); + + await screen.findByText("signup.spec.ts"); + expect(screen.queryByText("Start the app to run tests.")).toBeNull(); + expect( + ( + screen.getByRole("button", { + name: "Run all tests", + }) as HTMLButtonElement + ).disabled, + ).toBe(false); + }); + + it("still requires the preview when the run is not sandboxed", async () => { + // The fallback path runs Playwright against the user's preview, so the + // gate is still correct there. + mocks.appUrl = null; + mocks.settings = { disableSandboxedE2eTests: true }; + + renderPanel(); + + expect(await screen.findByText("Start the app to run tests.")).toBeTruthy(); + expect( + ( + screen.getByRole("button", { + name: "Run all tests", + }) as HTMLButtonElement + ).disabled, + ).toBe(true); + }); + + it("doesn't refuse the run while settings are still loading", async () => { + // `usesSandboxedE2eTests` answers false for absent settings, so the panel + // would flash the amber gate banner and a disabled Run button on every + // mount — a hard refusal for a state that may not apply at all. + mocks.appUrl = null; + mocks.settings = undefined as unknown as Record; + + renderPanel(); + + expect( + ( + (await screen.findByRole("button", { + name: "Run all tests", + })) as HTMLButtonElement + ).disabled, + ).toBe(false); + expect(screen.queryByText("Start the app to run tests.")).toBeNull(); + }); + describe("stopping a run", () => { /** Put the panel's app into `phase` as if a run had reached it. */ function setPhase( @@ -391,10 +451,44 @@ describe("TestsPanel", () => { expect(mocks.showError).toHaveBeenCalledWith(error); }); - it("names the Neon teardown, which restarts the preview", () => { + it("tells Neon users their preview keeps its real database", async () => { + // The pre-sandbox copy promised a double preview restart. Nothing + // restarts any more, so that disclosure would now be a lie. + mocks.app = { id: 1, testingEnabled: true, neonProjectId: "neon-proj" }; + renderPanel(); + + expect( + await screen.findByText(/Your preview keeps running against your real/), + ).toBeTruthy(); + expect(screen.queryByText(/restart the preview/i)).toBeNull(); + }); + + it("drops the sandbox disclosure when the sandbox is turned off", async () => { + mocks.app = { id: 1, testingEnabled: true, neonProjectId: "neon-proj" }; + mocks.settings = { disableSandboxedE2eTests: true }; + renderPanel(); + + await screen.findByText("signup.spec.ts"); + expect(screen.queryByText(/Your preview keeps running/)).toBeNull(); + }); + + it("promises no sandbox while settings are still loading", async () => { + // The Run gate treats loading as "sandbox available" so it doesn't refuse + // the run; this banner has to key off the same guard, or the panel + // briefly promises sandboxing to a user who has it turned off. + mocks.app = { id: 1, testingEnabled: true, neonProjectId: "neon-proj" }; + mocks.settings = undefined as unknown as Record; + renderPanel(); + + await screen.findByText("signup.spec.ts"); + expect(screen.queryByText(/Your preview keeps running/)).toBeNull(); + }); + + it("names the Neon teardown without promising a restore", () => { // This is the wait that can pass a minute (the branch delete retries with - // backoff), and it visibly reloads the user's preview. Calling it - // "Running…" — as the panel used to — reads as a hang. + // backoff). Calling it "Running…" — as the panel used to — reads as a + // hang, but the run had its own sandbox and its own server, so nothing + // of the user's is being put back either. const { store } = renderPanel(); setPhase(store, { phase: "cleaning-up", @@ -402,31 +496,47 @@ describe("TestsPanel", () => { }); expect( - screen.getByText(/Restoring your database and preview/), + screen.getByText(/Removing the temporary test database/), ).toBeTruthy(); + expect(screen.queryByText(/Restoring/i)).toBeNull(); expect( - screen.getByRole("button", { name: "Restoring your app" }).textContent, - ).toContain("Restoring…"); + screen.getByRole("button", { + name: "Removing the temporary test database", + }).textContent, + ).toContain("Cleaning up…"); }); - it("does not promise a database restore on the Supabase path", () => { - // That teardown only deletes the temporary test user. It never swaps - // `.env.local` and never restarts the app, so the Neon copy would lie. + it("names the sandbox cleanup on the Supabase path", () => { + // That teardown only deletes the temporary test user and the run's + // sandbox copy. It never swaps `.env.local` and never restarts the app. const { store } = renderPanel(); setPhase(store, { phase: "cleaning-up", isolation: { mode: "supabase-test-user" }, + sandboxed: true, }); - expect(screen.getByText(/Cleaning up the test data/)).toBeTruthy(); - expect(screen.queryByText(/Restoring your database/)).toBeNull(); + expect(screen.getByText(/Cleaning up the test sandbox/)).toBeTruthy(); + expect(screen.queryByText(/Restoring/i)).toBeNull(); expect( screen.getByRole("button", { name: "Cleaning up test data" }) .textContent, ).toContain("Cleaning up…"); - expect( - screen.queryByRole("button", { name: "Restoring your app" }), - ).toBeNull(); + }); + + it("claims no sandbox when the run never took one", () => { + // The fallback path (docker/cloud runtime, or the opt-out) creates no + // workspace, so naming one would be the same inaccurate cleanup copy the + // Neon "restoring your preview" wording was. + const { store } = renderPanel(); + setPhase(store, { + phase: "cleaning-up", + isolation: { mode: "supabase-test-user" }, + sandboxed: false, + }); + + expect(screen.getByText(/Cleaning up the test data/)).toBeTruthy(); + expect(screen.queryByText(/test sandbox/)).toBeNull(); }); it("does not carry a completed run's stop latch into the next run", () => { diff --git a/src/components/preview_panel/TestsPanel.tsx b/src/components/preview_panel/TestsPanel.tsx index 10af6903be..d24b62a09f 100644 --- a/src/components/preview_panel/TestsPanel.tsx +++ b/src/components/preview_panel/TestsPanel.tsx @@ -78,6 +78,7 @@ import { queryKeys } from "@/lib/queryKeys"; import { cn } from "@/lib/utils"; import { showError, showInfo, showSuccess } from "@/lib/toast"; import { findCaseResult, statusLabel, testKey } from "@/lib/testResultUtils"; +import { usesSandboxedE2eTests } from "@/lib/e2eSandbox"; import { usePreviewIframeController } from "@/preview_iframe/usePreviewIframe"; import { sameOriginStartPath } from "./previewAddressPath"; @@ -702,6 +703,27 @@ export function TestsPanel() { const parallel = settings?.testParallel ?? false; const devServerRunning = appUrl.appUrl !== null; + // A sandboxed run serves the app itself, on its own port, from its own copy — + // the user's preview is not involved, so requiring it would block the whole + // point of the feature. The fallback path (Docker/cloud runtime, or the + // opt-out) still runs Playwright against the preview and still needs it up. + // Recording is unaffected either way: it drives the live preview. + // + // While settings are still loading there is nothing to disclose yet: + // `usesSandboxedE2eTests` answers false for absent settings, which would + // flash the amber "Start the app to run tests." banner and a disabled Run + // button on every mount for a state that may not apply at all. + // + // Tri-state on purpose: `undefined` means settings haven't loaded, and that + // is neither a refusal nor a promise. `usesSandboxedE2eTests` answers false + // for absent settings, so reading it directly would flash the amber gate on + // every mount — and the Neon disclosure below has the opposite default, so + // reading `!disableSandboxedE2eTests` there would briefly promise sandboxing + // to a user who turned it off. One value, two explicit comparisons. + const sandboxAvailable = settings + ? usesSandboxedE2eTests(settings) + : undefined; + const testRunBlocked = sandboxAvailable === false && !devServerRunning; // Owns the run's whole lifecycle, teardown included. Gates every action that // must not interleave with it (Run, Record, Delete), because the per-app lock // is still held during `cleaning-up`. @@ -715,7 +737,11 @@ export function TestsPanel() { (runState.phase === "cleaning-up" && !runState.wasStopped); const isStopping = runState.phase === "stopping"; const isCleaningUp = runState.phase === "cleaning-up"; - const isRestoringApp = + // Nothing about the user's app is restored any more: the run had its own + // sandbox copy and its own server, so cleanup is deleting that sandbox and + // the temporary database it was pointed at. The real `.env.local` and the + // preview were never touched. + const isRemovingTestDatabase = isCleaningUp && runState.isolation?.mode === "neon-branch"; const specsQuery = useQuery({ queryKey: queryKeys.tests.list({ appId: selectedAppId }), @@ -746,10 +772,12 @@ export function TestsPanel() { }); const loadingSpecs = specsQuery.isLoading && specs.length === 0; - const showNeonRestartDisclosure = - specs.length > 0 && - !!app?.neonProjectId && - (settings?.runtimeMode2 ?? "host") === "host"; + // Host runs get the sandbox: a throwaway copy of the app, its own server, and + // a temporary Neon branch that only that copy points at. Worth saying, since + // the alternative a user would assume is "my tests hit my real database" — + // but it must not promise the preview restart the old env-swap path did. + const showNeonSandboxDisclosure = + specs.length > 0 && !!app?.neonProjectId && sandboxAvailable === true; // Pop the output drawer when a run starts for the app being viewed. Keyed // off the global atom's phase transition — not the raw IPC event — so it @@ -1377,8 +1405,8 @@ export function TestsPanel() { disabled={showStopping || isCleaningUp} aria-label={ isCleaningUp - ? isRestoringApp - ? "Restoring your app" + ? isRemovingTestDatabase + ? "Removing the temporary test database" : "Cleaning up test data" : showStopping ? "Stopping tests" @@ -1397,9 +1425,7 @@ export function TestsPanel() { )} {isCleaningUp - ? isRestoringApp - ? "Restoring…" - : "Cleaning up…" + ? "Cleaning up…" : showStopping ? "Stopping…" : "Stop"} @@ -1409,13 +1435,13 @@ export function TestsPanel() { specs.length > 0 && (