fix: FT-5226's socket-drop shape kills the canary and could kill a deploy

gotchas.md

@@ -314,6 +314,8 @@ process.on("unhandledRejection", (e) => console.error(e));
 
 **Symptom.** A client connecting to a PPg database **immediately after it is provisioned** fails on the first connection. Through Prisma Next's control client it surfaces as `CliStructuredError: Database connection failed`; the raw node-postgres error is `message: "Failed to connect to upstream database. Please contact Prisma support…"`, `err.code === undefined`, no `err.cause`. The **direct** endpoint fast-rejects (~0.4–0.6s); the **pooled** endpoint slow-times-out (~10s). Intermittent — the same DSN sometimes connects on attempt 1; it reproduces reliably only when connecting within a moment of provisioning.
 
+The same cold window has a **second presentation**, and it is the more dangerous one. Instead of rejecting, the proxy **completes** the connection — `pg` resolves `connect()` only after authentication succeeds and the server sends `ReadyForQuery` — and then drops the socket, so the failure lands on the *first query* as `Connection terminated unexpectedly`. Two consequences: a retry wrapped around the connect alone never engages, because the connect succeeded; and `pg` reports the drop by emitting an `'error'` event on the client, which is an uncaught exception in any process not listening for it, so it kills the process rather than returning an error you can catch. Both shapes turned up in the same "Cold-connect canary" job within two days (jobs 92958820634 and 93248605341, 2026-08-07 and 2026-08-09).
+
 **Cause.** The PPg edge accepts the TCP/TLS connection but the **upstream database is cold** (just provisioned / scaled to zero) and not yet ready, so the proxy rejects with the generic "upstream" error. Confirmed **not** TLS, **not** network (no `ECONNREFUSED`/`ETIMEDOUT`), **not** auth: on a **warmed** DB every SSL posture (`require`, `verify-full`, `no-verify`) and both endpoints connect, and PPg's cert is publicly trusted. Same cold/scale-to-zero family as FT-5219, different surface — FT-5219 is an *idle-close* crash of a persistent runtime client; this is the *first connect* being rejected at deploy time. A deploy-time migration hits the cold window ~every time because it connects the instant the DB is provisioned.
 
 **Workaround.** Bounded connection **retry** on connect — retry connect/transient failures (not real errors) for ~1 min; warm DBs connect immediately and the retry rides out the cold-start (observed connect at ~10s):
@@ -335,8 +337,9 @@ await withConnectionRetry(() => client.dbInit(...), { attempts: 12, delayMs: 500
 **References.**
 
 - Upstream: [FT-5226](https://linear.app/prisma-company/issue/FT-5226/first-connection-to-a-freshly-provisioned-postgres-is-rejected-while)
-- Workaround source: [`packages/app-cloud/src/prisma-next-migrate.ts`](packages/app-cloud/src/prisma-next-migrate.ts) (`withConnectionRetry`)
-- Removal guard: the CI canary (`scripts/cold-connect-canary.ts`, "Cold-connect canary" E2E job) fails only when every cold connect in a run of at least 14 succeeds — the rejection is intermittent, so a shorter unanimous streak is treated as luck, not a fix (same reasoning as PRO-217's 14-hold requirement below) — when the platform fixes FT-5226 it goes red, forcing removal of `withConnectionRetry` and itself (an inconclusive run passes with a warning annotation instead of blocking)
+- Workaround source: [`packages/1-prisma-cloud/1-extensions/target/src/pg-connection.ts`](packages/1-prisma-cloud/1-extensions/target/src/pg-connection.ts) (`withConnectionRetry`), used by [`prisma-next-migrate.ts`](packages/1-prisma-cloud/1-extensions/target/src/prisma-next-migrate.ts) and [`pg-warm-resource.ts`](packages/1-prisma-cloud/1-extensions/target/src/pg-warm-resource.ts). Both wrap connect **and** the operation, which is what makes them proof against the socket-drop presentation — wrapping the connect alone would not be. Wrapping is necessary but not sufficient: every client also needs an `'error'` listener, because an unhandled `'error'` event is raised outside the promise and `withConnectionRetry` cannot catch it.
+- The control client used for deploy-time migrations (`createPostgresControlClient` → `PostgresControlDriver`) has the same trap, and it bites on the dependency rather than on our code. `@prisma-next/driver-postgres@0.16.0` builds it around a long-lived `pg.Client` with no `'error'` listener (`dist/control.mjs:32`), so a socket drop there takes the deploy process down instead of failing the migration. Fixed upstream in [`prisma/prisma@0e51f1f4d`](https://github.com/prisma/prisma/commit/0e51f1f4d) (2026-07-22), which adds the listener and makes `close()` tolerate a dropped socket. 0.16.0 was published one day earlier and never got it; the fix ships in `@prisma/orm-target-postgres@8.0.0-rc.1` (`client.on("error", () => {})` at `dist/control-6WFTtLAM.mjs:33`, verified in the published tarball), which Composer adopts in the Prisma 8 upgrade. Nothing to do here once that lands — and no dependency patch was ever warranted.
+- Removal guard: the CI canary (`scripts/cold-connect-canary.ts`, "Cold-connect canary" E2E job) fails only when every cold connect in a run of at least 14 succeeds — the rejection is intermittent, so a shorter unanimous streak is treated as luck, not a fix (same reasoning as PRO-217's 14-hold requirement below) — when the platform fixes FT-5226 it goes red, forcing removal of `withConnectionRetry` and itself (an inconclusive run passes with a warning annotation instead of blocking). Exit 1 means that verdict and nothing else, because it is read as an instruction to delete working production code: a canary that cannot provision, a teardown that fails, and a stray async error all exit 0 with the reason logged. The stray-error case is not hypothetical — an unhandled `'error'` event from the canary's own `pg` client (the same defect as the idle-close entry below) made bun exit 1 a second *after* the correct "bug still present" verdict had printed, which reads in the log as a crash during project teardown
 - Related: [FT-5219](https://linear.app/prisma-company/issue/FT-5219) (idle-close, runtime), [PRO-212](https://linear.app/prisma-company/issue/PRO-212) (nested endpoint DSNs)
 
 ---

packages/1-prisma-cloud/1-extensions/target/src/__tests__/pg-warm-resource.test.ts

@@ -8,9 +8,11 @@
  *
  * Self-isolating: owns a uniquely-named database (never the shared `public`).
  */
+
 import { afterAll, beforeAll, describe, expect, test } from 'bun:test';
+import net from 'node:net';
 import * as Effect from 'effect/Effect';
-import { pgWarmProviderService } from '../pg-warm-resource.ts';
+import { pgWarmProviderService, warmDatabase } from '../pg-warm-resource.ts';
 import {
   createTestDatabase,
   startTestPostgres,
@@ -61,3 +63,57 @@ describe.skipIf(pg === undefined)('PgWarm reconcile warms a real database', () =
     expect(result.url).toBe(testDb.url);
   });
 });
+
+/**
+ * FT-5226's second shape: the cold upstream ACCEPTS the connection and then
+ * drops the socket. pg reports that by emitting 'error' on the client rather
+ * than only rejecting the query, and an 'error' event with no listener is an
+ * uncaught exception — raised outside the promise, so `withConnectionRetry`
+ * cannot catch it and it would take the deploy process down.
+ *
+ * Needs no real Postgres: the stub speaks just enough of the startup protocol
+ * to make pg's `connect()` resolve, which is what puts the client past
+ * `_connecting` and into the path that emits on the client.
+ */
+describe('a cold upstream that drops the socket after connecting', () => {
+  let server: net.Server;
+  let url: string;
+
+  beforeAll(async () => {
+    server = net.createServer((socket) => {
+      socket.on('error', () => {});
+      let startupAnswered = false;
+      socket.on('data', (buf) => {
+        // SSLRequest → refuse, so the startup continues in plaintext.
+        if (buf.length >= 8 && buf.readInt32BE(4) === 80877103) {
+          socket.write(Buffer.from('N'));
+          return;
+        }
+        // Never answer `select 1` — answering it would complete the query and
+        // the warm would succeed before the drop lands.
+        if (startupAnswered) return;
+        startupAnswered = true;
+        const authOk = Buffer.alloc(9);
+        authOk.write('R', 0, 'ascii');
+        authOk.writeInt32BE(8, 1);
+        authOk.writeInt32BE(0, 5);
+        const readyForQuery = Buffer.alloc(6);
+        readyForQuery.write('Z', 0, 'ascii');
+        readyForQuery.writeInt32BE(5, 1);
+        readyForQuery.write('I', 5, 'ascii');
+        socket.write(Buffer.concat([authOk, readyForQuery]));
+        setTimeout(() => socket.destroy(), 20);
+      });
+    });
+    await new Promise<void>((resolve) => server.listen(0, '127.0.0.1', resolve));
+    const { port } = server.address() as net.AddressInfo;
+    url = `postgres://warm:warm@127.0.0.1:${port}/warm?sslmode=disable`;
+  });
+  afterAll(() => server.close());
+
+  test('surfaces as a rejection instead of killing the process', async () => {
+    // attempts: 1 — the retry itself is proven in pg-connection.test.ts; what
+    // this asserts is that the drop comes back as a rejected promise at all.
+    await expect(warmDatabase(url, { attempts: 1 })).rejects.toThrow();
+  });
+});

packages/1-prisma-cloud/1-extensions/target/src/pg-warm-resource.ts

@@ -36,17 +36,30 @@ export type PgWarm = Resource<'PrismaCloud.PgWarm', PgWarmProps, PgWarmAttribute
 /** The `PgWarm` resource constructor — `yield* PgWarm(id, { url })` in a lowering. */
 export const PgWarm = Resource<PgWarm>('PrismaCloud.PgWarm');
 
-/** Connect (retrying the cold-start) and run `select 1`, then release the connection. */
-async function warmDatabase(url: string): Promise<void> {
+/**
+ * Connect (retrying the cold-start) and run `select 1`, then release the
+ * connection. Exported so tests can drive it directly; `retry` overrides the
+ * bounded retry's defaults.
+ */
+export async function warmDatabase(
+  url: string,
+  retry: { readonly attempts?: number; readonly delayMs?: number } = {},
+): Promise<void> {
   await withConnectionRetry(async () => {
     const client = new pg.Client({ connectionString: normalizeSslMode(url) });
+    // A cold upstream can accept the connect and then drop the socket (FT-5226's
+    // second shape). pg reports that by emitting 'error' on the client, which is
+    // an uncaught exception with no listener — raised outside this promise, so
+    // the retry above never sees it and it would kill the deploy instead. The
+    // query/end rejection is what the retry acts on.
+    client.on('error', (error) => console.error('pg warm client socket error', error));
     await client.connect();
     try {
       await client.query('select 1');
     } finally {
       await client.end();
     }
-  });
+  }, retry);
 }
 
 /**

scripts/cold-connect-canary.ts

@@ -13,25 +13,50 @@
  * canary; inconclusive → exit 0 with a CI warning annotation, so a flake
  * never blocks unrelated PRs. Sampling is adaptive: the first rejection ends
  * the run, and only an all-success streak keeps going to the full depth.
+ *
+ * Exit 1 is therefore a live instruction to delete production code, so it must
+ * mean a bug-gone verdict and nothing else. Every other outcome — a canary that
+ * cannot provision, a teardown that fails, a stray async error — exits 0 with
+ * the reason logged. See the uncaught-error handlers below for why that needs
+ * explicit work under bun.
  */
 import pg from 'pg';
 import { deleteProjectDeep, type HttpCall, type ProjectRef } from './ci-cleanup-utils.ts';
 import {
   type ColdConnectSample,
+  type ColdConnectVerdict,
   classifyColdConnectRun,
   classifyColdConnectSample,
   MIN_BUG_GONE_SAMPLES,
 } from './cold-connect-canary-classify.ts';
 
+// bun exits 1 on any uncaught error or unhandled rejection, whatever
+// process.exitCode says — which silently turns a stray socket error into this
+// script's "delete withConnectionRetry" signal. Absorbing them here keeps the
+// exit code equal to the verdict; the run's own failures are caught below, so
+// nothing that decides the verdict reaches these handlers.
+process.on('uncaughtException', (error) => {
+  console.error('Uncaught error — logged only, the verdict decides the exit code:', error);
+});
+process.on('unhandledRejection', (reason) => {
+  console.error('Unhandled rejection — logged only, the verdict decides the exit code:', reason);
+});
+
 const API = 'https://api.prisma.io/v1';
 const REGION = 'us-east-1';
 const SAMPLES = Number(process.env['COLD_CONNECT_SAMPLES'] ?? '5');
 
 const token = process.env['PRISMA_SERVICE_TOKEN'];
 const workspaceId = process.env['PRISMA_WORKSPACE_ID'];
 if (!token || !workspaceId) {
+  // Exit 0, not 1: a run with no credentials sampled nothing, and exit 1 would
+  // tell the reader to delete withConnectionRetry on the strength of it.
   console.error('PRISMA_SERVICE_TOKEN and PRISMA_WORKSPACE_ID are required');
-  process.exit(1);
+  console.log(
+    '::warning title=Cold-connect canary (FT-5226) could not run::PRISMA_SERVICE_TOKEN and ' +
+      'PRISMA_WORKSPACE_ID are required — no FT-5226 verdict this run; not blocking.',
+  );
+  process.exit(0);
 }
 
 const runId = process.env['GITHUB_RUN_ID'] ?? `${process.pid}${Math.floor(Math.random() * 1000)}`;
@@ -104,6 +129,13 @@ async function sampleColdConnect(projectId: string, index: number): Promise<Cold
   if (!dsn) throw new Error('connection returned no direct/pooled connection string');
 
   const client = new pg.Client({ connectionString: dsn, connectionTimeoutMillis: 10_000 });
+  // PPg sometimes accepts the connect and then drops the socket. pg reports
+  // that as an 'error' event on the client, and an 'error' event with no
+  // listener is an uncaught exception. The connect/query failure below is what
+  // classifies the sample; this listener only keeps the report from killing us.
+  client.on('error', (error: Error) => {
+    console.log(`  sample #${index}: client reported a socket error — ${error.message}`);
+  });
   let connectError: unknown;
   const started = Date.now();
   try {
@@ -126,16 +158,18 @@ async function sampleColdConnect(projectId: string, index: number): Promise<Cold
 
 let project: ProjectRef | undefined;
 
-try {
+/** Provisions the project, samples until the verdict is settled, and reports it. */
+async function runCanary(): Promise<ColdConnectVerdict> {
   const createdProject = await apiData('POST', '/projects', {
     name: projectName,
     workspaceId,
   });
-  project = {
+  const created: ProjectRef = {
     id: requireString(createdProject, 'id'),
     name: requireString(createdProject, 'name'),
   };
-  console.log(`Created project "${project.name}" (${project.id}); sampling ${SAMPLES} cold DBs…`);
+  project = created;
+  console.log(`Created project "${created.name}" (${created.id}); sampling ${SAMPLES} cold DBs…`);
 
   // One rejection settles the verdict, so stop there. An all-success streak
   // keeps sampling up to MIN_BUG_GONE_SAMPLES — below that, all-success is
@@ -145,7 +179,7 @@ try {
     samples.length < SAMPLES ||
     (samples.length < MIN_BUG_GONE_SAMPLES && samples.every((s) => s === 'success'))
   ) {
-    const sample = await sampleColdConnect(project.id, samples.length);
+    const sample = await sampleColdConnect(created.id, samples.length);
     samples.push(sample);
     if (sample === 'rejected') break;
   }
@@ -158,15 +192,42 @@ try {
       `::warning title=Cold-connect canary (FT-5226) inconclusive::${result.message} [${detail}]`,
     );
   }
-  process.exitCode = result.verdict === 'bug-gone' ? 1 : 0;
-} finally {
-  if (project) {
-    console.log(`Deleting project "${project.name}" (${project.id})…`);
+  return result.verdict;
+}
+
+/**
+ * Teardown is best-effort: a leftover project costs one workspace slot until
+ * the CI cleanup job's next sweep of the `canary` prefix, which is not worth
+ * overturning a verdict the run already reached.
+ */
+async function deleteCanaryProject(): Promise<void> {
+  if (!project) return;
+  console.log(`Deleting project "${project.name}" (${project.id})…`);
+  const leaked = `canary project "${project.name}" (${project.id}) — the CI cleanup job sweeps the "canary" prefix.`;
+  try {
     const deleted = await deleteProjectDeep(http, project, { log: (line) => console.error(line) });
-    if (!deleted) {
-      console.error(
-        `Failed to delete canary project "${project.name}" (${project.id}) — check for a leak.`,
-      );
-    }
+    if (!deleted) console.error(`Could not delete ${leaked}`);
+  } catch (error) {
+    console.error(`Deleting ${leaked}\n  the delete threw:`, error);
   }
 }
+
+let verdict: ColdConnectVerdict;
+try {
+  verdict = await runCanary();
+} catch (error) {
+  // No samples means no verdict, so there is nothing to report about FT-5226 —
+  // and claiming bug-gone here would tell someone to delete a workaround this
+  // run never tested. Same treatment as inconclusive: loud, but not blocking.
+  console.error('Cold-connect canary failed before reaching a verdict:', error);
+  const detail = (error instanceof Error ? error.message : String(error)).replace(/\s+/g, ' ');
+  console.log(
+    `::warning title=Cold-connect canary (FT-5226) could not run::${detail} — no FT-5226 ` +
+      'verdict this run; not blocking. Keep withConnectionRetry.',
+  );
+  verdict = 'inconclusive';
+} finally {
+  await deleteCanaryProject();
+}
+
+process.exitCode = verdict === 'bug-gone' ? 1 : 0;