From e172776941adb75902c4ea2e6146971437266be8 Mon Sep 17 00:00:00 2001 From: willbot Date: Sun, 9 Aug 2026 16:16:58 +0200 Subject: [PATCH 1/8] fix(ci): stop a dropped socket failing the cold-connect canary MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Prisma Postgres sometimes accepts a cold connection and then drops the socket. `pg` reports that by emitting an `'error'` event on the client, and an `'error'` event with no listener is an uncaught exception, so bun fixed the exit code at 1 while the canary went on to print the correct "bug still present" verdict and delete its project. The version banner bun prints at exit landed just after "Deleting project ...", which reads as a crash during teardown — it is not; teardown succeeded every time. Exit 1 is this canary's instruction to delete `withConnectionRetry` from production code, so a dropped socket was quietly producing that signal on a required check. Each probe client now listens for `'error'` and logs it. Stray uncaught errors and rejections are absorbed so they cannot decide the exit code. Teardown is best-effort, since the CI cleanup job already sweeps the `canary` prefix. A canary that cannot run at all now exits 0 with a warning annotation rather than 1, which was the same false signal. Signed-off-by: willbot Signed-off-by: Will Madden --- gotchas.md | 2 +- scripts/cold-connect-canary.ts | 89 ++++++++++++++++++++++++++++------ 2 files changed, 76 insertions(+), 15 deletions(-) diff --git a/gotchas.md b/gotchas.md index 833fb1ddf..dc2007bdc 100644 --- a/gotchas.md +++ b/gotchas.md @@ -336,7 +336,7 @@ await withConnectionRetry(() => client.dbInit(...), { attempts: 12, delayMs: 500 - Upstream: [FT-5226](https://linear.app/prisma-company/issue/FT-5226/first-connection-to-a-freshly-provisioned-postgres-is-rejected-while) - Workaround source: [`packages/app-cloud/src/prisma-next-migrate.ts`](packages/app-cloud/src/prisma-next-migrate.ts) (`withConnectionRetry`) -- Removal guard: the CI canary (`scripts/cold-connect-canary.ts`, "Cold-connect canary" E2E job) fails only when every cold connect in a run of at least 14 succeeds — the rejection is intermittent, so a shorter unanimous streak is treated as luck, not a fix (same reasoning as PRO-217's 14-hold requirement below) — when the platform fixes FT-5226 it goes red, forcing removal of `withConnectionRetry` and itself (an inconclusive run passes with a warning annotation instead of blocking) +- Removal guard: the CI canary (`scripts/cold-connect-canary.ts`, "Cold-connect canary" E2E job) fails only when every cold connect in a run of at least 14 succeeds — the rejection is intermittent, so a shorter unanimous streak is treated as luck, not a fix (same reasoning as PRO-217's 14-hold requirement below) — when the platform fixes FT-5226 it goes red, forcing removal of `withConnectionRetry` and itself (an inconclusive run passes with a warning annotation instead of blocking). Exit 1 means that verdict and nothing else, because it is read as an instruction to delete working production code: a canary that cannot provision, a teardown that fails, and a stray async error all exit 0 with the reason logged. The stray-error case is not hypothetical — an unhandled `'error'` event from the canary's own `pg` client (the same defect as the idle-close entry below) made bun exit 1 a second *after* the correct "bug still present" verdict had printed, which reads in the log as a crash during project teardown - Related: [FT-5219](https://linear.app/prisma-company/issue/FT-5219) (idle-close, runtime), [PRO-212](https://linear.app/prisma-company/issue/PRO-212) (nested endpoint DSNs) --- diff --git a/scripts/cold-connect-canary.ts b/scripts/cold-connect-canary.ts index f840968cb..6f35e391a 100644 --- a/scripts/cold-connect-canary.ts +++ b/scripts/cold-connect-canary.ts @@ -13,16 +13,35 @@ * canary; inconclusive → exit 0 with a CI warning annotation, so a flake * never blocks unrelated PRs. Sampling is adaptive: the first rejection ends * the run, and only an all-success streak keeps going to the full depth. + * + * Exit 1 is therefore a live instruction to delete production code, so it must + * mean a bug-gone verdict and nothing else. Every other outcome — a canary that + * cannot provision, a teardown that fails, a stray async error — exits 0 with + * the reason logged. See the uncaught-error handlers below for why that needs + * explicit work under bun. */ import pg from 'pg'; import { deleteProjectDeep, type HttpCall, type ProjectRef } from './ci-cleanup-utils.ts'; import { type ColdConnectSample, + type ColdConnectVerdict, classifyColdConnectRun, classifyColdConnectSample, MIN_BUG_GONE_SAMPLES, } from './cold-connect-canary-classify.ts'; +// bun exits 1 on any uncaught error or unhandled rejection, whatever +// process.exitCode says — which silently turns a stray socket error into this +// script's "delete withConnectionRetry" signal. Absorbing them here keeps the +// exit code equal to the verdict; the run's own failures are caught below, so +// nothing that decides the verdict reaches these handlers. +process.on('uncaughtException', (error) => { + console.error('Uncaught error — logged only, the verdict decides the exit code:', error); +}); +process.on('unhandledRejection', (reason) => { + console.error('Unhandled rejection — logged only, the verdict decides the exit code:', reason); +}); + const API = 'https://api.prisma.io/v1'; const REGION = 'us-east-1'; const SAMPLES = Number(process.env['COLD_CONNECT_SAMPLES'] ?? '5'); @@ -30,8 +49,14 @@ const SAMPLES = Number(process.env['COLD_CONNECT_SAMPLES'] ?? '5'); const token = process.env['PRISMA_SERVICE_TOKEN']; const workspaceId = process.env['PRISMA_WORKSPACE_ID']; if (!token || !workspaceId) { + // Exit 0, not 1: a run with no credentials sampled nothing, and exit 1 would + // tell the reader to delete withConnectionRetry on the strength of it. console.error('PRISMA_SERVICE_TOKEN and PRISMA_WORKSPACE_ID are required'); - process.exit(1); + console.log( + '::warning title=Cold-connect canary (FT-5226) could not run::PRISMA_SERVICE_TOKEN and ' + + 'PRISMA_WORKSPACE_ID are required — no FT-5226 verdict this run; not blocking.', + ); + process.exit(0); } const runId = process.env['GITHUB_RUN_ID'] ?? `${process.pid}${Math.floor(Math.random() * 1000)}`; @@ -104,6 +129,13 @@ async function sampleColdConnect(projectId: string, index: number): Promise { + console.log(` sample #${index}: client reported a socket error — ${error.message}`); + }); let connectError: unknown; const started = Date.now(); try { @@ -126,16 +158,18 @@ async function sampleColdConnect(projectId: string, index: number): Promise { const createdProject = await apiData('POST', '/projects', { name: projectName, workspaceId, }); - project = { + const created: ProjectRef = { id: requireString(createdProject, 'id'), name: requireString(createdProject, 'name'), }; - console.log(`Created project "${project.name}" (${project.id}); sampling ${SAMPLES} cold DBs…`); + project = created; + console.log(`Created project "${created.name}" (${created.id}); sampling ${SAMPLES} cold DBs…`); // One rejection settles the verdict, so stop there. An all-success streak // keeps sampling up to MIN_BUG_GONE_SAMPLES — below that, all-success is @@ -145,7 +179,7 @@ try { samples.length < SAMPLES || (samples.length < MIN_BUG_GONE_SAMPLES && samples.every((s) => s === 'success')) ) { - const sample = await sampleColdConnect(project.id, samples.length); + const sample = await sampleColdConnect(created.id, samples.length); samples.push(sample); if (sample === 'rejected') break; } @@ -158,15 +192,42 @@ try { `::warning title=Cold-connect canary (FT-5226) inconclusive::${result.message} [${detail}]`, ); } - process.exitCode = result.verdict === 'bug-gone' ? 1 : 0; -} finally { - if (project) { - console.log(`Deleting project "${project.name}" (${project.id})…`); + return result.verdict; +} + +/** + * Teardown is best-effort: a leftover project costs one workspace slot until + * the CI cleanup job's next sweep of the `canary` prefix, which is not worth + * overturning a verdict the run already reached. + */ +async function deleteCanaryProject(): Promise { + if (!project) return; + console.log(`Deleting project "${project.name}" (${project.id})…`); + const leaked = `canary project "${project.name}" (${project.id}) — the CI cleanup job sweeps the "canary" prefix.`; + try { const deleted = await deleteProjectDeep(http, project, { log: (line) => console.error(line) }); - if (!deleted) { - console.error( - `Failed to delete canary project "${project.name}" (${project.id}) — check for a leak.`, - ); - } + if (!deleted) console.error(`Could not delete ${leaked}`); + } catch (error) { + console.error(`Deleting ${leaked}\n the delete threw:`, error); } } + +let verdict: ColdConnectVerdict; +try { + verdict = await runCanary(); +} catch (error) { + // No samples means no verdict, so there is nothing to report about FT-5226 — + // and claiming bug-gone here would tell someone to delete a workaround this + // run never tested. Same treatment as inconclusive: loud, but not blocking. + console.error('Cold-connect canary failed before reaching a verdict:', error); + const detail = (error instanceof Error ? error.message : String(error)).replace(/\s+/g, ' '); + console.log( + `::warning title=Cold-connect canary (FT-5226) could not run::${detail} — no FT-5226 ` + + 'verdict this run; not blocking. Keep withConnectionRetry.', + ); + verdict = 'inconclusive'; +} finally { + await deleteCanaryProject(); +} + +process.exitCode = verdict === 'bug-gone' ? 1 : 0; From dd2a3d35c9151520739862d8942211742ef4e7bc Mon Sep 17 00:00:00 2001 From: willbot Date: Sun, 9 Aug 2026 16:23:19 +0200 Subject: [PATCH 2/8] docs(gotchas): record the socket-drop presentation of FT-5226 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The entry described only the connect-time rejection, which would lead the next reader to write a retry around the connect alone. The cold window also presents as a completed connection that is then dropped, so the failure lands on the first query — the canary hit both shapes in the same job within two days. Also repoints the workaround source, which named `packages/app-cloud/src/prisma-next-migrate.ts`. That path does not exist; `withConnectionRetry` lives in `packages/1-prisma-cloud/1-extensions/target/src/pg-connection.ts`. Signed-off-by: willbot Signed-off-by: Will Madden --- gotchas.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/gotchas.md b/gotchas.md index dc2007bdc..06699e491 100644 --- a/gotchas.md +++ b/gotchas.md @@ -314,6 +314,8 @@ process.on("unhandledRejection", (e) => console.error(e)); **Symptom.** A client connecting to a PPg database **immediately after it is provisioned** fails on the first connection. Through Prisma Next's control client it surfaces as `CliStructuredError: Database connection failed`; the raw node-postgres error is `message: "Failed to connect to upstream database. Please contact Prisma support…"`, `err.code === undefined`, no `err.cause`. The **direct** endpoint fast-rejects (~0.4–0.6s); the **pooled** endpoint slow-times-out (~10s). Intermittent — the same DSN sometimes connects on attempt 1; it reproduces reliably only when connecting within a moment of provisioning. +The same cold window has a **second presentation**, and it is the more dangerous one. Instead of rejecting, the proxy **completes** the connection — `pg` resolves `connect()` only after authentication succeeds and the server sends `ReadyForQuery` — and then drops the socket, so the failure lands on the *first query* as `Connection terminated unexpectedly`. Two consequences: a retry wrapped around the connect alone never engages, because the connect succeeded; and `pg` reports the drop by emitting an `'error'` event on the client, which is an uncaught exception in any process not listening for it, so it kills the process rather than returning an error you can catch. Both shapes turned up in the same "Cold-connect canary" job within two days (jobs 92958820634 and 93248605341, 2026-08-07 and 2026-08-09). + **Cause.** The PPg edge accepts the TCP/TLS connection but the **upstream database is cold** (just provisioned / scaled to zero) and not yet ready, so the proxy rejects with the generic "upstream" error. Confirmed **not** TLS, **not** network (no `ECONNREFUSED`/`ETIMEDOUT`), **not** auth: on a **warmed** DB every SSL posture (`require`, `verify-full`, `no-verify`) and both endpoints connect, and PPg's cert is publicly trusted. Same cold/scale-to-zero family as FT-5219, different surface — FT-5219 is an *idle-close* crash of a persistent runtime client; this is the *first connect* being rejected at deploy time. A deploy-time migration hits the cold window ~every time because it connects the instant the DB is provisioned. **Workaround.** Bounded connection **retry** on connect — retry connect/transient failures (not real errors) for ~1 min; warm DBs connect immediately and the retry rides out the cold-start (observed connect at ~10s): @@ -335,7 +337,7 @@ await withConnectionRetry(() => client.dbInit(...), { attempts: 12, delayMs: 500 **References.** - Upstream: [FT-5226](https://linear.app/prisma-company/issue/FT-5226/first-connection-to-a-freshly-provisioned-postgres-is-rejected-while) -- Workaround source: [`packages/app-cloud/src/prisma-next-migrate.ts`](packages/app-cloud/src/prisma-next-migrate.ts) (`withConnectionRetry`) +- Workaround source: [`packages/1-prisma-cloud/1-extensions/target/src/pg-connection.ts`](packages/1-prisma-cloud/1-extensions/target/src/pg-connection.ts) (`withConnectionRetry`), used by [`prisma-next-migrate.ts`](packages/1-prisma-cloud/1-extensions/target/src/prisma-next-migrate.ts) and [`pg-warm-resource.ts`](packages/1-prisma-cloud/1-extensions/target/src/pg-warm-resource.ts). Both wrap connect **and** the operation, which is what makes them proof against the socket-drop presentation — wrapping the connect alone would not be. - Removal guard: the CI canary (`scripts/cold-connect-canary.ts`, "Cold-connect canary" E2E job) fails only when every cold connect in a run of at least 14 succeeds — the rejection is intermittent, so a shorter unanimous streak is treated as luck, not a fix (same reasoning as PRO-217's 14-hold requirement below) — when the platform fixes FT-5226 it goes red, forcing removal of `withConnectionRetry` and itself (an inconclusive run passes with a warning annotation instead of blocking). Exit 1 means that verdict and nothing else, because it is read as an instruction to delete working production code: a canary that cannot provision, a teardown that fails, and a stray async error all exit 0 with the reason logged. The stray-error case is not hypothetical — an unhandled `'error'` event from the canary's own `pg` client (the same defect as the idle-close entry below) made bun exit 1 a second *after* the correct "bug still present" verdict had printed, which reads in the log as a crash during project teardown - Related: [FT-5219](https://linear.app/prisma-company/issue/FT-5219) (idle-close, runtime), [PRO-212](https://linear.app/prisma-company/issue/PRO-212) (nested endpoint DSNs) From 8ebdb523747d4e4869adc397e5018a4a02d6831f Mon Sep 17 00:00:00 2001 From: willbot Date: Sun, 9 Aug 2026 16:29:38 +0200 Subject: [PATCH 3/8] fix(prisma-cloud): stop a dropped socket killing the deploy in PgWarm PgWarm exists to make the first connection to a freshly provisioned database, which makes it the most likely place in the codebase to meet FT-5226's socket-drop shape: the cold upstream accepts the connection and then drops it. `pg` reports that by emitting an `'error'` event on the client, and an `'error'` event with no listener is an uncaught exception. It is raised outside the promise, so `withConnectionRetry` never sees it and the deploy process dies instead of retrying. There is no `uncaughtException` guard anywhere in the deploy path. `warmDatabase` now listens for `'error'` and logs it, matching what the pools in `prisma-next.ts` and `auth-options.ts` already do. It is exported with an optional retry override so the test can drive it without waiting out the default minute of retries. The test needs no real Postgres: a stub speaks enough of the startup protocol for `connect()` to resolve, then drops the socket. Removing the listener makes it fail at `pg/lib/client.js:217`, the emit path. Signed-off-by: willbot Signed-off-by: Will Madden --- .../src/__tests__/pg-warm-resource.test.ts | 58 ++++++++++++++++++- .../target/src/pg-warm-resource.ts | 19 +++++- 2 files changed, 73 insertions(+), 4 deletions(-) diff --git a/packages/1-prisma-cloud/1-extensions/target/src/__tests__/pg-warm-resource.test.ts b/packages/1-prisma-cloud/1-extensions/target/src/__tests__/pg-warm-resource.test.ts index 41eb5e9b7..46a9a0bf5 100644 --- a/packages/1-prisma-cloud/1-extensions/target/src/__tests__/pg-warm-resource.test.ts +++ b/packages/1-prisma-cloud/1-extensions/target/src/__tests__/pg-warm-resource.test.ts @@ -8,9 +8,11 @@ * * Self-isolating: owns a uniquely-named database (never the shared `public`). */ + import { afterAll, beforeAll, describe, expect, test } from 'bun:test'; +import net from 'node:net'; import * as Effect from 'effect/Effect'; -import { pgWarmProviderService } from '../pg-warm-resource.ts'; +import { pgWarmProviderService, warmDatabase } from '../pg-warm-resource.ts'; import { createTestDatabase, startTestPostgres, @@ -61,3 +63,57 @@ describe.skipIf(pg === undefined)('PgWarm reconcile warms a real database', () = expect(result.url).toBe(testDb.url); }); }); + +/** + * FT-5226's second shape: the cold upstream ACCEPTS the connection and then + * drops the socket. pg reports that by emitting 'error' on the client rather + * than only rejecting the query, and an 'error' event with no listener is an + * uncaught exception — raised outside the promise, so `withConnectionRetry` + * cannot catch it and it would take the deploy process down. + * + * Needs no real Postgres: the stub speaks just enough of the startup protocol + * to make pg's `connect()` resolve, which is what puts the client past + * `_connecting` and into the path that emits on the client. + */ +describe('a cold upstream that drops the socket after connecting', () => { + let server: net.Server; + let url: string; + + beforeAll(async () => { + server = net.createServer((socket) => { + socket.on('error', () => {}); + let startupAnswered = false; + socket.on('data', (buf) => { + // SSLRequest → refuse, so the startup continues in plaintext. + if (buf.length >= 8 && buf.readInt32BE(4) === 80877103) { + socket.write(Buffer.from('N')); + return; + } + // Never answer `select 1` — answering it would complete the query and + // the warm would succeed before the drop lands. + if (startupAnswered) return; + startupAnswered = true; + const authOk = Buffer.alloc(9); + authOk.write('R', 0, 'ascii'); + authOk.writeInt32BE(8, 1); + authOk.writeInt32BE(0, 5); + const readyForQuery = Buffer.alloc(6); + readyForQuery.write('Z', 0, 'ascii'); + readyForQuery.writeInt32BE(5, 1); + readyForQuery.write('I', 5, 'ascii'); + socket.write(Buffer.concat([authOk, readyForQuery])); + setTimeout(() => socket.destroy(), 20); + }); + }); + await new Promise((resolve) => server.listen(0, '127.0.0.1', resolve)); + const { port } = server.address() as net.AddressInfo; + url = `postgres://warm:warm@127.0.0.1:${port}/warm?sslmode=disable`; + }); + afterAll(() => server.close()); + + test('surfaces as a rejection instead of killing the process', async () => { + // attempts: 1 — the retry itself is proven in pg-connection.test.ts; what + // this asserts is that the drop comes back as a rejected promise at all. + await expect(warmDatabase(url, { attempts: 1 })).rejects.toThrow(); + }); +}); diff --git a/packages/1-prisma-cloud/1-extensions/target/src/pg-warm-resource.ts b/packages/1-prisma-cloud/1-extensions/target/src/pg-warm-resource.ts index 0d471c4fa..e355fa07e 100644 --- a/packages/1-prisma-cloud/1-extensions/target/src/pg-warm-resource.ts +++ b/packages/1-prisma-cloud/1-extensions/target/src/pg-warm-resource.ts @@ -36,17 +36,30 @@ export type PgWarm = Resource<'PrismaCloud.PgWarm', PgWarmProps, PgWarmAttribute /** The `PgWarm` resource constructor — `yield* PgWarm(id, { url })` in a lowering. */ export const PgWarm = Resource('PrismaCloud.PgWarm'); -/** Connect (retrying the cold-start) and run `select 1`, then release the connection. */ -async function warmDatabase(url: string): Promise { +/** + * Connect (retrying the cold-start) and run `select 1`, then release the + * connection. Exported so tests can drive it directly; `retry` overrides the + * bounded retry's defaults. + */ +export async function warmDatabase( + url: string, + retry: { readonly attempts?: number; readonly delayMs?: number } = {}, +): Promise { await withConnectionRetry(async () => { const client = new pg.Client({ connectionString: normalizeSslMode(url) }); + // A cold upstream can accept the connect and then drop the socket (FT-5226's + // second shape). pg reports that by emitting 'error' on the client, which is + // an uncaught exception with no listener — raised outside this promise, so + // the retry above never sees it and it would kill the deploy instead. The + // query/end rejection is what the retry acts on. + client.on('error', (error) => console.error('pg warm client socket error', error)); await client.connect(); try { await client.query('select 1'); } finally { await client.end(); } - }); + }, retry); } /** From 544ac8482229a8bcd5d6903dae6d1b01424f146c Mon Sep 17 00:00:00 2001 From: willbot Date: Sun, 9 Aug 2026 16:30:10 +0200 Subject: [PATCH 4/8] docs(gotchas): note the unguarded pg client in the prisma-next control driver MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Wrapping connect and the operation in `withConnectionRetry` is necessary but not sufficient — every client also needs an `'error'` listener, because an unhandled `'error'` event is raised outside the promise. `@prisma-next/driver-postgres` 0.16.0 has neither, so a socket drop during a deploy-time migration kills the process. Signed-off-by: willbot Signed-off-by: Will Madden --- gotchas.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/gotchas.md b/gotchas.md index 06699e491..6e12bf98b 100644 --- a/gotchas.md +++ b/gotchas.md @@ -337,7 +337,8 @@ await withConnectionRetry(() => client.dbInit(...), { attempts: 12, delayMs: 500 **References.** - Upstream: [FT-5226](https://linear.app/prisma-company/issue/FT-5226/first-connection-to-a-freshly-provisioned-postgres-is-rejected-while) -- Workaround source: [`packages/1-prisma-cloud/1-extensions/target/src/pg-connection.ts`](packages/1-prisma-cloud/1-extensions/target/src/pg-connection.ts) (`withConnectionRetry`), used by [`prisma-next-migrate.ts`](packages/1-prisma-cloud/1-extensions/target/src/prisma-next-migrate.ts) and [`pg-warm-resource.ts`](packages/1-prisma-cloud/1-extensions/target/src/pg-warm-resource.ts). Both wrap connect **and** the operation, which is what makes them proof against the socket-drop presentation — wrapping the connect alone would not be. +- Workaround source: [`packages/1-prisma-cloud/1-extensions/target/src/pg-connection.ts`](packages/1-prisma-cloud/1-extensions/target/src/pg-connection.ts) (`withConnectionRetry`), used by [`prisma-next-migrate.ts`](packages/1-prisma-cloud/1-extensions/target/src/prisma-next-migrate.ts) and [`pg-warm-resource.ts`](packages/1-prisma-cloud/1-extensions/target/src/pg-warm-resource.ts). Both wrap connect **and** the operation, which is what makes them proof against the socket-drop presentation — wrapping the connect alone would not be. Wrapping is necessary but not sufficient: every client also needs an `'error'` listener, because an unhandled `'error'` event is raised outside the promise and `withConnectionRetry` cannot catch it. +- Still exposed: `@prisma-next/driver-postgres` 0.16.0 builds the control client used for deploy-time migrations (`createPostgresControlClient` → `PostgresControlDriver`) around a long-lived `pg.Client` with no `'error'` listener anywhere in the package (`dist/control.mjs:32`). A socket drop there takes the deploy process down instead of failing the migration. Not fixable here — it needs an upstream fix or a process-level guard. - Removal guard: the CI canary (`scripts/cold-connect-canary.ts`, "Cold-connect canary" E2E job) fails only when every cold connect in a run of at least 14 succeeds — the rejection is intermittent, so a shorter unanimous streak is treated as luck, not a fix (same reasoning as PRO-217's 14-hold requirement below) — when the platform fixes FT-5226 it goes red, forcing removal of `withConnectionRetry` and itself (an inconclusive run passes with a warning annotation instead of blocking). Exit 1 means that verdict and nothing else, because it is read as an instruction to delete working production code: a canary that cannot provision, a teardown that fails, and a stray async error all exit 0 with the reason logged. The stray-error case is not hypothetical — an unhandled `'error'` event from the canary's own `pg` client (the same defect as the idle-close entry below) made bun exit 1 a second *after* the correct "bug still present" verdict had printed, which reads in the log as a crash during project teardown - Related: [FT-5219](https://linear.app/prisma-company/issue/FT-5219) (idle-close, runtime), [PRO-212](https://linear.app/prisma-company/issue/PRO-212) (nested endpoint DSNs) From 7e8bb85f0b31bb00aea8161e65ff3e42367b335f Mon Sep 17 00:00:00 2001 From: willbot Date: Sun, 9 Aug 2026 16:41:29 +0200 Subject: [PATCH 5/8] docs(gotchas): the control-driver gap is fixed upstream, we are pinned before it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Corrects the previous note, which said the unguarded control client needed an upstream fix. It already has one — prisma/prisma 0e51f1f4d, 2026-07-22 — but `@prisma-next/driver-postgres@0.16.0` shipped 2026-07-21 and is the last standalone release, so the fix only reaches us with the Prisma 8 upgrade. Signed-off-by: willbot Signed-off-by: Will Madden --- gotchas.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/gotchas.md b/gotchas.md index 6e12bf98b..a887e388b 100644 --- a/gotchas.md +++ b/gotchas.md @@ -338,7 +338,7 @@ await withConnectionRetry(() => client.dbInit(...), { attempts: 12, delayMs: 500 - Upstream: [FT-5226](https://linear.app/prisma-company/issue/FT-5226/first-connection-to-a-freshly-provisioned-postgres-is-rejected-while) - Workaround source: [`packages/1-prisma-cloud/1-extensions/target/src/pg-connection.ts`](packages/1-prisma-cloud/1-extensions/target/src/pg-connection.ts) (`withConnectionRetry`), used by [`prisma-next-migrate.ts`](packages/1-prisma-cloud/1-extensions/target/src/prisma-next-migrate.ts) and [`pg-warm-resource.ts`](packages/1-prisma-cloud/1-extensions/target/src/pg-warm-resource.ts). Both wrap connect **and** the operation, which is what makes them proof against the socket-drop presentation — wrapping the connect alone would not be. Wrapping is necessary but not sufficient: every client also needs an `'error'` listener, because an unhandled `'error'` event is raised outside the promise and `withConnectionRetry` cannot catch it. -- Still exposed: `@prisma-next/driver-postgres` 0.16.0 builds the control client used for deploy-time migrations (`createPostgresControlClient` → `PostgresControlDriver`) around a long-lived `pg.Client` with no `'error'` listener anywhere in the package (`dist/control.mjs:32`). A socket drop there takes the deploy process down instead of failing the migration. Not fixable here — it needs an upstream fix or a process-level guard. +- Still exposed: `@prisma-next/driver-postgres` 0.16.0 builds the control client used for deploy-time migrations (`createPostgresControlClient` → `PostgresControlDriver`) around a long-lived `pg.Client` with no `'error'` listener anywhere in the package (`dist/control.mjs:32`). A socket drop there takes the deploy process down instead of failing the migration. Fixed upstream — [`prisma/prisma@0e51f1f4d`](https://github.com/prisma/prisma/commit/0e51f1f4d) (2026-07-22) adds the listener and makes `close()` tolerate a dropped socket — but 0.16.0 was published 2026-07-21, one day earlier, and it is the last standalone release: the package is now `@internal/driver-postgres`, private, at `8.0.0-rc.1`, so the fix reaches us only with the Prisma 8 (0.16 → 0.17) upgrade. Until then the options are a `pnpm patch` (precedent: `patches/alchemy@2.0.0-beta.67.patch`) or a process-level `uncaughtException` guard in the deploy path. Exposure is limited in practice because `PgWarm` absorbs the cold-start window ahead of the migration. - Removal guard: the CI canary (`scripts/cold-connect-canary.ts`, "Cold-connect canary" E2E job) fails only when every cold connect in a run of at least 14 succeeds — the rejection is intermittent, so a shorter unanimous streak is treated as luck, not a fix (same reasoning as PRO-217's 14-hold requirement below) — when the platform fixes FT-5226 it goes red, forcing removal of `withConnectionRetry` and itself (an inconclusive run passes with a warning annotation instead of blocking). Exit 1 means that verdict and nothing else, because it is read as an instruction to delete working production code: a canary that cannot provision, a teardown that fails, and a stray async error all exit 0 with the reason logged. The stray-error case is not hypothetical — an unhandled `'error'` event from the canary's own `pg` client (the same defect as the idle-close entry below) made bun exit 1 a second *after* the correct "bug still present" verdict had printed, which reads in the log as a crash during project teardown - Related: [FT-5219](https://linear.app/prisma-company/issue/FT-5219) (idle-close, runtime), [PRO-212](https://linear.app/prisma-company/issue/PRO-212) (nested endpoint DSNs) From c92095dc2e127b698b0bf680d7c041fd3b92601c Mon Sep 17 00:00:00 2001 From: willbot Date: Sun, 9 Aug 2026 16:53:44 +0200 Subject: [PATCH 6/8] docs(gotchas): there is no Prisma 8 release to upgrade to yet MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The previous note implied the upgrade was merely large. It is unavailable: `@prisma-next/*` stops at 0.16.0 and npm has no 8.x `prisma` at all, not even a release candidate. Also drops `pnpm patch` from the stopgaps — we are not patching dependencies. Signed-off-by: willbot Signed-off-by: Will Madden --- gotchas.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/gotchas.md b/gotchas.md index a887e388b..fcb0dac47 100644 --- a/gotchas.md +++ b/gotchas.md @@ -338,7 +338,7 @@ await withConnectionRetry(() => client.dbInit(...), { attempts: 12, delayMs: 500 - Upstream: [FT-5226](https://linear.app/prisma-company/issue/FT-5226/first-connection-to-a-freshly-provisioned-postgres-is-rejected-while) - Workaround source: [`packages/1-prisma-cloud/1-extensions/target/src/pg-connection.ts`](packages/1-prisma-cloud/1-extensions/target/src/pg-connection.ts) (`withConnectionRetry`), used by [`prisma-next-migrate.ts`](packages/1-prisma-cloud/1-extensions/target/src/prisma-next-migrate.ts) and [`pg-warm-resource.ts`](packages/1-prisma-cloud/1-extensions/target/src/pg-warm-resource.ts). Both wrap connect **and** the operation, which is what makes them proof against the socket-drop presentation — wrapping the connect alone would not be. Wrapping is necessary but not sufficient: every client also needs an `'error'` listener, because an unhandled `'error'` event is raised outside the promise and `withConnectionRetry` cannot catch it. -- Still exposed: `@prisma-next/driver-postgres` 0.16.0 builds the control client used for deploy-time migrations (`createPostgresControlClient` → `PostgresControlDriver`) around a long-lived `pg.Client` with no `'error'` listener anywhere in the package (`dist/control.mjs:32`). A socket drop there takes the deploy process down instead of failing the migration. Fixed upstream — [`prisma/prisma@0e51f1f4d`](https://github.com/prisma/prisma/commit/0e51f1f4d) (2026-07-22) adds the listener and makes `close()` tolerate a dropped socket — but 0.16.0 was published 2026-07-21, one day earlier, and it is the last standalone release: the package is now `@internal/driver-postgres`, private, at `8.0.0-rc.1`, so the fix reaches us only with the Prisma 8 (0.16 → 0.17) upgrade. Until then the options are a `pnpm patch` (precedent: `patches/alchemy@2.0.0-beta.67.patch`) or a process-level `uncaughtException` guard in the deploy path. Exposure is limited in practice because `PgWarm` absorbs the cold-start window ahead of the migration. +- Still exposed: `@prisma-next/driver-postgres` 0.16.0 builds the control client used for deploy-time migrations (`createPostgresControlClient` → `PostgresControlDriver`) around a long-lived `pg.Client` with no `'error'` listener anywhere in the package (`dist/control.mjs:32`). A socket drop there takes the deploy process down instead of failing the migration. Fixed upstream — [`prisma/prisma@0e51f1f4d`](https://github.com/prisma/prisma/commit/0e51f1f4d) (2026-07-22) adds the listener and makes `close()` tolerate a dropped socket — but 0.16.0 was published 2026-07-21, one day earlier, and it is the last standalone release: the package is now `@internal/driver-postgres`, private, at `8.0.0-rc.1`, so the fix reaches us only with the Prisma 8 upgrade — and there is nothing to upgrade to yet: `@prisma-next/*` stops at 0.16.0 and no 8.x `prisma` has been published, not even a release candidate. Until one exists the only local option is a targeted `uncaughtException` guard in the deploy path matching `isTransientConnectionError`. Exposure is limited in practice because `PgWarm` absorbs the cold-start window ahead of the migration, so the residual risk is a drop mid-migration. - Removal guard: the CI canary (`scripts/cold-connect-canary.ts`, "Cold-connect canary" E2E job) fails only when every cold connect in a run of at least 14 succeeds — the rejection is intermittent, so a shorter unanimous streak is treated as luck, not a fix (same reasoning as PRO-217's 14-hold requirement below) — when the platform fixes FT-5226 it goes red, forcing removal of `withConnectionRetry` and itself (an inconclusive run passes with a warning annotation instead of blocking). Exit 1 means that verdict and nothing else, because it is read as an instruction to delete working production code: a canary that cannot provision, a teardown that fails, and a stray async error all exit 0 with the reason logged. The stray-error case is not hypothetical — an unhandled `'error'` event from the canary's own `pg` client (the same defect as the idle-close entry below) made bun exit 1 a second *after* the correct "bug still present" verdict had printed, which reads in the log as a crash during project teardown - Related: [FT-5219](https://linear.app/prisma-company/issue/FT-5219) (idle-close, runtime), [PRO-212](https://linear.app/prisma-company/issue/PRO-212) (nested endpoint DSNs) From bfd7aaf0c2958a1bc6d6995bdb2f92629917e5a2 Mon Sep 17 00:00:00 2001 From: willbot Date: Sun, 9 Aug 2026 17:59:30 +0200 Subject: [PATCH 7/8] docs(gotchas): the control-driver fix ships in 8.0.0-rc.1, which we are adopting My previous two notes were both wrong. The packages were renamed and regrouped: the driver is published as `@prisma/orm-target-postgres@8.0.0-rc.1`, not under `@prisma-next/*` or `prisma`, which is why searching those three scopes found nothing. Verified the listener is in the published tarball at `dist/control-6WFTtLAM.mjs:33`. Composer adopts it in the Prisma 8 upgrade, so no local workaround is needed. Signed-off-by: willbot Signed-off-by: Will Madden --- gotchas.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/gotchas.md b/gotchas.md index fcb0dac47..9331329c3 100644 --- a/gotchas.md +++ b/gotchas.md @@ -338,7 +338,7 @@ await withConnectionRetry(() => client.dbInit(...), { attempts: 12, delayMs: 500 - Upstream: [FT-5226](https://linear.app/prisma-company/issue/FT-5226/first-connection-to-a-freshly-provisioned-postgres-is-rejected-while) - Workaround source: [`packages/1-prisma-cloud/1-extensions/target/src/pg-connection.ts`](packages/1-prisma-cloud/1-extensions/target/src/pg-connection.ts) (`withConnectionRetry`), used by [`prisma-next-migrate.ts`](packages/1-prisma-cloud/1-extensions/target/src/prisma-next-migrate.ts) and [`pg-warm-resource.ts`](packages/1-prisma-cloud/1-extensions/target/src/pg-warm-resource.ts). Both wrap connect **and** the operation, which is what makes them proof against the socket-drop presentation — wrapping the connect alone would not be. Wrapping is necessary but not sufficient: every client also needs an `'error'` listener, because an unhandled `'error'` event is raised outside the promise and `withConnectionRetry` cannot catch it. -- Still exposed: `@prisma-next/driver-postgres` 0.16.0 builds the control client used for deploy-time migrations (`createPostgresControlClient` → `PostgresControlDriver`) around a long-lived `pg.Client` with no `'error'` listener anywhere in the package (`dist/control.mjs:32`). A socket drop there takes the deploy process down instead of failing the migration. Fixed upstream — [`prisma/prisma@0e51f1f4d`](https://github.com/prisma/prisma/commit/0e51f1f4d) (2026-07-22) adds the listener and makes `close()` tolerate a dropped socket — but 0.16.0 was published 2026-07-21, one day earlier, and it is the last standalone release: the package is now `@internal/driver-postgres`, private, at `8.0.0-rc.1`, so the fix reaches us only with the Prisma 8 upgrade — and there is nothing to upgrade to yet: `@prisma-next/*` stops at 0.16.0 and no 8.x `prisma` has been published, not even a release candidate. Until one exists the only local option is a targeted `uncaughtException` guard in the deploy path matching `isTransientConnectionError`. Exposure is limited in practice because `PgWarm` absorbs the cold-start window ahead of the migration, so the residual risk is a drop mid-migration. +- The control client used for deploy-time migrations (`createPostgresControlClient` → `PostgresControlDriver`) has the same trap, and it bites on the dependency rather than on our code. `@prisma-next/driver-postgres@0.16.0` builds it around a long-lived `pg.Client` with no `'error'` listener (`dist/control.mjs:32`), so a socket drop there takes the deploy process down instead of failing the migration. Fixed upstream in [`prisma/prisma@0e51f1f4d`](https://github.com/prisma/prisma/commit/0e51f1f4d) (2026-07-22), which adds the listener and makes `close()` tolerate a dropped socket. 0.16.0 was published one day earlier and never got it; the fix ships in `@prisma/orm-target-postgres@8.0.0-rc.1` (`client.on("error", () => {})` at `dist/control-6WFTtLAM.mjs:33`, verified in the published tarball), which Composer adopts in the Prisma 8 upgrade. Nothing to do here once that lands — and no dependency patch was ever warranted. - Removal guard: the CI canary (`scripts/cold-connect-canary.ts`, "Cold-connect canary" E2E job) fails only when every cold connect in a run of at least 14 succeeds — the rejection is intermittent, so a shorter unanimous streak is treated as luck, not a fix (same reasoning as PRO-217's 14-hold requirement below) — when the platform fixes FT-5226 it goes red, forcing removal of `withConnectionRetry` and itself (an inconclusive run passes with a warning annotation instead of blocking). Exit 1 means that verdict and nothing else, because it is read as an instruction to delete working production code: a canary that cannot provision, a teardown that fails, and a stray async error all exit 0 with the reason logged. The stray-error case is not hypothetical — an unhandled `'error'` event from the canary's own `pg` client (the same defect as the idle-close entry below) made bun exit 1 a second *after* the correct "bug still present" verdict had printed, which reads in the log as a crash during project teardown - Related: [FT-5219](https://linear.app/prisma-company/issue/FT-5219) (idle-close, runtime), [PRO-212](https://linear.app/prisma-company/issue/PRO-212) (nested endpoint DSNs) From 8d899c2aa96619ff13a30b18fb6c593a35a83f9e Mon Sep 17 00:00:00 2001 From: willbot Date: Sun, 9 Aug 2026 18:26:12 +0200 Subject: [PATCH 8/8] fix(test): narrow the stub socket chunk to a Buffer A `data` chunk is typed `string | Buffer`, so `readInt32BE` does not typecheck on it. We never set an encoding, but narrowing with `Buffer.isBuffer` is honest about it and keeps `test:types` green. Signed-off-by: willbot Signed-off-by: Will Madden --- .../1-extensions/target/src/__tests__/pg-warm-resource.test.ts | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/packages/1-prisma-cloud/1-extensions/target/src/__tests__/pg-warm-resource.test.ts b/packages/1-prisma-cloud/1-extensions/target/src/__tests__/pg-warm-resource.test.ts index 46a9a0bf5..1125cbff8 100644 --- a/packages/1-prisma-cloud/1-extensions/target/src/__tests__/pg-warm-resource.test.ts +++ b/packages/1-prisma-cloud/1-extensions/target/src/__tests__/pg-warm-resource.test.ts @@ -83,7 +83,8 @@ describe('a cold upstream that drops the socket after connecting', () => { server = net.createServer((socket) => { socket.on('error', () => {}); let startupAnswered = false; - socket.on('data', (buf) => { + socket.on('data', (chunk) => { + const buf = Buffer.isBuffer(chunk) ? chunk : Buffer.from(chunk); // SSLRequest → refuse, so the startup continues in plaintext. if (buf.length >= 8 && buf.readInt32BE(4) === 80877103) { socket.write(Buffer.from('N'));