Repository navigation
feat(typed-sql): add the typed SQL compiler core and dialect interface - #206
Conversation
Pure compileTypedSQL(q, dialect, facts) for statically typed SQL engines, with an unexported typedDialect interface, catalog facts and a final placeholder numbering pass that skips quoted identifiers. Every value is a bound argument and every identifier is validated and quoted, so the SQL text never depends on a value. Unsupported nodes return errors matching dal.ErrNotSupported. Not wired into the readers yet (SQL-04). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
|
[review r1 #206] independent adversarial review (Opus), head 30cccbf Round 1, independent review of head What I verified
FindingsMajor
Minor
Neither major changes the dialect interface, so SQL-03 can start against this branch while they are fixed. Both fixes are a few lines plus a test each. VERDICT: blockers=0 majors=2 minors=6 land=no |
…ect contract Review round 1 of the typed SQL compiler. - A scan-bounded or named-database source inside a join is refused with ErrNotSupported instead of being rendered as the whole local table; a lone source keeps working because the generic join restates its scan as ORDER BY and LIMIT. - ORDER BY over a grouped, constant-carrying alias nested in a larger expression is refused as HAVING already refuses it (one alias rewriter per ORDER BY item; only for statements with a GROUP BY). - dal.DocumentID() is refused until it is mapped to the primary key; dal.ID keeps naming a real column. - The argument limit is checked per statement, not per IN list. - Ordinal references compare expressions structurally with constants of the same kind and content, not by String(). - The dialect contract states the operand count/order rule and the no-literal/no-comment rule, and the compiler enforces both on every fragment; the numbering pass refuses string literals and comments. - compileTypedSQL documents exactly what the text depends on. - The value-independence proof is a helper that takes a typedDialect; a generated test covers two independently drawn constants. - Goldens for the journey statement, an aggregate argument with a constant and quote characters in qualifiers. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…t restates its scan A lone source with WithScan compiled as the whole table for every statement shape. It now compiles only for the leaf query DALgo builds (ORDER BY equal to the scan orders after alias resolution, LIMIT within the scan limit, no WHERE, aggregation or OFFSET) and otherwise returns ErrNotSupported. Also: probe divide with two different marker-carrying operands so operands with identical text cannot hide a swap; drop DALgo's grouping-validation text, which quotes the query's constants, from the compile error; broaden the property draws (whole, zero and negative floats, integers beyond 32 bits, digit-only, date-looking and empty strings) and prove seven content-typing dialects fail the proof; state each limit in the dialect contract. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…pture, and the money option An ORDER BY alias rewritten to a bare column name is read by PostgreSQL as an output column first, so when another select alias reuses that name the server sorts by the wrong column (and a scan restated through such an alias passes the scan check). typedCheckOrderByName refuses an item whose written bare name any other select-list output shares, unaliased expressions included. GROUP BY and HAVING are pinned safe by tests: GROUP BY prefers an input column and HAVING cannot see outputs. Also: refuse a query carrying the DTQL money option instead of dropping it; reword the fixed aggregation message so it does not claim which rule failed and points at dal.ValidateAggregation; correct the divide probe test and comment to the (1.0 / right * left) case; record the name-resolution assumption on typedDialect. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ure guard The guard compared the query's spelling, so a quoteIdent that folds case (the PostgreSQL FoldLower mode) wrote Total and total as one identifier and let a statement through that sorts by the wrong output. It now compares the quoted names the dialect wrote: typedSelectItem keeps the quoted output name and the quoted field name. Tests with a lower-casing fake dialect were red first. Also document what stays on the query's spelling and fail closed (alias lookup, source qualifiers, the scan comparison), and fix the divide comment. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…e row or a function of it A name that matches no column is not an error to PostgreSQL: a bare name equal to a source's identity is the whole row, and a qualified x.f is f(x). With the catalog facts for a source, a field must now be one of its columns (plain error otherwise); without them, a bare name written as the base source's identity is refused with ErrNotSupported, compared on the quoted names written. The dialect contract states both readings and that SQL-04 must pass facts for every source. A facts lookup now matches a column only when the catalog's own name is the folded query name, and a select-all is refused over a catalog column that is not in its folded form. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
|
Review by claude-opus-5-5@claude-code@wf_adde141d-f1b (head [review r6 #206] independent adversarial review (Opus), head Round 6. Land it: both round 5 findings are closed, I could not break the fix, and the required checks are green on this head. Three minors, none blocking. The remaining open reading (a qualified name on a source the facts do not know) belongs to SQL-04 and is listed under "For later tasks". What I verified
Earlier findings
FindingsMinor
For later tasks (not findings)
Answers to the author's notes
Lead's disposition of the remaining findings. Minors 1 to 3 are DEFERRED to the PostgreSQL dialect task (SQL-03), the next change to these files, reviewed independently. The items under "For later tasks" are written into the task texts of SQL-03, SQL-04 and SQL-08 as acceptance lines. VERDICT: blockers=0 majors=0 minors=3 land=yes |
SQL-03 (task `pg-sql-03-postgres-dialect`): the PostgreSQL dialect for the typed SQL compiler core (#206, v0.23.0), with golden tests, plus the three minors and the follow-ups the core's last review left for this task. Source commits: - 10616d9 feat(typed-sql): add the PostgreSQL dialect with golden tests Add the PostgreSQL typedDialect (identifier modes exact and fold to lower case, 63-byte limit on the folded name, constants typed by Go type, one catalog query for column facts) with golden files, go-sqlmock tests for the catalog query and the 100% coverage the package gates on. Close the three minors and the follow-ups the compiler core review left for this task, run the core fold tests against the real dialect, and rewrite the README section on the legacy text path. Pull request: #212 Review: #212 (comment)
What
Plan task SQL-02 (PostgreSQL launch round): the core of a dialect-parameterised compiler for statically typed SQL engines. It is a pure function with no wiring into the readers (that is SQL-04), proven by golden tests against a fake dialect. No database is needed.
New files only: the ten
typed_*.gofiles (typed_sql.go,typed_dialect.go,typed_placeholders.goand their tests).sqlite_emit.go,reader_*.goand every existing file are untouched.compileTypedSQL(q, dialect, facts) (sql, args, err)renders one SELECT: select list, FROM with INNER and LEFT joins (nested right subtrees parenthesised, same scope rules ascompileSQLRelationWithin), WHERE and HAVING (comparisons, AND/OR, IS NULL / IS NOT NULL, IN / NOT IN), GROUP BY, ORDER BY, aggregates, arithmetic, LIMIT and OFFSET.typedDialectinterface:quoteIdent,bind,limitOffset,orderItem,divide,aggregateResult,emptyIn,placeholderStyle,capabilities,catalogFacts, and a reservedwindowhook the compiler never calls.typedCatalogFacts: per source the ordered columns with type category, NOT NULL and non-deterministic-collation flags. The zero value is valid and gives correct, less optimised SQL, but it cannot check that a field is a column (see "A name is a column or it is refused"); SQL-04 must pass facts for every source it compiles.Safety properties (each has a test)
Every value is a bound argument; no byte of a value reaches the SQL text. The text depends on the shape of the query and on exactly these facts about its values, each of which chooses between fixed fragments and writes no content: a nil constant on the right of
==rendersIS NULLinstead of a marker; the length of an IN / NOT IN list sets how many markers it writes, and an empty list renders the dialect'semptyIn; a zero limit or offset leaves its clause out; whether a GROUP BY or ORDER BY expression that carries a constant equals a selected one (same shape, constants of the same kind and content) picks the select-list position or the inline bound expression; the Go type of a constant (never its content) picks the cast the dialect writes beside its marker. A property test compiles 300 random draws (hostile, empty, digit-only and date-looking strings; zero, negative and 64-bit integers; whole, zero, negative and fractional floats; times, bytes, IN lists, limit and offset) and asserts the text is byte-identical every time and the arguments match in order; a second test proves the equal-or-different-constants choice yields exactly one of two fixed texts. The proof takes a dialect, so SQL-03 runs it on the PostgreSQL dialect; seven dialects that type a marker by content (bigint for a whole float, numeric for a fraction, and so on) are each shown to fail it.Every identifier goes through
quoteIdent, then the compiler verifies the result is one well-formed quoted identifier (a faulty dialect cannot let a name escape its quotes). Identifier validation (non-empty, no NUL, valid UTF-8, length limit) never echoes the identifier.Placeholders are numbered once, over the final text, skipping quoted identifiers, and the count must equal the argument count. A statement may bind at most 65535 values in total (PostgreSQL's wire limit), counted over the whole statement, not per IN list.
The compiler enforces the dialect contract on every fragment a dialect writes. SQL-03 must obey it: a fragment holds no string literal and no comment (the numbering pass sees quoted identifiers only); an operand that carries a marker is written verbatim, exactly once, in the order it was passed (so
dividemay not swap or repeat a marker-carrying operand; the compiler also probesdividewith two different marker-carrying operands, sodividemust be a function of its operand texts alone: that catches a fragment such as(1.0 / right * left), which for two operands written the same is byte-identical to the same form with the operands in the order they were passed, so the text check cannot object, and would compute the inverse quotient);bindandlimitOffsetwrite exactly one marker per argument and every other method writes none; an error a dialect returns must not quote a value.No error message the compiler builds quotes a constant.
dal.ValidateAggregationwords its message with the query's own expressions, so its text is replaced by a fixed message. That validation rejects for more reasons than grouping (SUM(*),COUNT(DISTINCT *), DISTINCT on MIN or MAX, a wildcard in an aggregate select, an unsupported aggregate or condition), so the message does not claim which rule failed: it says the query breaks DALgo's aggregation rules and points atdal.ValidateAggregation(q), which names it.Unsupported nodes return errors matching
errors.Is(err, dal.ErrNotSupported)and no SQL: cursors, subqueries, FIRST / LAST, RIGHT / FULL / CROSS joins, non-equality ON, nested ON naming an outer alias, parented or group sources, unknown comparison and group operators, unsupported value types, Param,dal.DocumentID()(it has no column; mapping it to the primary key is not implemented;dal.ID(name, value)names a real column and compiles), aggregates the dialect cannot run natively, the DTQLmoneyoption (exact decimal results are computed by DALgo's federated engine; the statement would compute double precision), an ORDER BY name the server would read as a different select-list output, compared as written, and a bare name the server would read as the whole row (both below). Malformed queries (invalid aggregation, unknown source, bad identifier, negative limit) deliberately do not matchErrNotSupported.Source options the statement cannot honour are refused, never dropped. In a statement with joins, a scan-bounded source (
CollectionRef.WithScan) or a source in a named database is refused, because rendering it as the whole local table would join rows DALgo's generic join never sees. A lone scan-bounded source compiles only when the statement restates its scan: ORDER BY equal to the scan's orders, a LIMIT within the scan's limit, and no WHERE, GROUP BY, aggregate or OFFSET (the leaf query DALgo's generic join and federated executor send); anything else is refused and runs in the generic engine. An ORDER BY alias that resolves to a column whose name another select alias reuses is refused even when it equals the scan order, because the server would sort by the other output (the next section); that holds when the dialect folds case, where the two names differ only in case. The scan comparison itself stays on the query's spelling, so under a folding dialect the scan orderTotaland an ORDER BYTOTALare different and the statement is refused. A lone source in a named database compiles by its table name because the federated executor already routed it; SQL-04 must give the compiler the connection's own database name to check that.A bare ORDER BY name is never captured by a select-list output. PostgreSQL reads a name that stands alone in ORDER BY as an output column first, so an alias rewritten to a bare column is refused when any select-list output of that name is another expression:
SELECT Total AS a, Name AS Total ORDER BY awould sort by Name, andSELECT a AS x, COUNT(b) AS a GROUP BY a ORDER BY xwould sort by the count where DALgo sorts by the group key. An unaliased expression counts too, because the compiler names it by its text. A qualified name and a name nested in an expression are input columns and compile as before. The check compares the quoted names the dialect wrote, not the query's spelling, because the server compares what is written and a dialect'squoteIdentmay fold case (the PostgreSQL dialect's FoldLower mode writesTotalandtotalas"total"): with a lower-casingquoteIdent,SELECT Total AS a, Name AS total ORDER BY ais writtenORDER BY "total"and would sort by Name, so it is refused, whileSELECT Total AS a, TOTAL ORDER BY a, whose second output is the column itself, compiles. Alias lookup, source qualifiers and the scan comparison stay on the query's spelling, which is what DALgo reads; under a folding dialect they refuse or leave a name as the column and never pick another expression, and the written-name check catches the names that then meet. Wildcard exclusions match catalog names exactly, asdal.WildcardProjection.Excludesdefines. GROUP BY and HAVING need no refusal under any case rule: PostgreSQL reads a GROUP BY name as an input column before an output, and HAVING cannot see outputs; both are pinned by tests, for an exact and for a folding dialect, andtypedDialectrecords the assumption for the next dialect. Writing the column qualified would keep these queries native; refusing is the narrower change.A name is a column or it is refused. PostgreSQL does not fail on a name that matches no column, it reads it as something else: a bare name equal to a source's identity (its alias, else its table name) as the whole row, one composite value with every column in it (
SELECT x FROM Invoice AS x), and a qualified namex.fthat is no column ofxas the functionfapplied to the row (x.to_jsonisto_json(x);row_to_json,to_jsonb,concat,countandquote_literalread alike). The caller chooses both the alias and the field, and a field policy that checks field names only would pass both and leak every column inside the value. The compiler closes both with the catalog facts: when the facts know a field's source and list no column of that name, the statement fails with a plain error naming the column (notErrNotSupported: the table has no such column), for a bare and a qualified field, in every clause and in a join's ON; when the facts do not know the source, an unqualified field written as the base source's identity is refused withErrNotSupported, compared on the quoted names written, so a foldingquoteIdentis covered. An aliased source hides its table's own name, so only the alias counts. A qualified name without facts cannot be checked: thetypedDialectcontract says SQL-04 must pass facts for every source it compiles, each listing every column a query may name (a system column such asctidis refused unless the dialect lists it).A facts lookup matches a column only when the catalog's own name is the name the dialect writes. The lookup folded both the query's name and the catalog's names and took the first match, so with a folding dialect and a catalog that holds
Total(NOT NULL) andtotal(nullable),ORDER BY totalreadTotal's facts, dropped the NULLS clause and sorted the nullable column with NULL on the wrong side. It now compares the catalog's own name with the folded query name, and a select-all is refused when a catalog column's name is not its own folded form, because the statement could not name that column.Foldmust be the case rulequoteIdentapplies; the contract says so.Decisions a reviewer should check
dialect.limitOffset).dal.PlanAggregation(q, dialect.capabilities())gates aggregation, so a dialect that does not declare a capability can never be handed that query.AllColumnsExcept, drops the NULLS clause for NOT NULL columns not on the nullable side of an outer join, refuses a join whose key types differ (typedJoinKeysComparable), and refuses a field that is not one of its source's columns.JoinedSource.Algorithms) are not read: they are preferences and the server picks its own plan.1and1.0) compile as different expressions, which can only produce a server error, never wrong rows.Tests
Golden SQL plus arguments per clause, error table, property tests, direct unit tests of the helpers, a test that pins every place a bare name is written into ORDER BY, GROUP BY or HAVING, tests for the whole-row and function-of-the-row readings (refusals with and without facts, in SELECT, WHERE, GROUP BY, HAVING, ORDER BY and a join's ON, and the goldens that must still compile: a column named like its table or alias when the facts list it, a qualified column with facts), and the same tests against a fake dialect whose
quoteIdentlower-cases (refusals and the statements that still compile). SQL-03 should run them against its real FoldLower dialect; SQL-08 should pin the two readings against a real server, since they are stated here from PostgreSQL's documentation and parser, not from a run. 100% statement coverage of the module (CI gate),go vet,golangci-lintandstaticcheckclean.Changed in review round 5
ErrNotSupported). Tests were red first:SELECT x FROM Invoice AS x,SELECT Invoice FROM Invoice, the same names in WHERE, ORDER BY, GROUP BY, an aggregate and HAVING,SELECT X FROM Invoice AS xunder the folding fake,x.to_jsonand its relatives with facts that do not list them, and a join key the facts do not list.TestCompileTypedSQLCheckedJoinKeyTypesaccepted "a column missing from the facts is left to the server" and now rejects it, andTestCompileTypedSQLOrderBy's facts forAlbumdid not listArtistId, which its own join key reads, so they now do."total"written twice.typedDialectname-resolution contract lists the facts lookup as the third kind of lookup that does not compare written names, states the whole-row and function-of-the-row readings and that SQL-04 must pass facts for every source it compiles, and saysFoldmust be the case rulequoteIdentapplies.🤖 Generated with Claude Code