feat: add PostgreSQL persistence backend - #19
Open
jbuchbinder wants to merge 38 commits into
Open
Conversation
Add Persistence.Postgres project implementing IDbContext via Npgsql + Dapper. C# Implementation: - DbContext.cs: Full IDbContext with PostgresCompiler, JSONB operators, PostgresException mapping (23505/23503/23502), connection pooling with TransactionScope sharing, multi-row batch INSERT SQL (ported from MySQL backend): - 7 PL/pgSQL functions (BuildAttributeQuery, GetAttributeNames, GetDateAttributeValue, GetLocalizedValue, GetNumericAttributeValue, GetTextAttributeValue, IsMatchingCluster) - 44 PL/pgSQL stored procedures (all CRUD + derived/consolidated queries) Integration: - DbProvider="postgres" in Startup.cs - Solution + Server.csproj project references
Add Create-Model.pg.sql — ported from MySQL schema with: - 65 tables (AUTO_INCREMENT → GENERATED ALWAYS AS IDENTITY) - 64 indexes - 7 PL/pgSQL functions (inlined from Persistence.Postgres/Functions/) - 44 PL/pgSQL stored procedures (inlined from Persistence.Postgres/StoredProcedures/) - Reserved word quoting (Case, User, Binary, Key, Order, Schema) - MySQL → PG type mappings (TINYINT(1)→BOOLEAN, DATETIME(6)→TIMESTAMP(6), etc.) Update Dockerfile: - Add Persistence.Postgres csproj copy for multi-stage build - Copy Create-Model.pg.sql and Create-Model.mysql.sql into /sql/ Update Database/README.md with PostgreSQL section.
…ures - Add End and Limit to PG reserved word set for DDL quoting - Fix 5 SP files: quote bare End column references as "End" - Regenerate Create-Model.pg.sql with corrected quoting
PostgreSQL procedures cannot use RETURN QUERY (function-only construct). Replace with bare SELECT/EXECUTE, which is supported in PG 14+ procedures as the last statement to return result sets. Fixes 24 stored procedures: - GetCollectorResults, GetCollectorCustomResults - GetWageTypeResults, GetWageTypeCustomResults - GetConsolidated* (4 files) - GetDerived* (12 files) - GetEmployeeCaseValuesByTenant, GetLookupRangeValue - GetConsolidatedPayrunResults Regenerated Create-Model.pg.sql with all fixes.
Alpine Linux uses en_US.utf8 (no hyphen, lowercase) while the DbContext compares against en_US.UTF-8 (with hyphen). Normalize by stripping hyphens before case-insensitive comparison.
PostgreSQL lowercases unquoted identifiers, so the table created by 'CREATE TABLE Version (...)' is stored as 'version'. Quoting it as "Version" in queries causes 'relation does not exist' errors.
PostgreSQL lowercases unquoted identifiers, so columns created as MajorVersion, MinorVersion, SubVersion are stored as lowercased.
SqlKata PostgresCompiler wraps table names in double quotes, requiring case-sensitive matching. Quote all 65 table names in the DDL and in all 44 stored procedures + 7 functions to ensure consistent case matching.
Table names like User were already quoted in source SP files. The bulk
replacement was adding a second set of quotes (""User"" → "User").
Fix: deduplicate first, then quote bare names only.
The DDL now quotes all table names, so Version is stored as "Version". Update DbContext version query to match.
The DDL creates columns with unquoted identifiers (folded to lowercase by PostgreSQL), but QuoteIdentifier was emitting double-quoted PascalCase names like "Identifier" which don't match the actual lowercase column names. Changed to emit lowercase qutoed identifiers like "identifier".
…tifiers PostgreSQL folds unquoted identifiers to lowercase, but the C# code emitted double-quoted PascalCase names like "Identifier" which didn't match the actual lowercase column names. Changed QuoteIdentifier to emit lowercase-quoted identifiers like "identifier" to match the DDL.
DbType.DateTime2 forces Npgsql to use timestamp without time zone, which rejects DateTime values with Kind=UTC. Removing the explicit type lets Npgsql infer timestamptz from the column metadata or DateTime Kind, fixing the 'Cannot write DateTime with Kind=UTC to PostgreSQL type timestamp without time zone' error on all writes (employee creation, tenant creation, etc).
Npgsql 9.0 defaults untyped DateTime parameters to timestamp without time zone, even when Kind=UTC. DbType.DateTimeOffset forces Npgsql to use timestamp with time zone, which matches our TIMESTAMPTZ columns.
The PostgreSQL init DDL uses unquoted identifiers which are folded to lowercase by PostgreSQL. QuoteIdentifier must emit lowercase-quoted identifiers to match.
…ifiers The DDL init file (01-Create-Model.pg.sql) now uses double-quoted PascalCase identifiers like \"Tenant\" and \"Identifier\". QuoteIdentifier must emit PascalCase-quoted identifiers to match. The earlier lowercase workaround was for the old unquoted DDL which has since been fixed. The core DateTime fix (DbType.DateTimeOffset for AddCreated/AddUpdated) is the only change needed.
Raw SQL in DbContext used lowercase unquoted column names (majorversion, minorversion, subversion) which don't match the double-quoted PascalCase DDL columns (MajorVersion, MinorVersion, SubVersion).
Implements the missing API endpoint that returns WageTypeResult[] for a completed payrun job, querying by PayrunJobId through the newly exposed IWageTypeResultRepository in PayrunJobServiceSettings.
Adds entrypoint.sh that creates all stored procedures from Persistence.Postgres/StoredProcedures/*.pg.sql before starting the .NET app. Fixes 'procedure deletepayrunjob does not exist' and all other missing procedure errors.
C# PayrollRepositoryCollectorCommand passes includeClusters and excludeClusters before collectorNames, but the PG procedure had them in the opposite order. Also added double-quoted column names to match PE's PascalCase DDL.
Replaces CommandType.StoredProcedure with CALL "proc"(@p::text, ...) syntax. Eliminates PostgreSQL 'unknown' type errors caused by NULL parameters lacking type information during function resolution. Also fixes GetDerivedCollectors parameter order (includeClusters/ excludeClusters before collectorNames to match C# code).
…umns - entrypoint.sh: retry on failure, log counts, use ON_ERROR_STOP - All stored procedures already have PG-compatible double-quoted columns
entrypoint.sh needs psql to create stored procedures on startup. The dotnet/aspnet base image doesn't include PostgreSQL client tools. Also fixed connection string parsing to handle leading whitespace from YAML >- fold operator.
PostgreSQL folds unquoted identifiers to lowercase. The C# code passes PascalCase procedure names, but since PostgreSQL procedures are created with unquoted names, CALL "ProcName" fails. Use unquoted CALL ProcName instead.
Stored procedures use TIMESTAMP(6) without timezone. The CALL type cast must match exactly for PostgreSQL to resolve the procedure.
Dapper may strip @ from parameter names in ParameterNames. Try both '@name' and 'name' when looking up parameter types to avoid defaulting to ::text for all INTEGER/DATETIME params.
- Column quoting: all PascalCase identifiers double-quoted - Boolean comparisons: SharedRegulation = true/false (not 1/0) - LANGUAGE sql for Get* procedures (returns result sets via Npgsql) - IsMatchingCluster function: TEXT params, unquoted calls - MapSpParameters: infer NpgsqlDbType from CLR value types - CommandType.StoredProcedure preserved for Npgsql result sets - PayrollLayer entries added linking Base+Federal to all payrolls
Npgsql generates CALL proc("param" := $1, ...) with double-quoted
PascalCase parameter names. Procedure parameters must be defined
with matching double-quoted names ("tenantId" not p_tenantId).
Also quoted all parameter references in procedure bodies.
Only convert Get* stored procedures to SELECT * FROM function calls. Delete*/Update* procedures must continue using CommandType.StoredProcedure to generate CALL statements, avoiding 'deletepayrunjob is a procedure' error.
…ions - Reverted all .pg.sql files to c2f8c75 (original PostgreSQL port) - Only GetDerivedCollectors/WageTypes/PayrollRegulations are RETURNS TABLE functions - All other procedures remain as LANGUAGE plpgsql (correct for command-type operations) - BuildFunctionCallSql only activates for Get* query procedures - Payrun Phase 3 completes, Phase 4 employee resolution needs SqlKata debugging
…eeDivision - EmployeeRepository.QueryAsync uses raw SQL instead of SqlKata for division queries - Added DistinctBy deduplication for employee resolver - Deleted duplicate EmployeeDivision rows causing 'Unknown employee' errors - Fixed employee identifier passing from StartPayrunJob API - BuildFunctionCallSql restricted to Get* procedures only
When the SqlKata query returns 0 employees, directly query the EmployeeRepository with the same DivisionQuery. This bypasses the SqlKata compilation issue for the resolver path.
…reSQL queries
Root cause: ToTableColumn("Employee", "*") returns "Employee.*".
SqlKata's PostgresCompiler wraps this as a single quoted identifier
"Employee.*", which PostgreSQL treats as a literal column name that
doesn't exist. The query silently returns 0 rows.
Same issue affects JOIN conditions and WHERE clauses — "Employee"."Id"
is compiled as "Employee.Id" (single identifier).
Fixed by using SelectRaw/WhereRaw/Join with properly quoted
"Table"."Column" syntax.
The SqlKata PostgresCompiler wraps ToTableColumn("Table", "*") output
as a single quoted identifier "Table.*" instead of "Table".*, causing
all employee division queries to return 0 rows.
Bypassed entirely with raw SQL SELECT that produces correct "Employee".* output.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add Persistence.Postgres project implementing IDbContext via Npgsql + Dapper.
C# Implementation:
SQL (ported from MySQL backend):
Integration:
Description
Implements PostgreSQL support for Payroll Engine.
Type of change