Initial new parser implementation

build/docker/Dockerfile

@@ -1,6 +1,6 @@
 # --platform tells docker to always use the host platform for the build not the target platform. Runtime container will use target platform.
 # Use .NET 10 SDK to support SQL script generation tool, targeting net10.0 in builds
-FROM --platform=$BUILDPLATFORM mcr.microsoft.com/dotnet/sdk:10.0.302-azurelinux3.0 AS build
+FROM --platform=$BUILDPLATFORM mcr.microsoft.com/dotnet/sdk:10.0.400-azurelinux3.0 AS build
 ARG TARGETARCH
 ARG FHIR_VERSION
 ARG ASSEMBLY_VER
@@ -87,7 +87,7 @@ COPY . .
 RUN dotnet publish /repo/src/Microsoft.Health.Fhir.${FHIR_VERSION}.Web/Microsoft.Health.Fhir.${FHIR_VERSION}.Web.csproj -o "/build" --no-restore -p:AssemblyVersion="${ASSEMBLY_VER}" -p:FileVersion="${ASSEMBLY_VER}" -p:Version="${ASSEMBLY_VER}" -f net10.0 -a $TARGETARCH
 
 # Implicitly uses the target platform for the runtime image.
-FROM mcr.microsoft.com/dotnet/aspnet:10.0.10-azurelinux3.0 AS runtime
+FROM mcr.microsoft.com/dotnet/aspnet:10.0.11-azurelinux3.0 AS runtime
 
 ARG FHIR_VERSION
 

docs/arch/adr-2608-sql-search-parser-remake.md

@@ -0,0 +1,55 @@
+# ADR-2608: SQL Search Parser Remake
+
+**Status**: Proposed
+**Date**: 2026-08-21
+**Feature**: SqlSearchParser
+
+## Context
+
+The FHIR server's SQL search query generation pipeline relied on an expression tree architecture where incoming search parameters were first parsed into a generic `Expression` tree, then passed through 14+ chained rewriter/visitor passes (compartment rewriting, date equality semantics, flattening, untyped reference resolution, sort rewriting, partition elimination, predicate pushdown, string overflow handling, numeric range rewriting, include seeding, and more). The final `SqlQueryGenerator` visitor then converted the fully-rewritten expression tree into parameterized SQL, optionally cached as a stored procedure via `CustomQueries`.
+
+This approach had several problems:
+
+- **Debuggability**: Tracing how a FHIR search URL became a SQL query required stepping through 14+ visitor passes, each mutating the expression tree in non-obvious ways. Intermediate states were opaque and difficult to inspect.
+- **Complexity**: Each new search feature (chained searches, reverse chains, SMART scopes, compartments) required adding or modifying rewriter passes that interacted with all other passes, creating a combinatorial explosion of edge cases.
+- **Indirection**: The expression tree abstraction was designed to be storage-agnostic, but in practice the SQL Server backend was the only consumer. The abstraction added layers of indirection without practical benefit.
+- **Performance tuning**: The generated SQL was constrained by what the visitor pattern could express. Optimizations like sharing expensive reference CTEs across multiple chain parameters were architecturally difficult to implement.
+
+## Options Considered
+
+1. **Incremental refactoring of the expression tree pipeline** — Simplify existing rewriters and improve logging *(rejected: the fundamental problem is the multi-pass visitor architecture itself; incremental fixes would not address debuggability or the indirection cost)*
+
+2. **Direct SQL generation from query parameters** — Bypass the expression tree entirely and generate CTEs directly from the parsed query parameters using type-specific SQL parsers *(viable)*
+
+3. **Replace expression tree with a SQL-specific IR** — Keep the expression parsing but introduce a SQL-specific intermediate representation before generation *(rejected: still two translation layers when one suffices; the query parameters already carry all needed information)*
+
+## Decision
+
+We chose **direct SQL generation from query parameters** (Option 2). The new `SearchParameterSqlParser` in `SqlSearchParser/` takes `QueryParams` (a dictionary of search parameter names to values) directly from `SearchOptionsFactory` and produces a raw SQL query string composed of CTEs.
+
+The new pipeline flow is:
+
+```
+HTTP Request
+  → SearchOptionsFactory (parses URL into QueryParams dictionary)
+  → SearchParameterSqlParser.ParseMultiple (generates SQL directly)
+  → CTE-based SQL query string
+  → SqlConnection.ExecuteReader
+```
+
+Each search parameter type has a dedicated parser (`DateTimeSqlParser`, `TokenSqlParser`, `ReferenceSqlParser`, `StringSqlParser`, etc.) that knows how to generate the appropriate CTE for its table. Special parsers handle cross-cutting concerns: `ChainedSqlParser` for forward chains, `ReversedChainSqlParser` for reverse chains, `CompartmentSqlParser` for compartment searches, `SmartCompartmentSqlParser` for SMART scopes, and `IncludeSqlParser` for `_include`/`_revinclude`.
+
+Key architectural features of the new approach:
+
+- **Chain grouping**: Multiple chain parameters sharing the same reference lookup are grouped via `ChainSearchGroup`, allowing the expensive reference CTE to be generated once and reused. An intersection CTE enforces AND semantics across grouped chains.
+- **Linear CTE pipeline**: Each parser appends its CTE to a `SqlQueryBuilder`, with `LastCteName` threading results forward. No multi-pass rewriting needed.
+- **Direct SQL control**: Optimizations like sort-aware paging, continuation token handling, and partition elimination are applied inline during generation rather than as separate visitor passes.
+
+## Consequences
+
+- **Debuggability is dramatically improved.** A standalone `SqlSearchDebugger` tool (in `tools/`) can show the mapping from FHIR URL to SQL query without connecting to a database. The single-pass generation makes it straightforward to trace how each parameter contributes to the final query.
+- **New search features are easier to add.** Adding SMART scope support, for example, required writing one new parser class (`SmartCompartmentSqlParser`) and a few lines in `ParseMultiple`, rather than inserting a new rewriter into a 14-pass chain.
+- **Performance optimizations are more natural.** Chain grouping with shared reference CTEs was a direct architectural addition, not a fight against the visitor pattern.
+- **The expression tree pipeline is retained but dormant.** The old `CreateDefaultSearchExpression` method and its rewriters remain in the codebase (commented out) as a fallback reference. Some expression-based validation (e.g., SMART scope type checking in `ExpressionAccessControl`) still operates on expressions built by `SearchOptionsFactory`.
+- **Storage abstraction is reduced.** The new parser is SQL Server-specific by design. If a second storage backend needed the same search semantics, it would need its own query generator rather than reusing the expression tree. In practice, this trade-off is acceptable since the Cosmos DB backend has its own query pipeline already.
+- **The old `SqlQueryGenerator`, all 14+ rewriter classes, and the `CustomQueries` stored procedure cache are no longer exercised.** These can be removed once the new parser is validated in production.

global.json

@@ -1,6 +1,6 @@
 {
     "sdk": {
-        "version": "10.0.302"
+        "version": "10.0.400"
     },
     "test": {
         "runner": "Microsoft.Testing.Platform"

nuget.config

@@ -4,7 +4,7 @@
     <add key="repositoryPath" value="packages" />
   </config>
   <packageSources>
-    <add key="nuget.org" value="https://api.nuget.org/v3/index.json" />
+    <add key="nuget.org" value="https://packagefeedproxy.microsoft.io/nuget/v3/index.json" />
     <add key="Microsoft Health OSS" value="https://microsofthealthoss.pkgs.visualstudio.com/FhirServer/_packaging/Public/nuget/v3/index.json" />
   </packageSources>
   <packageSourceMapping>

src/Microsoft.Health.Fhir.Core/Features/Definition/SearchParameterDefinitionManager.cs

@@ -386,6 +386,11 @@ private async Task LoadSearchParamsFromDataStore(CancellationToken cancellationT
                 searchOptions.UnsupportedSearchParams = new List<Tuple<string, string>>();
                 searchOptions.Expression = Expression.SearchParameter(SearchParameterInfo.ResourceTypeSearchParameter, Expression.StringEquals(FieldName.TokenCode, null, KnownResourceTypes.SearchParameter, false));
                 searchOptions.MaxItemCount = 10;
+
+                searchOptions.QueryParams = new Dictionary<string, IList<string>>();
+                searchOptions.QueryParams.Add("_count", new List<string> { "10" });
+                searchOptions.QueryParams.Add("_type", new List<string> { KnownResourceTypes.SearchParameter });
+
                 searchOptions.ResourceVersionTypes = ResourceVersionType.Latest;
                 if (continuationToken != null)
                 {

src/Microsoft.Health.Fhir.Core/Features/KnownQueryParameterNames.cs

@@ -148,5 +148,7 @@ public static class KnownQueryParameterNames
         public const string ReturnDetails = "_details";
 
         public const string MetaHistory = "_meta-history";
+
+        public const string ReverseChain = "_has";
     }
 }

src/Microsoft.Health.Fhir.Core/Features/Search/SearchOptions.cs

@@ -57,6 +57,7 @@ internal SearchOptions(SearchOptions other)
             IsAsyncOperation = other.IsAsyncOperation;
             SkipAppendIntersectionWithPredecessor = other.SkipAppendIntersectionWithPredecessor;
             ContainsIterativeInclude = other.ContainsIterativeInclude;
+            QueryParams = new Dictionary<string, IList<string>>(other.QueryParams);
         }
 
         /// <summary>
@@ -145,7 +146,7 @@ internal set
         /// <summary>
         /// Gets the collection of search parameters used for filtering and querying resources.
         /// </summary>
-        public IReadOnlyList<SearchParameterInfo> SearchParameters { get; internal set; } = new List<SearchParameterInfo>();
+        public IList<SearchParameterInfo> SearchParameters { get; internal set; } = new List<SearchParameterInfo>();
 
         /// <summary>
         /// Gets the list of search parameters that were not used in the search.
@@ -187,6 +188,10 @@ internal set
         /// </summary>
         public bool SkipAppendIntersectionWithPredecessor { get; set; }
 
+#pragma warning disable CA2227 // Collection properties should be read only
+        public IDictionary<string, IList<string>> QueryParams { get; set; }
+#pragma warning restore CA2227 // Collection properties should be read only
+
         /// <summary>
         /// Gets or sets a value indicating whether the search contains iterative includes.
         /// </summary>

src/Microsoft.Health.Fhir.Core/Properties/AssemblyInfo.cs

@@ -52,3 +52,4 @@
 [assembly: InternalsVisibleTo("Microsoft.Health.Fhir.R4.ResourceParser")]
 
 [assembly: InternalsVisibleTo("Microsoft.Health.Fhir.SqlServer.UnitTests")]
+[assembly: InternalsVisibleTo("SqlSearchDebugger")]

src/Microsoft.Health.Fhir.Shared.Core/Features/Operations/MemberMatch/MemberMatchService.cs

@@ -66,7 +66,7 @@ public async Task<ResourceElement> FindMatch(ResourceElement coverage, ResourceE
             searchOptions.MaxItemCount = 2;
             searchOptions.Sort = new List<(SearchParameterInfo, SortOrder)>();
             searchOptions.UnsupportedSearchParams = new List<Tuple<string, string>>();
-            searchOptions.Expression = CreateSearchExpression(coverage, patient);
+            searchOptions.QueryParams = BuildQueryParams(coverage, patient);
 
             SearchResult results = null;
             try
@@ -94,59 +94,41 @@ public async Task<ResourceElement> FindMatch(ResourceElement coverage, ResourceE
             return CreatePatientWithIdentity(patient, results);
         }
 
-        private ResourceElement CreatePatientWithIdentity(ResourceElement patient, SearchResult results)
+        private Dictionary<string, IList<string>> BuildQueryParams(ResourceElement coverage, ResourceElement patient)
         {
-            var searchMatchOnly = results.Results.Where(x => x.SearchEntryMode == ValueSets.SearchEntryMode.Match).ToList();
-            if (searchMatchOnly.Count > 1)
-            {
-                throw new MemberMatchMatchingException(Core.Resources.MemberMatchMultipleMatchesFound);
-            }
+            var queryParams = new Dictionary<string, IList<string>>();
 
-            if (searchMatchOnly.Count == 0)
-            {
-                throw new MemberMatchMatchingException(Core.Resources.MemberMatchNoMatchFound);
-            }
+            // Resource type filter - search for Patient resources
+            queryParams["_type"] = new List<string> { KnownResourceTypes.Patient };
 
-            var match = searchMatchOnly[0];
-            var element = _resourceDeserializer.Deserialize(match.Resource);
-            var foundPatient = element.ToPoco<Patient>();
-            var id = foundPatient.Identifier.Where(x => x.Type != null && x.Type.Coding != null && x.Type.Coding.Exists(x => x.Code == "MB")).FirstOrDefault();
-            if (id == null)
-            {
-                throw new MemberMatchMatchingException(Core.Resources.MemberMatchNoMatchFound);
-            }
-
-            var resultPatient = patient.ToPoco<Patient>();
-            var resultId = new Identifier(id.System, id.Value);
-            resultId.Type = new CodeableConcept("http://terminology.hl7.org/CodeSystem/v2-0203", "UMB", "Member Match");
-            resultPatient.Identifier.Add(resultId);
-            var result = resultPatient.ToResourceElement();
-            return result;
-        }
-
-        private MultiaryExpression CreateSearchExpression(ResourceElement coverage, ResourceElement patient)
-        {
-            IReadOnlyCollection<SearchIndexEntry> coverageValues = _searchIndexer.Extract(coverage);
+            // Add patient search parameters
             IReadOnlyCollection<SearchIndexEntry> patientValues = _searchIndexer.Extract(patient);
-            var expressions = new List<Expression>();
-            var reverseChainExpressions = new List<Expression>();
-            expressions.Add(Expression.SearchParameter(_resourceTypeSearchParameter, Expression.StringEquals(FieldName.TokenCode, null, KnownResourceTypes.Patient, false)));
             foreach (SearchIndexEntry patientValue in patientValues)
             {
                 if (IgnoreInSearch(patientValue))
                 {
                     continue;
                 }
 
-                var modifier = string.Empty;
+                var paramName = patientValue.SearchParameter.Code;
                 if (patientValue.SearchParameter.Type == ValueSets.SearchParamType.String)
                 {
-                    modifier = ":exact";
+                    paramName += ":exact";
                 }
 
-                expressions.Add(_expressionParser.Parse(new[] { KnownResourceTypes.Patient }, patientValue.SearchParameter.Code + modifier, patientValue.Value.ToString()));
+                var value = patientValue.Value.ToString();
+                if (queryParams.TryGetValue(paramName, out var existingValues))
+                {
+                    existingValues.Add(value);
+                }
+                else
+                {
+                    queryParams[paramName] = new List<string> { value };
+                }
             }
 
+            // Add coverage search parameters as reverse chain (_has:Coverage:beneficiary:<param>=<value>)
+            IReadOnlyCollection<SearchIndexEntry> coverageValues = _searchIndexer.Extract(coverage);
             foreach (var coverageValue in coverageValues)
             {
                 if (IgnoreInSearch(coverageValue))
@@ -160,26 +142,49 @@ private MultiaryExpression CreateSearchExpression(ResourceElement coverage, Reso
                     modifier = ":exact";
                 }
 
-                reverseChainExpressions.Add(_expressionParser.Parse(new[] { KnownResourceTypes.Coverage }, coverageValue.SearchParameter.Code + modifier, coverageValue.Value.ToString()));
-            }
-
-            if (reverseChainExpressions.Count != 0)
-            {
-                Expression reverseChainedExpression;
-                if (reverseChainExpressions.Count == 1)
+                var hasKey = $"_has:Coverage:beneficiary:{coverageValue.SearchParameter.Code}{modifier}";
+                var value = coverageValue.Value.ToString();
+                if (queryParams.TryGetValue(hasKey, out var existingValues))
                 {
-                    reverseChainedExpression = reverseChainExpressions[0];
+                    existingValues.Add(value);
                 }
                 else
                 {
-                    reverseChainedExpression = Expression.And(reverseChainExpressions);
+                    queryParams[hasKey] = new List<string> { value };
                 }
+            }
+
+            return queryParams;
+        }
+
+        private ResourceElement CreatePatientWithIdentity(ResourceElement patient, SearchResult results)
+        {
+            var searchMatchOnly = results.Results.Where(x => x.SearchEntryMode == ValueSets.SearchEntryMode.Match).ToList();
+            if (searchMatchOnly.Count > 1)
+            {
+                throw new MemberMatchMatchingException(Core.Resources.MemberMatchMultipleMatchesFound);
+            }
+
+            if (searchMatchOnly.Count == 0)
+            {
+                throw new MemberMatchMatchingException(Core.Resources.MemberMatchNoMatchFound);
+            }
 
-                ChainedExpression expression = Expression.Chained(new[] { KnownResourceTypes.Coverage }, _coverageBeneficiaryParameter, new[] { KnownResourceTypes.Patient }, true, reverseChainedExpression);
-                expressions.Add(expression);
+            var match = searchMatchOnly[0];
+            var element = _resourceDeserializer.Deserialize(match.Resource);
+            var foundPatient = element.ToPoco<Patient>();
+            var id = foundPatient.Identifier.Where(x => x.Type != null && x.Type.Coding != null && x.Type.Coding.Exists(x => x.Code == "MB")).FirstOrDefault();
+            if (id == null)
+            {
+                throw new MemberMatchMatchingException(Core.Resources.MemberMatchNoMatchFound);
             }
 
-            return Expression.And(expressions);
+            var resultPatient = patient.ToPoco<Patient>();
+            var resultId = new Identifier(id.System, id.Value);
+            resultId.Type = new CodeableConcept("http://terminology.hl7.org/CodeSystem/v2-0203", "UMB", "Member Match");
+            resultPatient.Identifier.Add(resultId);
+            var result = resultPatient.ToResourceElement();
+            return result;
         }
 
         private static bool IgnoreInSearch(SearchIndexEntry searchEntry) =>