Test suite refactoring. Misc bug fixes to joins

.github/workflows/build+test.yml

@@ -1,62 +1,39 @@
 name: build+test
-on: [push,pull_request]
+
+on: [push, pull_request]
+
 jobs:
   quanta-ci-job:
     runs-on: ubuntu-latest
     steps:
-      - name: Start LocalStack
-        env:
-          LOCALSTACK_API_KEY: ${{ secrets.LOCALSTACK_API_KEY }}
-        run: |
-          pip install localstack awscli-local[ver1] # install LocalStack cli and awslocal
-          docker pull localstack/localstack         # Make sure to pull the latest version of the image
-          localstack start -d                       # Start LocalStack in the background
-
-          echo "Waiting for LocalStack startup..."  # Wait 30 seconds for the LocalStack container
-          localstack wait -t 30                     # to become ready before timing out 
-          echo "Startup complete"      
-
+      - name: Check out repository
+        uses: actions/checkout@v4
 
-      - name: Start consul
+      - name: Start Consul
         run: |
-
           wget -O- https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
           echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
-          sudo apt update && sudo apt install consul
-
+          sudo apt update
+          sudo apt install consul
           consul agent -dev &
-
-          # docker pull consul 
-          # -d 
-          # docker run -p 8500:8500  -p 8600:8600/udp -p 4000:4000 -p 4001:4001 -p 4002:4002 --name=badger consul agent -dev -client="0.0.0.0" 
-
-          echo "consul Startup complete"
-
-      - name: Run some Tests against LocalStack
-        run: |
-          awslocal s3 mb s3://test
-          awslocal s3 ls
-          echo "Test Execution complete!"  
-
-      - uses: actions/checkout@v3
+          echo "Consul startup complete"
 
       - name: Set up Go
         uses: actions/setup-go@v3
         with:
-          go-version: 1.21
+          go-version: "1.21"
 
       - name: Build
         run: |
           ls -lah
           go build -v ./...
 
       - name: Test
-        run: | 
+        run: |
           ls -lah
-          # go test -v ./...
-          ./test/run-go-tests.sh 
-      - name: Test-Integration
-        run: | 
+          ./test/run-go-tests.sh
+
+      - name: Test integration
+        run: |
           ls -lah
-          # go test -v ./...
-          ./test/run-go-integration-tests.sh 
+          ./test/run-go-integration-tests.sh

docs/ARCHITECTURE.md

@@ -0,0 +1,146 @@
+# ARCHITECTURE.md
+
+# Quanta Architecture Overview
+
+## Overview
+
+Quanta is a distributed bitmap-oriented analytical processing engine optimized for:
+
+- high-speed filtering
+- distributed joins
+- analytical aggregation
+- streaming ingestion
+- bitmap and BSI processing
+
+---
+
+# Major Components
+
+## Data Nodes
+
+Quanta data nodes:
+
+- own shards
+- store bitmap and BSI data
+- execute bitmap operations
+- participate in distributed query execution
+
+Data nodes are shard-aware and coordinated through Consul.
+
+---
+
+## Query Proxy
+
+The query proxy:
+
+- accepts SQL requests
+- performs SQL parsing/planning
+- generates logical query plans
+- coordinates distributed query execution
+
+The proxy currently exposes a MySQL-compatible interface.
+
+---
+
+## Logical Query Layer
+
+SQL queries are translated into an intermediate logical query representation before execution.
+
+This layer separates:
+
+- SQL syntax/planning
+from:
+- bitmap execution semantics
+
+---
+
+## Bitmap Execution Engine
+
+The bitmap execution engine is the core architectural focus of Quanta.
+
+Current capabilities include:
+
+- bitmap filtering
+- BSI processing
+- joins
+- aggregate operations
+- top-N analytics
+- distributed shard execution
+
+---
+
+## Consul Integration
+
+Consul is used for:
+
+- node discovery
+- cluster coordination
+- shard metadata
+- service registration
+
+---
+
+## Ingestion
+
+Quanta supports:
+
+- streaming ingestion
+- SQL INSERT
+- bulk loading
+- Parquet export/import workflows
+
+Current ingestion work is focused on:
+
+- simplified local workflows
+- TPC-H loading
+- streaming demonstrations
+
+---
+
+# Query Flow
+
+```text
+SQL
+  ↓
+SQL parser/planner
+  ↓
+Logical query representation
+  ↓
+Distributed bitmap execution
+  ↓
+Shard/node aggregation
+  ↓
+Result assembly
+```
+
+---
+
+# Quanta-in-a-Box
+
+Quanta-in-a-Box (QIAB) provides:
+
+- local 3-node cluster startup
+- local proxy startup
+- integration test environment
+- reproducible development workflows
+
+The current implementation uses:
+
+- start-local
+- Ensure_Cluster
+- in-process node startup
+
+QIAB is the preferred development and conformance environment. It is not the
+intended limit of production topology. Containerized and multi-host deployment
+requirements are documented in [`DEPLOYMENT.md`](DEPLOYMENT.md).
+
+---
+
+# Current Architectural Priorities
+
+- startup simplification
+- query correctness stabilization
+- TPC-H analytical validation
+- streaming demo workflows
+- SQL support expansion
+- ingestion cleanup