From c2fb238b68a12a7f3f0f53e3f5e4b21112d12b00 Mon Sep 17 00:00:00 2001 From: Andrei Pechkurov Date: Thu, 9 Jul 2026 17:33:21 +0300 Subject: [PATCH 1/8] Document live views (OSS and Enterprise) Adds documentation for the new live view feature: incrementally maintained window-function results over a single WAL-backed base table, queried like a regular table. New pages: - concepts/live-views.md: what live views are, how they work (refresh vs flush cadences, in-memory tier, freshness), supported window functions, anchored windows, backfill, base-table lifecycle, monitoring, limitations, tradeoffs, and Enterprise features. - query/sql/create-live-view.md: CREATE LIVE VIEW reference (FLUSH EVERY, IN MEMORY, PARTITION BY, BACKFILL, anchored windows, constraints, errors). - query/sql/drop-live-view.md: DROP LIVE VIEW reference. - configuration/live-views.md: cairo.live.view.* server settings. Updated pages: - query/functions/meta.md: live_views() catalogue function; table_type 'L'. - query/sql/show.md: SHOW CREATE LIVE VIEW. - security/rbac.md: CREATE LIVE VIEW / DROP LIVE VIEW permissions. - concepts/views.md, configuration/overview.md, operations/backup.md, sidebars.js: cross-links, index rows, and navigation. Co-Authored-By: Claude Opus 4.8 (1M context) --- documentation/concepts/live-views.md | 325 ++++++++++++++++++++ documentation/concepts/views.md | 1 + documentation/configuration/live-views.md | 105 +++++++ documentation/configuration/overview.md | 1 + documentation/operations/backup.md | 2 +- documentation/query/functions/meta.md | 61 +++- documentation/query/sql/create-live-view.md | 276 +++++++++++++++++ documentation/query/sql/drop-live-view.md | 72 +++++ documentation/query/sql/show.md | 18 ++ documentation/security/rbac.md | 2 + documentation/sidebars.js | 8 + 11 files changed, 869 insertions(+), 2 deletions(-) create mode 100644 documentation/concepts/live-views.md create mode 100644 documentation/configuration/live-views.md create mode 100644 documentation/query/sql/create-live-view.md create mode 100644 documentation/query/sql/drop-live-view.md diff --git a/documentation/concepts/live-views.md b/documentation/concepts/live-views.md new file mode 100644 index 0000000000..afb287aa64 --- /dev/null +++ b/documentation/concepts/live-views.md @@ -0,0 +1,325 @@ +--- +title: Live views +sidebar_label: Live views +description: + Live views incrementally maintain window-function results over a base table so + that running totals, moving averages, and rankings can be read like a regular + table without recomputing on every query. +--- + +A live view is a QuestDB table that stores the incrementally maintained result +of a window-function query over a single base table. As new rows arrive in the +base table, the window functions run once per new row and the output is appended +to the view. Querying the live view then scans precomputed rows instead of +reprocessing the base table on every read. + +Live views target workloads where the same window aggregate is read frequently +against high-rate ingestion: rolling VWAP, cumulative volume, running ranks, or +day-over-day comparisons that would otherwise recompute a window over millions +of rows on each query. + +:::note + +Live views are a new feature. The supported SQL surface is deliberately narrow +in this first version. See [Limitations](#limitations) for the shapes that are +rejected at creation time. + +::: + +## Live views vs materialized views + +Both feature types pre-compute a query and refresh it incrementally, but they +serve different query shapes: + +| Aspect | Live view | [Materialized view](/docs/concepts/materialized-views/) | +| ------ | --------- | --------------------- | +| Query shape | Window functions (`OVER`) | `SAMPLE BY` / time-based `GROUP BY` | +| Output cardinality | One row per base row | One row per time bucket | +| Typical use | Running totals, moving averages, rankings | OHLC bars, downsampled summaries | +| Base tables | A single WAL-backed table | One or more tables (JOINs allowed) | +| Freshness control | `FLUSH EVERY`, `IN MEMORY` | `REFRESH` strategy | + +Use a materialized view when you want to aggregate rows into time buckets. Use a +live view when you want to keep a row-per-input result of a window computation. + +## Quick example + +Given a `trades` table of incoming trades: + +```questdb-sql title="Base table" +CREATE TABLE trades ( + symbol SYMBOL, + side SYMBOL, + price DOUBLE, + amount DOUBLE, + timestamp TIMESTAMP +) TIMESTAMP(timestamp) PARTITION BY DAY WAL; +``` + +Create a live view that keeps a 300-row moving average of price per symbol: + +```questdb-sql title="Live view with a moving average" +CREATE LIVE VIEW trades_ma +FLUSH EVERY 1s +IN MEMORY 5s +AS +SELECT + timestamp, + symbol, + price, + avg(price) OVER ( + PARTITION BY symbol + ORDER BY timestamp + ROWS 300 PRECEDING + ) AS moving_avg +FROM trades; +``` + +Query it like any table: + +```questdb-sql title="Query the live view" +SELECT * FROM trades_ma +WHERE timestamp IN '$today'; +``` + +The view updates incrementally as new rows arrive in `trades`. Each new trade +produces one output row carrying its moving average. + +## How live views work + +A live view is its own WAL-backed table maintained by a background refresh +worker. The worker reads new committed rows from the base table and runs the +view's window functions over them, appending the output. Two independent +cadences govern how that output becomes visible and durable: + +- **Refresh** runs continuously. As the worker computes output rows, it appends + them to an in-memory tier. This is what keeps the view fresh. +- **Flush** runs on the `FLUSH EVERY` cadence. It persists the in-memory rows to + the live view's own WAL-backed disk tier and advances a durability checkpoint. + +Reads combine both tiers. Recent rows are served from the in-memory tier and the +older prefix from disk, so a query sees the freshest computed rows without +waiting for a flush. + +```questdb-sql title="Show the live view definition" +SHOW CREATE LIVE VIEW trades_ma; +``` + +### Freshness + +Because refresh publishes to the in-memory tier ahead of flush, a direct +`SELECT` that reads the full output rows sees data as soon as it is refreshed. +This is independent of `FLUSH EVERY`, which is a durability and +write-amplification control, not a freshness control. + +Some read shapes are served from the disk tier only and therefore trail by up to +one `FLUSH EVERY` interval: + +- Reads that project or aggregate the view's columns rather than reading full + output rows +- Reads filtered to a timestamp interval +- A live view used as the right-hand side of an [`ASOF JOIN`](/docs/query/sql/asof-join/) + +Keep `FLUSH EVERY` small (for example `1s`) so this lag stays negligible. + +:::tip + +A live view falling behind sustained ingestion stays correct but grows stale. +There is no automatic throttle. Monitor `lag_seqtxn` and `lag_micros` in +[`live_views()`](/docs/query/functions/meta/#live_views) to detect a view that +cannot keep up. + +::: + +## Supported window functions + +Live views maintain the window functions whose result can be computed +incrementally in a single forward pass over a partitioned frame: + +- **Ranking**: `row_number`, `rank`, `dense_rank` +- **Cumulative and bounded aggregates**: `sum`, `avg`, `count`, `min`, `max`, + `ksum`, `first_value`, `last_value`, `nth_value` +- **Offset**: `lag` +- **Statistics**: `variance`, `stddev`, covariance, correlation, and EWMA + +Every window function must have a `PARTITION BY` clause. Both bounded `ROWS` and +bounded `RANGE` frames are supported. + +String, `VARCHAR`, `BINARY`, `ARRAY`, and `SYMBOL` columns can appear as +pass-through output columns and as `count` arguments, but there are no +string- or array-valued window functions. + +The following shapes cannot be maintained by an append-only incremental refresh +and are rejected at creation time: + +- Multi-pass or look-ahead functions: `percent_rank`, `cume_dist`, `ntile`, + `lead` +- Window functions without `PARTITION BY` +- Unbounded frames on non-anchored windows + +## Anchored windows + +An anchored window resets its cumulative aggregate on a boundary, which is useful +for running totals that restart each day or on a period boundary. Declare it in a +named window with either the `ANCHOR DAILY` shorthand or an `ANCHOR EXPRESSION` +clause: + +```questdb-sql title="Cumulative daily volume per symbol" +CREATE LIVE VIEW trades_daily_volume +FLUSH EVERY 1s +AS +SELECT + timestamp, + symbol, + sum(amount) OVER w AS cumulative_volume +FROM trades +WINDOW w AS ( + PARTITION BY symbol + ORDER BY timestamp + ANCHOR DAILY +); +``` + +An anchored window must be partitioned, cannot use a bounded frame, and its +anchor expression must be deterministic. + +## Backfill + +By default a live view only reflects data that arrives after it is created. Rows +in the base table below the view's creation-time lower bound are not processed. + +Add the `BACKFILL` clause to materialize the base table's existing history before +the view starts live-tailing: + +```questdb-sql title="Backfill existing history" +CREATE LIVE VIEW trades_ma +FLUSH EVERY 1s +BACKFILL +AS +SELECT + timestamp, + symbol, + avg(price) OVER (PARTITION BY symbol ORDER BY timestamp ROWS 300 PRECEDING) + AS moving_avg +FROM trades; +``` + +The backfill sweep is resumable: it checkpoints its progress and continues after +a restart. + +## Base table lifecycle + +A live view is tied to a single WAL-backed base table and tracks the exact set of +base columns its query references: + +- Changes to columns the view does not reference pass through transparently and + the view keeps refreshing. +- Dropping, renaming, or changing the type of a referenced column invalidates the + view. +- Renaming, dropping, or truncating the base table invalidates the view. +- `DROP PARTITION`, `TRUNCATE`, and base TTL eviction freeze the already-emitted + rows and the view continues forward from where it was. + +An invalidated view keeps serving its existing data and reports the reason in +[`live_views()`](/docs/query/functions/meta/#live_views). It stops refreshing. + +Live views over [deduplicated](/docs/concepts/deduplication/) base tables are +supported. A keep-last `UPSERT` replacement at an earlier timestamp is reflected +in the view. A view over a deduplicated base is one `FLUSH EVERY` cycle behind +rather than sub-cycle fresh, because its refresh is coupled to base apply. + +## Monitoring + +The [`live_views()`](/docs/query/functions/meta/#live_views) function exposes the +state, refresh lag, in-memory footprint, and backfill progress of every live +view: + +```questdb-sql title="List all live views" +SELECT view_name, base_table_name, view_status, lag_seqtxn, lag_micros +FROM live_views(); +``` + +Live views also appear in [`tables()`](/docs/query/functions/meta/#tables) with +`table_type = 'L'`, and are recognized by `SHOW CREATE LIVE VIEW`, `EXPLAIN`, +`pg_class`, and `information_schema.tables`. + +## Limitations + +Live views have a deliberately narrow surface in this first version. Statements +outside it are rejected at creation time with a specific error: + +- **Single base table only.** No JOINs, subqueries, or CTEs in the view query. +- **No pre-aggregation.** `SAMPLE BY` and `GROUP BY` are not allowed between the + base table and the window functions. A view like "5-minute candles with a + rolling VWAP" must pre-aggregate upstream. +- **No live-view-on-live-view.** A live view cannot be the base of another live + view. +- **Deterministic queries only.** Non-deterministic functions such as `now()`, + `sysdate()`, `systimestamp()`, and `rnd_*()` are rejected in the projection, + the `WHERE` filter, and window-function arguments. +- **No TTL on the view.** Live-view disk growth is unbounded in this version. + Size retention on the base table instead. + +## Tradeoffs + +- **Storage grows with output.** The computed rows are stored on the live view's + disk tier in addition to the base table's rows. For wide projections or long + retention the view's footprint can exceed the base table. +- **No admission control.** A view that cannot keep up with ingestion stays + correct but stale, with no automatic throttle or drop. +- **Per-partition state for partitioned windows grows with distinct partition + cardinality.** A base table with high-cardinality partition keys (UUIDs, + session ids) holds one state entry per key seen, so native-memory use grows + over the life of the view. The `in_mem_bytes` column in + [`live_views()`](/docs/query/functions/meta/#live_views) reports this + footprint as a peak-sticky high-water mark. + +## Enterprise features + +QuestDB Enterprise adds access control, replication, and backup support for live +views. + +### Permissions + +Two dedicated permissions govern live-view DDL, modelled on the materialized-view +permissions: + +- `CREATE LIVE VIEW` is a database-level permission. +- `DROP LIVE VIEW` is checked against the target view. + +Querying a live view uses the standard table-level `SELECT` permission, since a +live view is a regular table token. See +[Role-based access control](/docs/security/rbac/) for the full permission model. + +### Replication + +A live view replicates physically like a materialized view. Its disk tier is a +regular WAL-backed table, so its rows transfer to replicas through the existing +object-store WAL path. A read-only replica never refreshes the view itself. It +reconstructs the primary's un-flushed in-memory rows in RAM so that reads on the +replica match the primary's freshness. Promoting a replica to primary resumes +refresh from the durable watermark. + +### Backup and restore + +A live view is captured by the object-store backup like a materialized view: its +table data rides the standard table path and its definition sidecars are carried +in the backup manifest. On restore, the un-flushed in-memory rows are re-derived +from the base table, which is the same bounded recompute a promote performs. + +## Related documentation + +- **SQL commands** + - [`CREATE LIVE VIEW`](/docs/query/sql/create-live-view/): Create a live view + - [`DROP LIVE VIEW`](/docs/query/sql/drop-live-view/): Remove a live view + +- **Related concepts** + - [Materialized views](/docs/concepts/materialized-views/): Incrementally + maintained `SAMPLE BY` aggregates + - [Views](/docs/concepts/views/): Virtual tables computed at query time + - [Window functions](/docs/query/functions/window-functions/overview/): The `OVER` functions a + live view maintains + +- **Configuration** + - [Live views configs](/docs/configuration/live-views/): Server configuration + options for live views diff --git a/documentation/concepts/views.md b/documentation/concepts/views.md index 0995c637c3..fdb136ae0e 100644 --- a/documentation/concepts/views.md +++ b/documentation/concepts/views.md @@ -313,6 +313,7 @@ SELECT table_name, table_type FROM tables() | `T` | Regular table | | `V` | View | | `M` | Materialized view | +| `L` | [Live view](/docs/concepts/live-views/) | ## Views vs materialized views diff --git a/documentation/configuration/live-views.md b/documentation/configuration/live-views.md new file mode 100644 index 0000000000..bfa3a247ae --- /dev/null +++ b/documentation/configuration/live-views.md @@ -0,0 +1,105 @@ +--- +title: Live views +description: Configuration settings for live views in QuestDB. +--- + +These settings control live view SQL support and the background refresh job that +maintains live views incrementally. For a conceptual overview, see +[Live views](/docs/concepts/live-views/). + +Live view refresh shares the materialized view refresh worker pool, so the +worker-pool settings under +[Materialized views](/docs/configuration/materialized-views/) +(`mat.view.refresh.worker.count`, `.affinity`, `.haltOnError`) also govern live +view refresh. There are no dedicated live-view worker-pool properties. + +## cairo.live.view.checkpoint.max.duration.micros + +- **Default**: `300000000` (5 minutes) +- **Reloadable**: no + +Time budget, in microseconds, for a single checkpoint write turn. Checkpoints let +a restart or out-of-order replay resume without rebuilding the whole view. + +## cairo.live.view.checkpoint.rows + +- **Default**: `1000000` +- **Reloadable**: no + +Number of newly flushed rows after which the refresh worker writes a head +checkpoint. Smaller values shorten restart replay at the cost of more checkpoint +writes. + +## cairo.live.view.enabled + +- **Default**: `true` +- **Reloadable**: no + +Enables or disables SQL support and the refresh job for live views. When +disabled, `CREATE LIVE VIEW` fails with `live views are disabled`. + +## cairo.live.view.flush.retry.max + +- **Default**: `5` +- **Reloadable**: no + +Maximum number of consecutive flush attempts before a view is marked invalid. A +flush persists the in-memory rows to the view's disk tier. + +## cairo.live.view.flush.retry.max.duration.micros + +- **Default**: `60000000` (60 seconds) +- **Reloadable**: no + +Maximum total time, in microseconds, spent retrying a stalled flush before the +view is marked invalid. + +## cairo.live.view.in.memory.buffer.growth.bytes + +- **Default**: `16777216` (16 MiB) +- **Reloadable**: no + +Increment by which the in-memory tier's buffer arena grows when it needs more +capacity. Accepts a size suffix such as `16M`. + +## cairo.live.view.in.memory.buffer.initial.bytes + +- **Default**: `65536` (64 KiB) +- **Reloadable**: no + +Initial size of a live view's in-memory tier buffer. Accepts a size suffix such +as `64K`. + +## cairo.live.view.in.memory.max + +- **Default**: `3600000000` (60 minutes) +- **Reloadable**: no + +Upper bound on the `IN MEMORY` retention window. A `CREATE LIVE VIEW` whose +`IN MEMORY` (or defaulted `FLUSH EVERY`) exceeds this value is rejected. + +## cairo.live.view.partition.compact.threshold + +- **Default**: `100000` +- **Reloadable**: no + +Row-count threshold at which an anchored live view compacts a partition's +per-function state. Compaction fires when a partition's anchor-map entry count +exceeds this value and the frontier has advanced. Applies only to anchored views +whose anchor is a monotone, fixed-duration-unit timestamp expression. + +## cairo.live.view.refresh.turn.max.commits + +- **Default**: `64` +- **Reloadable**: no + +Maximum number of base-table commits a refresh worker processes in a single turn +before yielding to other views. + +## cairo.live.view.refresh.turn.max.duration.micros + +- **Default**: `50000` (50 milliseconds) +- **Reloadable**: no + +Maximum wall-clock time, in microseconds, a refresh worker spends on one view per +turn before yielding. diff --git a/documentation/configuration/overview.md b/documentation/configuration/overview.md index 5d2aa018a6..f9b9d0f930 100644 --- a/documentation/configuration/overview.md +++ b/documentation/configuration/overview.md @@ -531,6 +531,7 @@ http.net.connection.sndbuf=2m | [HTTP server](/docs/configuration/http-server/) | Web Console and REST API | | | [IAM](/docs/configuration/iam/) | Identity and Access Management | ✓ | | [Ingestion (ILP/HTTP)](/docs/configuration/ingestion/) | InfluxDB Line Protocol settings | | +| [Live views](/docs/configuration/live-views/) | Live view refresh settings | | | [Logging & Metrics](/docs/configuration/logging-metrics/) | Log levels and metrics | | | [Materialized views](/docs/configuration/materialized-views/) | Materialized view refresh settings | | | [Minimal HTTP server](/docs/configuration/http-min-server/) | Health check and metrics endpoint | | diff --git a/documentation/operations/backup.md b/documentation/operations/backup.md index 5af15cacc0..e21d5b3f7d 100644 --- a/documentation/operations/backup.md +++ b/documentation/operations/backup.md @@ -362,7 +362,7 @@ primary/replica backups below). - **Database-wide only**: Backup captures the entire database. You cannot exclude tables or backup selected tables individually. Every backup includes - all user tables, materialized views, and metadata. + all user tables, materialized views, live views, and metadata. - **One backup at a time**: Only one backup can run at any given time. Starting a new backup while one is running will return an error. - **Primary and replica backups are separate**: Each QuestDB instance has its diff --git a/documentation/query/functions/meta.md b/documentation/query/functions/meta.md index ada4004dd0..869aa5263c 100644 --- a/documentation/query/functions/meta.md +++ b/documentation/query/functions/meta.md @@ -134,6 +134,65 @@ If you want to re-read metadata for all user tables, simply use an asterisk: SELECT hydrate_table_metadata('*'); ``` +## live_views + +`live_views()` returns the list of all [live views](/docs/concepts/live-views/) +in the database, along with their status, refresh lag, in-memory footprint, and +backfill progress. + +**Arguments:** + +- `live_views()` does not require arguments. + +**Return value:** + +Returns a `table` with the following columns: + +| Column | Type | Description | +| ------ | ---- | ----------- | +| `view_name` | STRING | Live view name | +| `view_table_dir_name` | STRING | View directory name on disk | +| `base_table_name` | STRING | Base table name | +| `view_sql` | STRING | Query used to maintain the view | +| `view_status` | STRING | View status: `active`, `backfilling`, or `invalid` | +| `invalidation_reason` | STRING | Message explaining why the view was marked invalid | +| `flush_every_interval` | LONG | `FLUSH EVERY` interval value | +| `flush_every_interval_unit` | STRING | `FLUSH EVERY` unit: `MILLISECOND`, `SECOND`, `MINUTE`, `HOUR`, or `DAY` | +| `in_memory_interval` | LONG | `IN MEMORY` interval value | +| `in_memory_interval_unit` | STRING | `IN MEMORY` unit: `MILLISECOND`, `SECOND`, `MINUTE`, `HOUR`, or `DAY` | +| `in_mem_bytes` | LONG | Native footprint of the in-memory tier, a peak-sticky high-water mark | +| `in_mem_rows` | LONG | Live row count held in the in-memory tier | +| `o3_rejected_count` | LONG | Count of out-of-order base commits routed through replay | +| `below_lower_bound_count` | LONG | Count of base rows dropped below the view's lower bound | +| `lag_seqtxn` | LONG | Base transactions the view is behind (`base_table_txn - last_processed_seqtxn`) | +| `lag_micros` | LONG | Time the view is behind the base table, in microseconds | +| `last_processed_seqtxn` | LONG | Last base transaction processed by the refresh worker | +| `applied_watermark` | LONG | Last base transaction durably applied to the view's disk tier | +| `lv_consumed_seqtxn` | LONG | Base WAL purge floor held by this view | +| `view_lower_bound_timestamp` | TIMESTAMP | Lower timestamp bound below which base rows are ignored | +| `writer_stall_micros` | LONG | Time a flush has been stalled waiting to write, in microseconds | +| `backfill_target_seqtxn` | LONG | Target base transaction for an in-progress backfill | +| `head_checkpoint_lv_seqtxn` | LONG | View transaction of the latest head checkpoint | +| `head_checkpoint_max_ts` | TIMESTAMP | Maximum timestamp covered by the latest head checkpoint | +| `head_checkpoint_state_bytes` | LONG | Size of the latest head checkpoint's window state | + +The `in_mem_bytes` and `in_mem_rows` columns are complementary. `in_mem_bytes` is +the peak-sticky arena footprint that does not shrink after a burst, while +`in_mem_rows` is the live row count that drops as rows age out of the `IN MEMORY` +window. Together they distinguish a view actively buffering rows from one holding +capacity retained from a past burst. + +**Examples:** + +```questdb-sql title="List all live views" +SELECT view_name, base_table_name, view_status, lag_seqtxn, lag_micros +FROM live_views(); +``` + +| view_name | base_table_name | view_status | lag_seqtxn | lag_micros | +| --------- | --------------- | ----------- | ---------- | ---------- | +| trades_ma | trades | active | 0 | 0 | + ## materialized_views `materialized_views()` returns the list of all materialized views in the @@ -618,7 +677,7 @@ Returns a `table` with the following columns: | Column | Type | Description | |--------|------|-------------| | `table_suspended` | BOOLEAN | Whether a WAL table is suspended (`false` for non-WAL tables) | -| `table_type` | CHAR | Table type: `T` (table), `M` (materialized view), `V` (view) | +| `table_type` | CHAR | Table type: `T` (table), `M` (materialized view), `V` (view), `L` (live view) | | `table_row_count` | LONG | Approximate row count at last tracked write | | `table_min_timestamp` | TIMESTAMP | Minimum timestamp of data in the table (updated on WAL merge) | | `table_max_timestamp` | TIMESTAMP | Maximum timestamp of data in the table (updated on WAL merge) | diff --git a/documentation/query/sql/create-live-view.md b/documentation/query/sql/create-live-view.md new file mode 100644 index 0000000000..3b59ebb5ba --- /dev/null +++ b/documentation/query/sql/create-live-view.md @@ -0,0 +1,276 @@ +--- +title: CREATE LIVE VIEW +sidebar_label: CREATE LIVE VIEW +description: + Documentation for the CREATE LIVE VIEW SQL keyword in QuestDB. +--- + +Creates a live view that incrementally maintains the result of a window-function +query over a single base table and can be queried like a regular table. For a +conceptual overview, see [Live views](/docs/concepts/live-views/). + +## Syntax + +```questdb-sql title="CREATE LIVE VIEW" +CREATE LIVE VIEW [ IF NOT EXISTS ] viewName +FLUSH EVERY duration +[ IN MEMORY duration ] +[ PARTITION BY ( YEAR | MONTH | WEEK | DAY | HOUR ) ] +[ BACKFILL ] +AS [ ( ] query [ ) ] +[ OWNED BY ownerName ] +``` + +Where: + +- `duration`: a single token with a unit of `ms`, `s`, `m`, `h`, or `d`, for + example `100ms`, `5s`, or `30m`. +- `query`: a `SELECT` over one WAL-backed base table whose projection contains + [window functions](/docs/query/functions/window-functions/overview/). + +`FLUSH EVERY` is required and must come first. `IN MEMORY`, `PARTITION BY`, and +`BACKFILL` are optional and may appear in any order. All view-level clauses +precede `AS`. + +## Parameters + +| Parameter | Description | +| --------- | ----------- | +| `viewName` | Name for the live view | +| `IF NOT EXISTS` | Create only if a view with this name does not already exist | +| `FLUSH EVERY` | How often computed rows are persisted to disk. Required | +| `IN MEMORY` | Window of recent rows kept in RAM for fresh reads. Defaults to `FLUSH EVERY` | +| `PARTITION BY` | Partitioning unit for the view's disk tier. Defaults to the base table's scheme | +| `BACKFILL` | Materialize the base table's existing history before live-tailing | +| `query` | A window-function `SELECT` over a single WAL-backed base table | +| `OWNED BY` | Assign ownership (Enterprise) | + +## Clauses + +### FLUSH EVERY + +`FLUSH EVERY` sets how often the view's computed rows are persisted from the +in-memory tier to the view's own WAL-backed disk tier. It controls durability and +write amplification, not read freshness: a direct `SELECT` reads the freshest +computed rows regardless of the flush cadence. + +A smaller interval persists more often, shortening crash recovery at the cost of +more write volume. A larger interval reduces write volume but lengthens recovery +and increases the staleness of the read shapes that are served from disk only +(see [Freshness](/docs/concepts/live-views/#freshness)). + +The minimum is `100ms`. The maximum is +[`cairo.live.view.in.memory.max`](/docs/configuration/live-views/#cairoliveviewinmemorymax) +(60 minutes by default), because `IN MEMORY` defaults to `FLUSH EVERY`. + +```questdb-sql +CREATE LIVE VIEW trades_ma +FLUSH EVERY 1s +AS +SELECT timestamp, symbol, + avg(price) OVER (PARTITION BY symbol ORDER BY timestamp ROWS 300 PRECEDING) + AS moving_avg +FROM trades; +``` + +### IN MEMORY + +`IN MEMORY` sets how long a window of recent output rows is retained in RAM to +serve fast, fresh reads. Reads of recent data are served from the in-memory tier +and older data from disk. It defaults to `FLUSH EVERY`. + +`IN MEMORY` must be at least `FLUSH EVERY` and at most +[`cairo.live.view.in.memory.max`](/docs/configuration/live-views/#cairoliveviewinmemorymax). + +```questdb-sql +CREATE LIVE VIEW trades_ma +FLUSH EVERY 1s +IN MEMORY 5s +AS +SELECT timestamp, symbol, + avg(price) OVER (PARTITION BY symbol ORDER BY timestamp ROWS 300 PRECEDING) + AS moving_avg +FROM trades; +``` + +### PARTITION BY + +`PARTITION BY` sets the partitioning of the view's disk tier. If omitted, the +view inherits the base table's partitioning scheme. + +```questdb-sql +CREATE LIVE VIEW trades_ma +FLUSH EVERY 1s +PARTITION BY HOUR +AS +SELECT timestamp, symbol, + avg(price) OVER (PARTITION BY symbol ORDER BY timestamp ROWS 300 PRECEDING) + AS moving_avg +FROM trades; +``` + +### BACKFILL + +By default a live view processes only the rows that arrive after it is created. +Add `BACKFILL` to materialize the base table's existing history first. The sweep +is resumable across restarts. + +```questdb-sql +CREATE LIVE VIEW trades_ma +FLUSH EVERY 1s +BACKFILL +AS +SELECT timestamp, symbol, + avg(price) OVER (PARTITION BY symbol ORDER BY timestamp ROWS 300 PRECEDING) + AS moving_avg +FROM trades; +``` + +## Anchored windows + +An anchored window resets its cumulative aggregate on a boundary. Declare it in a +named `WINDOW` with either the `ANCHOR DAILY` shorthand or an +`ANCHOR EXPRESSION` clause: + +```questdb-sql title="Cumulative daily volume, reset each day" +CREATE LIVE VIEW trades_daily_volume +FLUSH EVERY 1s +AS +SELECT timestamp, symbol, + sum(amount) OVER w AS cumulative_volume +FROM trades +WINDOW w AS (PARTITION BY symbol ORDER BY timestamp ANCHOR DAILY); +``` + +```questdb-sql title="Anchor on an arbitrary expression" +CREATE LIVE VIEW trades_hourly_volume +FLUSH EVERY 1s +AS +SELECT timestamp, symbol, + sum(amount) OVER w AS bucket_volume +FROM trades +WINDOW w AS ( + PARTITION BY symbol + ORDER BY timestamp + ANCHOR EXPRESSION timestamp_floor('1h', timestamp) +); +``` + +An anchored window must be partitioned, must `ORDER BY` the designated timestamp +ascending, cannot use a bounded frame, and its anchor expression must be +deterministic. + +## Query constraints + +The view query is validated at creation time and must: + +- Read a single WAL-backed base table that has a designated timestamp. No JOINs, + subqueries, or CTEs. +- Contain [window functions](/docs/query/functions/window-functions/overview/) that can be + maintained incrementally (see + [supported functions](/docs/concepts/live-views/#supported-window-functions)). +- Give every window function a `PARTITION BY` clause. +- Not use `SAMPLE BY`, `GROUP BY`, `ORDER BY`, or `LIMIT` in the view query. +- Not use non-deterministic functions such as `now()`, `sysdate()`, + `systimestamp()`, or `rnd_*()`. +- Not read another live view. + +## Complete example + +```questdb-sql title="Base table" +CREATE TABLE trades ( + symbol SYMBOL, + side SYMBOL, + price DOUBLE, + amount DOUBLE, + timestamp TIMESTAMP +) TIMESTAMP(timestamp) PARTITION BY DAY WAL; +``` + +```questdb-sql title="Fully specified live view" +CREATE LIVE VIEW IF NOT EXISTS trades_ma +FLUSH EVERY 1s +IN MEMORY 5s +PARTITION BY HOUR +BACKFILL +AS +SELECT + timestamp, + symbol, + price, + avg(price) OVER ( + PARTITION BY symbol + ORDER BY timestamp + ROWS 300 PRECEDING + ) AS moving_avg +FROM trades; +``` + +This creates a view that: + +- Persists computed rows to disk every second (`FLUSH EVERY 1s`) +- Keeps 5 seconds of recent rows in RAM for fresh reads (`IN MEMORY 5s`) +- Partitions its disk tier by hour (`PARTITION BY HOUR`) +- Materializes the existing history in `trades` before live-tailing (`BACKFILL`) +- Keeps a 300-row moving average of price per symbol + +## Metadata + +Query view metadata with [`live_views()`](/docs/query/functions/meta/#live_views): + +```questdb-sql +SELECT view_name, base_table_name, view_status, lag_seqtxn +FROM live_views(); +``` + +## Permissions (Enterprise) + +Creating a live view requires the database-level `CREATE LIVE VIEW` permission +and `SELECT` on the base table: + +```questdb-sql title="Grant permission to create live views" +GRANT CREATE LIVE VIEW TO user1; +``` + +```questdb-sql title="Grant SELECT on the base table" +GRANT SELECT ON trades TO user1; +``` + +When you create a live view you automatically receive all permissions on it, +including `DROP LIVE VIEW`, with the `GRANT` option. + +### OWNED BY clause + +Assign ownership to a user, group, or service account: + +```questdb-sql +CREATE GROUP analysts; +CREATE LIVE VIEW trades_ma +FLUSH EVERY 1s +AS +SELECT timestamp, symbol, + avg(price) OVER (PARTITION BY symbol ORDER BY timestamp ROWS 300 PRECEDING) + AS moving_avg +FROM trades +OWNED BY analysts; +``` + +## Errors + +| Error | Cause | +| ----- | ----- | +| `live views are disabled` | Live-view support is turned off (`cairo.live.view.enabled=false`) | +| `live view already exists` | A live view of this name exists and `IF NOT EXISTS` was not specified | +| `table or view with the requested name already exists` | The name is taken by a table, view, or materialized view | +| `live view FLUSH EVERY must be at least 100ms` | The `FLUSH EVERY` interval is below the minimum | +| `live view select must be a simple scan of a single WAL base table; joins, subqueries, GROUP BY, ORDER BY and LIMIT are not supported yet` | The view query is not a simple scan of one base table | +| `live view base table must have a designated timestamp` | The base table has no designated timestamp | +| `non-deterministic function cannot be used in materialized view` | The query uses `now()`, `rnd_*()`, or a similar non-deterministic function | +| `permission denied` | Missing required permission (Enterprise) | + +## See also + +- [Live views concept](/docs/concepts/live-views/) +- [DROP LIVE VIEW](/docs/query/sql/drop-live-view/) +- [Window functions](/docs/query/functions/window-functions/overview/) +- [live_views()](/docs/query/functions/meta/#live_views) diff --git a/documentation/query/sql/drop-live-view.md b/documentation/query/sql/drop-live-view.md new file mode 100644 index 0000000000..dd684e04b3 --- /dev/null +++ b/documentation/query/sql/drop-live-view.md @@ -0,0 +1,72 @@ +--- +title: DROP LIVE VIEW +sidebar_label: DROP LIVE VIEW +description: + Documentation for the DROP LIVE VIEW SQL keyword in QuestDB. +--- + +Permanently deletes a live view and all of its data. For a conceptual overview, +see [Live views](/docs/concepts/live-views/). + +## Syntax + +```questdb-sql title="DROP LIVE VIEW" +DROP LIVE VIEW [ IF EXISTS ] viewName +``` + +## Parameters + +| Parameter | Description | +| --------- | ----------- | +| `viewName` | Name of the live view to drop | +| `IF EXISTS` | Suppress the error if the view does not exist | + +## Examples + +```questdb-sql title="Drop a live view" +DROP LIVE VIEW trades_ma; +``` + +```questdb-sql title="Drop only if it exists (no error if missing)" +DROP LIVE VIEW IF EXISTS trades_ma; +``` + +## Behavior + +| Aspect | Description | +| ------ | ----------- | +| Permanence | Deletion is permanent and not recoverable | +| Space reclamation | Disk space is reclaimed asynchronously | +| Active queries | Existing read queries may delay space reclamation | +| Permissions | On Enterprise, the view's access-control grants are removed with it | + +:::warning + +This operation cannot be undone. The view and all of its precomputed data are +permanently deleted. + +::: + +## Permissions (Enterprise) + +Dropping a live view requires the `DROP LIVE VIEW` permission on the specific +view: + +```questdb-sql title="Grant drop permission" +GRANT DROP LIVE VIEW ON trades_ma TO user1; +``` + +The view creator automatically receives this permission with the `GRANT` option. + +## Errors + +| Error | Cause | +| ----- | ----- | +| `live view name expected` | The name refers to a table or view that is not a live view | +| `live view does not exist` | The view does not exist and `IF EXISTS` was not specified | +| `permission denied` | Missing `DROP LIVE VIEW` permission (Enterprise) | + +## See also + +- [Live views concept](/docs/concepts/live-views/) +- [CREATE LIVE VIEW](/docs/query/sql/create-live-view/) diff --git a/documentation/query/sql/show.md b/documentation/query/sql/show.md index 0d19bae2bb..8f3fe6b860 100644 --- a/documentation/query/sql/show.md +++ b/documentation/query/sql/show.md @@ -18,6 +18,7 @@ SHOW { TABLES | PARTITIONS FROM tableName | CREATE TABLE tableName | CREATE VIEW viewName + | CREATE LIVE VIEW viewName | USER [userName] | USERS | GROUPS [userName] @@ -36,6 +37,8 @@ SHOW { TABLES - `SHOW PARTITIONS` returns the partition information for the selected table. - `SHOW CREATE TABLE` returns a DDL query that allows you to recreate the table. - `SHOW CREATE VIEW` returns a DDL query that allows you to recreate a view. +- `SHOW CREATE LIVE VIEW` returns a DDL query that allows you to recreate a live + view. - `SHOW USER` shows user secret (enterprise-only) - `SHOW GROUPS` shows all groups the user belongs or all groups in the system (enterprise-only) @@ -201,6 +204,21 @@ SHOW CREATE VIEW my_view; This returns the `CREATE VIEW` statement that would recreate the view, including any `DECLARE` parameters if the view is parameterized. +### SHOW CREATE LIVE VIEW + +```questdb-sql title="retrieving live view ddl" +SHOW CREATE LIVE VIEW trades_ma; +``` + +| ddl | +| --- | +| CREATE LIVE VIEW 'trades_ma' FLUSH EVERY 1s IN MEMORY 5s PARTITION BY DAY AS (
SELECT timestamp, symbol, avg(price) OVER (PARTITION BY symbol ORDER BY timestamp ROWS 300 PRECEDING) AS moving_avg FROM trades
); | + +This returns the `CREATE LIVE VIEW` statement that would recreate the +[live view](/docs/concepts/live-views/), including its `FLUSH EVERY`, +`IN MEMORY`, `PARTITION BY`, and `BACKFILL` clauses. On QuestDB Enterprise the +output also carries an `OWNED BY` clause identifying the view's owner. + ### SHOW PARTITIONS ```questdb-sql diff --git a/documentation/security/rbac.md b/documentation/security/rbac.md index 9bb9897a0e..d92a6dd2e2 100644 --- a/documentation/security/rbac.md +++ b/documentation/security/rbac.md @@ -580,6 +580,7 @@ SELECT * FROM all_permissions(); | CANCEL ANY COPY | Database | Cancel COPY operations | | CREATE TABLE | Database | Create tables | | CREATE MATERIALIZED VIEW | Database | Create materialized views | +| CREATE LIVE VIEW | Database | Create live views | | DEDUP ENABLE | Database | Table | Enable deduplication | | DEDUP DISABLE | Database | Table | Disable deduplication | | DETACH PARTITION | Database | Table | Detach partitions | @@ -589,6 +590,7 @@ SELECT * FROM all_permissions(); | DROP PARTITION | Database | Table | Drop partitions | | DROP TABLE | Database | Table | Drop tables | | DROP MATERIALIZED VIEW | Database | Table | Drop materialized views | +| DROP LIVE VIEW | Database | Table | Drop live views | | ENABLE STORAGE POLICY | Database | Table | Enable storage policies | | INSERT | Database | Table | Insert data | | REFRESH MATERIALIZED VIEW | Database | Table | Refresh materialized views | diff --git a/documentation/sidebars.js b/documentation/sidebars.js index f4734f2e9e..e6bc0bfde5 100644 --- a/documentation/sidebars.js +++ b/documentation/sidebars.js @@ -334,6 +334,7 @@ module.exports = { id: "query/sql/acl/create-group", type: "doc", }, + "query/sql/create-live-view", "query/sql/create-mat-view", { id: "query/sql/acl/create-service-account", @@ -355,6 +356,7 @@ module.exports = { id: "query/sql/acl/drop-group", type: "doc", }, + "query/sql/drop-live-view", "query/sql/drop-mat-view", { id: "query/sql/acl/drop-service-account", @@ -533,6 +535,11 @@ module.exports = { type: "doc", label: "Materialized Views", }, + { + id: "concepts/live-views", + type: "doc", + label: "Live Views", + }, "concepts/deduplication", "concepts/ttl", "concepts/storage-policy", @@ -589,6 +596,7 @@ module.exports = { "configuration/http-server", "configuration/iam", "configuration/ingestion", + "configuration/live-views", "configuration/logging-metrics", "configuration/materialized-views", "configuration/http-min-server", From 9ec2c1bb49d3e28c7060070596df13e2fd97f294 Mon Sep 17 00:00:00 2001 From: Andrei Pechkurov Date: Thu, 9 Jul 2026 18:55:30 +0300 Subject: [PATCH 2/8] Fix live-views docs review findings Address internal-contradiction and precision issues in the live-views pages, verified against OSS PR questdb/questdb#6939: - Relabel the comparison-table "Freshness control" row to "Freshness / durability"; FLUSH EVERY is a durability knob, not a freshness one. - Make the flagship read example an unfiltered SELECT *, and note that timestamp-filtered reads trail by up to one FLUSH EVERY interval. - Remove "truncating" from the base-table invalidation bullet: TRUNCATE is freeze-and-continue (walked past like DROP PARTITION / TTL), not an invalidation. - Qualify the query constraint as a top-level ORDER BY; the window ORDER BY inside OVER(...) is required. - Narrow "all view-level clauses precede AS" to the four pre-AS clauses; note OWNED BY follows the query. - Add a note explaining the non-determinism error names "materialized view" because it reuses the shared guard (accurate server output). Co-Authored-By: Claude Opus 4.8 (1M context) --- documentation/concepts/live-views.md | 13 ++++++++----- documentation/query/sql/create-live-view.md | 15 ++++++++++++--- 2 files changed, 20 insertions(+), 8 deletions(-) diff --git a/documentation/concepts/live-views.md b/documentation/concepts/live-views.md index afb287aa64..90f562f8f8 100644 --- a/documentation/concepts/live-views.md +++ b/documentation/concepts/live-views.md @@ -37,7 +37,7 @@ serve different query shapes: | Output cardinality | One row per base row | One row per time bucket | | Typical use | Running totals, moving averages, rankings | OHLC bars, downsampled summaries | | Base tables | A single WAL-backed table | One or more tables (JOINs allowed) | -| Freshness control | `FLUSH EVERY`, `IN MEMORY` | `REFRESH` strategy | +| Freshness / durability | `FLUSH EVERY`, `IN MEMORY` | `REFRESH` strategy | Use a materialized view when you want to aggregate rows into time buckets. Use a live view when you want to keep a row-per-input result of a window computation. @@ -78,12 +78,15 @@ FROM trades; Query it like any table: ```questdb-sql title="Query the live view" -SELECT * FROM trades_ma -WHERE timestamp IN '$today'; +SELECT * FROM trades_ma; ``` The view updates incrementally as new rows arrive in `trades`. Each new trade -produces one output row carrying its moving average. +produces one output row carrying its moving average. A direct `SELECT` of the +full output rows sees data as soon as it is refreshed. Filtering a read to a +timestamp interval (for example `WHERE timestamp IN '$today'`) is served from the +disk tier and can trail by up to one `FLUSH EVERY` interval; see +[Freshness](#freshness). ## How live views work @@ -216,7 +219,7 @@ base columns its query references: the view keeps refreshing. - Dropping, renaming, or changing the type of a referenced column invalidates the view. -- Renaming, dropping, or truncating the base table invalidates the view. +- Renaming or dropping the base table invalidates the view. - `DROP PARTITION`, `TRUNCATE`, and base TTL eviction freeze the already-emitted rows and the view continues forward from where it was. diff --git a/documentation/query/sql/create-live-view.md b/documentation/query/sql/create-live-view.md index 3b59ebb5ba..a8b7bc1215 100644 --- a/documentation/query/sql/create-live-view.md +++ b/documentation/query/sql/create-live-view.md @@ -29,8 +29,8 @@ Where: [window functions](/docs/query/functions/window-functions/overview/). `FLUSH EVERY` is required and must come first. `IN MEMORY`, `PARTITION BY`, and -`BACKFILL` are optional and may appear in any order. All view-level clauses -precede `AS`. +`BACKFILL` are optional and may appear in any order. These four clauses all +precede `AS`; the optional `OWNED BY` clause follows the query. ## Parameters @@ -170,7 +170,8 @@ The view query is validated at creation time and must: maintained incrementally (see [supported functions](/docs/concepts/live-views/#supported-window-functions)). - Give every window function a `PARTITION BY` clause. -- Not use `SAMPLE BY`, `GROUP BY`, `ORDER BY`, or `LIMIT` in the view query. +- Not use `SAMPLE BY`, `GROUP BY`, a top-level `ORDER BY`, or `LIMIT` in the view + query. The `ORDER BY` inside a window's `OVER (...)` is required and allowed. - Not use non-deterministic functions such as `now()`, `sysdate()`, `systimestamp()`, or `rnd_*()`. - Not read another live view. @@ -268,6 +269,14 @@ OWNED BY analysts; | `non-deterministic function cannot be used in materialized view` | The query uses `now()`, `rnd_*()`, or a similar non-deterministic function | | `permission denied` | Missing required permission (Enterprise) | +:::note + +The non-determinism check is the same guard materialized views use, so its error +message names "materialized view" even when it is raised for a live view. The +rule and its effect are identical for both view types. + +::: + ## See also - [Live views concept](/docs/concepts/live-views/) From a9eb6fc170c8b66c7ad863c4ab95e9e3ddbea70a Mon Sep 17 00:00:00 2001 From: Andrei Pechkurov Date: Thu, 9 Jul 2026 19:59:02 +0300 Subject: [PATCH 3/8] Use EMA/VWEMA terminology in live-views supported functions QuestDB's window-function docs name this capability EMA / VWEMA (modes of avg()); "EWMA" appeared only here. Align the terminology for consistency and searchability. Co-Authored-By: Claude Opus 4.8 (1M context) --- documentation/concepts/live-views.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/documentation/concepts/live-views.md b/documentation/concepts/live-views.md index 90f562f8f8..1ebecb3b42 100644 --- a/documentation/concepts/live-views.md +++ b/documentation/concepts/live-views.md @@ -143,7 +143,7 @@ incrementally in a single forward pass over a partitioned frame: - **Cumulative and bounded aggregates**: `sum`, `avg`, `count`, `min`, `max`, `ksum`, `first_value`, `last_value`, `nth_value` - **Offset**: `lag` -- **Statistics**: `variance`, `stddev`, covariance, correlation, and EWMA +- **Statistics**: `variance`, `stddev`, covariance, correlation, EMA, and VWEMA Every window function must have a `PARTITION BY` clause. Both bounded `ROWS` and bounded `RANGE` frames are supported. From fbf502982b4aaf816bdd6ea3b1ebce081b69f095 Mon Sep 17 00:00:00 2001 From: Andrei Pechkurov Date: Mon, 13 Jul 2026 12:30:06 +0300 Subject: [PATCH 4/8] Address live view review feedback --- documentation/concepts/live-views.md | 56 ++++++++++++++++++-- documentation/concepts/materialized-views.md | 6 +++ documentation/query/functions/meta.md | 12 ++++- documentation/query/sql/create-live-view.md | 2 + 4 files changed, 69 insertions(+), 7 deletions(-) diff --git a/documentation/concepts/live-views.md b/documentation/concepts/live-views.md index 1ebecb3b42..1689981d0e 100644 --- a/documentation/concepts/live-views.md +++ b/documentation/concepts/live-views.md @@ -128,9 +128,8 @@ Keep `FLUSH EVERY` small (for example `1s`) so this lag stays negligible. :::tip A live view falling behind sustained ingestion stays correct but grows stale. -There is no automatic throttle. Monitor `lag_seqtxn` and `lag_micros` in -[`live_views()`](/docs/query/functions/meta/#live_views) to detect a view that -cannot keep up. +There is no automatic throttle. See [Monitoring](#monitoring) for how to +interpret lag and detect a view that cannot keep up. ::: @@ -225,6 +224,21 @@ base columns its query references: An invalidated view keeps serving its existing data and reports the reason in [`live_views()`](/docs/query/functions/meta/#live_views). It stops refreshing. +Invalidation is permanent: reversing the schema change does not automatically +revalidate the view, and `ALTER LIVE VIEW ... RESUME WAL` only recovers a +suspended WAL writer. + +To recover, inspect `invalidation_reason`, repair the base-table schema, and +save the definition before dropping the view: + +```questdb-sql +SHOW CREATE LIVE VIEW trades_ma; +``` + +Then drop and recreate the live view. Add `BACKFILL` to the recreated definition +if it must include history that is still present in the base table. Dropping the +invalid view removes its materialized rows, including rows whose source history +is no longer retained by the base table. Live views over [deduplicated](/docs/concepts/deduplication/) base tables are supported. A keep-last `UPSERT` replacement at an earlier timestamp is reflected @@ -242,6 +256,28 @@ SELECT view_name, base_table_name, view_status, lag_seqtxn, lag_micros FROM live_views(); ``` +`lag_seqtxn` is the number of committed base-table WAL transactions beyond the +view's durable processed watermark. It counts transactions, not rows: one +transaction may contain one row or millions. A value of `0` means that the +durable tier is caught up. A temporary non-zero value is expected between +`FLUSH EVERY` cycles, especially when ingestion produces many small commits. + +There is no universal acceptable non-zero value. Sample the metric over time and +compare it with the view's normal flush-cycle baseline. A bounded sawtooth that +returns to zero around flushes is normal. A value that stays elevated for +multiple flush intervals or keeps increasing indicates that the view cannot +keep up. + +`lag_micros` reports the elapsed time since the last successful flush. It is a +flush-activity indicator, not the timestamp difference between base and view +rows, and can continue growing while an idle view has `lag_seqtxn = 0`. + +When lag grows persistently, check `view_status` and `writer_stall_micros` for a +failed or blocked flush. Also verify CPU and I/O capacity, reduce the number or +cost of maintained views, or increase the shared +[`mat.view.refresh.worker.count`](/docs/configuration/materialized-views/) +setting. + Live views also appear in [`tables()`](/docs/query/functions/meta/#tables) with `table_type = 'L'`, and are recognized by `SHOW CREATE LIVE VIEW`, `EXPLAIN`, `pg_class`, and `information_schema.tables`. @@ -251,12 +287,22 @@ Live views also appear in [`tables()`](/docs/query/functions/meta/#tables) with Live views have a deliberately narrow surface in this first version. Statements outside it are rejected at creation time with a specific error: +| Base object | Derived object | Supported | +| ----------- | -------------- | --------- | +| Live view | Regular view | Yes | +| Live view | Materialized view | No | +| Live view | Live view | No | +| Regular view | Live view | No; a regular view is not a WAL table | +| Materialized view | Live view | Yes | + +A full rebuild of a materialized view invalidates a live view that uses it as +its base. Recreate the live view after the rebuild, following the recovery steps +in [Base table lifecycle](#base-table-lifecycle). + - **Single base table only.** No JOINs, subqueries, or CTEs in the view query. - **No pre-aggregation.** `SAMPLE BY` and `GROUP BY` are not allowed between the base table and the window functions. A view like "5-minute candles with a rolling VWAP" must pre-aggregate upstream. -- **No live-view-on-live-view.** A live view cannot be the base of another live - view. - **Deterministic queries only.** Non-deterministic functions such as `now()`, `sysdate()`, `systimestamp()`, and `rnd_*()` are rejected in the projection, the `WHERE` filter, and window-function arguments. diff --git a/documentation/concepts/materialized-views.md b/documentation/concepts/materialized-views.md index a0722c58b7..cf7ed4ac26 100644 --- a/documentation/concepts/materialized-views.md +++ b/documentation/concepts/materialized-views.md @@ -97,6 +97,10 @@ Materialized views are ideal for: - **Historical summaries**: Data that doesn't need real-time accuracy - **OHLC calculations**: Candlestick charts, time-bucketed analytics +Use a [live view](/docs/concepts/live-views/) instead when you need to +incrementally maintain a row-per-input window computation, such as a moving +average, running total, or ranking. + Use regular [views](/docs/concepts/views/) instead when: - Query execution cost is acceptable for your workload @@ -571,6 +575,8 @@ the replica's view was not fully up-to-date. - **Related Concepts** - [Views](/docs/concepts/views/): Virtual tables that compute results at query time + - [Live views](/docs/concepts/live-views/): Incrementally maintained + row-per-input window-function results - **SQL Commands** diff --git a/documentation/query/functions/meta.md b/documentation/query/functions/meta.md index 869aa5263c..412bc1370f 100644 --- a/documentation/query/functions/meta.md +++ b/documentation/query/functions/meta.md @@ -164,8 +164,8 @@ Returns a `table` with the following columns: | `in_mem_rows` | LONG | Live row count held in the in-memory tier | | `o3_rejected_count` | LONG | Count of out-of-order base commits routed through replay | | `below_lower_bound_count` | LONG | Count of base rows dropped below the view's lower bound | -| `lag_seqtxn` | LONG | Base transactions the view is behind (`base_table_txn - last_processed_seqtxn`) | -| `lag_micros` | LONG | Time the view is behind the base table, in microseconds | +| `lag_seqtxn` | LONG | Committed base WAL transactions beyond the view's durable processed watermark (`base_table_txn - last_processed_seqtxn`) | +| `lag_micros` | LONG | Microseconds since the last successful flush | | `last_processed_seqtxn` | LONG | Last base transaction processed by the refresh worker | | `applied_watermark` | LONG | Last base transaction durably applied to the view's disk tier | | `lv_consumed_seqtxn` | LONG | Base WAL purge floor held by this view | @@ -182,6 +182,14 @@ the peak-sticky arena footprint that does not shrink after a burst, while window. Together they distinguish a view actively buffering rows from one holding capacity retained from a past burst. +`lag_seqtxn` counts transactions, not rows. A value of `0` means the durable +tier is caught up; a temporary non-zero value is expected between `FLUSH EVERY` +cycles. Alert on a value that remains above its normal flush-cycle baseline or +keeps increasing across samples rather than on a universal fixed threshold. +`lag_micros` measures flush activity and may grow while an idle view has +`lag_seqtxn = 0`. For operational guidance, see +[Monitoring live views](/docs/concepts/live-views/#monitoring). + **Examples:** ```questdb-sql title="List all live views" diff --git a/documentation/query/sql/create-live-view.md b/documentation/query/sql/create-live-view.md index a8b7bc1215..08ae6e8d66 100644 --- a/documentation/query/sql/create-live-view.md +++ b/documentation/query/sql/create-live-view.md @@ -265,6 +265,8 @@ OWNED BY analysts; | `table or view with the requested name already exists` | The name is taken by a table, view, or materialized view | | `live view FLUSH EVERY must be at least 100ms` | The `FLUSH EVERY` interval is below the minimum | | `live view select must be a simple scan of a single WAL base table; joins, subqueries, GROUP BY, ORDER BY and LIMIT are not supported yet` | The view query is not a simple scan of one base table | +| `base table must be a WAL table` | The base object is a non-WAL table or a regular view | +| `live views are not allowed as base tables in V1` | The base object is another live view | | `live view base table must have a designated timestamp` | The base table has no designated timestamp | | `non-deterministic function cannot be used in materialized view` | The query uses `now()`, `rnd_*()`, or a similar non-deterministic function | | `permission denied` | Missing required permission (Enterprise) | From f880ea53354cda40bbe23df3fd97fe721fbe23f5 Mon Sep 17 00:00:00 2001 From: Andrei Pechkurov Date: Thu, 16 Jul 2026 11:51:07 +0300 Subject: [PATCH 5/8] Document live view START FROM clause --- documentation/concepts/live-views.md | 36 +++++++++------ documentation/query/functions/meta.md | 6 +-- documentation/query/sql/create-live-view.md | 50 ++++++++++++++++----- documentation/query/sql/show.md | 4 +- 4 files changed, 65 insertions(+), 31 deletions(-) diff --git a/documentation/concepts/live-views.md b/documentation/concepts/live-views.md index 1689981d0e..9f4904348c 100644 --- a/documentation/concepts/live-views.md +++ b/documentation/concepts/live-views.md @@ -62,6 +62,7 @@ Create a live view that keeps a 300-row moving average of price per symbol: CREATE LIVE VIEW trades_ma FLUSH EVERY 1s IN MEMORY 5s +START FROM NOW AS SELECT timestamp, @@ -169,6 +170,7 @@ clause: ```questdb-sql title="Cumulative daily volume per symbol" CREATE LIVE VIEW trades_daily_volume FLUSH EVERY 1s +START FROM NOW AS SELECT timestamp, @@ -185,18 +187,23 @@ WINDOW w AS ( An anchored window must be partitioned, cannot use a bounded frame, and its anchor expression must be deterministic. -## Backfill +## Start boundary and historical data -By default a live view only reflects data that arrives after it is created. Rows -in the base table below the view's creation-time lower bound are not processed. +Every live view has a mandatory `START FROM` clause that defines an event-time +lower bound for its rows: -Add the `BACKFILL` clause to materialize the base table's existing history before -the view starts live-tailing: +- `START FROM NOW` resolves to the creation time. +- `START FROM BEGINNING` includes all base-table history. +- `START FROM 'timestamp'` includes rows at or after the specified timestamp. -```questdb-sql title="Backfill existing history" +The boundary is inclusive and is evaluated against the base table's designated +timestamp, not commit time. If qualifying rows already exist when the view is +created, QuestDB seeds them before switching to continuous refresh. + +```questdb-sql title="Include all existing history" CREATE LIVE VIEW trades_ma FLUSH EVERY 1s -BACKFILL +START FROM BEGINNING AS SELECT timestamp, @@ -206,8 +213,9 @@ SELECT FROM trades; ``` -The backfill sweep is resumable: it checkpoints its progress and continues after -a restart. +The seed sweep is resumable: it checkpoints its progress and continues after a +restart. `START FROM NOW` can still seed existing future-dated rows whose +designated timestamps are at or above the resolved creation-time boundary. ## Base table lifecycle @@ -235,10 +243,10 @@ save the definition before dropping the view: SHOW CREATE LIVE VIEW trades_ma; ``` -Then drop and recreate the live view. Add `BACKFILL` to the recreated definition -if it must include history that is still present in the base table. Dropping the -invalid view removes its materialized rows, including rows whose source history -is no longer retained by the base table. +Then drop and recreate the live view. Use `START FROM BEGINNING` or an explicit +timestamp if it must include history that is still present in the base table. +Dropping the invalid view removes its materialized rows, including rows whose +source history is no longer retained by the base table. Live views over [deduplicated](/docs/concepts/deduplication/) base tables are supported. A keep-last `UPSERT` replacement at an earlier timestamp is reflected @@ -248,7 +256,7 @@ rather than sub-cycle fresh, because its refresh is coupled to base apply. ## Monitoring The [`live_views()`](/docs/query/functions/meta/#live_views) function exposes the -state, refresh lag, in-memory footprint, and backfill progress of every live +state, refresh lag, in-memory footprint, and seed progress of every live view: ```questdb-sql title="List all live views" diff --git a/documentation/query/functions/meta.md b/documentation/query/functions/meta.md index 412bc1370f..a813f6355b 100644 --- a/documentation/query/functions/meta.md +++ b/documentation/query/functions/meta.md @@ -138,7 +138,7 @@ SELECT hydrate_table_metadata('*'); `live_views()` returns the list of all [live views](/docs/concepts/live-views/) in the database, along with their status, refresh lag, in-memory footprint, and -backfill progress. +seed progress. **Arguments:** @@ -154,7 +154,7 @@ Returns a `table` with the following columns: | `view_table_dir_name` | STRING | View directory name on disk | | `base_table_name` | STRING | Base table name | | `view_sql` | STRING | Query used to maintain the view | -| `view_status` | STRING | View status: `active`, `backfilling`, or `invalid` | +| `view_status` | STRING | View status: `active`, `seeding`, or `invalid` | | `invalidation_reason` | STRING | Message explaining why the view was marked invalid | | `flush_every_interval` | LONG | `FLUSH EVERY` interval value | | `flush_every_interval_unit` | STRING | `FLUSH EVERY` unit: `MILLISECOND`, `SECOND`, `MINUTE`, `HOUR`, or `DAY` | @@ -171,7 +171,7 @@ Returns a `table` with the following columns: | `lv_consumed_seqtxn` | LONG | Base WAL purge floor held by this view | | `view_lower_bound_timestamp` | TIMESTAMP | Lower timestamp bound below which base rows are ignored | | `writer_stall_micros` | LONG | Time a flush has been stalled waiting to write, in microseconds | -| `backfill_target_seqtxn` | LONG | Target base transaction for an in-progress backfill | +| `seed_target_seqtxn` | LONG | Target base transaction for an in-progress initial seed | | `head_checkpoint_lv_seqtxn` | LONG | View transaction of the latest head checkpoint | | `head_checkpoint_max_ts` | TIMESTAMP | Maximum timestamp covered by the latest head checkpoint | | `head_checkpoint_state_bytes` | LONG | Size of the latest head checkpoint's window state | diff --git a/documentation/query/sql/create-live-view.md b/documentation/query/sql/create-live-view.md index 08ae6e8d66..69746bc8c3 100644 --- a/documentation/query/sql/create-live-view.md +++ b/documentation/query/sql/create-live-view.md @@ -16,7 +16,7 @@ CREATE LIVE VIEW [ IF NOT EXISTS ] viewName FLUSH EVERY duration [ IN MEMORY duration ] [ PARTITION BY ( YEAR | MONTH | WEEK | DAY | HOUR ) ] -[ BACKFILL ] +START FROM ( NOW | BEGINNING | 'timestamp' ) AS [ ( ] query [ ) ] [ OWNED BY ownerName ] ``` @@ -28,9 +28,10 @@ Where: - `query`: a `SELECT` over one WAL-backed base table whose projection contains [window functions](/docs/query/functions/window-functions/overview/). -`FLUSH EVERY` is required and must come first. `IN MEMORY`, `PARTITION BY`, and -`BACKFILL` are optional and may appear in any order. These four clauses all -precede `AS`; the optional `OWNED BY` clause follows the query. +`FLUSH EVERY` is required and must come first. `START FROM` is also required and +may appear in any order with the optional `IN MEMORY` and `PARTITION BY` +clauses. These clauses all precede `AS`; the optional `OWNED BY` clause follows +the query. ## Parameters @@ -41,7 +42,7 @@ precede `AS`; the optional `OWNED BY` clause follows the query. | `FLUSH EVERY` | How often computed rows are persisted to disk. Required | | `IN MEMORY` | Window of recent rows kept in RAM for fresh reads. Defaults to `FLUSH EVERY` | | `PARTITION BY` | Partitioning unit for the view's disk tier. Defaults to the base table's scheme | -| `BACKFILL` | Materialize the base table's existing history before live-tailing | +| `START FROM` | Inclusive event-time boundary: `NOW`, `BEGINNING`, or a timestamp literal. Required | | `query` | A window-function `SELECT` over a single WAL-backed base table | | `OWNED BY` | Assign ownership (Enterprise) | @@ -66,6 +67,7 @@ The minimum is `100ms`. The maximum is ```questdb-sql CREATE LIVE VIEW trades_ma FLUSH EVERY 1s +START FROM NOW AS SELECT timestamp, symbol, avg(price) OVER (PARTITION BY symbol ORDER BY timestamp ROWS 300 PRECEDING) @@ -86,6 +88,7 @@ and older data from disk. It defaults to `FLUSH EVERY`. CREATE LIVE VIEW trades_ma FLUSH EVERY 1s IN MEMORY 5s +START FROM NOW AS SELECT timestamp, symbol, avg(price) OVER (PARTITION BY symbol ORDER BY timestamp ROWS 300 PRECEDING) @@ -102,6 +105,7 @@ view inherits the base table's partitioning scheme. CREATE LIVE VIEW trades_ma FLUSH EVERY 1s PARTITION BY HOUR +START FROM NOW AS SELECT timestamp, symbol, avg(price) OVER (PARTITION BY symbol ORDER BY timestamp ROWS 300 PRECEDING) @@ -109,16 +113,35 @@ SELECT timestamp, symbol, FROM trades; ``` -### BACKFILL +### START FROM -By default a live view processes only the rows that arrive after it is created. -Add `BACKFILL` to materialize the base table's existing history first. The sweep -is resumable across restarts. +`START FROM` defines the inclusive event-time boundary for rows in the live +view. It is mandatory and accepts: + +- `NOW`: resolve the engine clock once when the view is created. +- `BEGINNING`: include all base-table history. +- A quoted timestamp literal: include rows whose designated timestamp is equal + to or later than that value. + +The boundary applies to the base table's designated timestamp, not to commit +time. QuestDB performs a resumable initial seed for qualifying rows already +present at creation, then continues refreshing from new base commits. ```questdb-sql CREATE LIVE VIEW trades_ma FLUSH EVERY 1s -BACKFILL +START FROM BEGINNING +AS +SELECT timestamp, symbol, + avg(price) OVER (PARTITION BY symbol ORDER BY timestamp ROWS 300 PRECEDING) + AS moving_avg +FROM trades; +``` + +```questdb-sql title="Start from an explicit timestamp" +CREATE LIVE VIEW trades_ma_from_april +FLUSH EVERY 1s +START FROM '2026-04-01T00:00:00.000000Z' AS SELECT timestamp, symbol, avg(price) OVER (PARTITION BY symbol ORDER BY timestamp ROWS 300 PRECEDING) @@ -135,6 +158,7 @@ named `WINDOW` with either the `ANCHOR DAILY` shorthand or an ```questdb-sql title="Cumulative daily volume, reset each day" CREATE LIVE VIEW trades_daily_volume FLUSH EVERY 1s +START FROM NOW AS SELECT timestamp, symbol, sum(amount) OVER w AS cumulative_volume @@ -145,6 +169,7 @@ WINDOW w AS (PARTITION BY symbol ORDER BY timestamp ANCHOR DAILY); ```questdb-sql title="Anchor on an arbitrary expression" CREATE LIVE VIEW trades_hourly_volume FLUSH EVERY 1s +START FROM NOW AS SELECT timestamp, symbol, sum(amount) OVER w AS bucket_volume @@ -193,7 +218,7 @@ CREATE LIVE VIEW IF NOT EXISTS trades_ma FLUSH EVERY 1s IN MEMORY 5s PARTITION BY HOUR -BACKFILL +START FROM BEGINNING AS SELECT timestamp, @@ -212,7 +237,7 @@ This creates a view that: - Persists computed rows to disk every second (`FLUSH EVERY 1s`) - Keeps 5 seconds of recent rows in RAM for fresh reads (`IN MEMORY 5s`) - Partitions its disk tier by hour (`PARTITION BY HOUR`) -- Materializes the existing history in `trades` before live-tailing (`BACKFILL`) +- Includes all existing history in `trades` (`START FROM BEGINNING`) - Keeps a 300-row moving average of price per symbol ## Metadata @@ -248,6 +273,7 @@ Assign ownership to a user, group, or service account: CREATE GROUP analysts; CREATE LIVE VIEW trades_ma FLUSH EVERY 1s +START FROM NOW AS SELECT timestamp, symbol, avg(price) OVER (PARTITION BY symbol ORDER BY timestamp ROWS 300 PRECEDING) diff --git a/documentation/query/sql/show.md b/documentation/query/sql/show.md index 8f3fe6b860..086419fefe 100644 --- a/documentation/query/sql/show.md +++ b/documentation/query/sql/show.md @@ -212,11 +212,11 @@ SHOW CREATE LIVE VIEW trades_ma; | ddl | | --- | -| CREATE LIVE VIEW 'trades_ma' FLUSH EVERY 1s IN MEMORY 5s PARTITION BY DAY AS (
SELECT timestamp, symbol, avg(price) OVER (PARTITION BY symbol ORDER BY timestamp ROWS 300 PRECEDING) AS moving_avg FROM trades
); | +| CREATE LIVE VIEW 'trades_ma' FLUSH EVERY 1s IN MEMORY 5s PARTITION BY DAY START FROM NOW AS (
SELECT timestamp, symbol, avg(price) OVER (PARTITION BY symbol ORDER BY timestamp ROWS 300 PRECEDING) AS moving_avg FROM trades
); | This returns the `CREATE LIVE VIEW` statement that would recreate the [live view](/docs/concepts/live-views/), including its `FLUSH EVERY`, -`IN MEMORY`, `PARTITION BY`, and `BACKFILL` clauses. On QuestDB Enterprise the +`IN MEMORY`, `PARTITION BY`, and `START FROM` clauses. On QuestDB Enterprise the output also carries an `OWNED BY` clause identifying the view's owner. ### SHOW PARTITIONS From 34b989e57c6227a752eef26685320bf3fcc8f5fd Mon Sep 17 00:00:00 2001 From: Andrei Pechkurov Date: Thu, 16 Jul 2026 11:59:48 +0300 Subject: [PATCH 6/8] Document recent live view observability changes --- documentation/concepts/live-views.md | 6 ++++ documentation/configuration/live-views.md | 44 +++++++++++++++++++++++ documentation/query/functions/meta.md | 8 +++-- 3 files changed, 55 insertions(+), 3 deletions(-) diff --git a/documentation/concepts/live-views.md b/documentation/concepts/live-views.md index 9f4904348c..eb99902909 100644 --- a/documentation/concepts/live-views.md +++ b/documentation/concepts/live-views.md @@ -286,6 +286,12 @@ cost of maintained views, or increase the shared [`mat.view.refresh.worker.count`](/docs/configuration/materialized-views/) setting. +For out-of-order replay cost, compare `o3_resume_replay_rows` with +`o3_boundary_replay_rows`. Resume replays start from a retained checkpoint and +remain bounded to the affected tail. Boundary replays rebuild from the view's +`START FROM` boundary and are more expensive. Both counters reset on restart; +tune the checkpoint-retention settings when boundary rebuilds are frequent. + Live views also appear in [`tables()`](/docs/query/functions/meta/#tables) with `table_type = 'L'`, and are recognized by `SHOW CREATE LIVE VIEW`, `EXPLAIN`, `pg_class`, and `information_schema.tables`. diff --git a/documentation/configuration/live-views.md b/documentation/configuration/live-views.md index bfa3a247ae..6cec9eabfc 100644 --- a/documentation/configuration/live-views.md +++ b/documentation/configuration/live-views.md @@ -21,6 +21,35 @@ view refresh. There are no dedicated live-view worker-pool properties. Time budget, in microseconds, for a single checkpoint write turn. Checkpoints let a restart or out-of-order replay resume without rebuilding the whole view. +## cairo.live.view.checkpoint.retention.count + +- **Default**: `8` +- **Reloadable**: no + +Maximum number of checkpoints retained per live view for bounded out-of-order +replay. A deeper checkpoint ring lets a replay resume closer to a late row, at +the cost of additional disk usage. The newest checkpoint is always retained. A +value of `0` or less disables the count bound. + +## cairo.live.view.checkpoint.retention.max.bytes + +- **Default**: `67108864` (64 MiB) +- **Reloadable**: no + +Maximum total serialized size of retained checkpoints per live view. When the +ring exceeds this budget, QuestDB prunes the oldest checkpoints. The newest +checkpoint is always retained. A value of `0` or less disables the byte bound. + +## cairo.live.view.checkpoint.retention.micros + +- **Default**: `0` (disabled) +- **Reloadable**: no + +Optional event-time horizon for retained checkpoints. Checkpoints older than +this distance from the newest checkpoint are pruned. Keep this disabled for +low-rate or coarse-checkpoint views unless you specifically need an additional +age bound, because a short horizon can leave only the newest replay anchor. + ## cairo.live.view.checkpoint.rows - **Default**: `1000000` @@ -88,6 +117,21 @@ per-function state. Compaction fires when a partition's anchor-map entry count exceeds this value and the frontier has advanced. Applies only to anchored views whose anchor is a monotone, fixed-duration-unit timestamp expression. +## cairo.live.view.refresh.memory.limit.bytes + +- **Default**: `0` (unlimited) +- **Reloadable**: yes + +Per-live-view limit on the peak memory used during a refresh cycle. It includes +persistent window state and transient buffers such as Parquet row-group decode +buffers. Exceeding the limit invalidates the view immediately; recovery requires +dropping and recreating it. + +Size the limit above the refresh workload's allocation floor. In particular, a +bounded `ROWS` frame allocates at least one +`cairo.sql.window.store.page.size` page (1 MiB by default), and a Parquet-backed +refresh may decode a complete row group. + ## cairo.live.view.refresh.turn.max.commits - **Default**: `64` diff --git a/documentation/query/functions/meta.md b/documentation/query/functions/meta.md index a813f6355b..df366bee87 100644 --- a/documentation/query/functions/meta.md +++ b/documentation/query/functions/meta.md @@ -154,7 +154,7 @@ Returns a `table` with the following columns: | `view_table_dir_name` | STRING | View directory name on disk | | `base_table_name` | STRING | Base table name | | `view_sql` | STRING | Query used to maintain the view | -| `view_status` | STRING | View status: `active`, `seeding`, or `invalid` | +| `view_status` | STRING | Lifecycle status: `creating`, `active`, `seeding`, `invalid`, `dropping`, `version_unsupported`, or `state_unreadable` | | `invalidation_reason` | STRING | Message explaining why the view was marked invalid | | `flush_every_interval` | LONG | `FLUSH EVERY` interval value | | `flush_every_interval_unit` | STRING | `FLUSH EVERY` unit: `MILLISECOND`, `SECOND`, `MINUTE`, `HOUR`, or `DAY` | @@ -162,8 +162,8 @@ Returns a `table` with the following columns: | `in_memory_interval_unit` | STRING | `IN MEMORY` unit: `MILLISECOND`, `SECOND`, `MINUTE`, `HOUR`, or `DAY` | | `in_mem_bytes` | LONG | Native footprint of the in-memory tier, a peak-sticky high-water mark | | `in_mem_rows` | LONG | Live row count held in the in-memory tier | -| `o3_rejected_count` | LONG | Count of out-of-order base commits routed through replay | -| `below_lower_bound_count` | LONG | Count of base rows dropped below the view's lower bound | +| `o3_rejected_count` | LONG | Count of late out-of-order rows rejected below the view lower bound; resets on restart | +| `below_lower_bound_count` | LONG | Count of in-order base rows dropped below the view lower bound; resets on restart | | `lag_seqtxn` | LONG | Committed base WAL transactions beyond the view's durable processed watermark (`base_table_txn - last_processed_seqtxn`) | | `lag_micros` | LONG | Microseconds since the last successful flush | | `last_processed_seqtxn` | LONG | Last base transaction processed by the refresh worker | @@ -175,6 +175,8 @@ Returns a `table` with the following columns: | `head_checkpoint_lv_seqtxn` | LONG | View transaction of the latest head checkpoint | | `head_checkpoint_max_ts` | TIMESTAMP | Maximum timestamp covered by the latest head checkpoint | | `head_checkpoint_state_bytes` | LONG | Size of the latest head checkpoint's window state | +| `o3_resume_replay_rows` | LONG | Rows re-emitted by bounded out-of-order replays resumed from retained checkpoints; resets on restart | +| `o3_boundary_replay_rows` | LONG | Rows re-emitted by full boundary-rebuild out-of-order replays; resets on restart | The `in_mem_bytes` and `in_mem_rows` columns are complementary. `in_mem_bytes` is the peak-sticky arena footprint that does not shrink after a burst, while From 9ae8a4f9bd566fea55376f62912f06fa103a4c65 Mon Sep 17 00:00:00 2001 From: Andrei Pechkurov Date: Mon, 27 Jul 2026 11:54:39 +0300 Subject: [PATCH 7/8] Update live view documentation --- documentation/concepts/live-views.md | 118 ++++++++++++-------- documentation/configuration/live-views.md | 79 +++++++------ documentation/query/functions/meta.md | 107 ++++++++++++------ documentation/query/sql/create-live-view.md | 88 +++++++++------ 4 files changed, 246 insertions(+), 146 deletions(-) diff --git a/documentation/concepts/live-views.md b/documentation/concepts/live-views.md index eb99902909..5d49e26e4a 100644 --- a/documentation/concepts/live-views.md +++ b/documentation/concepts/live-views.md @@ -136,36 +136,39 @@ interpret lag and detect a view that cannot keep up. ## Supported window functions -Live views maintain the window functions whose result can be computed -incrementally in a single forward pass over a partitioned frame: +Live views maintain window functions whose result can be computed incrementally +in a single forward pass and whose state can be checkpointed: -- **Ranking**: `row_number`, `rank`, `dense_rank` -- **Cumulative and bounded aggregates**: `sum`, `avg`, `count`, `min`, `max`, +- **Anchored ranking**: `row_number`, `rank`, `dense_rank` +- **Bounded or anchored aggregates**: `sum`, `avg`, `count`, `min`, `max`, `ksum`, `first_value`, `last_value`, `nth_value` - **Offset**: `lag` - **Statistics**: `variance`, `stddev`, covariance, correlation, EMA, and VWEMA -Every window function must have a `PARTITION BY` clause. Both bounded `ROWS` and -bounded `RANGE` frames are supported. +Stateful window functions must have a `PARTITION BY` clause. Both bounded `ROWS` +and bounded `RANGE` frames are supported. Ranking functions must use an anchored +window because an out-of-order row would otherwise change every later rank. String, `VARCHAR`, `BINARY`, `ARRAY`, and `SYMBOL` columns can appear as -pass-through output columns and as `count` arguments, but there are no -string- or array-valued window functions. +pass-through output columns and as `count` arguments, but there are no string- +or array-valued window functions. The following shapes cannot be maintained by an append-only incremental refresh and are rejected at creation time: - Multi-pass or look-ahead functions: `percent_rank`, `cume_dist`, `ntile`, `lead` -- Window functions without `PARTITION BY` -- Unbounded frames on non-anchored windows +- Stateful window functions without `PARTITION BY` +- Unanchored ranking functions +- Frames that start at `UNBOUNDED PRECEDING`, except stateless `last_value` + shapes ## Anchored windows -An anchored window resets its cumulative aggregate on a boundary, which is useful -for running totals that restart each day or on a period boundary. Declare it in a -named window with either the `ANCHOR DAILY` shorthand or an `ANCHOR EXPRESSION` -clause: +An anchored window resets its cumulative aggregate on a boundary, which is +useful for running totals that restart each day or on a period boundary. Declare +it in a named window with either `ANCHOR DAILY 'HH:MM' ['timezone']` or an +`ANCHOR EXPRESSION` clause: ```questdb-sql title="Cumulative daily volume per symbol" CREATE LIVE VIEW trades_daily_volume @@ -180,12 +183,18 @@ FROM trades WINDOW w AS ( PARTITION BY symbol ORDER BY timestamp - ANCHOR DAILY + ANCHOR DAILY '00:00' ); ``` -An anchored window must be partitioned, cannot use a bounded frame, and its -anchor expression must be deterministic. +`ANCHOR DAILY` resets at the specified wall-clock time. Without a time zone, the +time is interpreted in UTC; add an IANA time zone such as `'America/New_York'` +when the boundary follows local civil time. + +An anchored window must be a named window, must partition by base-table columns, +must order by the designated timestamp ascending, and cannot use a bounded +frame. A live view supports at most one anchored window. An anchor expression +must be deterministic and return `TIMESTAMP`, `LONG`, or `INT`. ## Start boundary and historical data @@ -255,8 +264,8 @@ rather than sub-cycle fresh, because its refresh is coupled to base apply. ## Monitoring -The [`live_views()`](/docs/query/functions/meta/#live_views) function exposes the -state, refresh lag, in-memory footprint, and seed progress of every live +The [`live_views()`](/docs/query/functions/meta/#live_views) function exposes +the state, refresh lag, in-memory footprint, and seed progress of every live view: ```questdb-sql title="List all live views" @@ -273,8 +282,8 @@ durable tier is caught up. A temporary non-zero value is expected between There is no universal acceptable non-zero value. Sample the metric over time and compare it with the view's normal flush-cycle baseline. A bounded sawtooth that returns to zero around flushes is normal. A value that stays elevated for -multiple flush intervals or keeps increasing indicates that the view cannot -keep up. +multiple flush intervals or keeps increasing indicates that the view cannot keep +up. `lag_micros` reports the elapsed time since the last successful flush. It is a flush-activity indicator, not the timestamp difference between base and view @@ -286,11 +295,21 @@ cost of maintained views, or increase the shared [`mat.view.refresh.worker.count`](/docs/configuration/materialized-views/) setting. -For out-of-order replay cost, compare `o3_resume_replay_rows` with -`o3_boundary_replay_rows`. Resume replays start from a retained checkpoint and -remain bounded to the affected tail. Boundary replays rebuild from the view's -`START FROM` boundary and are more expensive. Both counters reset on restart; -tune the checkpoint-retention settings when boundary rebuilds are frequent. +For out-of-order repair cost, compare `o3_replay_scan_rows`, +`o3_resume_replay_rows`, and `o3_boundary_replay_rows`. The first counts base +rows scanned; the other two split rows emitted by the resume-from-checkpoint and +rebuild paths. `checkpoint_repair_plan` reports whether the view has a finite +`range`, `rows`, or `anchor` repair plan. The last disposition and denial +columns show which path actually ran and why a local rebuild was not selected. +These counters reset on restart. + +The `checkpoint_timeline_*` columns describe the persistent timeline used for +restart and out-of-order repair. In particular, +`checkpoint_timeline_sharing_ratio` shows how effectively checkpoint state is +shared, while `checkpoint_gc_lag_generations` and +`checkpoint_obsolete_segment_bytes` can expose delayed cleanup. See +[`live_views()`](/docs/query/functions/meta/#live_views) for the complete +catalog. Live views also appear in [`tables()`](/docs/query/functions/meta/#tables) with `table_type = 'L'`, and are recognized by `SHOW CREATE LIVE VIEW`, `EXPLAIN`, @@ -301,22 +320,26 @@ Live views also appear in [`tables()`](/docs/query/functions/meta/#tables) with Live views have a deliberately narrow surface in this first version. Statements outside it are rejected at creation time with a specific error: -| Base object | Derived object | Supported | -| ----------- | -------------- | --------- | -| Live view | Regular view | Yes | -| Live view | Materialized view | No | -| Live view | Live view | No | -| Regular view | Live view | No; a regular view is not a WAL table | -| Materialized view | Live view | Yes | +| Base object | Derived object | Supported | +| ----------------- | ----------------- | ------------------------------------- | +| Live view | Regular view | Yes | +| Live view | Materialized view | No | +| Live view | Live view | No | +| Regular view | Live view | No; a regular view is not a WAL table | +| Materialized view | Live view | Yes | A full rebuild of a materialized view invalidates a live view that uses it as its base. Recreate the live view after the rebuild, following the recovery steps in [Base table lifecycle](#base-table-lifecycle). - **Single base table only.** No JOINs, subqueries, or CTEs in the view query. +- **Explicit output columns only.** `SELECT *` and other wildcard projections + are rejected because the output schema is fixed when the view is created. - **No pre-aggregation.** `SAMPLE BY` and `GROUP BY` are not allowed between the base table and the window functions. A view like "5-minute candles with a rolling VWAP" must pre-aggregate upstream. +- **No designated-timestamp filter.** A `WHERE` clause may filter other columns, + but cannot filter the base table's designated timestamp yet. - **Deterministic queries only.** Non-deterministic functions such as `now()`, `sysdate()`, `systimestamp()`, and `rnd_*()` are rejected in the projection, the `WHERE` filter, and window-function arguments. @@ -332,10 +355,11 @@ in [Base table lifecycle](#base-table-lifecycle). correct but stale, with no automatic throttle or drop. - **Per-partition state for partitioned windows grows with distinct partition cardinality.** A base table with high-cardinality partition keys (UUIDs, - session ids) holds one state entry per key seen, so native-memory use grows - over the life of the view. The `in_mem_bytes` column in - [`live_views()`](/docs/query/functions/meta/#live_views) reports this - footprint as a peak-sticky high-water mark. + session ids) holds state per key. Use + [`cairo.live.view.refresh.memory.limit.bytes`](/docs/configuration/live-views/#cairoliveviewrefreshmemorylimitbytes) + to bound the per-view refresh footprint. The `in_mem_bytes` column in + [`live_views()`](/docs/query/functions/meta/#live_views) separately reports + the peak-sticky capacity of the recent-row tier. ## Enterprise features @@ -356,19 +380,23 @@ live view is a regular table token. See ### Replication -A live view replicates physically like a materialized view. Its disk tier is a -regular WAL-backed table, so its rows transfer to replicas through the existing -object-store WAL path. A read-only replica never refreshes the view itself. It -reconstructs the primary's un-flushed in-memory rows in RAM so that reads on the -replica match the primary's freshness. Promoting a replica to primary resumes -refresh from the durable watermark. +Live-view definitions replicate, but their derived rows do not. Every node +refreshes and flushes its own node-local live-view table: + +- The primary refreshes directly from the base table's WAL. +- A read-only replica refreshes from its locally applied copy of the base table. +- Live-view WAL is not uploaded or transferred between nodes. + +A role switch continues the local refresh state; it does not reconstruct or +transfer the former primary's live-view rows. Replica freshness therefore also +depends on base-table replication and apply lag. ### Backup and restore A live view is captured by the object-store backup like a materialized view: its table data rides the standard table path and its definition sidecars are carried -in the backup manifest. On restore, the un-flushed in-memory rows are re-derived -from the base table, which is the same bounded recompute a promote performs. +in the backup manifest. After restore, unflushed rows are re-derived from the +base table. ## Related documentation diff --git a/documentation/configuration/live-views.md b/documentation/configuration/live-views.md index 6cec9eabfc..204de67516 100644 --- a/documentation/configuration/live-views.md +++ b/documentation/configuration/live-views.md @@ -13,42 +13,52 @@ worker-pool settings under (`mat.view.refresh.worker.count`, `.affinity`, `.haltOnError`) also govern live view refresh. There are no dedicated live-view worker-pool properties. +## cairo.live.view.checkpoint.compaction.interval + +- **Default**: `0` (disabled) +- **Reloadable**: no + +Number of checkpoint seals between physical compaction attempts. Compaction +repacks live state pages from sparse data segments so obsolete segments can be +reclaimed. A value of `0` disables compaction. + ## cairo.live.view.checkpoint.max.duration.micros - **Default**: `300000000` (5 minutes) - **Reloadable**: no -Time budget, in microseconds, for a single checkpoint write turn. Checkpoints let -a restart or out-of-order replay resume without rebuilding the whole view. +Maximum wall-clock interval, in microseconds, between head-checkpoint writes. +This caps restart and out-of-order replay work for low-rate views that do not +reach the row-count trigger. -## cairo.live.view.checkpoint.retention.count +## cairo.live.view.checkpoint.repair.replay.max.rows -- **Default**: `8` +- **Default**: `1000000` - **Reloadable**: no -Maximum number of checkpoints retained per live view for bounded out-of-order -replay. A deeper checkpoint ring lets a replay resume closer to a late row, at -the cost of additional disk usage. The newest checkpoint is always retained. A -value of `0` or less disables the count bound. +Maximum base rows a localized out-of-order repair replays in one refresh turn. A +larger repair pauses at a timestamp-group boundary and continues in a later turn +without publishing partial output. The refresh-turn duration limit also applies. +A value of `0` or less disables this row budget. -## cairo.live.view.checkpoint.retention.max.bytes +## cairo.live.view.checkpoint.repair.scan.max.keys -- **Default**: `67108864` (64 MiB) +- **Default**: `100000` - **Reloadable**: no -Maximum total serialized size of retained checkpoints per live view. When the -ring exceeds this budget, QuestDB prunes the oldest checkpoints. The newest -checkpoint is always retained. A value of `0` or less disables the byte bound. +Maximum partition keys inspected while planning a localized `ROWS` repair. If +planning crosses the limit, QuestDB uses an unlocalized repair instead. A value +of `0` or less disables this key budget. -## cairo.live.view.checkpoint.retention.micros +## cairo.live.view.checkpoint.repair.scan.max.rows -- **Default**: `0` (disabled) +- **Default**: `1000000` - **Reloadable**: no -Optional event-time horizon for retained checkpoints. Checkpoints older than -this distance from the newest checkpoint are pruned. Keep this disabled for -low-rate or coarse-checkpoint views unless you specifically need an additional -age bound, because a short horizon can leave only the newest replay anchor. +Maximum base rows scanned while discovering the bounds of a localized `ROWS` +repair. This includes rows later discarded by the view's `WHERE` filter. If the +limit is crossed, QuestDB uses the conservative unlocalized bound. A value of +`0` or less disables this scan budget. ## cairo.live.view.checkpoint.rows @@ -88,8 +98,12 @@ view is marked invalid. - **Default**: `16777216` (16 MiB) - **Reloadable**: no -Increment by which the in-memory tier's buffer arena grows when it needs more -capacity. Accepts a size suffix such as `16M`. +Fast-path growth budget for the in-memory tier. Once the published buffer's +footprint reaches this size, refresh falls back to a swap that evicts expired +rows and may shrink the buffer instead of continuing to append in place. Raise +it for `IN MEMORY` windows expected to exceed the default. A value of `0` or +less forces this compaction path on every publish. Accepts a size suffix such as +`16M`. ## cairo.live.view.in.memory.buffer.initial.bytes @@ -112,10 +126,9 @@ Upper bound on the `IN MEMORY` retention window. A `CREATE LIVE VIEW` whose - **Default**: `100000` - **Reloadable**: no -Row-count threshold at which an anchored live view compacts a partition's -per-function state. Compaction fires when a partition's anchor-map entry count -exceeds this value and the frontier has advanced. Applies only to anchored views -whose anchor is a monotone, fixed-duration-unit timestamp expression. +Anchor-map tombstone threshold that triggers compaction. Compaction also runs +when tombstones exceed half of the anchor map, regardless of this absolute +threshold. ## cairo.live.view.refresh.memory.limit.bytes @@ -123,14 +136,14 @@ whose anchor is a monotone, fixed-duration-unit timestamp expression. - **Reloadable**: yes Per-live-view limit on the peak memory used during a refresh cycle. It includes -persistent window state and transient buffers such as Parquet row-group decode -buffers. Exceeding the limit invalidates the view immediately; recovery requires -dropping and recreating it. +persistent window state, the `IN MEMORY` recent-row tier, staging memory, and +transient buffers such as Parquet row-group decode buffers. Exceeding the limit +invalidates the view immediately; recovery requires dropping and recreating it. Size the limit above the refresh workload's allocation floor. In particular, a -bounded `ROWS` frame allocates at least one -`cairo.sql.window.store.page.size` page (1 MiB by default), and a Parquet-backed -refresh may decode a complete row group. +bounded `ROWS` frame allocates at least one `cairo.sql.window.store.page.size` +page (1 MiB by default), and a Parquet-backed refresh may decode a complete row +group. ## cairo.live.view.refresh.turn.max.commits @@ -145,5 +158,5 @@ before yielding to other views. - **Default**: `50000` (50 milliseconds) - **Reloadable**: no -Maximum wall-clock time, in microseconds, a refresh worker spends on one view per -turn before yielding. +Maximum wall-clock time, in microseconds, a refresh worker spends on one view +per turn before yielding. diff --git a/documentation/query/functions/meta.md b/documentation/query/functions/meta.md index df366bee87..edd667c5c3 100644 --- a/documentation/query/functions/meta.md +++ b/documentation/query/functions/meta.md @@ -148,48 +148,83 @@ seed progress. Returns a `table` with the following columns: -| Column | Type | Description | -| ------ | ---- | ----------- | -| `view_name` | STRING | Live view name | -| `view_table_dir_name` | STRING | View directory name on disk | -| `base_table_name` | STRING | Base table name | -| `view_sql` | STRING | Query used to maintain the view | -| `view_status` | STRING | Lifecycle status: `creating`, `active`, `seeding`, `invalid`, `dropping`, `version_unsupported`, or `state_unreadable` | -| `invalidation_reason` | STRING | Message explaining why the view was marked invalid | -| `flush_every_interval` | LONG | `FLUSH EVERY` interval value | -| `flush_every_interval_unit` | STRING | `FLUSH EVERY` unit: `MILLISECOND`, `SECOND`, `MINUTE`, `HOUR`, or `DAY` | -| `in_memory_interval` | LONG | `IN MEMORY` interval value | -| `in_memory_interval_unit` | STRING | `IN MEMORY` unit: `MILLISECOND`, `SECOND`, `MINUTE`, `HOUR`, or `DAY` | -| `in_mem_bytes` | LONG | Native footprint of the in-memory tier, a peak-sticky high-water mark | -| `in_mem_rows` | LONG | Live row count held in the in-memory tier | -| `o3_rejected_count` | LONG | Count of late out-of-order rows rejected below the view lower bound; resets on restart | -| `below_lower_bound_count` | LONG | Count of in-order base rows dropped below the view lower bound; resets on restart | -| `lag_seqtxn` | LONG | Committed base WAL transactions beyond the view's durable processed watermark (`base_table_txn - last_processed_seqtxn`) | -| `lag_micros` | LONG | Microseconds since the last successful flush | -| `last_processed_seqtxn` | LONG | Last base transaction processed by the refresh worker | -| `applied_watermark` | LONG | Last base transaction durably applied to the view's disk tier | -| `lv_consumed_seqtxn` | LONG | Base WAL purge floor held by this view | -| `view_lower_bound_timestamp` | TIMESTAMP | Lower timestamp bound below which base rows are ignored | -| `writer_stall_micros` | LONG | Time a flush has been stalled waiting to write, in microseconds | -| `seed_target_seqtxn` | LONG | Target base transaction for an in-progress initial seed | -| `head_checkpoint_lv_seqtxn` | LONG | View transaction of the latest head checkpoint | -| `head_checkpoint_max_ts` | TIMESTAMP | Maximum timestamp covered by the latest head checkpoint | -| `head_checkpoint_state_bytes` | LONG | Size of the latest head checkpoint's window state | -| `o3_resume_replay_rows` | LONG | Rows re-emitted by bounded out-of-order replays resumed from retained checkpoints; resets on restart | -| `o3_boundary_replay_rows` | LONG | Rows re-emitted by full boundary-rebuild out-of-order replays; resets on restart | - -The `in_mem_bytes` and `in_mem_rows` columns are complementary. `in_mem_bytes` is -the peak-sticky arena footprint that does not shrink after a burst, while -`in_mem_rows` is the live row count that drops as rows age out of the `IN MEMORY` -window. Together they distinguish a view actively buffering rows from one holding -capacity retained from a past burst. +| Column | Type | Description | +| ---------------------------------------------- | --------- | ---------------------------------------------------------------------------------------------------------------------- | +| `view_name` | STRING | Live view name | +| `view_table_dir_name` | STRING | View directory name on disk | +| `base_table_name` | STRING | Base table name | +| `view_sql` | STRING | Query used to maintain the view | +| `view_status` | STRING | Lifecycle status: `creating`, `active`, `seeding`, `invalid`, `dropping`, `version_unsupported`, or `state_unreadable` | +| `invalidation_reason` | STRING | Message explaining why the view was marked invalid | +| `flush_every_interval` | LONG | `FLUSH EVERY` interval value | +| `flush_every_interval_unit` | STRING | `FLUSH EVERY` unit: `MILLISECOND`, `SECOND`, `MINUTE`, `HOUR`, or `DAY` | +| `in_memory_interval` | LONG | `IN MEMORY` interval value | +| `in_memory_interval_unit` | STRING | `IN MEMORY` unit: `MILLISECOND`, `SECOND`, `MINUTE`, `HOUR`, or `DAY` | +| `in_mem_bytes` | LONG | Native capacity of the in-memory tier; a peak-sticky high-water mark | +| `in_mem_rows` | LONG | Live row count in the published in-memory tier | +| `o3_rejected_count` | LONG | Late out-of-order rows rejected below the view lower bound; resets on restart | +| `below_lower_bound_count` | LONG | In-order rows dropped below the view lower bound; resets on restart | +| `lag_seqtxn` | LONG | Committed base WAL transactions beyond `last_processed_seqtxn` | +| `lag_micros` | LONG | Microseconds since the last successful flush | +| `last_processed_seqtxn` | LONG | Last base transaction processed by the refresh worker | +| `applied_watermark` | LONG | Last base transaction durably applied to the view's disk tier | +| `lv_consumed_seqtxn` | LONG | Base WAL purge floor held by this view | +| `view_lower_bound_timestamp` | TIMESTAMP | Resolved `START FROM` boundary; `NULL` for `BEGINNING` | +| `writer_stall_micros` | LONG | Current uninterrupted flush-writer stall duration, or `0` | +| `seed_target_seqtxn` | LONG | Target base transaction during initial seeding; otherwise `NULL` | +| `o3_resume_replay_rows` | LONG | Rows emitted by repairs resumed from a checkpoint; resets on restart | +| `o3_boundary_replay_rows` | LONG | Rows emitted by rebuild repairs; resets on restart | +| `o3_replay_scan_rows` | LONG | Base rows scanned by both repair paths; resets on restart | +| `checkpoint_timeline_generation` | LONG | Current published checkpoint-timeline generation | +| `checkpoint_timeline_entries` | LONG | Checkpoint roots in the current generation | +| `checkpoint_timeline_normalized_base_seqtxn` | LONG | Base transaction through which the timeline is normalized | +| `checkpoint_timeline_logical_bytes` | LONG | Bytes the roots would use as independent complete state images | +| `checkpoint_timeline_physical_bytes` | LONG | Bytes physically stored for the current timeline generation | +| `checkpoint_timeline_shared_bytes` | LONG | Logical bytes avoided through state sharing | +| `checkpoint_timeline_sharing_ratio` | DOUBLE | Shared bytes divided by logical bytes | +| `checkpoint_timeline_row_position_delta_bytes` | LONG | Bytes used by the row-position delta index | +| `checkpoint_data_segment_count` | LONG | Data segments found by the latest checkpoint purge sweep | +| `checkpoint_obsolete_segment_bytes` | LONG | Obsolete segment bytes found by the latest purge sweep | +| `checkpoint_oldest_pinned_generation` | LONG | Oldest checkpoint generation still retained | +| `checkpoint_gc_lag_generations` | LONG | Generations between the current and oldest retained generation | +| `checkpoint_last_write_micros` | LONG | Duration of the latest checkpoint write | +| `checkpoint_last_restore_micros` | LONG | Duration of the latest checkpoint restore | +| `checkpoint_last_write_new_bytes` | LONG | New bytes written by the latest checkpoint publication | +| `checkpoint_last_lookup_depth` | LONG | Metadata-tree depth of the latest checkpoint lookup | +| `checkpoint_repair_in_progress` | BOOLEAN | Whether a localized repair is suspended across refresh turns | +| `checkpoint_repair_correction_timestamp` | TIMESTAMP | Earliest timestamp whose output may have changed in the active repair | +| `checkpoint_repair_low_timestamp` | TIMESTAMP | Inclusive base timestamp from which the active repair scans | +| `checkpoint_repair_high_timestamp` | TIMESTAMP | Repair convergence boundary; `NULL` when it runs to end of data | +| `checkpoint_repair_roots_versioned` | LONG | Checkpoint roots versioned by repairs; resets on restart | +| `checkpoint_repair_new_bytes` | LONG | Bytes written by repairs; resets on restart | +| `checkpoint_repair_resumes` | LONG | Times a repair resumed in a later refresh turn; resets on restart | +| `checkpoint_repair_failures` | LONG | Repair failures; resets on restart | +| `checkpoint_repair_plan` | STRING | Available localized plan: `range`, `rows`, `anchor`, a `+` combination, `none`, or `NULL` before compilation | +| `checkpoint_repair_last_disposition` | STRING | Last executor used: `localized rebuild`, `boundary rebuild`, or `resume from anchor` | +| `checkpoint_repair_last_denial` | STRING | Reason the last repair did not use a localized rebuild; otherwise `NULL` | + +The `in_mem_bytes` and `in_mem_rows` columns are complementary. `in_mem_bytes` +is the peak-sticky arena footprint that does not shrink after a burst, while +`in_mem_rows` is the live row count that drops as rows age out of the +`IN MEMORY` window. Together they distinguish a view actively buffering rows +from one holding capacity retained from a past burst. `lag_seqtxn` counts transactions, not rows. A value of `0` means the durable tier is caught up; a temporary non-zero value is expected between `FLUSH EVERY` cycles. Alert on a value that remains above its normal flush-cycle baseline or keeps increasing across samples rather than on a universal fixed threshold. `lag_micros` measures flush activity and may grow while an idle view has -`lag_seqtxn = 0`. For operational guidance, see +`lag_seqtxn = 0`. + +The checkpoint columns describe the current persistent timeline, its storage and +lookup cost, and out-of-order repair activity. Timeline fields are `NULL` before +the first generation is published. The data-segment and obsolete-byte fields +reflect the latest purge sweep and remain `NULL` until a sweep has run. +`checkpoint_repair_plan` describes the query's available repair bounds, while +the disposition and denial columns report what the most recent repair actually +did. + +For operational guidance, see [Monitoring live views](/docs/concepts/live-views/#monitoring). **Examples:** diff --git a/documentation/query/sql/create-live-view.md b/documentation/query/sql/create-live-view.md index 69746bc8c3..fe7247cc79 100644 --- a/documentation/query/sql/create-live-view.md +++ b/documentation/query/sql/create-live-view.md @@ -151,9 +151,16 @@ FROM trades; ## Anchored windows -An anchored window resets its cumulative aggregate on a boundary. Declare it in a -named `WINDOW` with either the `ANCHOR DAILY` shorthand or an -`ANCHOR EXPRESSION` clause: +An anchored window resets its functions on a boundary. Declare it in a named +`WINDOW` with one of these forms: + +```questdb-sql +ANCHOR DAILY 'HH:MM' [ 'timezone' ] +ANCHOR EXPRESSION expression +``` + +`ANCHOR DAILY` requires a quoted 24-hour time. An optional IANA time zone makes +the reset follow local civil time; without one, the boundary is in UTC. ```questdb-sql title="Cumulative daily volume, reset each day" CREATE LIVE VIEW trades_daily_volume @@ -163,9 +170,16 @@ AS SELECT timestamp, symbol, sum(amount) OVER w AS cumulative_volume FROM trades -WINDOW w AS (PARTITION BY symbol ORDER BY timestamp ANCHOR DAILY); +WINDOW w AS ( + PARTITION BY symbol + ORDER BY timestamp + ANCHOR DAILY '00:00' +); ``` +For example, `ANCHOR DAILY '09:30' 'America/New_York'` resets at the New York +market open and follows daylight-saving transitions. + ```questdb-sql title="Anchor on an arbitrary expression" CREATE LIVE VIEW trades_hourly_volume FLUSH EVERY 1s @@ -181,9 +195,16 @@ WINDOW w AS ( ); ``` -An anchored window must be partitioned, must `ORDER BY` the designated timestamp -ascending, cannot use a bounded frame, and its anchor expression must be -deterministic. +An anchored window must: + +- Be a named window; `ANCHOR` is not supported in an inline `OVER (...)`. +- Use `PARTITION BY` with base-table columns directly. +- `ORDER BY` the designated timestamp ascending. +- Use the default unbounded frame; `ANCHOR` cannot be combined with a bounded + `ROWS` or `RANGE` frame. + +A live view supports at most one anchored window. An `ANCHOR EXPRESSION` must be +deterministic, non-constant, and return `TIMESTAMP`, `LONG`, or `INT`. ## Query constraints @@ -191,12 +212,21 @@ The view query is validated at creation time and must: - Read a single WAL-backed base table that has a designated timestamp. No JOINs, subqueries, or CTEs. -- Contain [window functions](/docs/query/functions/window-functions/overview/) that can be - maintained incrementally (see +- Contain [window functions](/docs/query/functions/window-functions/overview/) + that can be maintained incrementally (see [supported functions](/docs/concepts/live-views/#supported-window-functions)). -- Give every window function a `PARTITION BY` clause. -- Not use `SAMPLE BY`, `GROUP BY`, a top-level `ORDER BY`, or `LIMIT` in the view - query. The `ORDER BY` inside a window's `OVER (...)` is required and allowed. +- Give every stateful window function a `PARTITION BY` clause. +- Use a bounded `ROWS` or `RANGE` frame, or a named anchored window. Ranking + functions (`row_number`, `rank`, and `dense_rank`) must be anchored. Frames + starting at `UNBOUNDED PRECEDING` are rejected except for stateless + `last_value` shapes. +- Not use `SAMPLE BY`, `GROUP BY`, a top-level `ORDER BY`, or `LIMIT` in the + view query. The `ORDER BY` inside a window's `OVER (...)` is required and + allowed. +- List output columns explicitly; wildcard projections such as `SELECT *` are + not allowed. +- Not filter on the base table's designated timestamp. Other deterministic + `WHERE` predicates are supported. - Not use non-deterministic functions such as `now()`, `sysdate()`, `systimestamp()`, or `rnd_*()`. - Not read another live view. @@ -284,26 +314,20 @@ OWNED BY analysts; ## Errors -| Error | Cause | -| ----- | ----- | -| `live views are disabled` | Live-view support is turned off (`cairo.live.view.enabled=false`) | -| `live view already exists` | A live view of this name exists and `IF NOT EXISTS` was not specified | -| `table or view with the requested name already exists` | The name is taken by a table, view, or materialized view | -| `live view FLUSH EVERY must be at least 100ms` | The `FLUSH EVERY` interval is below the minimum | -| `live view select must be a simple scan of a single WAL base table; joins, subqueries, GROUP BY, ORDER BY and LIMIT are not supported yet` | The view query is not a simple scan of one base table | -| `base table must be a WAL table` | The base object is a non-WAL table or a regular view | -| `live views are not allowed as base tables in V1` | The base object is another live view | -| `live view base table must have a designated timestamp` | The base table has no designated timestamp | -| `non-deterministic function cannot be used in materialized view` | The query uses `now()`, `rnd_*()`, or a similar non-deterministic function | -| `permission denied` | Missing required permission (Enterprise) | - -:::note - -The non-determinism check is the same guard materialized views use, so its error -message names "materialized view" even when it is raised for a live view. The -rule and its effect are identical for both view types. - -::: +| Error | Cause | +| ------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------- | +| `live views are disabled` | Live-view support is turned off (`cairo.live.view.enabled=false`) | +| `live view already exists` | A live view of this name exists and `IF NOT EXISTS` was not specified | +| `table or view with the requested name already exists` | The name is taken by a table, view, or materialized view | +| `live view FLUSH EVERY must be at least 100ms` | The `FLUSH EVERY` interval is below the minimum | +| `live view select must be a simple scan of a single WAL base table; joins, subqueries, GROUP BY, ORDER BY and LIMIT are not supported yet` | The view query is not a simple scan of one base table | +| `base table must be a WAL table` | The base object is a non-WAL table or a regular view | +| `live views are not allowed as base tables in V1` | The base object is another live view | +| `live view base table must have a designated timestamp` | The base table has no designated timestamp | +| `wildcard column select is not allowed in live view queries` | The top-level projection contains `*` | +| `live view unbounded window must have an ANCHOR clause` | A stateful partitioned window uses the default unbounded frame without an anchor | +| `non-deterministic function cannot be used in live view` | The query uses `now()`, `rnd_*()`, or a similar non-deterministic function | +| `permission denied` | Missing required permission (Enterprise) | ## See also From 489b9a6682e077e205553db30bd5984d8948c4ae Mon Sep 17 00:00:00 2001 From: javier Date: Wed, 5 Aug 2026 13:17:02 +0200 Subject: [PATCH 8/8] Describe live views as beta and compare view types Reword the live view note to state the feature ships as beta, and repeat the note on the CREATE LIVE VIEW reference page. Extend the views comparison table with a live view column and add guidance on when to choose a live view over a view or materialized view. --- documentation/concepts/live-views.md | 2 +- documentation/concepts/views.md | 30 ++++++++++++++------- documentation/query/sql/create-live-view.md | 9 +++++++ 3 files changed, 30 insertions(+), 11 deletions(-) diff --git a/documentation/concepts/live-views.md b/documentation/concepts/live-views.md index 5d49e26e4a..008c125ad2 100644 --- a/documentation/concepts/live-views.md +++ b/documentation/concepts/live-views.md @@ -20,7 +20,7 @@ of rows on each query. :::note -Live views are a new feature. The supported SQL surface is deliberately narrow +Live views are currently released as beta. The supported SQL surface is deliberately narrow in this first version. See [Limitations](#limitations) for the shapes that are rejected at creation time. diff --git a/documentation/concepts/views.md b/documentation/concepts/views.md index fdb136ae0e..411b38ff65 100644 --- a/documentation/concepts/views.md +++ b/documentation/concepts/views.md @@ -315,17 +315,17 @@ SELECT table_name, table_type FROM tables() | `M` | Materialized view | | `L` | [Live view](/docs/concepts/live-views/) | -## Views vs materialized views +## Views vs materialized views vs live views Understanding when to use each type is important for performance: -| Feature | View | Materialized View | -| ------- | ---- | ----------------- | -| Data storage | None (virtual) | Physical storage | -| Query execution | On every access | Pre-computed | -| Data freshness | Always current | Depends on refresh | -| Performance | Query-time cost | Read-time benefit | -| Storage cost | Zero | Proportional to result size | +| Feature | View | Materialized View | Live View | +| ------- | ---- | ----------------- | --------- | +| Data storage | None (virtual) | Physical storage | Memory + Physical | +| Query execution | On every access | Pre-computed | Pre-computed | +| Data freshness | Always current | Depends on refresh | Very low latency if query is served from the `IN MEMORY` tier, up to one `FLUSH EVERY` interval if from disk tier | +| Performance | Query-time cost | Read-time benefit | Read-time benefit | +| Storage cost | Zero | Proportional to result size | Memory + Proportional to result size | ### When to use views @@ -342,9 +342,18 @@ Understanding when to use each type is important for performance: - Dashboard queries that run repeatedly - Historical summaries that don't need real-time accuracy + For detailed comparisons and examples, see [Materialized Views](/docs/concepts/materialized-views/). +### When to use live views + +Use a [live view](/docs/concepts/live-views/) instead when you need to +incrementally maintain a row-per-input window computation, such as a moving +average, running total, or ranking. + +The output of a live view is one row per base row. + ## Security with views Views provide a security boundary between users and underlying data. @@ -461,7 +470,7 @@ EXPLAIN SELECT * FROM my_view WHERE symbol = 'AAPL' - Use indexed columns in filters for best performance - Use parameterized views for common filter patterns - Avoid deeply nested view hierarchies (>3-4 levels) for maintainability -- Consider materialized views for expensive aggregations that run frequently +- Consider materialized views or live views for expensive aggregations that run frequently ## Limitations @@ -484,5 +493,6 @@ EXPLAIN SELECT * FROM my_view WHERE symbol = 'AAPL' - [`DROP VIEW`](/docs/query/sql/drop-view/): Remove a view - **Related Concepts** - - [Materialized Views](/docs/concepts/materialized-views/): Pre-computed query results + - [Materialized Views](/docs/concepts/materialized-views/): Incrementally maintained `SAMPLE BY` aggregates + - [Live views](/docs/concepts/live-views/): Incrementally maintained row-per-input window-function results - [DECLARE](/docs/query/sql/declare/): Parameter declaration for views diff --git a/documentation/query/sql/create-live-view.md b/documentation/query/sql/create-live-view.md index fe7247cc79..86cf349a4f 100644 --- a/documentation/query/sql/create-live-view.md +++ b/documentation/query/sql/create-live-view.md @@ -9,6 +9,15 @@ Creates a live view that incrementally maintains the result of a window-function query over a single base table and can be queried like a regular table. For a conceptual overview, see [Live views](/docs/concepts/live-views/). +:::note + +Live views are currently released as beta. The supported SQL surface is deliberately narrow +in this first version. See +[Limitations](/docs/concepts/live-views/#limitations) for the shapes that are +rejected at creation time. + +::: + ## Syntax ```questdb-sql title="CREATE LIVE VIEW"