Skip to content

Commit a2b7d44

Browse files
committed
refactor(storage): specialize canonical reads to body spans
Read tool projection ranges directly from one immutable body and remove unused cross-header streaming. Share captured source validation and range dispatch through CanonicalRead on the original SQLite worker. Cover original and malformed canonical artifact tokens through the real capture path, including continued reads after malformed-token rejection. Generated-by: Codex
1 parent e93efbc commit a2b7d44

11 files changed

Lines changed: 195 additions & 298 deletions

File tree

‎platform/event-log/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -73,7 +73,7 @@ Canonical databases use UTF-8. Successful tool results store their original `mod
7373

7474
Bodies are directly seekable raw bytes, shared by SHA-256 of their original token with exact byte comparison on a repeated identity. Reads validate logical lengths and selected ranges. Framed compression was evaluated against raw storage and rejected because it repeatedly slowed supported window and sequential reads; it is not a persisted format. Space savings come from compact directories and sharing identical projection tokens, not from compressing unique bodies.
7575

76-
Complete-event JSON queries join the referenced body into the selected SQL row within the original snapshot and reconstruct the original bytes. Typed prefix readers order headers and body references, keeping large payloads out of SQLite's sort records. One prepared primary-key lookup resolves each referenced body on the same connection and transaction; inline events require no body lookup. Readers decode the borrowed header and original projection bytes directly on the original SQLite worker. They avoid copying the body into a temporary string and finish decoding before advancing either row. The existing event/byte preflight and transaction cover the complete result; no SQLite borrow or unfinished decoder leaves that worker. Typed headers reuse fact field types and defaults, ignore unknown metadata, and verify that the projection placeholder is exactly at its stored offset. Root evidence hashes the three original byte slices as one length-prefixed JSON field. Metadata predicates use the header and retain their indexes. Semantic spans retain original canonical coordinates; tool spans map those logical coordinates to header/body ranges, while inline model parts retain physical SQLite BLOB bounds. Resolution stays on the original SQLite worker. Hot range reads use its existing prepared-statement cache. Body references follow the existing history-owner collection proof; clearing or deleting the final reference removes its body in the same transaction. The event row publishes the header, logical length, offset and reference atomically; live references cannot be redirected. Neither a body digest nor a derived directory grants access or extends lifetime. Derived archive placeholders and other fragment bodies keep their chunked storage.
76+
Complete-event JSON queries join the referenced body into the selected SQL row within the original snapshot and reconstruct the original bytes. Typed prefix readers order headers and body references, keeping large payloads out of SQLite's sort records. One prepared primary-key lookup resolves each referenced body on the same connection and transaction; inline events require no body lookup. Readers decode the borrowed header and original projection bytes directly on the original SQLite worker. They avoid copying the body into a temporary string and finish decoding before advancing either row. The existing event/byte preflight and transaction cover the complete result; no SQLite borrow or unfinished decoder leaves that worker. Typed headers reuse fact field types and defaults, ignore unknown metadata, and verify that the projection placeholder is exactly at its stored offset. Root evidence hashes the three original byte slices as one length-prefixed JSON field. Metadata predicates use the header and retain their indexes. Semantic spans retain original canonical coordinates; tool spans translate those logical coordinates into offsets within their immutable body, while inline model parts retain physical SQLite BLOB bounds. Resolution stays on the original SQLite worker. Hot range reads use its existing prepared-statement cache. Body references follow the existing history-owner collection proof; clearing or deleting the final reference removes its body in the same transaction. The event row publishes the header, logical length, offset and reference atomically; live references cannot be redirected. Neither a body digest nor a derived directory grants access or extends lifetime. Derived archive placeholders and other fragment bodies keep their chunked storage.
7777

7878
Semantic row, payload and child directories are finite derived descriptors encoded with Postcard. Writers encode them once in the publication transaction. Pages copy their stored bytes alongside fixed little-endian locator fields into the caller's charged allocation; consumers decode the original typed descriptors without navigating whole bodies. Original byte positions and leaf commitments remain unchanged. Directory encoding never supplies a canonical fact or source identity. The SQL migration converts former JSON descriptors atomically; its conversion functions exist only during migration.
7979

‎platform/event-log/README.zh-CN.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -67,7 +67,7 @@ Control 中的调用意图、项目登记、workspace attachment、Session 路
6767

6868
正文保存为可直接 seek 的原始字节,按原始 token 的 SHA-256 共享,重复身份还需逐字节核对。读取校验逻辑长度与选中区间。对照实验中,分帧压缩稳定拖慢了受支持的窗口和顺序读取,因此未作为持久化格式保留。空间收益来自紧凑目录与相同投影 token 的共享,不能把压缩实验中独有正文的压缩率套到最终实现。
6969

70-
完整事件的 JSON 查询在原快照内的同一 SQL 行 JOIN 引用的正文,并重建原始字节。类型化前缀 reader 只排列 header 和正文引用,避免大正文进入 SQLite 排序记录;随后在同一连接、同一事务内复用一条 prepared 主键查询读取引用的正文,内联事件无需查询正文。Reader 在原 SQLite worker 上直接解码借用的 header 和原始投影字节,避免把正文复制到临时字符串中,并在推进任一行之前完成解码。原有事件/字节预检和事务覆盖完整结果;SQLite 借用和未完成的解码都不会离开该 worker。Header 复用事实字段的类型和默认值、忽略未知元数据,并验证投影占位符恰好位于已存偏移。Root 证据把原始三段字节作为一个带长度前缀的 JSON 字段计算摘要;元数据谓词继续读取 header 并使用原索引。语义区间保留原规范坐标,工具区间映射至实际 header/body,内联 model part 则保留 SQLite BLOB 的物理边界。读取仍由原 SQLite worker 执行。热区间读取复用其已有 statement cache。正文引用的存活由既有历史所有者回收证明决定,最后一个引用清除或删除时,在同一事务删除正文。事件行原子发布 header、逻辑长度、偏移和引用,存活引用不可重定向;正文摘要和派生目录都不授予访问权,也不延长存活期。归档占位内容和其他片段正文继续使用分块存储。
70+
完整事件的 JSON 查询在原快照内的同一 SQL 行 JOIN 引用的正文,并重建原始字节。类型化前缀 reader 只排列 header 和正文引用,避免大正文进入 SQLite 排序记录;随后在同一连接、同一事务内复用一条 prepared 主键查询读取引用的正文,内联事件无需查询正文。Reader 在原 SQLite worker 上直接解码借用的 header 和原始投影字节,避免把正文复制到临时字符串中,并在推进任一行之前完成解码。原有事件/字节预检和事务覆盖完整结果;SQLite 借用和未完成的解码都不会离开该 worker。Header 复用事实字段的类型和默认值、忽略未知元数据,并验证投影占位符恰好位于已存偏移。Root 证据把原始三段字节作为一个带长度前缀的 JSON 字段计算摘要;元数据谓词继续读取 header 并使用原索引。语义区间保留原规范坐标,工具区间把原规范坐标换算为不可变正文内部的偏移,内联 model part 则保留 SQLite BLOB 的物理边界。读取仍由原 SQLite worker 执行。热区间读取复用其已有 statement cache。正文引用的存活由既有历史所有者回收证明决定,最后一个引用清除或删除时,在同一事务删除正文。事件行原子发布 header、逻辑长度、偏移和引用,存活引用不可重定向;正文摘要和派生目录都不授予访问权,也不延长存活期。归档占位内容和其他片段正文继续使用分块存储。
7171

7272
语义行、payload 和子项目录是有限的派生描述符,采用 Postcard 编码。writer 在发布事务内编码一次;分页把已存字节和固定小端序定位字段直接复制到调用者计费的缓冲区,消费端恢复原有类型,无需读取整个正文。原始字节位置与叶子摘要保持不变,目录编码不参与规范事实或来源身份。SQL 迁移在同一事务内转换原 JSON 描述符;转换函数仅在迁移期间存在。
7373

‎platform/event-log/src/canonical.rs‎

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,5 @@ mod write;
2727

2828
pub(crate) use read::{COLUMNS, JOINS, decode_event_on_handle, json, json_on_handle, one};
2929
pub(crate) use recipe::projection;
30-
pub(crate) use source::{
31-
SPAN_COLUMNS, SpanMetadata, with_span, with_span_metadata, with_span_on_row,
32-
};
30+
pub(crate) use source::{SPAN_COLUMNS, SpanMetadata, with_span, with_span_metadata};
3331
pub(crate) use write::prepare;

‎platform/event-log/src/canonical/migration.rs‎

Lines changed: 13 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -210,20 +210,29 @@ mod tests {
210210
.unwrap()
211211
.unwrap();
212212
let ranges = [
213-
0..64,
214-
(source.offset - 2)..(source.offset + 32),
215-
(source.offset + source.json.len() - 2)..(source.offset + source.json.len() + 4),
216-
(original.len() - 64)..original.len(),
213+
source.offset..source.offset + 64,
214+
source.offset + 32..source.offset + 96,
215+
source.offset + source.json.len() - 64..source.offset + source.json.len(),
216+
source.offset + source.json.len()..source.offset + source.json.len(),
217217
];
218218
let expected_ranges: Vec<_> = ranges
219219
.iter()
220220
.map(|range| original.as_bytes()[range.clone()].to_vec())
221221
.collect();
222+
let outside_body = [source.offset - 1, source.offset + source.json.len()];
222223
crate::connection::worker_query(&mut db, move |worker| {
223224
Box::pin(async move {
224225
// SAFETY: the callback exclusively owns this original SQLite worker.
225226
let raw =
226227
unsafe { rusqlite::Connection::from_handle(worker.as_raw_handle().as_ptr()) }?;
228+
for offset in outside_body {
229+
assert!(
230+
crate::canonical::with_span::<()>(&raw, 1, offset as u64, 1, |_| {
231+
panic!("tool range must remain inside its canonical body");
232+
})
233+
.is_err()
234+
);
235+
}
227236
for (range, expected) in ranges.into_iter().zip(expected_ranges) {
228237
let actual = crate::canonical::with_span(
229238
&raw,

‎platform/event-log/src/canonical/recipe.rs‎

Lines changed: 4 additions & 79 deletions
Original file line numberDiff line numberDiff line change
@@ -17,11 +17,11 @@
1717
* under the License.
1818
*/
1919

20-
//! Lossless canonical event recipes and byte-range mapping.
20+
//! Lossless canonical event recipes over a header and its optional body slot.
2121
use crate::StoreError;
2222
use serde::Deserialize;
2323
use serde_json::value::RawValue;
24-
use std::{borrow::Cow, ops::Range};
24+
use std::borrow::Cow;
2525

2626
fn invalid(reason: &'static str) -> StoreError {
2727
StoreError::Sqlite(rusqlite::Error::InvalidParameterName(reason.into()))
@@ -124,12 +124,7 @@ pub(crate) struct Slot {
124124
pub offset: u64,
125125
pub bytes: u64,
126126
}
127-
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
128-
pub(crate) enum Segment {
129-
Header { offset: u64, bytes: u64 },
130-
Body { offset: u64, bytes: u64 },
131-
}
132-
/// Logical positions map to physical spans without loading the header body.
127+
/// The logical event size and optional body slot determine the stored header size.
133128
#[derive(Clone, Copy, Debug)]
134129
pub(crate) struct Layout {
135130
bytes: u64,
@@ -159,50 +154,6 @@ impl Layout {
159154
None => self.bytes,
160155
}
161156
}
162-
/// A logical byte range touches at most the prefix, body, and suffix.
163-
pub fn segments(&self, range: Range<u64>) -> Result<[Option<Segment>; 3], StoreError> {
164-
if range.start > range.end || range.end > self.bytes {
165-
return Err(invalid("canonical byte range is outside its event"));
166-
}
167-
let mut segments = [None; 3];
168-
if range.is_empty() {
169-
return Ok(segments);
170-
}
171-
let Some(slot) = self.slot else {
172-
segments[0] = Some(Segment::Header {
173-
offset: range.start,
174-
bytes: range.end - range.start,
175-
});
176-
return Ok(segments);
177-
};
178-
let body_end = slot
179-
.offset
180-
.checked_add(slot.bytes)
181-
.ok_or_else(|| invalid("canonical slot range overflows"))?;
182-
let prefix_end = range.end.min(slot.offset);
183-
if range.start < prefix_end {
184-
segments[0] = Some(Segment::Header {
185-
offset: range.start,
186-
bytes: prefix_end - range.start,
187-
});
188-
}
189-
let body_start = range.start.max(slot.offset);
190-
let body_limit = range.end.min(body_end);
191-
if body_start < body_limit {
192-
segments[1] = Some(Segment::Body {
193-
offset: body_start - slot.offset,
194-
bytes: body_limit - body_start,
195-
});
196-
}
197-
let suffix_start = range.start.max(body_end);
198-
if suffix_start < range.end {
199-
segments[2] = Some(Segment::Header {
200-
offset: suffix_start - body_end + slot.offset + 4,
201-
bytes: range.end - suffix_start,
202-
});
203-
}
204-
Ok(segments)
205-
}
206157
}
207158

208159
/// The header and optional typed body slot describe exactly one event.
@@ -357,7 +308,7 @@ mod tests {
357308
}
358309
}
359310
#[test]
360-
fn extraction_and_every_byte_range_preserve_original_bytes() {
311+
fn extraction_and_reconstruction_preserve_original_bytes() {
361312
let projection = projection("tool_settled", ORIGINAL).unwrap().unwrap();
362313
let header = projection.header(ORIGINAL).unwrap();
363314
let recipe = Recipe::new(
@@ -375,31 +326,6 @@ mod tests {
375326
.unwrap(),
376327
ORIGINAL
377328
);
378-
for start in 0..=ORIGINAL.len() {
379-
for end in start..=ORIGINAL.len() {
380-
let mut result = Vec::new();
381-
for segment in recipe
382-
.layout
383-
.segments(start as u64..end as u64)
384-
.unwrap()
385-
.into_iter()
386-
.flatten()
387-
{
388-
let (source, offset, bytes) = match segment {
389-
Segment::Header { offset, bytes } => (header.as_bytes(), offset, bytes),
390-
Segment::Body { offset, bytes } => {
391-
(projection.json.as_bytes(), offset, bytes)
392-
}
393-
};
394-
result.extend_from_slice(&source[offset as usize..(offset + bytes) as usize]);
395-
}
396-
assert_eq!(
397-
result,
398-
&ORIGINAL.as_bytes()[start..end],
399-
"range {start}..{end}"
400-
);
401-
}
402-
}
403329
}
404330
#[test]
405331
fn escaped_success_tag_preserves_its_original_encoding() {
@@ -437,7 +363,6 @@ mod tests {
437363
let recipe = Recipe::new("null", 4, None).unwrap();
438364
assert_eq!(recipe.reconstruct(None).unwrap(), "null");
439365
assert!(recipe.reconstruct(Some(b"null")).is_err());
440-
assert!(recipe.layout.segments(0..5).is_err());
441366
}
442367
#[test]
443368
fn invalid_recipes_do_not_silently_become_inline_events() {

0 commit comments

Comments
 (0)