Skip to content

Commit 3b5dcec

Browse files
committed
refactor(storage): decode canonical prefixes on the SQLite worker
Decode borrowed headers and original projection slices before advancing each SQLite row. Remove the temporary body string, transfer queue and independent decoder while preserving the original snapshot, budgets and raw evidence. Align the cleanup scheduling test with upstream's readable Root contract, retaining sibling drainage, confirmed outcomes and failed shutdown evidence. Generated-by: Codex
1 parent c7fe107 commit 3b5dcec

8 files changed

Lines changed: 38 additions & 273 deletions

File tree

‎platform/event-log/README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -73,7 +73,7 @@ Canonical databases use UTF-8. Successful tool results store their original `mod
7373

7474
Bodies are directly seekable raw bytes, shared by SHA-256 of their original token with exact byte comparison on a repeated identity. Reads validate logical lengths and selected ranges. Framed compression was evaluated against raw storage and rejected because it repeatedly slowed supported window and sequential reads; it is not a persisted format. Space savings come from compact directories and sharing identical projection tokens, not from compressing unique bodies.
7575

76-
Complete-event queries join the referenced body into the selected SQL row within the original snapshot. JSON readers reconstruct the original bytes; typed prefix readers prepare the header and original projection bytes on the SQLite worker, then decode owned values on a joined blocking task while reading continues. They avoid a full temporary JSON string and keep the original transaction until both stages finish, including error and panic paths. The transfer contains only the prefix admitted by the existing event/byte budgets; no SQLite borrow crosses to the decoder. Typed headers reuse fact field types and defaults, ignore unknown metadata, and verify that the projection placeholder is exactly at its stored offset. Root evidence hashes the three original byte slices as one length-prefixed JSON field. Metadata predicates use the header and retain their indexes. Semantic spans retain original canonical coordinates; tool spans map those logical coordinates to header/body ranges, while inline model parts retain physical SQLite BLOB bounds. Resolution stays on the original SQLite worker. Hot range reads use its existing prepared-statement cache. Body references follow the existing history-owner collection proof; clearing or deleting the final reference removes its body in the same transaction. The event row publishes the header, logical length, offset and reference atomically; live references cannot be redirected. Neither a body digest nor a derived directory grants access or extends lifetime. Derived archive placeholders and other fragment bodies keep their chunked storage.
76+
Complete-event queries join the referenced body into the selected SQL row within the original snapshot. JSON readers reconstruct the original bytes; typed prefix readers decode the borrowed header and original projection bytes directly on the original SQLite worker. They avoid copying the body into a temporary string and finish decoding before advancing its row. The existing event/byte preflight and transaction cover the complete result; no SQLite borrow or unfinished decoder leaves that worker. Typed headers reuse fact field types and defaults, ignore unknown metadata, and verify that the projection placeholder is exactly at its stored offset. Root evidence hashes the three original byte slices as one length-prefixed JSON field. Metadata predicates use the header and retain their indexes. Semantic spans retain original canonical coordinates; tool spans map those logical coordinates to header/body ranges, while inline model parts retain physical SQLite BLOB bounds. Resolution stays on the original SQLite worker. Hot range reads use its existing prepared-statement cache. Body references follow the existing history-owner collection proof; clearing or deleting the final reference removes its body in the same transaction. The event row publishes the header, logical length, offset and reference atomically; live references cannot be redirected. Neither a body digest nor a derived directory grants access or extends lifetime. Derived archive placeholders and other fragment bodies keep their chunked storage.
7777

7878
Semantic row, payload and child directories are finite derived descriptors encoded with Postcard. Writers encode them once in the publication transaction. Pages copy their stored bytes alongside fixed little-endian locator fields into the caller's charged allocation; consumers decode the original typed descriptors without navigating whole bodies. Original byte positions and leaf commitments remain unchanged. Directory encoding never supplies a canonical fact or source identity. The SQL migration converts former JSON descriptors atomically; its conversion functions exist only during migration.
7979

‎platform/event-log/README.zh-CN.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -67,7 +67,7 @@ Control 中的调用意图、项目登记、workspace attachment、Session 路
6767

6868
正文保存为可直接 seek 的原始字节,按原始 token 的 SHA-256 共享,重复身份还需逐字节核对。读取校验逻辑长度与选中区间。对照实验中,分帧压缩稳定拖慢了受支持的窗口和顺序读取,因此未作为持久化格式保留。空间收益来自紧凑目录与相同投影 token 的共享,不能把压缩实验中独有正文的压缩率套到最终实现。
6969

70-
完整事件查询在原快照内的同一 SQL 行 JOIN 引用的正文。JSON reader 重建原始字节;类型化前缀 reader 在原 SQLite worker 上准备 header 和原始投影字节,由等待完成的阻塞任务解码已拥有的数据,同时继续读取。它避免构造完整临时 JSON 字符串;包括错误和 panic 在内,都等待两段结束后才结束原事务。传递范围仅限已通过原有事件/字节预算的前缀,不把 SQLite 借用交给解码任务。Header 复用事实字段的类型和默认值、忽略未知元数据,并验证投影占位符恰好位于已存偏移。Root 证据把原始三段字节作为一个带长度前缀的 JSON 字段计算摘要;元数据谓词继续读取 header 并使用原索引。语义区间保留原规范坐标,工具区间映射至实际 header/body,内联 model part 则保留 SQLite BLOB 的物理边界。读取仍由原 SQLite worker 执行。热区间读取复用其已有 statement cache。正文引用的存活由既有历史所有者回收证明决定,最后一个引用清除或删除时,在同一事务删除正文。事件行原子发布 header、逻辑长度、偏移和引用,存活引用不可重定向;正文摘要和派生目录都不授予访问权,也不延长存活期。归档占位内容和其他片段正文继续使用分块存储。
70+
完整事件查询在原快照内的同一 SQL 行 JOIN 引用的正文。JSON reader 重建原始字节;类型化前缀 reader 在原 SQLite worker 上直接解码借用的 header 和原始投影字节,避免把正文复制到临时字符串中,并在推进当前行之前完成解码。原有事件/字节预检和事务覆盖完整结果;SQLite 借用和未完成的解码都不会离开该 worker。Header 复用事实字段的类型和默认值、忽略未知元数据,并验证投影占位符恰好位于已存偏移。Root 证据把原始三段字节作为一个带长度前缀的 JSON 字段计算摘要;元数据谓词继续读取 header 并使用原索引。语义区间保留原规范坐标,工具区间映射至实际 header/body,内联 model part 则保留 SQLite BLOB 的物理边界。读取仍由原 SQLite worker 执行。热区间读取复用其已有 statement cache。正文引用的存活由既有历史所有者回收证明决定,最后一个引用清除或删除时,在同一事务删除正文。事件行原子发布 header、逻辑长度、偏移和引用,存活引用不可重定向;正文摘要和派生目录都不授予访问权,也不延长存活期。归档占位内容和其他片段正文继续使用分块存储。
7171

7272
语义行、payload 和子项目录是有限的派生描述符,采用 Postcard 编码。writer 在发布事务内编码一次;分页把已存字节和固定小端序定位字段直接复制到调用者计费的缓冲区,消费端恢复原有类型,无需读取整个正文。原始字节位置与叶子摘要保持不变,目录编码不参与规范事实或来源身份。SQL 迁移在同一事务内转换原 JSON 描述符;转换函数仅在迁移期间存在。
7373

‎platform/event-log/src/canonical.rs‎

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -25,8 +25,7 @@ mod recipe;
2525
mod source;
2626
mod write;
2727

28-
pub(crate) use event::PreparedEvent;
29-
pub(crate) use read::{COLUMNS, JOINS, json, json_on_handle, one, prepare_event_on_handle};
28+
pub(crate) use read::{COLUMNS, JOINS, decode_event_on_handle, json, json_on_handle, one};
3029
pub(crate) use recipe::projection;
3130
pub(crate) use source::{
3231
SPAN_COLUMNS, SpanMetadata, with_span, with_span_metadata, with_span_on_row,

‎platform/event-log/src/canonical/event.rs‎

Lines changed: 17 additions & 76 deletions
Original file line numberDiff line numberDiff line change
@@ -53,64 +53,7 @@ struct HeaderOutcome<'a> {
5353
artifacts: Vec<ArtifactEvidence>,
5454
}
5555

56-
pub(crate) struct PreparedEvent(Prepared);
57-
58-
enum Prepared {
59-
Inline(String),
60-
Tool {
61-
header: Box<ToolHeader>,
62-
projection: String,
63-
},
64-
}
65-
struct ToolHeader {
66-
id: String,
67-
recorded_at: SystemTime,
68-
invocation: Invocation,
69-
operation_id: String,
70-
raw: RawToolResultRef,
71-
artifacts: Vec<ArtifactEvidence>,
72-
}
73-
74-
impl PreparedEvent {
75-
pub(super) fn inline(json: &str) -> Self {
76-
Self(Prepared::Inline(json.to_owned()))
77-
}
78-
pub(crate) fn decode(self) -> Result<RuntimeEvent, StoreError> {
79-
match self.0 {
80-
Prepared::Inline(json) => Ok(serde_json::from_str(&json)?),
81-
Prepared::Tool { header, projection } => {
82-
let model_projection: DurableToolProjection = serde_json::from_str(&projection)?;
83-
let ToolHeader {
84-
id,
85-
recorded_at,
86-
invocation,
87-
operation_id,
88-
raw,
89-
artifacts,
90-
} = *header;
91-
Ok(RuntimeEvent {
92-
id,
93-
recorded_at,
94-
invocation,
95-
fact: Fact::ToolSettled {
96-
operation_id,
97-
outcome: ToolOutcome::Succeeded {
98-
raw,
99-
model_projection,
100-
artifacts,
101-
},
102-
},
103-
})
104-
}
105-
}
106-
}
107-
}
108-
109-
pub(super) fn prepare(
110-
header: &str,
111-
offset: usize,
112-
body: &[u8],
113-
) -> Result<PreparedEvent, StoreError> {
56+
pub(super) fn decode(header: &str, offset: usize, body: &[u8]) -> Result<RuntimeEvent, StoreError> {
11457
let Header {
11558
id,
11659
recorded_at,
@@ -142,17 +85,20 @@ pub(super) fn prepare(
14285
}
14386
let body = std::str::from_utf8(body)
14487
.map_err(|error| std::io::Error::new(std::io::ErrorKind::InvalidData, error))?;
145-
Ok(PreparedEvent(Prepared::Tool {
146-
header: Box::new(ToolHeader {
147-
id,
148-
recorded_at,
149-
invocation,
88+
let model_projection: DurableToolProjection = serde_json::from_str(body)?;
89+
Ok(RuntimeEvent {
90+
id,
91+
recorded_at,
92+
invocation,
93+
fact: Fact::ToolSettled {
15094
operation_id,
151-
raw,
152-
artifacts,
153-
}),
154-
projection: body.to_owned(),
155-
}))
95+
outcome: ToolOutcome::Succeeded {
96+
raw,
97+
model_projection,
98+
artifacts,
99+
},
100+
},
101+
})
156102
}
157103

158104
#[cfg(test)]
@@ -205,8 +151,7 @@ mod tests {
205151
}),
206152
)
207153
.unwrap();
208-
let (prepared, parts) = recipe.prepare(Some(projection.json.as_bytes())).unwrap();
209-
let event = prepared.decode().unwrap();
154+
let (event, parts) = recipe.decode(Some(projection.json.as_bytes())).unwrap();
210155
assert_eq!(event, expected);
211156
assert_eq!(parts.concat(), json.as_bytes());
212157
}
@@ -230,7 +175,7 @@ mod tests {
230175
}),
231176
)
232177
.unwrap();
233-
assert!(recipe.prepare(Some(projection.json.as_bytes())).is_err());
178+
assert!(recipe.decode(Some(projection.json.as_bytes())).is_err());
234179
}
235180

236181
#[test]
@@ -252,11 +197,7 @@ mod tests {
252197
] {
253198
let position = changed.find("\"model_projection\": null").unwrap()
254199
+ "\"model_projection\": ".len();
255-
assert!(
256-
prepare(&changed, position, projection.json.as_bytes())
257-
.and_then(PreparedEvent::decode)
258-
.is_err()
259-
);
200+
assert!(decode(&changed, position, projection.json.as_bytes()).is_err());
260201
}
261202
}
262203
}

‎platform/event-log/src/canonical/read.rs‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -126,12 +126,12 @@ pub(crate) fn json(row: &SqliteRow) -> Result<Cow<'_, str>, StoreError> {
126126
pub(crate) fn json_on_handle<'a>(row: &'a rusqlite::Row<'_>) -> Result<Cow<'a, str>, StoreError> {
127127
reconstruct(fields_on_handle(row)?)
128128
}
129-
pub(crate) fn prepare_event_on_handle<'a>(
129+
pub(crate) fn decode_event_on_handle<'a>(
130130
row: &'a rusqlite::Row<'_>,
131131
visit: impl FnOnce([&'a [u8]; 3]),
132-
) -> Result<super::event::PreparedEvent, StoreError> {
132+
) -> Result<maka_agent_facts::event::RuntimeEvent, StoreError> {
133133
let (recipe, body) = material(fields_on_handle(row)?)?;
134-
let (event, parts) = recipe.prepare(body)?;
134+
let (event, parts) = recipe.decode(body)?;
135135
visit(parts);
136136
Ok(event)
137137
}

‎platform/event-log/src/canonical/recipe.rs‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -244,13 +244,13 @@ impl<'a> Recipe<'a> {
244244
Ok(())
245245
})
246246
}
247-
pub fn prepare(
247+
pub fn decode(
248248
&self,
249249
body: Option<&'a [u8]>,
250-
) -> Result<(super::event::PreparedEvent, [&'a [u8]; 3]), StoreError> {
250+
) -> Result<(maka_agent_facts::event::RuntimeEvent, [&'a [u8]; 3]), StoreError> {
251251
match (self.layout.slot, body) {
252252
(None, None) => Ok((
253-
super::event::PreparedEvent::inline(self.header),
253+
serde_json::from_str(self.header)?,
254254
[self.header.as_bytes(), &[], &[]],
255255
)),
256256
(Some(slot), Some(body)) => {
@@ -259,7 +259,7 @@ impl<'a> Recipe<'a> {
259259
}
260260
let position = usize::try_from(slot.offset)
261261
.map_err(|_| invalid("slot offset exceeds memory range"))?;
262-
let event = super::event::prepare(self.header, position, body)?;
262+
let event = super::event::decode(self.header, position, body)?;
263263
Ok((
264264
event,
265265
[

‎platform/event-log/src/prefix.rs‎

Lines changed: 11 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -19,12 +19,11 @@
1919

2020
use maka_agent_facts::event::LogPrefix;
2121
use maka_agent_facts::event::LogScope;
22+
use maka_agent_facts::event::StoredEvent;
2223
use sqlx::Connection;
2324

2425
use crate::{EventLog, StoreError, context::selection::Selection, sequence_number};
2526

26-
mod decode;
27-
2827
impl EventLog {
2928
/// Read one complete bounded execution prefix, or fail without returning a partial
3029
/// history that a caller could accidentally treat as complete. Session-owned
@@ -86,28 +85,31 @@ impl EventLog {
8685
let root_scope = session.is_none();
8786
let (events, root_digest) = if count == 0 {
8887
(Vec::new(), root_scope.then(crate::context::evidence::RootDigest::new))
89-
} else { decode::run(count as usize, |sender| crate::connection::worker_query(&mut transaction, move |worker| Box::pin(async move {
88+
} else { crate::connection::worker_query(&mut transaction, move |worker| Box::pin(async move {
9089
// SAFETY: this callback exclusively owns the original SQLite worker.
9190
// The wrapper and all borrowed rows/statements close before return.
9291
let raw = unsafe { rusqlite::Connection::from_handle(worker.as_raw_handle().as_ptr()) }?;
9392
let result = (|| {
9493
let mut digest = session.is_none().then(crate::context::evidence::RootDigest::new);
94+
let mut events = Vec::new();
9595
let mut statement = raw.prepare(&query)?;
9696
let mut rows = statement.query(rusqlite::params![session, selected_lineage])?;
9797
while let Some(row) = rows.next()? {
9898
let sequence: i64 = row.get("sequence")?;
99-
let prepared = crate::canonical::prepare_event_on_handle(row, |parts| {
99+
let event = crate::canonical::decode_event_on_handle(row, |parts| {
100100
if let Some(digest) = &mut digest { digest.record(sequence, &parts); }
101101
})?;
102-
if sender.send((sequence_number(sequence)?, prepared)).is_err() {
103-
return Ok(decode::Produced::DecoderStopped);
104-
}
102+
events.push(StoredEvent { sequence: sequence_number(sequence)?, event });
103+
}
104+
if events.len() as u64 != sequence_number(count)? {
105+
return Err(std::io::Error::new(std::io::ErrorKind::InvalidData,
106+
"prefix decoder stopped before completing the admitted snapshot").into());
105107
}
106-
Ok(decode::Produced::Complete(digest))
108+
Ok((events, digest))
107109
})();
108110
drop(raw);
109111
result
110-
}))).await? };
112+
})).await? };
111113
let digest = match root_digest {
112114
Some(digest) => digest.finish(high_water)?,
113115
None => crate::context::evidence::selected(

0 commit comments

Comments
 (0)