diff --git a/CHANGELOG.md b/CHANGELOG.md index abc3a7a7..01c53222 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,16 @@ Entries that change an on-disk format or a response shape say so. ## [Unreleased] ### Added +- `taguru extract --source-id ID`, `--date WHEN`, `--tag TAG` (#466 + S1, ADR 0017): bake the promotion runbook's source conventions into + the written batch — the `session:{agent}:{id}` header source (with + the `/{doc}` stem suffix across several documents, collisions + refused), and the passage line's `date` (`YYYY-MM-DD` or epoch + seconds) and `tags`. All three are manifest computation inputs + (`serde(default)` — older manifests keep matching default runs) but + deliberately not checkpoint inputs: a metadata change rewrites the + batch while reusing every cached chunk answer. `--date`/`--tag` with + `--no-passage` is a usage error (metadata rides the passage line). - `taguru extract --coverage` / `TAGURU_EXTRACT_COVERAGE` (#496 S4, ADR 0016): report every sentence that holds a candidate pair (two or more deterministically segmented document names) yet is covered by diff --git a/adr/0017-runbook-metadata-at-extract-time.md b/adr/0017-runbook-metadata-at-extract-time.md new file mode 100644 index 00000000..88460cc7 --- /dev/null +++ b/adr/0017-runbook-metadata-at-extract-time.md @@ -0,0 +1,83 @@ +# 0017. Promotion-runbook metadata at extract time + +- **Status**: Accepted +- **Date**: 2026-08-09 +- **Issue**: #466 (S1) +- **Related**: #465 (the runbook whose conventions these flags encode), + ADR 0011 (why `date` is load-bearing), ADR 0005 (the batch contract + the metadata rides), issue #167 (the passage-line metadata fields) +- **Supersedes**: — / **Superseded by**: — + +Once Accepted, this document's Decision is immutable: a changed decision gets a +new `adr/000N-*.md` that names this one in *Supersedes*, never an edit here. + +## 1. Scope + +How `taguru extract` writes the promotion runbook's source conventions +— session source id, assertion date, topic tags — into the batches it +emits (#466 S1). Out of scope: bundling the promotion sequence itself +(a verb or MCP tool, #466's remaining splits), and the SDK producers. + +## 2. Context + +The 2026-08-08 runbook rehearsal (#466's gate record) found the single +most mechanical step of every promotion: extract knows only document +paths, so the operator rewrites each emitted batch's `source` to the +runbook's `session:{agent}:{id}`, and hand-adds the `date` and `tags` +the scratch conventions require — every time, purely mechanically, +with a text editor against a generated file. The import wire format +has carried all three fields since #167; extract simply never had a +way to be told them. + +## 3. Decision + +**`--source-id ID`, `--date WHEN`, and `--tag TAG` (repeatable) bake +the runbook's conventions into the written batch. All three are +manifest computation inputs and none is a checkpoint input.** + +1. **`--source-id` replaces the header's source**: verbatim for a + single document; with several, each document gets `ID/{file stem}` + — the runbook's own `/{doc}` convention, made automatic because + import's retract-then-apply is per source id, and one id covering + two documents would silently fold them. Two documents whose stems + collide fail the second with the reason, before any model call. + The manifest stays keyed by the document PATH — the path names the + input; this names the output. +2. **`--date` and `--tag` ride the passage line**, exactly where the + wire format carries source metadata. Requiring the passage is + therefore enforced as a usage error against `--no-passage`, not a + silent drop — an associations-only source stores no metadata and is + invisible to every windowed read (docs/promotion.html's own + warning). `--date` accepts epoch seconds (the wire unit) or + `YYYY-MM-DD` (what a session note records — that day's UTC + midnight, round-tripped through the rendering direction so a + non-existent date is refused rather than normalized). +3. **Manifest inputs, not checkpoint inputs**: all three are baked + into the emitted file, so a change must rewrite the batch (the + `context`/`description` precedent — a skip would leave the old id + or date in place). But none of them reaches the prompt — the model + is still shown the document path — so cached chunk answers stay + reusable across a metadata change: the rewrite costs zero model + calls for checkpointed units. The fingerprint records the EFFECTIVE + written source (suffix included), so revising the suffix scheme + re-extracts too; `""`/`0`/`[]` are the off values, keeping pre-S1 + manifest entries matching default runs. +4. **Flag-only, no env counterparts**: a session id and its date are + per-invocation values, not deployment settings — the + `--context`/`--description` precedent, so `KNOWN_KEYS` and the + config dialect are untouched. + +## 4. Consequences + +- The runbook's step 2 loses its hand-editing: extract now emits an + import-ready promotion batch directly, and the `#496` controls + (`--vocabulary` for the resolve-first rule, `--coverage` for the + review's mechanical floor) compose with it — the flags are + orthogonal by construction. +- Batches, manifests, and checkpoints from before this change parse + and match unchanged (`serde(default)` on the new manifest fields; + the no-flags batch is byte-for-byte identical). +- Rust-only, like every extract control; the SDK producers inherit the + whole set together in their own follow-up. +- The remaining #466 splits (an MCP promotion tool over the graph + path; a CLI text-path preset) build on this without changes here. diff --git a/docs/extract.html b/docs/extract.html index eebb811f..b5b3b9a3 100644 --- a/docs/extract.html +++ b/docs/extract.html @@ -163,6 +163,18 @@

The CLI shape

usage, latency, parse/validation issues), one per document written (association/alias/duplicate/dropped/ uncovered counts); off by default, ignored under --dry-run +--source-id ID write ID as the batch header's source instead of the + document path — the promotion runbook's + session:{agent}:{id} convention; several documents each + get ID/{file stem}, and a collision fails (import + retracts-then-applies per source id). Changing it + rewrites the batch but reuses cached chunk answers + (ADR 0017) +--date WHEN the session's own date, written on the passage line: + YYYY-MM-DD (UTC midnight) or positive epoch seconds; + needs the passage +--tag TAG tag the batch's source (repeatable, deduplicated), + written on the passage line; needs the passage --context NAME the context every batch file targets --description TEXT attach a create block (used only when the context is absent) --schema FILE a context schema document (see below) — the same shape diff --git a/docs/promotion.html b/docs/promotion.html index d2da0e3a..52c4b490 100644 --- a/docs/promotion.html +++ b/docs/promotion.html @@ -115,9 +115,16 @@

The promotion procedure

scratch has grown.
  • Extract the keepers with taguru extract over the session passages (or hand-write the batch), targeting the permanent context — keeping - the session:{agent}:{id} source ids and the dates. Resolve - spellings against the permanent context first (resolve / - resolve_label): reuse its vocabulary, never fork it.
    • + the session:{agent}:{id} source ids and the dates: + --source-id, --date, and --tag bake all three + conventions into the emitted batch directly (ADR 0017), so no hand-editing of the + generated file remains. Resolve spellings against the permanent context rather than + forking its vocabulary: export it (taguru export --out DIR) and hand the + export to --vocabulary (ADR 0015), which steers the extraction toward + the context's existing spellings and admits them through validation — the manual + resolve/resolve_label pass remains for hand-written + batches. --coverage (ADR 0016) reports what the extraction left behind, + sentence by sentence — the mechanical floor under the review this step owes.
  • Import via POST /import / taguru import — retract-then-apply per source makes re-promoting the same session idempotent.
  • Audit the landing zone: taguru consolidation --context NAME diff --git a/src/clock.rs b/src/clock.rs index 88343ab9..402d4172 100644 --- a/src/clock.rs +++ b/src/clock.rs @@ -30,6 +30,21 @@ pub(crate) fn iso8601_utc(unix_secs: u64) -> String { format!("{y:04}-{m:02}-{d:02}T{hh:02}:{mm:02}:{ss:02}Z") } +/// [`civil_from_days`]'s inverse (Hinnant's `days_from_civil`, same +/// source): (year, month, day) → days since the Unix epoch. Consumed +/// by `extract --date`'s `YYYY-MM-DD` spelling (#466 S1), which +/// round-trips the result through [`iso8601_utc`] to refuse +/// non-existent dates rather than silently normalizing them. +pub(crate) fn days_from_civil(y: i64, m: u32, d: u32) -> i64 { + let y = if m <= 2 { y - 1 } else { y }; + let era = if y >= 0 { y } else { y - 399 } / 400; + let yoe = (y - era * 400) as u64; + let mp = if m > 2 { m - 3 } else { m + 9 } as u64; + let doy = (153 * mp + 2) / 5 + d as u64 - 1; + let doe = yoe * 365 + yoe / 4 - yoe / 100 + doy; + era * 146097 + doe as i64 - 719468 +} + fn civil_from_days(z: i64) -> (i64, u32, u32) { let z = z + 719468; let era = if z >= 0 { z } else { z - 146096 } / 146097; @@ -55,4 +70,16 @@ mod tests { assert_eq!(iso8601_utc(1709164800), "2024-02-29T00:00:00Z"); assert_eq!(iso8601_utc(951868800), "2000-03-01T00:00:00Z"); } + + #[test] + fn days_from_civil_inverts_the_rendering_direction() { + assert_eq!(days_from_civil(1970, 1, 1), 0); + assert_eq!(days_from_civil(2026, 7, 26), 1785057262 / 86400); + assert_eq!(days_from_civil(2024, 2, 29), 1709164800 / 86400); + assert_eq!(days_from_civil(2000, 3, 1), 951868800 / 86400); + assert_eq!(days_from_civil(1969, 12, 31), -1); + // The negative-year era branch (`y - 399`): year 0 is a leap + // year (≡ 2000 mod 400), so -1-03-01 → 0-03-01 spans 366 days. + assert_eq!(days_from_civil(0, 3, 1) - days_from_civil(-1, 3, 1), 366); + } } diff --git a/src/extract.rs b/src/extract.rs index ccc674ff..eaf37245 100644 --- a/src/extract.rs +++ b/src/extract.rs @@ -158,6 +158,8 @@ use vocabulary::{ContextVocabulary, context_names_block, load_vocabulary}; #[cfg(test)] use aggregate::{cross_output_issues, schema_output_issues}; #[cfg(test)] +use args::parse_date; +#[cfg(test)] use candidates::{CANDIDATE_CAP, CANDIDATE_MAX_BYTES}; #[cfg(test)] use chat_client::build_chat_body; @@ -183,6 +185,7 @@ usage: taguru extract [--dry-run] [--force] [--no-passage] [--questions N] [--structured-output MODE] [--max-output-tokens N] [--lossy] [--candidates] [--vocabulary PATH] [--coverage] [--diagnostics-out FILE] [--schema FILE] + [--source-id ID] [--date WHEN] [--tag TAG]... --context NAME [--description TEXT] --out DIR FILE|DIR... Reads documents (.md/.txt; a directory expands to its files, sorted by @@ -280,6 +283,21 @@ chat endpoint: across runs. Default (unset): no sidecar, stdout/ stderr unchanged. Ignored under --dry-run, which calls nothing to record. + --source-id ID write ID as the batch header's source instead of the + document path — the promotion runbook's + session:{agent}:{id} convention (docs/promotion.html). + With several documents, each gets ID/{file stem}; two + documents landing on one source id is an error (import + retracts-then-applies per source id). Changing it + rewrites the batch but reuses cached chunk answers + --date WHEN the session's own date, written on the batch's passage + line (the assertion time windowed reads and the + staleness audit run on): YYYY-MM-DD (UTC midnight) or + positive epoch seconds. Needs the passage + --tag TAG tag the batch's source (repeatable, deduplicated) — + written on the passage line; how a later session finds + its trail via passage search's tags filter. Needs the + passage --context NAME the context every batch file targets --description TEXT add a create block (used only if the context is absent) --schema FILE the target context's schema document (same shape as @@ -718,6 +736,11 @@ pub fn run(args: &[String]) -> i32 { .as_ref() .map(|vocabulary| vocabulary.labels.clone()) .unwrap_or_default(), + source_id: args.source_id, + date: args.date, + tags: args.tags, + multi_document: files.len() > 1, + claimed_source_ids: BTreeMap::new(), claimed: BTreeMap::new(), parallel, lossy, diff --git a/src/extract/args.rs b/src/extract/args.rs index 831f092a..040ccea2 100644 --- a/src/extract/args.rs +++ b/src/extract/args.rs @@ -64,6 +64,17 @@ pub(super) struct Args { pub(super) schema: Option, pub(super) context: String, pub(super) description: Option, + /// #466 S1 (ADR 0017): the promotion runbook's `session:{agent}:{id}` + /// source id, replacing the document path in the written batch + /// header. `None` keeps the path (today's batch, byte for byte). + pub(super) source_id: Option, + /// #466 S1: the session's own date (epoch seconds), emitted on the + /// written batch's passage line — the assertion time every windowed + /// read and the staleness audit run on. `None` emits no field. + pub(super) date: Option, + /// #466 S1: the session's topic tags, emitted on the written + /// batch's passage line. Empty emits no field. + pub(super) tags: Vec, pub(super) out: PathBuf, pub(super) paths: Vec, } @@ -87,6 +98,9 @@ impl Args { let mut schema: Option = None; let mut context: Option = None; let mut description: Option = None; + let mut source_id: Option = None; + let mut date: Option = None; + let mut tags: Vec = Vec::new(); let mut out: Option = None; let mut paths: Vec = Vec::new(); let mut rest = args.iter(); @@ -285,6 +299,90 @@ impl Args { )); } }, + "--source-id" => match rest.next() { + Some(id) if source_id.is_none() && !id.is_empty() => { + source_id = Some(id.clone()); + } + Some(id) if id.is_empty() => { + return Err(crate::config::subcommand_usage_error( + "extract", + "--source-id must not be empty", + )); + } + Some(_) => { + return Err(crate::config::subcommand_usage_error( + "extract", + "--source-id given twice", + )); + } + None => { + return Err(crate::config::subcommand_usage_error( + "extract", + "--source-id needs a source id (session:{agent}:{id})", + )); + } + }, + "--date" => match rest.next() { + Some(_) if date.is_some() => { + return Err(crate::config::subcommand_usage_error( + "extract", + "--date given twice", + )); + } + Some(when) => match parse_date(when) { + Some(seconds) => date = Some(seconds), + None => { + return Err(crate::config::subcommand_usage_error( + "extract", + "--date takes YYYY-MM-DD (UTC midnight) or positive epoch seconds", + )); + } + }, + None => { + return Err(crate::config::subcommand_usage_error( + "extract", + "--date needs a date (YYYY-MM-DD or epoch seconds)", + )); + } + }, + "--tag" => match rest.next() { + Some(tag) if tag.trim().is_empty() => { + return Err(crate::config::subcommand_usage_error( + "extract", + "--tag must not be empty", + )); + } + Some(tag) if tag.len() > crate::api::MAX_TAG_BYTES => { + return Err(crate::config::subcommand_usage_error( + "extract", + &format!( + "tag of {} bytes exceeds the {}-byte cap", + tag.len(), + crate::api::MAX_TAG_BYTES + ), + )); + } + Some(tag) => { + if !tags.contains(tag) { + tags.push(tag.clone()); + } + if tags.len() > crate::api::MAX_TAGS_PER_SOURCE { + return Err(crate::config::subcommand_usage_error( + "extract", + &format!( + "more than {} tags where a source carries at most that many", + crate::api::MAX_TAGS_PER_SOURCE + ), + )); + } + } + None => { + return Err(crate::config::subcommand_usage_error( + "extract", + "--tag needs a tag", + )); + } + }, "--out" => match rest.next() { Some(dir) if out.is_none() => out = Some(PathBuf::from(dir)), Some(_) => { @@ -352,6 +450,30 @@ impl Args { questions would attach to)", )); } + // Source metadata rides the batch's passage line (the import + // wire format has nowhere else to put it — an associations-only + // source stores no metadata and is invisible to every windowed + // read, docs/promotion.html's own warning), so stripping the + // passage while asking for metadata is a contradiction, not a + // silent drop. + if (date.is_some() || !tags.is_empty()) && no_passage { + return Err(crate::config::subcommand_usage_error( + "extract", + "--date/--tag ride the passage line (--no-passage strips it, and an \ + associations-only source stores no metadata)", + )); + } + if let Some(id) = &source_id + && id.len() > MAX_NAME_BYTES + { + return Err(crate::config::subcommand_usage_error( + "extract", + &format!( + "source id of {} bytes exceeds the {MAX_NAME_BYTES}-byte cap", + id.len() + ), + )); + } // TAGURU_CONFIG fallback (issue #248 item 2): --config wins, // but a deployment file baked in via the environment still // applies when it's absent — the same priority serve/health/ @@ -375,12 +497,62 @@ impl Args { schema, context, description, + source_id, + date, + tags, out, paths, }) } } +/// `--date`'s two spellings: bare digits are epoch seconds (the wire +/// format's own unit), `YYYY-MM-DD` is that day's UTC midnight — +/// what a session note actually records. Zero is rejected in both +/// spellings so the manifest can keep 0 as its off sentinel, the +/// `max_output_tokens` precedent. +pub(super) fn parse_date(text: &str) -> Option { + if !text.is_empty() && text.bytes().all(|b| b.is_ascii_digit()) { + return text.parse::().ok().filter(|&seconds| seconds > 0); + } + let mut parts = text.split('-'); + let year = parts.next()?.parse::().ok()?; + let month = parts.next()?.parse::().ok()?; + let day = parts.next()?.parse::().ok()?; + // Separate checks rather than one `||` chain: the round-trip below + // already refuses any out-of-range month/day (it renders as a + // different date), so a chained condition's operator is + // mutation-equivalent noise — and the range checks' real job is + // keeping the civil arithmetic's inputs in-domain (day 0 would + // underflow `d - 1` there). + if parts.next().is_some() { + return None; + } + // The year cap is what keeps the civil arithmetic in-domain: an + // i64-scale year overflows `era * 146097` before the round-trip + // below could refuse it, and a panic on external input is not a + // rejection. Four digits is also simply the YYYY-MM-DD contract. + if !(1..=9999).contains(&year) { + return None; + } + if !(1..=12).contains(&month) { + return None; + } + if !(1..=31).contains(&day) { + return None; + } + let days = crate::clock::days_from_civil(year, month, day); + // Round-trip through the rendering direction: a lexically plausible + // but non-existent date (2026-02-30) normalizes to a different day, + // so refusing the mismatch refuses the typo. + let seconds = u64::try_from(days.checked_mul(86400)?).ok()?; + if crate::clock::iso8601_utc(seconds).starts_with(&format!("{year:04}-{month:02}-{day:02}T")) { + Some(seconds).filter(|&seconds| seconds > 0) + } else { + None + } +} + /// What one document's pipeline concluded; [`run`] only counts these /// into the summary line. pub(super) enum Outcome { diff --git a/src/extract/manifest.rs b/src/extract/manifest.rs index 29e6e81e..8dade583 100644 --- a/src/extract/manifest.rs +++ b/src/extract/manifest.rs @@ -93,6 +93,27 @@ pub(super) struct ManifestEntry { /// computation input. #[serde(default)] pub(super) vocabulary_digest: String, + /// `--source-id`'s EFFECTIVE written source for this document + /// (`""` = off, the path was written) — #466 S1, ADR 0017: baked + /// into the emitted header like `context`/`description`, so a + /// change must rewrite the batch rather than skip with the old id + /// in place. The effective value (suffix included), not the flag, + /// so a revision of the multi-document suffix scheme re-extracts + /// too. Prompt-neutral on purpose: the model is still shown the + /// document path, which is why this field is NOT in the checkpoint + /// fingerprint — cached chunk answers stay reusable across a + /// source-id change. + #[serde(default)] + pub(super) source_id: String, + /// `--date` of the run that wrote this batch (0 = no field + /// emitted): baked into the passage line, same reasoning — and the + /// same checkpoint-fingerprint exemption — as `source_id`. + #[serde(default)] + pub(super) date: u64, + /// `--tag`s of the run that wrote this batch (empty = no field + /// emitted): likewise. + #[serde(default)] + pub(super) tags: Vec, pub(super) output: String, } @@ -130,6 +151,9 @@ impl Manifest { schema_digest: &str, candidates: &str, vocabulary_digest: &str, + source_id: &str, + date: u64, + tags: &[String], ) -> bool { self.documents.get(source).is_some_and(|entry| { entry.sha256 == sha256 @@ -146,6 +170,9 @@ impl Manifest { && entry.schema_digest == schema_digest && entry.candidates == candidates && entry.vocabulary_digest == vocabulary_digest + && entry.source_id == source_id + && entry.date == date + && entry.tags == tags }) } @@ -166,6 +193,9 @@ impl Manifest { schema_digest: &str, candidates: &str, vocabulary_digest: &str, + source_id: &str, + date: u64, + tags: &[String], output: &str, ) { self.documents.insert( @@ -185,6 +215,9 @@ impl Manifest { schema_digest: schema_digest.to_string(), candidates: candidates.to_string(), vocabulary_digest: vocabulary_digest.to_string(), + source_id: source_id.to_string(), + date, + tags: tags.to_vec(), output: output.to_string(), }, ); diff --git a/src/extract/render.rs b/src/extract/render.rs index 5ea94a11..01c24324 100644 --- a/src/extract/render.rs +++ b/src/extract/render.rs @@ -12,6 +12,8 @@ pub(super) fn render_batch( description: Option<&str>, extraction: &Extraction, passage: Option<&str>, + date: Option, + tags: &[String], ) -> String { let mut header = serde_json::json!({ "taguru_batch": 1, @@ -23,7 +25,18 @@ pub(super) fn render_batch( } let mut lines = vec![header.to_string()]; if let Some(text) = passage { - lines.push(serde_json::json!({ "passage": text }).to_string()); + // #466 S1 (ADR 0017): the runbook's source metadata rides the + // passage line, exactly where the import wire format carries it + // (docs/import.html). Absent fields are omitted, keeping the + // no-flags batch byte for byte today's. + let mut line = serde_json::json!({ "passage": text }); + if let Some(date) = date { + line["date"] = serde_json::json!(date); + } + if !tags.is_empty() { + line["tags"] = serde_json::json!(tags); + } + lines.push(line.to_string()); for (paragraph, question) in &extraction.questions { lines.push( serde_json::json!({ "paragraph": paragraph, "question": question }).to_string(), diff --git a/src/extract/run.rs b/src/extract/run.rs index 512f64e9..e6a17d1c 100644 --- a/src/extract/run.rs +++ b/src/extract/run.rs @@ -11,6 +11,28 @@ use super::*; pub(super) struct Run { pub(super) context: String, pub(super) description: Option, + /// `--source-id` (#466 S1, ADR 0017): the promotion runbook's + /// session source id, written into the batch header in place of + /// the document path. `None` = the path, today's batch byte for + /// byte. The MANIFEST stays keyed by the document path either way + /// — the path names the input, this names the output. + pub(super) source_id: Option, + /// `--date` (#466 S1): epoch seconds for the passage line's + /// `date` field (`None` = no field). + pub(super) date: Option, + /// `--tag` (#466 S1): tags for the passage line (empty = no field). + pub(super) tags: Vec, + /// Whether this run extracts more than one document — under + /// `--source-id` that appends the runbook's `/{doc}` suffix + /// (the file stem) so per-source retract-then-apply cannot make + /// two documents silently replace each other. + pub(super) multi_document: bool, + /// Written-source claims (the batch HEADER's source), mirroring + /// `claimed`'s file-name check one level up: two documents whose + /// effective source ids collide would clobber each other at import + /// (retract-then-apply is per source id), so the second one fails + /// here instead. + pub(super) claimed_source_ids: BTreeMap, pub(super) force: bool, pub(super) dry_run: bool, pub(super) no_passage: bool, @@ -102,6 +124,26 @@ impl Run { questions_requested: self.questions > 0, }) } + + /// The source id the batch header carries: the document path + /// (today's behavior), or `--source-id`'s override — verbatim for + /// a single document, with the runbook's `/{doc}` suffix (the file + /// stem) when the run extracts several, since one session id + /// covering two documents would make import's per-source + /// retract-then-apply fold them into one another. + pub(super) fn written_source(&self, path: &Path, source: &str) -> String { + match &self.source_id { + None => source.to_string(), + Some(id) if !self.multi_document => id.clone(), + Some(id) => { + let stem = path + .file_stem() + .map(|stem| stem.to_string_lossy()) + .unwrap_or_default(); + format!("{id}/{stem}") + } + } + } } /// [`Run::extract_chunks`]'s result: either every chunk completed, or a @@ -180,8 +222,34 @@ impl Run { self.claimed.insert(file_name.clone(), source.to_string()); let out_path = self.out.join(&file_name); + let written_source = self.written_source(path, source); + if written_source.len() > MAX_NAME_BYTES { + return Err(format!( + "its source id '{written_source}' is {} bytes, over the \ + {MAX_NAME_BYTES}-byte source cap", + written_source.len() + )); + } + if let Some(other) = self.claimed_source_ids.get(&written_source) { + return Err(format!( + "its source id '{written_source}' collides with '{other}' — import's \ + retract-then-apply is per source id, so one would silently replace the \ + other; rename one of the documents" + )); + } + self.claimed_source_ids + .insert(written_source.clone(), source.to_string()); + let text = read_document(path)?; let hash = sha256_hex(text.as_bytes()); + // The fingerprint's source-id value is the EFFECTIVE written + // source, but only under the flag — "" when off, so pre-S1 + // manifest entries (no field) keep matching default runs. + let source_id_value = if self.source_id.is_some() { + written_source.as_str() + } else { + "" + }; if !self.force && self.manifest.matches( source, @@ -198,6 +266,9 @@ impl Run { &self.schema_digest, candidates_manifest_value(self.candidates), &self.vocabulary_digest, + source_id_value, + self.date.unwrap_or(0), + &self.tags, ) && out_path.is_file() { @@ -328,10 +399,12 @@ impl Run { ); let body = render_batch( &self.context, - source, + &written_source, self.description.as_deref(), &extraction, (!self.no_passage).then_some(text.as_str()), + self.date, + &self.tags, ); if let Err(message) = crate::ingest::parse_batch(Cursor::new(body.as_bytes())) { return Err(format!( @@ -357,6 +430,9 @@ impl Run { &self.schema_digest, candidates_manifest_value(self.candidates), &self.vocabulary_digest, + source_id_value, + self.date.unwrap_or(0), + &self.tags, &file_name, ); // The batch is durably written and manifest-recorded — the diff --git a/src/extract/tests.rs b/src/extract/tests.rs index e0238092..98e87d46 100644 --- a/src/extract/tests.rs +++ b/src/extract/tests.rs @@ -1109,6 +1109,9 @@ fn manifests_reextract_when_the_candidates_mode_changes() { "", candidates_manifest_value(true), "", + "", + 0, + &[], "a.md.jsonl", ); assert!(manifest.matches( @@ -1125,12 +1128,31 @@ fn manifests_reextract_when_the_candidates_mode_changes() { false, "", candidates_manifest_value(true), - "" + "", + "", + 0, + &[] )); // Turning the control off — or a future algorithm revision — is a // computation-input change like any other. assert!(!manifest.matches( - "a.md", "hash-1", "model-1", "sake", 0, false, "", 0, "", 0, false, "", "", "" + "a.md", + "hash-1", + "model-1", + "sake", + 0, + false, + "", + 0, + "", + 0, + false, + "", + "", + "", + "", + 0, + &[] )); // Pre-S2 entries (no field) default to "" and keep matching // default-off runs. @@ -1150,10 +1172,29 @@ fn manifests_reextract_when_the_candidates_mode_changes() { "", "", "", + "", + 0, + &[], "b.md.jsonl", ); assert!(legacy.matches( - "b.md", "hash-2", "model-1", "sake", 0, false, "", 0, "", 0, false, "", "", "" + "b.md", + "hash-2", + "model-1", + "sake", + 0, + false, + "", + 0, + "", + 0, + false, + "", + "", + "", + "", + 0, + &[] )); assert!(!legacy.matches( "b.md", @@ -1169,7 +1210,10 @@ fn manifests_reextract_when_the_candidates_mode_changes() { false, "", candidates_manifest_value(true), - "" + "", + "", + 0, + &[] )); } @@ -1408,16 +1452,67 @@ fn manifests_reextract_when_the_vocabulary_digest_changes() { "", "", "digest-a", + "", + 0, + &[], "a.md.jsonl", ); assert!(manifest.matches( - "a.md", "hash-1", "model-1", "sake", 0, false, "", 0, "", 0, false, "", "", "digest-a" + "a.md", + "hash-1", + "model-1", + "sake", + 0, + false, + "", + 0, + "", + 0, + false, + "", + "", + "digest-a", + "", + 0, + &[] )); assert!(!manifest.matches( - "a.md", "hash-1", "model-1", "sake", 0, false, "", 0, "", 0, false, "", "", "digest-b" + "a.md", + "hash-1", + "model-1", + "sake", + 0, + false, + "", + 0, + "", + 0, + false, + "", + "", + "digest-b", + "", + 0, + &[] )); assert!(!manifest.matches( - "a.md", "hash-1", "model-1", "sake", 0, false, "", 0, "", 0, false, "", "", "" + "a.md", + "hash-1", + "model-1", + "sake", + 0, + false, + "", + 0, + "", + 0, + false, + "", + "", + "", + "", + 0, + &[] )); } @@ -2034,6 +2129,8 @@ fn rendered_batches_pass_the_import_parser() { Some("酒蔵の記憶"), &extraction, Some("一段落目。\n\n二段落目。"), + None, + &[], ); // A passage with newlines still serializes to one line each: // header, passage, question, fact, alias. @@ -2065,7 +2162,7 @@ fn a_stripped_passage_strips_the_paragraph_locators_too() { 0, 2, ); - let body = render_batch("sake", "docs/aomine.md", None, &extraction, None); + let body = render_batch("sake", "docs/aomine.md", None, &extraction, None, None, &[]); assert!( !body.contains("\"paragraph\""), "no passage line, no locators: {body}" @@ -2094,6 +2191,8 @@ fn a_paragraph_survives_extract_through_ingest_into_a_queried_attribution() { Some("配線テスト"), &extraction, Some("一段落目。\n\n二段落目。"), + None, + &[], ); let batch = crate::ingest::parse_batch(Cursor::new(body.as_bytes())) .expect("extract must never emit what import refuses"); @@ -2139,71 +2238,12 @@ fn manifests_skip_only_exact_recomputations() { "", "", "", + "", + 0, + &[], "a.md.jsonl", ); assert!(manifest.matches( - "a.md", "hash-1", "model-1", "sake", 0, false, "", 0, "", 0, false, "", "", "" - )); - assert!(!manifest.matches( - "a.md", "hash-2", "model-1", "sake", 0, false, "", 0, "", 0, false, "", "", "" - )); - assert!(!manifest.matches( - "a.md", "hash-1", "model-2", "sake", 0, false, "", 0, "", 0, false, "", "", "" - )); - assert!(!manifest.matches( - "b.md", "hash-1", "model-1", "sake", 0, false, "", 0, "", 0, false, "", "", "" - )); - // A re-pointed --context must re-extract, not keep files whose - // headers still name the old target. - assert!(!manifest.matches( - "a.md", "hash-1", "model-1", "vats", 0, false, "", 0, "", 0, false, "", "", "" - )); - // Toggling --no-passage changes whether the batch carries the - // source passage at all — a skip would keep the stale shape. - assert!(!manifest.matches( - "a.md", "hash-1", "model-1", "sake", 0, true, "", 0, "", 0, false, "", "", "" - )); - // A changed --description is baked into the batch header, so it - // must re-extract too rather than skip with the old one. - assert!(!manifest.matches( - "a.md", "hash-1", "model-1", "sake", 0, false, "new desc", 0, "", 0, false, "", "", "" - )); - // A changed --fact-budget is folded into the system prompt like - // --questions, so it must re-extract too rather than skip. - assert!(!manifest.matches( - "a.md", "hash-1", "model-1", "sake", 0, false, "", 5, "", 0, false, "", "", "" - )); - // A changed --structured-output or --max-output-tokens changes - // what the model can answer — computation inputs like the rest. - assert!(!manifest.matches( - "a.md", "hash-1", "model-1", "sake", 0, false, "", 0, "auto", 0, false, "", "", "" - )); - assert!(!manifest.matches( - "a.md", "hash-1", "model-1", "sake", 0, false, "", 0, "", 2048, false, "", "", "" - )); - // Issue #199: a changed --lossy changes what the batch's facts - // even are (dropped vs. corrected), so it must re-extract too. - assert!(!manifest.matches( - "a.md", "hash-1", "model-1", "sake", 0, false, "", 0, "", 0, true, "", "", "" - )); - - // A prompt bump invalidates entries recorded under the old one. - manifest - .documents - .get_mut("a.md") - .expect("just recorded") - .prompt_version = PROMPT_VERSION + 1; - assert!(!manifest.matches( - "a.md", "hash-1", "model-1", "sake", 0, false, "", 0, "", 0, false, "", "", "" - )); - - let dir = std::env::temp_dir().join(format!("taguru-manifest-{}", std::process::id())); - let _ = fs::remove_dir_all(&dir); - fs::create_dir_all(&dir).unwrap(); - let path = dir.join(MANIFEST_NAME); - assert!(Manifest::load(&path).documents.is_empty()); - let mut manifest = Manifest::default(); - manifest.record( "a.md", "hash-1", "model-1", @@ -2218,79 +2258,33 @@ fn manifests_skip_only_exact_recomputations() { "", "", "", - "a.md.jsonl", - ); - manifest.save(&path).unwrap(); - assert!(Manifest::load(&path).matches( - "a.md", "hash-1", "model-1", "sake", 0, false, "", 0, "", 0, false, "", "", "" - )); - fs::write(&path, "not json").unwrap(); - assert!(Manifest::load(&path).documents.is_empty()); - - // An entry written before the context/no_passage/description/ - // fact_budget fields existed still loads — and mismatches, so - // it re-extracts exactly once. - fs::write( - &path, - r#"{"documents": {"a.md": {"sha256": "hash-1", "model": "model-1", - "prompt_version": 1, "output": "a.md.jsonl"}}}"#, - ) - .unwrap(); - let legacy = Manifest::load(&path); - assert_eq!(legacy.documents.len(), 1); - assert!(!legacy.matches( - "a.md", "hash-1", "model-1", "sake", 0, false, "", 0, "", 0, false, "", "", "" - )); - - // An entry written before the structured_output/ - // max_output_tokens/lossy fields existed (all other fields - // current) must keep matching an all-defaults run — the new - // controls default to their zero/false values precisely so old - // manifests don't force a spurious re-extraction of everything. - fs::write( - &path, - format!( - r#"{{"documents": {{"a.md": {{"sha256": "hash-1", "model": "model-1", - "prompt_version": {PROMPT_VERSION}, "context": "sake", - "output": "a.md.jsonl"}}}}}}"# - ), - ) - .unwrap(); - let pre_ladder = Manifest::load(&path); - assert!(pre_ladder.matches( - "a.md", "hash-1", "model-1", "sake", 0, false, "", 0, "", 0, false, "", "", "" + "", + 0, + &[] )); - assert!(!pre_ladder.matches( + assert!(!manifest.matches( "a.md", - "hash-1", + "hash-2", "model-1", "sake", 0, false, "", 0, - "json-schema", + "", 0, false, "", "", - "" - )); - // Issue #199: an entry from before --lossy existed defaults to - // `false` (strict) and must NOT match a --lossy run. - assert!(!pre_ladder.matches( - "a.md", "hash-1", "model-1", "sake", 0, false, "", 0, "", 0, true, "", "", "" + "", + "", + 0, + &[] )); - let _ = fs::remove_dir_all(&dir); -} - -#[test] -fn manifests_reextract_when_the_schema_digest_changes() { - let mut manifest = Manifest::default(); - manifest.record( + assert!(!manifest.matches( "a.md", "hash-1", - "model-1", + "model-2", "sake", 0, false, @@ -2299,33 +2293,16 @@ fn manifests_reextract_when_the_schema_digest_changes() { "", 0, false, - "digest-1", "", "", - "a.md.jsonl", - ); - assert!(manifest.matches( - "a.md", "hash-1", "model-1", "sake", 0, false, "", 0, "", 0, false, "digest-1", "", "" - )); - // A different --schema document — even with everything else - // identical — must re-extract: the prompt's schema block and - // self-validation both changed under it. - assert!(!manifest.matches( - "a.md", "hash-1", "model-1", "sake", 0, false, "", 0, "", 0, false, "digest-2", "", "" + "", + "", + 0, + &[] )); - // Dropping --schema entirely (querying with "") must also - // re-extract a schema-recorded entry, not just swap it. assert!(!manifest.matches( - "a.md", "hash-1", "model-1", "sake", 0, false, "", 0, "", 0, false, "", "", "" - )); - - // An entry written before `--schema` existed defaults to "" — - // matches a schema-less rerun, mismatches once --schema is - // engaged, the same precedent structured_output/lossy set. - let mut legacy = Manifest::default(); - legacy.record( "b.md", - "hash-2", + "hash-1", "model-1", "sake", 0, @@ -2338,29 +2315,502 @@ fn manifests_reextract_when_the_schema_digest_changes() { "", "", "", - "b.md.jsonl", - ); - assert!(legacy.matches( - "b.md", "hash-2", "model-1", "sake", 0, false, "", 0, "", 0, false, "", "", "" - )); - assert!(!legacy.matches( - "b.md", "hash-2", "model-1", "sake", 0, false, "", 0, "", 0, false, "digest-1", "", "" + "", + 0, + &[] )); -} - -#[test] -fn request_options_default_adds_no_keys_to_the_body() { - let messages = [serde_json::json!({"role": "user", "content": "hi"})]; - // The pre-ladder body, byte for byte: serde_json orders keys - // alphabetically, so nothing about the base three moves when - // the optional keys are absent. - assert_eq!( - build_chat_body("m", &messages, &RequestOptions::default()), - r#"{"messages":[{"content":"hi","role":"user"}],"model":"m","temperature":0}"# - ); - let with_options = build_chat_body( - "m", - &messages, + // A re-pointed --context must re-extract, not keep files whose + // headers still name the old target. + assert!(!manifest.matches( + "a.md", + "hash-1", + "model-1", + "vats", + 0, + false, + "", + 0, + "", + 0, + false, + "", + "", + "", + "", + 0, + &[] + )); + // Toggling --no-passage changes whether the batch carries the + // source passage at all — a skip would keep the stale shape. + assert!(!manifest.matches( + "a.md", + "hash-1", + "model-1", + "sake", + 0, + true, + "", + 0, + "", + 0, + false, + "", + "", + "", + "", + 0, + &[] + )); + // A changed --description is baked into the batch header, so it + // must re-extract too rather than skip with the old one. + assert!(!manifest.matches( + "a.md", + "hash-1", + "model-1", + "sake", + 0, + false, + "new desc", + 0, + "", + 0, + false, + "", + "", + "", + "", + 0, + &[] + )); + // A changed --fact-budget is folded into the system prompt like + // --questions, so it must re-extract too rather than skip. + assert!(!manifest.matches( + "a.md", + "hash-1", + "model-1", + "sake", + 0, + false, + "", + 5, + "", + 0, + false, + "", + "", + "", + "", + 0, + &[] + )); + // A changed --structured-output or --max-output-tokens changes + // what the model can answer — computation inputs like the rest. + assert!(!manifest.matches( + "a.md", + "hash-1", + "model-1", + "sake", + 0, + false, + "", + 0, + "auto", + 0, + false, + "", + "", + "", + "", + 0, + &[] + )); + assert!(!manifest.matches( + "a.md", + "hash-1", + "model-1", + "sake", + 0, + false, + "", + 0, + "", + 2048, + false, + "", + "", + "", + "", + 0, + &[] + )); + // Issue #199: a changed --lossy changes what the batch's facts + // even are (dropped vs. corrected), so it must re-extract too. + assert!(!manifest.matches( + "a.md", + "hash-1", + "model-1", + "sake", + 0, + false, + "", + 0, + "", + 0, + true, + "", + "", + "", + "", + 0, + &[] + )); + + // A prompt bump invalidates entries recorded under the old one. + manifest + .documents + .get_mut("a.md") + .expect("just recorded") + .prompt_version = PROMPT_VERSION + 1; + assert!(!manifest.matches( + "a.md", + "hash-1", + "model-1", + "sake", + 0, + false, + "", + 0, + "", + 0, + false, + "", + "", + "", + "", + 0, + &[] + )); + + let dir = std::env::temp_dir().join(format!("taguru-manifest-{}", std::process::id())); + let _ = fs::remove_dir_all(&dir); + fs::create_dir_all(&dir).unwrap(); + let path = dir.join(MANIFEST_NAME); + assert!(Manifest::load(&path).documents.is_empty()); + let mut manifest = Manifest::default(); + manifest.record( + "a.md", + "hash-1", + "model-1", + "sake", + 0, + false, + "", + 0, + "", + 0, + false, + "", + "", + "", + "", + 0, + &[], + "a.md.jsonl", + ); + manifest.save(&path).unwrap(); + assert!(Manifest::load(&path).matches( + "a.md", + "hash-1", + "model-1", + "sake", + 0, + false, + "", + 0, + "", + 0, + false, + "", + "", + "", + "", + 0, + &[] + )); + fs::write(&path, "not json").unwrap(); + assert!(Manifest::load(&path).documents.is_empty()); + + // An entry written before the context/no_passage/description/ + // fact_budget fields existed still loads — and mismatches, so + // it re-extracts exactly once. + fs::write( + &path, + r#"{"documents": {"a.md": {"sha256": "hash-1", "model": "model-1", + "prompt_version": 1, "output": "a.md.jsonl"}}}"#, + ) + .unwrap(); + let legacy = Manifest::load(&path); + assert_eq!(legacy.documents.len(), 1); + assert!(!legacy.matches( + "a.md", + "hash-1", + "model-1", + "sake", + 0, + false, + "", + 0, + "", + 0, + false, + "", + "", + "", + "", + 0, + &[] + )); + + // An entry written before the structured_output/ + // max_output_tokens/lossy fields existed (all other fields + // current) must keep matching an all-defaults run — the new + // controls default to their zero/false values precisely so old + // manifests don't force a spurious re-extraction of everything. + fs::write( + &path, + format!( + r#"{{"documents": {{"a.md": {{"sha256": "hash-1", "model": "model-1", + "prompt_version": {PROMPT_VERSION}, "context": "sake", + "output": "a.md.jsonl"}}}}}}"# + ), + ) + .unwrap(); + let pre_ladder = Manifest::load(&path); + assert!(pre_ladder.matches( + "a.md", + "hash-1", + "model-1", + "sake", + 0, + false, + "", + 0, + "", + 0, + false, + "", + "", + "", + "", + 0, + &[] + )); + assert!(!pre_ladder.matches( + "a.md", + "hash-1", + "model-1", + "sake", + 0, + false, + "", + 0, + "json-schema", + 0, + false, + "", + "", + "", + "", + 0, + &[] + )); + // Issue #199: an entry from before --lossy existed defaults to + // `false` (strict) and must NOT match a --lossy run. + assert!(!pre_ladder.matches( + "a.md", + "hash-1", + "model-1", + "sake", + 0, + false, + "", + 0, + "", + 0, + true, + "", + "", + "", + "", + 0, + &[] + )); + let _ = fs::remove_dir_all(&dir); +} + +#[test] +fn manifests_reextract_when_the_schema_digest_changes() { + let mut manifest = Manifest::default(); + manifest.record( + "a.md", + "hash-1", + "model-1", + "sake", + 0, + false, + "", + 0, + "", + 0, + false, + "digest-1", + "", + "", + "", + 0, + &[], + "a.md.jsonl", + ); + assert!(manifest.matches( + "a.md", + "hash-1", + "model-1", + "sake", + 0, + false, + "", + 0, + "", + 0, + false, + "digest-1", + "", + "", + "", + 0, + &[] + )); + // A different --schema document — even with everything else + // identical — must re-extract: the prompt's schema block and + // self-validation both changed under it. + assert!(!manifest.matches( + "a.md", + "hash-1", + "model-1", + "sake", + 0, + false, + "", + 0, + "", + 0, + false, + "digest-2", + "", + "", + "", + 0, + &[] + )); + // Dropping --schema entirely (querying with "") must also + // re-extract a schema-recorded entry, not just swap it. + assert!(!manifest.matches( + "a.md", + "hash-1", + "model-1", + "sake", + 0, + false, + "", + 0, + "", + 0, + false, + "", + "", + "", + "", + 0, + &[] + )); + + // An entry written before `--schema` existed defaults to "" — + // matches a schema-less rerun, mismatches once --schema is + // engaged, the same precedent structured_output/lossy set. + let mut legacy = Manifest::default(); + legacy.record( + "b.md", + "hash-2", + "model-1", + "sake", + 0, + false, + "", + 0, + "", + 0, + false, + "", + "", + "", + "", + 0, + &[], + "b.md.jsonl", + ); + assert!(legacy.matches( + "b.md", + "hash-2", + "model-1", + "sake", + 0, + false, + "", + 0, + "", + 0, + false, + "", + "", + "", + "", + 0, + &[] + )); + assert!(!legacy.matches( + "b.md", + "hash-2", + "model-1", + "sake", + 0, + false, + "", + 0, + "", + 0, + false, + "digest-1", + "", + "", + "", + 0, + &[] + )); +} + +#[test] +fn request_options_default_adds_no_keys_to_the_body() { + let messages = [serde_json::json!({"role": "user", "content": "hi"})]; + // The pre-ladder body, byte for byte: serde_json orders keys + // alphabetically, so nothing about the base three moves when + // the optional keys are absent. + assert_eq!( + build_chat_body("m", &messages, &RequestOptions::default()), + r#"{"messages":[{"content":"hi","role":"user"}],"model":"m","temperature":0}"# + ); + let with_options = build_chat_body( + "m", + &messages, &RequestOptions { response_format: Some(json_object_response_format()), max_tokens: Some(512), @@ -3197,3 +3647,280 @@ fn a_gap_quote_is_capped_at_a_char_boundary() { "{quote}" ); } + +// ---- promotion runbook conventions (#466 S1, ADR 0017) ---- + +#[test] +fn dates_parse_as_epoch_seconds_or_utc_civil_days() { + assert_eq!(parse_date("1785974400"), Some(1785974400)); + // A civil date is that day's UTC midnight, round-tripped through + // the rendering direction. + let seconds = parse_date("2026-08-06").expect("a real date parses"); + assert_eq!(crate::clock::iso8601_utc(seconds), "2026-08-06T00:00:00Z"); + assert_eq!(parse_date("1970-01-01"), None); // 0 is the manifest's off sentinel + assert_eq!(parse_date("0"), None); + assert_eq!(parse_date("2026-02-30"), None); // normalizes ≠ as-written → refused + assert_eq!(parse_date("2026-13-01"), None); + assert_eq!(parse_date("session-note"), None); + assert_eq!(parse_date(""), None); +} + +#[test] +fn runbook_flags_parse_and_their_contradictions_are_usage_errors() { + fn parse(words: &[&str]) -> Result { + Args::parse(&words.iter().map(|s| s.to_string()).collect::>()) + } + let base = ["--context", "c", "--out", "o"]; + let mut ok = base.to_vec(); + ok.extend([ + "--source-id", + "session:claude:abc", + "--date", + "2026-08-06", + "--tag", + "ops", + "--tag", + "リリース", + "--tag", + "ops", // duplicates fold instead of erroring or double-writing + "doc.md", + ]); + let parsed = parse(&ok).expect("the runbook flags parse"); + assert_eq!(parsed.source_id.as_deref(), Some("session:claude:abc")); + assert_eq!( + parsed.date, + Some(parse_date("2026-08-06").expect("a real date parses")) + ); + assert_eq!(parsed.tags, vec!["ops".to_string(), "リリース".to_string()]); + + let mut twice = base.to_vec(); + twice.extend(["--source-id", "a", "--source-id", "b", "doc.md"]); + assert!(matches!(parse(&twice), Err(2))); + let mut empty = base.to_vec(); + empty.extend(["--source-id", "", "doc.md"]); + assert!(matches!(parse(&empty), Err(2))); + let mut bad_date = base.to_vec(); + bad_date.extend(["--date", "yesterday", "doc.md"]); + assert!(matches!(parse(&bad_date), Err(2))); + // Metadata rides the passage line, so stripping the passage while + // asking for it is a contradiction, not a silent drop. + let mut stripped = base.to_vec(); + stripped.extend(["--no-passage", "--date", "2026-08-06", "doc.md"]); + assert!(matches!(parse(&stripped), Err(2))); + let mut stripped_tag = base.to_vec(); + stripped_tag.extend(["--no-passage", "--tag", "ops", "doc.md"]); + assert!(matches!(parse(&stripped_tag), Err(2))); +} + +#[test] +fn the_passage_line_carries_date_and_tags_exactly_when_given() { + let extraction = merge( + vec![parse_model_output( + r#"{"associations": [{"subject": "a", "label": "l", "object": "b", "weight": 1.0}]}"#, + ) + .unwrap()], + 0, + 1, + ); + let plain = render_batch("c", "s", None, &extraction, Some("本文。"), None, &[]); + let passage_line = plain.lines().nth(1).expect("header then passage"); + // No flags → the passage line stays byte-for-byte pre-S1. + assert_eq!(passage_line, r#"{"passage":"本文。"}"#); + + let tagged = render_batch( + "c", + "session:claude:abc", + None, + &extraction, + Some("本文。"), + Some(1785974400), + &["ops".to_string(), "リリース".to_string()], + ); + let header: serde_json::Value = serde_json::from_str(tagged.lines().next().unwrap()).unwrap(); + assert_eq!(header["source"], "session:claude:abc"); + let passage: serde_json::Value = serde_json::from_str(tagged.lines().nth(1).unwrap()).unwrap(); + assert_eq!(passage["date"], 1785974400u64); + assert_eq!(passage["tags"], serde_json::json!(["ops", "リリース"])); + // What extract writes, import accepts. + crate::ingest::parse_batch(Cursor::new(tagged.as_bytes())).unwrap(); +} + +#[test] +fn manifests_rewrite_when_the_runbook_metadata_changes() { + let mut manifest = Manifest::default(); + manifest.record( + "a.md", + "hash-1", + "model-1", + "sake", + 0, + false, + "", + 0, + "", + 0, + false, + "", + "", + "", + "session:claude:abc", + 1785974400, + &["ops".to_string()], + "a.md.jsonl", + ); + let matches_with = |source_id: &str, date: u64, tags: &[String]| { + manifest.matches( + "a.md", "hash-1", "model-1", "sake", 0, false, "", 0, "", 0, false, "", "", "", + source_id, date, tags, + ) + }; + assert!(matches_with( + "session:claude:abc", + 1785974400, + &["ops".to_string()] + )); + // Any of the three changing must rewrite the batch — they are all + // baked into the emitted file. + assert!(!matches_with( + "session:claude:xyz", + 1785974400, + &["ops".to_string()] + )); + assert!(!matches_with("session:claude:abc", 0, &["ops".to_string()])); + assert!(!matches_with("session:claude:abc", 1785974400, &[])); + + // Pre-S1 entries (no fields) keep matching default runs. + let mut legacy = Manifest::default(); + legacy.record( + "b.md", + "hash-2", + "model-1", + "sake", + 0, + false, + "", + 0, + "", + 0, + false, + "", + "", + "", + "", + 0, + &[], + "b.md.jsonl", + ); + let json = serde_json::to_string(&legacy).unwrap(); + let stripped = json + .replace(r#""source_id":"","#, "") + .replace(r#""date":0,"#, "") + .replace(r#""tags":[],"#, ""); + // The replaces must actually have removed the fields — a silent + // no-match would leave all three in place and this test would + // "pass" without exercising the legacy shape at all. + for key in [r#""source_id""#, r#""date""#, r#""tags""#] { + assert!(!stripped.contains(key), "{key} survived in: {stripped}"); + } + let reloaded: Manifest = serde_json::from_str(&stripped).unwrap(); + assert!(reloaded.matches( + "b.md", + "hash-2", + "model-1", + "sake", + 0, + false, + "", + 0, + "", + 0, + false, + "", + "", + "", + "", + 0, + &[] + )); +} + +#[test] +fn runbook_flag_boundaries_hold_exactly() { + fn parse(words: Vec) -> Result { + Args::parse(&words) + } + fn base() -> Vec { + ["--context", "c", "--out", "o"] + .iter() + .map(|s| s.to_string()) + .collect() + } + // A duplicate --date is a usage error, never last-wins. + let mut dated = base(); + dated.extend( + ["--date", "2026-08-06", "--date", "2026-08-07", "doc.md"] + .iter() + .map(|s| s.to_string()), + ); + assert!(matches!(parse(dated), Err(2))); + // An empty tag is refused, not stored. + let mut empty_tag = base(); + empty_tag.extend(["--tag", "", "doc.md"].iter().map(|s| s.to_string())); + assert!(matches!(parse(empty_tag), Err(2))); + // Tag bytes: exactly at the cap passes, one over fails. + let mut at_cap = base(); + at_cap.extend([ + "--tag".to_string(), + "t".repeat(crate::api::MAX_TAG_BYTES), + "doc.md".to_string(), + ]); + assert!(parse(at_cap).is_ok()); + let mut over_cap = base(); + over_cap.extend([ + "--tag".to_string(), + "t".repeat(crate::api::MAX_TAG_BYTES + 1), + "doc.md".to_string(), + ]); + assert!(matches!(parse(over_cap), Err(2))); + // Tag count: exactly the per-source cap passes, one more fails. + let mut full = base(); + for i in 0..crate::api::MAX_TAGS_PER_SOURCE { + full.extend(["--tag".to_string(), format!("t{i}")]); + } + full.push("doc.md".to_string()); + assert!(parse(full).is_ok()); + let mut overfull = base(); + for i in 0..=crate::api::MAX_TAGS_PER_SOURCE { + overfull.extend(["--tag".to_string(), format!("t{i}")]); + } + overfull.push("doc.md".to_string()); + assert!(matches!(parse(overfull), Err(2))); + // Source id bytes: exactly at the name cap passes, one over fails. + let mut id_at_cap = base(); + id_at_cap.extend([ + "--source-id".to_string(), + "s".repeat(MAX_NAME_BYTES), + "doc.md".to_string(), + ]); + assert!(parse(id_at_cap).is_ok()); + let mut id_over = base(); + id_over.extend([ + "--source-id".to_string(), + "s".repeat(MAX_NAME_BYTES + 1), + "doc.md".to_string(), + ]); + assert!(matches!(parse(id_over), Err(2))); + // A fourth dash-separated part is refused even when the first + // three name a real date; day 0 is refused before the civil + // arithmetic ever sees it. + assert_eq!(parse_date("2026-08-06-07"), None); + assert_eq!(parse_date("2026-08-00"), None); + // The year cap keeps the civil arithmetic in-domain: an i64-scale + // year would overflow inside days_from_civil (a panic on external + // input, not a rejection), and five digits is outside the + // YYYY-MM-DD contract anyway. + assert_eq!(parse_date("9223372036854775807-01-01"), None); + assert_eq!(parse_date("10000-01-01"), None); + assert_eq!(parse_date("0000-01-01"), None); + assert_eq!(parse_date("-0001-01-01"), None); +} diff --git a/tests/http_api/extract.rs b/tests/http_api/extract.rs index f2a2d970..cad90add 100644 --- a/tests/http_api/extract.rs +++ b/tests/http_api/extract.rs @@ -1119,6 +1119,256 @@ fn extract_coverage_reports_uncovered_candidate_pair_sentences() { let _ = std::fs::remove_dir_all(&out); } +/// The `/{stem}` suffix can push a within-cap `--source-id` over the +/// 1024-byte source cap only at extract time — exactly at the cap +/// both documents pass; one byte over fails them before any model +/// call. +#[test] +fn extract_suffixed_source_ids_respect_the_name_cap() { + let docs = batch_dir("extract-source-cap-docs"); + std::fs::write(docs.join("aa.md"), "壱。").unwrap(); + std::fs::write(docs.join("bb.md"), "弐。").unwrap(); + let out = batch_dir("extract-source-cap-out"); + + // 1022 + "/" + 2-byte stem = 1025: one over MAX_NAME_BYTES (1024). + let over = "s".repeat(1022); + let (code, stdout, stderr) = run_extract( + &out, + &[ + ("TAGURU_EXTRACT_URL", "http://127.0.0.1:9/v1"), + ("TAGURU_EXTRACT_MODEL", "stub-model"), + ], + &[ + "--context", + "ops", + "--source-id", + &over, + docs.to_str().unwrap(), + ], + ); + assert_eq!(code, 1, "stdout: {stdout}\nstderr: {stderr}"); + assert!(stderr.contains("source cap"), "{stderr}"); + + // 1021 + "/" + 2 = 1024: exactly at the cap, both documents land. + let at_cap = "s".repeat(1021); + let reply = |name: &str, text: &str| { + json!({"associations": [ + {"subject": name, "label": "l", "object": text, "weight": 1.0, "paragraph": 0} + ]}) + .to_string() + }; + let (url, requests) = stub_chat_server(vec![reply("壱", "壱。"), reply("弐", "弐。")]); + let (code, stdout, stderr) = run_extract( + &out, + &[ + ("TAGURU_EXTRACT_URL", url.as_str()), + ("TAGURU_EXTRACT_MODEL", "stub-model"), + ], + &[ + "--context", + "ops", + "--source-id", + &at_cap, + docs.to_str().unwrap(), + ], + ); + assert_eq!(code, 0, "stdout: {stdout}\nstderr: {stderr}"); + requests.join().unwrap(); + + let _ = std::fs::remove_dir_all(&docs); + let _ = std::fs::remove_dir_all(&out); +} + +/// The `--source-id` usage errors name their actual cause: a +/// duplicate says "given twice", an empty id says "must not be +/// empty" — both exit 2, so only the wording tells the operator +/// which mistake they made. +#[test] +fn extract_source_id_usage_errors_name_their_cause() { + let out = batch_dir("extract-source-id-usage-out"); + let (code, _, stderr) = run_extract( + &out, + &[], + &[ + "--context", + "c", + "--source-id", + "a", + "--source-id", + "b", + "doc.md", + ], + ); + assert_eq!(code, 2, "{stderr}"); + assert!(stderr.contains("--source-id given twice"), "{stderr}"); + let (code, _, stderr) = + run_extract(&out, &[], &["--context", "c", "--source-id", "", "doc.md"]); + assert_eq!(code, 2, "{stderr}"); + assert!(stderr.contains("--source-id must not be empty"), "{stderr}"); + let _ = std::fs::remove_dir_all(&out); +} + +/// #466 S1 (ADR 0017): `--source-id`/`--date`/`--tag` bake the +/// promotion runbook's conventions into the written batch — the +/// session source id (with the `/{doc}` stem suffix across several +/// documents), the passage line's date and tags — and all three are +/// manifest computation inputs: same flags skip, a changed date +/// rewrites. A source-id collision between two documents fails the +/// second instead of letting import fold them into one another. +#[test] +fn extract_bakes_the_runbook_conventions_into_the_batch() { + let docs = batch_dir("extract-runbook-docs"); + std::fs::write(docs.join("s1.md"), "青嶺酒造は1907年に創業した。").unwrap(); + std::fs::write(docs.join("s2.md"), "杜氏は高瀬。").unwrap(); + let out = batch_dir("extract-runbook-out"); + + let replies = [ + json!({"associations": [ + {"subject": "青嶺酒造", "label": "創業年", "object": "1907年", "weight": 1.0, "paragraph": 0} + ]}) + .to_string(), + json!({"associations": [ + {"subject": "高瀬", "label": "役職", "object": "杜氏", "weight": 1.0, "paragraph": 0} + ]}) + .to_string(), + ]; + let flags = [ + "--context", + "ops", + "--source-id", + "session:claude:abc", + "--date", + "2026-08-06", + "--tag", + "ops", + "--tag", + "リリース", + ]; + let (url, requests) = stub_chat_server(replies.to_vec()); + let provider = [ + ("TAGURU_EXTRACT_URL", url.as_str()), + ("TAGURU_EXTRACT_MODEL", "stub-model"), + ]; + let mut args: Vec<&str> = flags.to_vec(); + let docs_arg = docs.to_str().unwrap().to_string(); + args.push(&docs_arg); + let (code, stdout, stderr) = run_extract(&out, &provider, &args); + assert_eq!(code, 0, "stdout: {stdout}\nstderr: {stderr}"); + requests.join().unwrap(); + + // Several documents: each header carries ID/{stem}; the passage + // line carries the date (2026-08-06 UTC midnight) and the tags. + for (file, expected_source) in [ + ("s1.md", "session:claude:abc/s1"), + ("s2.md", "session:claude:abc/s2"), + ] { + let batch_file = std::fs::read_dir(&out) + .unwrap() + .filter_map(Result::ok) + .map(|entry| entry.path()) + .find(|path| { + path.file_name() + .is_some_and(|name| name.to_string_lossy().contains(&file.replace(".md", ""))) + && path.extension().is_some_and(|ext| ext == "jsonl") + }) + .unwrap_or_else(|| panic!("a batch file for {file}")); + let body = std::fs::read_to_string(&batch_file).unwrap(); + let header: Value = serde_json::from_str(body.lines().next().unwrap()).unwrap(); + assert_eq!(header["source"], expected_source, "{body}"); + let passage: Value = serde_json::from_str(body.lines().nth(1).unwrap()).unwrap(); + assert_eq!(passage["date"], 1785974400u64, "{body}"); + assert_eq!(passage["tags"], json!(["ops", "リリース"]), "{body}"); + } + + // Same flags again: both documents skip (the metadata is in the + // fingerprint and unchanged). No model call — the stub above + // accepted exactly two connections. + let (code, stdout, stderr) = run_extract(&out, &provider, &args); + assert_eq!(code, 0, "stdout: {stdout}\nstderr: {stderr}"); + assert_eq!(stdout.matches("unchanged, skipped").count(), 2, "{stdout}"); + + // A changed date must rewrite — a skip would leave the old date in + // the emitted file. + let (url, requests) = stub_chat_server(replies.to_vec()); + let mut redated: Vec<&str> = args.clone(); + let position = redated.iter().position(|a| *a == "2026-08-06").unwrap(); + redated[position] = "2026-08-07"; + let (code, stdout, stderr) = run_extract( + &out, + &[ + ("TAGURU_EXTRACT_URL", url.as_str()), + ("TAGURU_EXTRACT_MODEL", "stub-model"), + ], + &redated, + ); + assert_eq!(code, 0, "stdout: {stdout}\nstderr: {stderr}"); + assert!(!stdout.contains("unchanged, skipped"), "{stdout}"); + requests.join().unwrap(); + + // A single document takes the id verbatim — no suffix to invent. + let (url, requests) = stub_chat_server(vec![replies[0].clone()]); + let single = docs.join("s1.md"); + let mut single_args: Vec<&str> = flags.to_vec(); + let single_arg = single.to_str().unwrap().to_string(); + single_args.push(&single_arg); + let (code, stdout, stderr) = run_extract( + &out, + &[ + ("TAGURU_EXTRACT_URL", url.as_str()), + ("TAGURU_EXTRACT_MODEL", "stub-model"), + ], + &single_args, + ); + assert_eq!(code, 0, "stdout: {stdout}\nstderr: {stderr}"); + requests.join().unwrap(); + let body = std::fs::read_to_string(out.join(format!( + "{}.jsonl", + single.to_str().unwrap().replace(['/', ':'], "__") + ))) + .unwrap(); + let header: Value = serde_json::from_str(body.lines().next().unwrap()).unwrap(); + assert_eq!(header["source"], "session:claude:abc"); + + // Two documents whose stems collide would land on ONE source id — + // import's per-source retract-then-apply would fold them, so the + // second fails before any call is spent on it. + let nested_a = docs.join("a"); + let nested_b = docs.join("b"); + std::fs::create_dir_all(&nested_a).unwrap(); + std::fs::create_dir_all(&nested_b).unwrap(); + std::fs::write(nested_a.join("x.md"), "壱。").unwrap(); + std::fs::write(nested_b.join("x.md"), "弐。").unwrap(); + let (url, requests) = stub_chat_server(vec![ + json!({"associations": [ + {"subject": "壱", "label": "l", "object": "壱。", "weight": 1.0, "paragraph": 0} + ]}) + .to_string(), + ]); + let collide_out = batch_dir("extract-runbook-collide-out"); + let (code, stdout, stderr) = run_extract( + &collide_out, + &[ + ("TAGURU_EXTRACT_URL", url.as_str()), + ("TAGURU_EXTRACT_MODEL", "stub-model"), + ], + &[ + "--context", + "ops", + "--source-id", + "session:claude:abc", + nested_a.join("x.md").to_str().unwrap(), + nested_b.join("x.md").to_str().unwrap(), + ], + ); + assert_eq!(code, 1, "stdout: {stdout}\nstderr: {stderr}"); + assert!(stderr.contains("collides"), "{stderr}"); + requests.join().unwrap(); + + let _ = std::fs::remove_dir_all(&docs); + let _ = std::fs::remove_dir_all(&out); + let _ = std::fs::remove_dir_all(&collide_out); +} + /// TAGURU_EXTRACT_COVERAGE resolves like its boolean siblings: `1` /// turns the report on, `0` keeps it off (and keeps every report line /// free of an "uncovered" note — the count note must not render at