diff --git a/CHANGELOG.md b/CHANGELOG.md index 35ca48c..f0f1b79 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -6,6 +6,38 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ## [Unreleased] +## [0.3.0] - 2026-05-31 + +Column scoping — focus detection on the columns that matter in a wide corpus, +deterministically and without guessing. + +### Added + +- `scan` / `explain` gain **`--columns C,..`** (analyze only these columns) and + **`--exclude C,..`** (analyze every column except these). The two are mutually + exclusive. Projection is applied before detection and to the baseline as well, + so drift comparison stays consistent. +- Column scoping is **explicit, never heuristic.** anomalyx will not guess which + columns are "interesting" — a silent auto-skip would itself be a guess, and + would wrongly drop exactly the near-unique numeric measurements the marquee + detectors rely on (packet `durationNanos`, span durations, latencies). You name + the scope; the result stays deterministic and reproducible. +- An unknown column name in `--columns`/`--exclude` on the primary corpus is a + hard error (exit `2`) — a typo can never silently scope a scan down to nothing + and read as "clean". The baseline is projected leniently (it is a different + corpus and need not carry every scoped column). + +### Notes + +- This directly tames wide, identifier-heavy corpora. On a real 20k-entry + `journalctl -o json` capture, `scan` emits ~10k mostly-noise `point` findings + across journald's many ID/counter/timestamp fields; `scan --exclude` of those + fields (or `--columns` of the meaningful ones) collapses that to a couple + hundred focused findings without touching detector configuration. +- New `RecordSet::select` / `RecordSet::without` projection primitives in + `ax-core`. No envelope or `config_version` change — column scope is an + input-side projection, so the determinism contract is unchanged. + ## [0.2.2] - 2026-05-31 ### Fixed @@ -106,7 +138,8 @@ Initial release — a contract-first anomaly-detection CLI over arbitrary corpor gates on every push. - Dual-licensed under MIT OR Apache-2.0. -[Unreleased]: https://github.com/copyleftdev/anomalyx/compare/v0.2.2...HEAD +[Unreleased]: https://github.com/copyleftdev/anomalyx/compare/v0.3.0...HEAD +[0.3.0]: https://github.com/copyleftdev/anomalyx/compare/v0.2.2...v0.3.0 [0.2.2]: https://github.com/copyleftdev/anomalyx/compare/v0.2.1...v0.2.2 [0.2.1]: https://github.com/copyleftdev/anomalyx/compare/v0.2.0...v0.2.1 [0.2.0]: https://github.com/copyleftdev/anomalyx/compare/v0.1.0...v0.2.0 diff --git a/Cargo.lock b/Cargo.lock index 19d8107..76cdd90 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -69,7 +69,7 @@ dependencies = [ [[package]] name = "anomalyx" -version = "0.2.2" +version = "0.3.0" dependencies = [ "anomalyx-core", "anomalyx-detect", @@ -80,7 +80,7 @@ dependencies = [ [[package]] name = "anomalyx-core" -version = "0.2.2" +version = "0.3.0" dependencies = [ "proptest", "serde", @@ -90,7 +90,7 @@ dependencies = [ [[package]] name = "anomalyx-detect" -version = "0.2.2" +version = "0.3.0" dependencies = [ "anomalyx-core", "proptest", @@ -101,7 +101,7 @@ dependencies = [ [[package]] name = "anomalyx-normalize" -version = "0.2.2" +version = "0.3.0" dependencies = [ "anomalyx-core", "apache-avro", @@ -129,7 +129,7 @@ dependencies = [ [[package]] name = "anomalyx-validate" -version = "0.2.2" +version = "0.3.0" dependencies = [ "anomalyx-core", "anomalyx-detect", diff --git a/Cargo.toml b/Cargo.toml index fb03d76..ab9a798 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -9,7 +9,7 @@ members = [ ] [workspace.package] -version = "0.2.2" +version = "0.3.0" edition = "2021" rust-version = "1.90" license = "MIT OR Apache-2.0" @@ -22,9 +22,9 @@ authors = ["copyleftdev"] # `ax-*` names were taken), but the import/extern name stays `ax_core` etc. via # the dependency key + `package` rename — so no source code changes are needed. # Keep versions in sync with workspace.package.version above. -ax-core = { path = "crates/ax-core", version = "0.2.2", package = "anomalyx-core" } -ax-normalize = { path = "crates/ax-normalize", version = "0.2.2", package = "anomalyx-normalize" } -ax-detect = { path = "crates/ax-detect", version = "0.2.2", package = "anomalyx-detect" } +ax-core = { path = "crates/ax-core", version = "0.3.0", package = "anomalyx-core" } +ax-normalize = { path = "crates/ax-normalize", version = "0.3.0", package = "anomalyx-normalize" } +ax-detect = { path = "crates/ax-detect", version = "0.3.0", package = "anomalyx-detect" } serde = { version = "1", features = ["derive"] } serde_json = "1" diff --git a/README.md b/README.md index 4e274ea..dc92fc4 100644 --- a/README.md +++ b/README.md @@ -38,6 +38,12 @@ With `--cadence COL`, column `COL` is read as event times and assessed for metronomic (automated) regularity. Without it the cadence detector is absent — which column means "time" is never guessed. +With `--columns C,..` (or its complement `--exclude C,..`) detection is scoped to +a chosen set of columns — the answer to identifier noise on wide corpora (e.g. +`journalctl -o json | anomalyx scan --exclude JOB_ID,_PID,__REALTIME_TIMESTAMP`). +The scope is explicit, never a heuristic guess; an unknown column name is a hard +error so a typo can't silently scan nothing. See [scan modes](docs/src/modes.md). + Exit codes are part of the contract: **`0`** clean · **`1`** anomalies found · **`2`** tool error. diff --git a/crates/anomalyx/src/main.rs b/crates/anomalyx/src/main.rs index 341ad0a..effc525 100644 --- a/crates/anomalyx/src/main.rs +++ b/crates/anomalyx/src/main.rs @@ -70,12 +70,14 @@ fn usage() -> &'static str { USAGE:\n\ \x20 anomalyx describe Protocol metadata\n\ \x20 anomalyx schema JSON Schema of scan output\n\ - \x20 anomalyx scan [--baseline B] [--period N] [--cadence COL] [PATH]\n\ - \x20 anomalyx explain [--baseline B] [--period N] [--cadence COL] [PATH]\n\ + \x20 anomalyx scan [--baseline B] [--period N] [--cadence COL] [--columns C,..|--exclude C,..] [PATH]\n\ + \x20 anomalyx explain [--baseline B] [--period N] [--cadence COL] [--columns C,..|--exclude C,..] [PATH]\n\ \n\ --baseline B compare against B (distributional drift + schema-diff)\n\ --period N treat rows as a time series of period N (contextual/seasonal)\n\ --cadence COL assess column COL for metronomic timing (cadence)\n\ + --columns C,.. analyze only these columns (focus a wide corpus)\n\ + --exclude C,.. analyze every column except these\n\ EXIT: 0 clean · 1 anomalies found · 2 error" } @@ -87,9 +89,22 @@ struct ScanArgs { baseline: Option, period: Option, cadence: Option, + /// `--columns`: analyze only these columns (allowlist). + columns: Option>, + /// `--exclude`: analyze every column except these (denylist). + exclude: Option>, positional: Vec, } +/// Splits a `--columns`/`--exclude` value into trimmed, non-empty column names. +fn parse_column_list(v: &str) -> Vec { + v.split(',') + .map(str::trim) + .filter(|s| !s.is_empty()) + .map(str::to_string) + .collect() +} + fn parse_scan_args(args: &[String]) -> Result { let mut parsed = ScanArgs::default(); let mut it = args.iter(); @@ -116,12 +131,64 @@ fn parse_scan_args(args: &[String]) -> Result { .ok_or_else(|| AxError::Config("--cadence requires a column name".into()))?; parsed.cadence = Some(v.clone()); } + "--columns" => { + let v = it.next().ok_or_else(|| { + AxError::Config("--columns requires a comma-separated list".into()) + })?; + let cols = parse_column_list(v); + if cols.is_empty() { + return Err(AxError::Config( + "--columns requires at least one column name".into(), + )); + } + parsed.columns = Some(cols); + } + "--exclude" => { + let v = it.next().ok_or_else(|| { + AxError::Config("--exclude requires a comma-separated list".into()) + })?; + let cols = parse_column_list(v); + if cols.is_empty() { + return Err(AxError::Config( + "--exclude requires at least one column name".into(), + )); + } + parsed.exclude = Some(cols); + } _ => parsed.positional.push(arg.clone()), } } + if parsed.columns.is_some() && parsed.exclude.is_some() { + return Err(AxError::Config( + "use --columns or --exclude, not both".into(), + )); + } Ok(parsed) } +/// Applies any `--columns`/`--exclude` projection to a record set before +/// detection. When `validate` is set (the primary corpus), an unknown column +/// name is a hard error (exit 2) — a typo must never silently scope the scan +/// down to nothing and read as "clean". The baseline is projected leniently +/// (`validate = false`): it is a different corpus and need not carry every +/// scoped column. +fn scope_columns(rs: RecordSet, args: &ScanArgs, validate: bool) -> Result { + if validate { + if let Some(names) = args.columns.as_ref().or(args.exclude.as_ref()) { + for n in names { + if rs.column(n).is_none() { + return Err(AxError::Config(format!("no such column '{n}'"))); + } + } + } + } + Ok(match (&args.columns, &args.exclude) { + (Some(keep), _) => rs.select(keep), + (_, Some(drop)) => rs.without(drop), + _ => rs, + }) +} + /// Builds the detector config, applying any `--period` / `--cadence` overrides. fn config_for(args: &ScanArgs) -> DetectConfig { DetectConfig { @@ -162,8 +229,10 @@ fn load_baseline(path: &Option) -> Result, AxError> { fn cmd_scan(rest: &[String]) -> Result { let args = parse_scan_args(rest)?; let (source, bytes) = read_input(args.positional.first())?; - let rs = ax_normalize::normalize(&source, &bytes)?; - let baseline = load_baseline(&args.baseline)?; + let rs = scope_columns(ax_normalize::normalize(&source, &bytes)?, &args, true)?; + let baseline = load_baseline(&args.baseline)? + .map(|b| scope_columns(b, &args, false)) + .transpose()?; let cfg = config_for(&args); let ctx = match &baseline { @@ -202,8 +271,10 @@ fn cmd_explain(rest: &[String]) -> Result { let handle = Handle::parse(handle_str).ok_or_else(|| AxError::BadHandle(handle_str.clone()))?; let (source, bytes) = read_input(args.positional.get(1))?; - let rs = ax_normalize::normalize(&source, &bytes)?; - let baseline = load_baseline(&args.baseline)?; + let rs = scope_columns(ax_normalize::normalize(&source, &bytes)?, &args, true)?; + let baseline = load_baseline(&args.baseline)? + .map(|b| scope_columns(b, &args, false)) + .transpose()?; let evidence = resolve_handle(&rs, &handle)?; @@ -463,6 +534,102 @@ mod tests { assert!(parse_scan_args(&strings(&["--period", "notanumber"])).is_err()); } + #[test] + fn parse_column_list_trims_and_drops_empties() { + assert_eq!(parse_column_list("a, b ,c"), strings(&["a", "b", "c"])); + // surrounding/empty entries are dropped, not kept as "" + assert_eq!(parse_column_list(",a,,"), strings(&["a"])); + assert!(parse_column_list(" , ").is_empty()); + } + + #[test] + fn parse_scan_args_parses_columns_and_exclude() { + let a = parse_scan_args(&strings(&["--columns", "p,q", "cur.csv"])).unwrap(); + assert_eq!(a.columns, Some(strings(&["p", "q"]))); + assert_eq!(a.exclude, None); + let b = parse_scan_args(&strings(&["--exclude", "noise", "cur.csv"])).unwrap(); + assert_eq!(b.exclude, Some(strings(&["noise"]))); + assert_eq!(b.columns, None); + } + + #[test] + fn parse_scan_args_rejects_columns_and_exclude_together() { + assert!(parse_scan_args(&strings(&["--columns", "a", "--exclude", "b"])).is_err()); + } + + #[test] + fn parse_scan_args_rejects_missing_or_empty_column_lists() { + assert!(parse_scan_args(&strings(&["--columns"])).is_err()); + assert!(parse_scan_args(&strings(&["--exclude"])).is_err()); + assert!(parse_scan_args(&strings(&["--columns", " , "])).is_err()); + assert!(parse_scan_args(&strings(&["--exclude", ","])).is_err()); + } + + fn abc() -> RecordSet { + RecordSet::new( + "-", + "csv", + vec![ + Column::new("a", vec![Value::Int(1), Value::Int(2)]), + Column::new("b", vec![Value::Int(3), Value::Int(4)]), + Column::new("c", vec![Value::Int(5), Value::Int(6)]), + ], + ) + } + + #[test] + fn scope_columns_select_keeps_only_named() { + let args = ScanArgs { + columns: Some(strings(&["a", "c"])), + ..ScanArgs::default() + }; + let rs = scope_columns(abc(), &args, true).unwrap(); + let names: Vec<&str> = rs.columns.iter().map(|c| c.name.as_str()).collect(); + assert_eq!(names, ["a", "c"]); + } + + #[test] + fn scope_columns_exclude_drops_named() { + let args = ScanArgs { + exclude: Some(strings(&["b"])), + ..ScanArgs::default() + }; + let rs = scope_columns(abc(), &args, true).unwrap(); + let names: Vec<&str> = rs.columns.iter().map(|c| c.name.as_str()).collect(); + assert_eq!(names, ["a", "c"]); + } + + #[test] + fn scope_columns_validates_unknown_name_on_primary() { + // A typo'd column on the primary corpus is a hard error, never a silent + // empty scan that reads as "clean". + let args = ScanArgs { + columns: Some(strings(&["a", "typo"])), + ..ScanArgs::default() + }; + assert!(scope_columns(abc(), &args, true).is_err()); + // The same unknown name on the baseline (validate = false) is tolerated: + // a different corpus need not carry every scoped column. + let scoped = scope_columns(abc(), &args, false).unwrap(); + let names: Vec<&str> = scoped.columns.iter().map(|c| c.name.as_str()).collect(); + assert_eq!(names, ["a"]); + } + + #[test] + fn scope_columns_unknown_exclude_name_also_validated() { + let args = ScanArgs { + exclude: Some(strings(&["typo"])), + ..ScanArgs::default() + }; + assert!(scope_columns(abc(), &args, true).is_err()); + } + + #[test] + fn scope_columns_noop_without_flags() { + let rs = scope_columns(abc(), &ScanArgs::default(), true).unwrap(); + assert_eq!(rs.width(), 3); + } + #[test] fn config_for_applies_period_override() { let a = ScanArgs { diff --git a/crates/ax-core/src/record.rs b/crates/ax-core/src/record.rs index 055babc..ae43256 100644 --- a/crates/ax-core/src/record.rs +++ b/crates/ax-core/src/record.rs @@ -7,6 +7,7 @@ use crate::value::{ColType, Value}; use serde::{Deserialize, Serialize}; +use std::collections::BTreeSet; /// One named column with an inferred type and its cells in row order. #[derive(Debug, Clone, Serialize, Deserialize)] @@ -93,6 +94,41 @@ impl RecordSet { pub fn column(&self, name: &str) -> Option<&Column> { self.columns.iter().find(|c| c.name == name) } + + /// A copy keeping only the columns named in `names`, in their original + /// column order (not the order given). Names with no matching column are + /// silently skipped — callers that must reject an unknown name should + /// validate with [`Self::column`] first. Provenance is preserved. + /// + /// This is the column-scoping primitive behind `scan --columns`: it lets a + /// caller focus detection on a handful of meaningful columns in a wide + /// corpus (e.g. journald's dozens of identifier/counter fields). + pub fn select(&self, names: &[String]) -> RecordSet { + let keep: BTreeSet<&str> = names.iter().map(String::as_str).collect(); + self.retain(|name| keep.contains(name)) + } + + /// A copy dropping the columns named in `names`, preserving the order and + /// provenance of the rest. The complement of [`Self::select`], behind + /// `scan --exclude`. + pub fn without(&self, names: &[String]) -> RecordSet { + let drop: BTreeSet<&str> = names.iter().map(String::as_str).collect(); + self.retain(|name| !drop.contains(name)) + } + + /// Shared projection: a copy keeping columns whose name satisfies `keep`. + fn retain(&self, keep: impl Fn(&str) -> bool) -> RecordSet { + RecordSet { + source: self.source.clone(), + format: self.format.clone(), + columns: self + .columns + .iter() + .filter(|c| keep(&c.name)) + .cloned() + .collect(), + } + } } #[cfg(test)] @@ -148,4 +184,60 @@ mod tests { assert!(rs.column("a").is_some()); assert!(rs.column("z").is_none()); } + + fn abc() -> RecordSet { + RecordSet::new( + "src.csv", + "csv", + vec![ + Column::new("a", vec![Value::Int(1)]), + Column::new("b", vec![Value::Int(2)]), + Column::new("c", vec![Value::Int(3)]), + ], + ) + } + + #[test] + fn select_keeps_named_columns_in_original_order() { + // Requested order is "c,a" but the projection preserves the corpus's own + // column order (a then c); the dropped column (b) is gone; provenance kept. + let rs = abc().select(&["c".to_string(), "a".to_string()]); + let names: Vec<&str> = rs.columns.iter().map(|c| c.name.as_str()).collect(); + assert_eq!(names, ["a", "c"]); + assert_eq!(rs.source, "src.csv"); + assert_eq!(rs.format, "csv"); + assert_eq!(rs.rows(), 1); + } + + #[test] + fn select_skips_unknown_names() { + // An unknown name contributes nothing — only the real "a" survives. + let rs = abc().select(&["a".to_string(), "nope".to_string()]); + let names: Vec<&str> = rs.columns.iter().map(|c| c.name.as_str()).collect(); + assert_eq!(names, ["a"]); + } + + #[test] + fn select_empty_yields_no_columns() { + assert_eq!(abc().select(&[]).width(), 0); + } + + #[test] + fn without_drops_named_columns_and_keeps_the_rest() { + let rs = abc().without(&["b".to_string()]); + let names: Vec<&str> = rs.columns.iter().map(|c| c.name.as_str()).collect(); + assert_eq!(names, ["a", "c"]); + assert_eq!(rs.source, "src.csv"); + assert_eq!(rs.format, "csv"); + } + + #[test] + fn without_empty_keeps_everything() { + assert_eq!(abc().without(&[]).width(), 3); + } + + #[test] + fn without_unknown_name_is_a_noop() { + assert_eq!(abc().without(&["zzz".to_string()]).width(), 3); + } } diff --git a/docs/src/modes.md b/docs/src/modes.md index be4f894..131c8af 100644 --- a/docs/src/modes.md +++ b/docs/src/modes.md @@ -2,7 +2,9 @@ A plain `scan` runs the single-corpus detectors (point, structural shape checks). Three flags activate the rest; when a flag is absent, the detectors it -would enable report [honest absence](./determinism.md) rather than guessing. +would enable report [honest absence](./determinism.md) rather than guessing. A +fourth pair of flags — `--columns` / `--exclude` — narrows *which* columns are +analyzed at all. ## `--baseline B` — drift & schema diff @@ -45,3 +47,36 @@ means "time" is never guessed. > Rows are treated in their given order as the time axis. If your data isn't > already time-ordered, sort it first. + +## `--columns C,..` / `--exclude C,..` — column scope + +Restrict detection to a chosen set of columns (`--columns`, an allowlist) or to +everything *but* a set (`--exclude`, a denylist). The two are mutually exclusive. +The projection is applied before any detector runs, and to the `--baseline` too, +so drift comparison stays consistent. + +```console +# focus a wide log on the columns that carry signal +$ journalctl -o json | anomalyx scan --columns PRIORITY,_SYSTEMD_UNIT + +# or keep everything except journald's identifier/counter/timestamp noise +$ journalctl -o json | anomalyx scan \ + --exclude JOB_ID,_PID,__MONOTONIC_TIMESTAMP,__REALTIME_TIMESTAMP,N_RESTARTS +``` + +This is the answer to *identifier noise* on wide corpora. The `point` detector +will dutifully flag statistical outliers in every numeric column — including +`JOB_ID`, PIDs, monotonic timestamps and restart counters, where an "outlier" is +real but meaningless. On a raw 20k-entry journald capture that's ~10k findings of +noise; excluding those fields collapses it to a couple hundred that matter. + +The scope is **explicit, never heuristic.** anomalyx will not auto-guess which +columns are "interesting" — that would be a guess, and the obvious guess +(drop near-unique columns) would wrongly discard exactly the near-unique numeric +*measurements* the marquee detectors depend on (packet `durationNanos`, span +durations, latencies). You name the scope; the result stays deterministic. + +> A column named in `--columns`/`--exclude` that doesn't exist in the corpus is +> a hard error (exit `2`), so a typo can't silently scope a scan down to nothing +> and read as "clean". (The baseline is projected leniently — it's a different +> corpus and need not carry every scoped column.)