feat(ocsf): OCSF-to-proto generator (ocsf-protogen) [AI-1408]#174
Merged
Conversation
✅ Snyk checks have passed. No issues have been found so far.
💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse. |
sago2k8
changed the title
Loads the OCSF 1.8.0 export/v2 compiled schema (classes, objects, dictionary, types, enums) into typed Go. Fixture fetched from schema.ocsf.io/1.8.0/export/v2/schema. Errors on a missing schema version. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
OCSF *_t types resolve via their base chain (timestamp_t->int64, float_t->double, json_t->google.protobuf.Value, string-derived->string). Enums reuse OCSF 0=Unknown as the proto zero value, prefix value names to avoid collisions, guard against int32 overflow, and fall back to a string field for string-keyed enums. Dispatch sentinels are grouped constants. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Append-only (message,attribute)->tag map: lowest-free allocation skipping the 19000-19999 proto-reserved range, retire-to-reserved so tags are never reused, load-time validation (positive tags, no dupes, no reserved-range tags), and CheckCompat rejecting changed, dropped-unreserved, reused-reserved, or un-reserved tags. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Deterministic proto3 emission: nested enums for int-keyed enums, stable field numbers via the tag map, protovalidate for required fields and at_least_one/just_one constraints, conditional well-known imports, and reporting of objects emitted as stubs. Unresolvable non-object types error rather than silently degrading. The proto package is the OCSF major version (ocsf.v1); minor bumps are additive within it. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Reflection-based marshaller producing schema-valid OCSF JSON: enums always emit as integers (singular, nested, repeated, map), 64-bit ints as JSON numbers (exact beyond 2^53), snake_case keys, and unset fields omitted. Fixes the three ways stock protojson diverges from OCSF, proven by a divergence test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
generate mode emits the proto and saves the tag map; --check gates on output drift and newly-stubbed objects; --compat-check runs CheckCompat between the base-ref and current field-number maps to enforce wire stability. Includes the committed baseline proto and field-numbers.json. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
End-to-end acceptance test: the generated 6003/3004 protos compile to Go,
populated events round-trip through proto binary, and their OCSF-JSON export
is validated live via POST /{version}/api/v2/validate (gating on error_count,
checking HTTP status, skipping only on genuine connectivity failure). Events
declare metadata.profiles for profile-gated attributes. genpb regenerates
from the module root into conformance/genpb.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds the ocsf module to the test and lint matrices and an ocsf-checks job running buf lint/build, ocsf-protogen --check, the git-based --compat-check against the base ref, and the live OCSF conformance test. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds the ocsf module's README and BSL LICENSE, a CODEOWNERS entry (@redpanda-data/team-ai), and a .licenseupdater.yaml entry, satisfying the repo Project Structure Check. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Emit now returns []GeneratedFile (one .proto per selected class plus a shared ocsf/v<N>/objects.proto) instead of a single monolithic file. --out becomes a module-root directory; --check compares the full generated file set against the committed tree (add/remove/change). Field numbers are keyed by message+attribute and are byte-identical to the prior baseline. Regenerated golden tree, cmd testdata tree, and genpb bindings; dropped the now-unneeded PACKAGE_DIRECTORY_MATCH lint waiver. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The generic ocsf.v<N> protos carry no Schema-Registry annotation, so there
is no schema text a consumer can register directly. Add EmitSRSchemas, which
emits one self-contained <class>.sr.proto per class for explicit SR
registration by consumers.
Each file inlines the class message plus its full transitive object closure:
the class message is emitted first so it lands at Confluent message-index 0
(what decoders rely on). All buf.validate output is stripped (no
per-field required annotations, no message-level CEL blocks) and the objects
file is never imported. Field numbers reuse the wire tagmap via the idempotent
Assign, so tags are identical to the main Emit output.
emitMessage gains an emitOptions{omitValidate} parameter threaded from the SR
path; the main callers pass the default (validate on), leaving existing golden
output unchanged. The CLI gains --sr-schema-out <dir>, integrated with --check
for drift detection. The sr/ golden dir is excluded from the buf module since
each .sr.proto is a standalone artifact, not part of the module.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A single audit-log topic needs ONE message covering every selected class, not one message per class. MergeClasses unions the attribute maps of N classes into a single flat definition, the schema-level half of the merged single-event layout. Merge rules per attribute name: - Type, object type, and is_array must agree across every declaring class; a mismatch is a hard generation error since the proto field type would otherwise depend on class selection. - Integer-keyed enums union by value. When the same numeric value carries different captions in two classes (a class-scoped enum), the attribute is demoted to its plain scalar type instead of failing: OCSF defines such semantics via the (class_uid, value) pair, which the merged message keeps. Enum-presence and key-kind mismatches demote too. - activity_id is demoted unconditionally. Its values genuinely collide across OCSF classes (1 = Create in api_activity, Logon in authentication, Assign Privileges in group_management), and demoting it later would break generated-code consumers, so it is int32 from day one; readable per-class names live in the disjoint-by-construction TypeUid union. - Requirement merges to "required" only when required in EVERY class; a blanket annotation would wrongly reject events of classes that do not use the attribute. Per-class requiredness becomes gated CEL in the emitter. The returned Merged also carries per-attribute owner classes, feeding the emitter's class-ownership validation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
EmitMerged turns a MergeClasses result into the single-topic layout: one flat message (e.g. AuditEvent) holding the union of all selected classes' attributes, next to the same shared objects.proto the per-class layout uses (byte-identical: same closure, same tagmap keys). EmitMergedSRSchema emits the one self-contained <name>.sr.proto for the topic, merged message first so it stays at Confluent message-index 0. Class awareness that the per-class layout got from separate message types is regenerated as protovalidate message-level CEL: - type_uid == class_uid * 100 + activity_id (the OCSF invariant that makes the demoted int32 activity_id safe), - ownership gates: attributes owned by a strict subset of classes may only be set when class_uid matches an owner, - per-class conditional requiredness for attributes required by one class but not all (repeated attributes excluded, matching the per-class rule), - per-class at_least_one/just_one constraints, gated on class_uid. emitMessage gains fieldComments (demoted class-scoped enums get an explanatory comment) and extraCEL (pre-rendered option blocks), both suppressed by omitValidate on the SR path; the constraint helpers are refactored to expose expression-level variants for the gated forms. Field numbers are a fresh append-only lineage keyed by the merged message name; a test proves adding a class to the selection never moves an existing tag. Golden output for api_activity + entity_management is committed under testdata/golden-merged and passes buf lint/build. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
--merged-message <Name> (e.g. AuditEvent) emits the merged single-event layout IN ADDITION to the per-class files, into the same output tree and sharing the same tagmap. Both paths emit an identical objects.proto; the duplicate is dropped after an equality check so any divergence surfaces as an error instead of a silent overwrite. With --sr-schema-out, the merged <snake_name>.sr.proto lands next to the per-class SR schemas. --check covers the merged files with the same drift and stray-file detection. The committed testdata baseline is regenerated with the flag: it gains ocsf/v1/audit_event.proto, and field-numbers.json gains the AuditEvent lineage (purely additive; --compat-check against the previous map passes). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Pass --merged-message AuditEvent to the baseline-drift check so CI verifies the committed audit_event.proto and the AuditEvent tagmap lineage exactly like the per-class files (the existing --compat-check already covers the new lineage since the tagmap is shared). Add buf lint/build over the golden-merged tree used by the gen tests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The exporter only went one way; audit events also need to come back from OCSF JSON (topic consumers, Postgres JSONB columns, events produced by other OCSF tools). FromOCSFJSON is the exact inverse of ToOCSFJSON: - snake_case keys resolve by proto field name; enums decode from OCSF integer values, keeping unknown numbers verbatim (proto3 enums are open, so newer-revision events survive), - 64-bit integers decode from unquoted JSON numbers without float64 precision loss, plus quoted decimal strings for protojson interop; fractional or exponent forms into integer fields are rejected rather than truncated, - google.protobuf.Value/Struct/ListValue fields take their natural JSON; for Value fields JSON null decodes to a set NullValue (protojson semantics), preserving the present-with-null vs absent distinction, - JSON null on other fields leaves them unset; null is rejected as a message body or array element (encoding/json would otherwise no-op and silently zero the value), - unknown keys land in the message's unmapped field (merged with any explicit unmapped object), mirroring OCSF's forward-compat model; a message without one errors, naming the keys. Conformance now proves both directions on the real generated classes: proto -> JSON -> proto is proto.Equal-lossless, JSON -> proto -> JSON is byte-identical, and spliced unknown keys survive via unmapped. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Generate Go bindings for the merged layout (buf.gen.merged.yaml over golden-merged; objects.proto maps to the same Go package as the per-class bindings and is byte-identical, so only audit_event.pb.go is new) and prove the four properties the single-topic design rests on: 1. Wire round-trip on the AuditEvent bindings for events of both classes. 2. JSON equivalence: an AuditEvent exports OCSF JSON deeply equal to the per-class message carrying the same event, and per-class JSON decodes into the equivalent AuditEvent — one topic loses nothing. 3. JSON round-trip both ways (lossless / byte-identical). 4. protovalidate (new test dependency buf.build/go/protovalidate) enforces the generated class-aware CEL: valid events of both classes pass, and type_uid inconsistency, foreign-class fields, and missing per-class required fields are each rejected. The merged events also validate CLEAN against the live OCSF server (schema.ocsf.io api/v2/validate), same skip-on-connectivity contract as the per-class conformance test. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Found by fuzzing the round-trip: a proto string field carrying invalid UTF-8 exported "successfully" with the invalid bytes silently replaced by U+FFFD by encoding/json — data corruption reported as success, and an asymmetric round-trip. proto3 requires string fields to be valid UTF-8 (proto.Marshal enforces the same), so ToOCSFJSON now rejects invalid UTF-8 in string fields and string map keys with an explicit error. Regression tests land with the adversarial suite in the next commit. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Hostile-input coverage for ToOCSFJSON/FromOCSFJSON on the sample message and on the real generated classes: - malformed JSON (truncated, garbage, wrong top-level types, control characters in keys) must error, never panic; - type confusion for every field kind, with the error naming the field; - integer boundaries: int64/int32 min/max accepted exactly, off-by-one overflow rejected, fractional and exponent forms into integer fields rejected instead of truncated; NaN/Inf export rejected; - hostile strings (unicode, JSON-injection attempts, 1 MiB payloads, null bytes) round-trip byte-exactly without smuggling fields past the class discriminator; - 1000-deep nesting, duplicate keys, null array elements, and full truncation/mutation sweeps over real event JSON; - seeded property smoke tests: hundreds of randomized events (SampleEvent and AuditEvent across both classes) must round-trip proto -> JSON -> proto losslessly and JSON -> proto -> JSON byte-identically; - three fuzz targets whose seed corpus runs on every go test run; extended runs (>1M execs) are clean. Fuzzing found and this suite pins down three real defects: the invalid UTF-8 corruption fixed in the previous commit, JSON null dropping a set google.protobuf.Value on import, and a nil-map panic when unknown keys merge into an explicit null unmapped (crash input committed under testdata/fuzz as a permanent regression corpus). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Describe the two generated layouts (per-class and merged single-event), the activity_id demotion rationale, the FromOCSFJSON round-trip contract, and the --merged-message / --sr-schema-out flags. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
--merged-sr-subject <subject> emits the (redpanda.api.common.v1.schema_registry) message option (and its import, resolved from buf.build/redpandadata/common) on the merged event message. Consumers that run protoc-gen-go-sr-normalize in their buf pass then get the self-contained SR schema text and subject as generated Go constants (<Name>SRSchema / <Name>SRSubject) directly from the emitted proto — no hand-copied schema files. The --sr-schema-out path remains as the non-buf route to the same schema text. The annotation is a message option, so field numbers and the tagmap are untouched; the committed baseline diff is the option block plus one import. The .sr.proto emitters never carry the option: they must stay self-contained, and a test now pins that. A conformance test reads the extension back from the compiled AuditEvent descriptor exactly the way protoc-gen-go-sr-normalize does, proving the annotation survives generation end to end. --merged-sr-subject requires --merged-message and is enforced. The golden-merged and cmd testdata buf modules gain the redpandadata/common dep; CI passes the new flag in the baseline-drift check. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Exporter/importer scenarios added:
- Unicode escapes: escaped JSON keys must resolve to proto fields,
surrogate pairs round-trip, and a lone surrogate (which encoding/json
replaces with U+FFFD) must decode to valid UTF-8 and re-export cleanly.
- Document-level oddities: BOM prefix, comments, and concatenated JSON
documents are rejected; exotic-whitespace documents parse.
- More number formats: -0 into int64, float-valued enums (5.0) rejected
rather than coerced, float64 overflow (1e309) errors instead of
becoming +Inf, plus-sign/hex/NaN/Infinity literals rejected.
- Presence semantics pinned: implicit-presence zeros are accepted on
import and canonicalized away on export; explicit-presence (optional)
empty values keep byte-exact round-trips.
- 1000-deep array nesting, backslash added to the mutation sweep, a
concurrent round-trip smoke test for the race detector, and new fuzz
seeds for all of the above.
Conformance (real AuditEvent) scenarios added:
- unmapped stress: 500+ hostile unknown keys (empty, unicode, quoted,
4 KiB names) all land in unmapped and stay idempotent.
- unmapped conflict semantics pinned: non-object unmapped plus unknown
keys errors; top-level unknown keys win over duplicate entries in an
explicit unmapped object; scalar unmapped without unknowns decodes.
- Importer-to-CEL loop: hostile-but-well-formed JSON (foreign-class
field, type_uid mismatch, missing class-required field) must DECODE
losslessly and then fail protovalidate on exactly the right generated
rule, asserted via structured violation rule ids rather than error
text. The empty event decodes, exports back to {}, and fails
blanket-required validation.
- Property test variant splicing random unknown keys into random events:
decode, unmapped population, and post-splice idempotence.
Suite passes under -race; fuzz targets re-run clean on the extended
seed corpus.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Findings from a dedicated test-coverage review, in priority order: - Six proto field kinds (map, bytes, float, uint32/64, fixed, sint) had zero reachability anywhere in the module: OCSF's type system never produces them, but ToOCSFJSON/FromOCSFJSON are documented generic over proto.Message. The sample fixture now declares all of them, the randomized generator populates them, and dedicated cases cover base64 round-trip and malformed input, uint64/fixed32 bounds and overflow, negative-into-unsigned, sint64 extremes, float32 quantization, hostile map keys, null/mistyped map values, duplicate-key semantics, and key-sorted deterministic map export. - Tag stability is now proven across a real on-disk tagmap save/load cycle with a widened class set (the exact --compat-check workflow), including tagmap.CheckCompat on the persisted maps; the previous test only covered one in-memory tagmap. - New genpb sync guard: field name→tag maps parsed from the committed golden protos must equal the compiled descriptors of the committed Go bindings, so editing a golden without regenerating genpb can no longer leave conformance tests passing against stale bindings. - The string-keyed and mixed-kind branches of the enum union in MergeClasses (previously dead in tests) get conflict, union, and kind-mismatch cases. - TestMergedProtovalidate now asserts structured violation rule ids instead of error-text substrings, matching the adversarial suite. - SR subjects are validated against [A-Za-z0-9._-] at emit time instead of trusting Go %q escaping to be proto-literal-legal for exotic input, with hostile-subject rejection tests. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Three low-severity findings from a dedicated correctness review (which found no high-severity issues and confirmed enum-slice aliasing, CEL semantics, importer reflection, UTF-8 rejection, objects.proto dedupe, and wire stability all sound): - decodeUint parsed via fmt.Sscanf %d, which accepts a fractional JSON number like 12.9 and silently truncates it to 12. Latent for OCSF types (no OCSF attribute maps to an unsigned kind) but FromOCSFJSON is generic over proto.Message; now strconv.ParseUint rejects it, with regression tests on the uint64 and fixed32 fixture fields. - diffSRSchemas only checked generated-vs-committed one way: dropping a class from --classes left its stale <class>.sr.proto committed and CI green. Stray *.sr.proto files now fail --check, mirroring the main tree's stray-file detection. - --merged-message naming a selected class's PascalCase message name (e.g. ApiActivity) would silently share one (message, attribute) tag lineage between two distinct messages. Now rejected with an explicit error via the exported gen.ClassMessageName. Also bumps the regenerated sample.pb.go license header year, which the Format Check's licenseupdater rewrites (the previous push's only CI failure). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The conformance suite's protovalidate dependency was added at @latest (v1.2.0, requiring cel-go 0.28). Consumers that pin this module as a Go tool (cloudv2) then get MVS-bumped to protovalidate v1.2.0, whose violation MESSAGE WORDING changed (e.g. "value must contain at least 1 item(s)" became "must contain at least 1 item(s)"), breaking every downstream test that asserts exact violation strings — ~15 failures across unrelated cloudv2 services. Pin to v1.0.0 / cel-go 0.26.1, matching cloudv2's resolution. The suite's own assertions use structured rule ids, not message text, and pass unchanged on both versions; v1.0.0 just stops this module from forcing a protovalidate behavior change on consumers. Revisit when the ecosystem moves to v1.2+ deliberately. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Same consumer-compatibility rationale as the protovalidate-go pin: the buf.build/gen/go/bufbuild/protovalidate/protocolbuffers/go snapshot carries the rule MESSAGE TEXT protovalidate renders, and the 2026-04 snapshot rewrote it (e.g. "value must be a valid HTTP header value" became "must be a valid HTTP header value"). Any consumer pinning this module gets MVS-bumped and every downstream test asserting exact violation strings breaks. Pin to the 2025-09 snapshot cloudv2 main resolves. Our own assertions use structured rule ids and field paths, not upstream message text; the full suite passes identically. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
birdayz
approved these changes
Jul 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Epic: AI-1408
Summary
Adds
ocsf, a new module that turns the compiled OCSF (Open Cybersecurity Schema Framework) security-event schema into wire-stable proto3, plus an OCSF-JSON exporter and a generator CLI. This is the foundation for emitting Redpanda authorization and audit activity as OCSF events; consumers pin an OCSF version and generate only the classes they need.The OCSF project ships no proto binding, and the one community generator (
valllabh/ocsf-tool) renumbers fields on every schema change (wire-unstable), drops requirement levels, and doesn't compose profiles. This module fixes those.What's here
internal/ocsf/schema— loads the OCSF 1.8.0export/v2compiled schema into typed Go.internal/ocsf/gen— OCSF→proto type/enum mapping and deterministic proto3 emission (nested enums, protovalidate, conditional imports).internal/ocsf/tagmap— the wire-stability core: an append-only(message, attribute) → tagmap with a compatibility check.internal/ocsf/exporter— marshals a generated proto message to schema-valid OCSF JSON.cmd/ocsf-protogen— the CLI (--check,--compat-check).internal/ocsf/conformance— end-to-end acceptance tests.Highlights
ocsf-protogen --compat-checkdiffs the committed tag map against the base branch in CI and fails on any incompatible change.protojsonis invalid OCSF on three counts (enum names instead of integers, 64-bit ints as quoted strings, camelCase keys). The reflection-based exporter always emits integer enums, numeric int64 (exact beyond 2^53), and snake_case keys, with a divergence test that proves it differs fromprotojsonon exactly those points.POST /{version}/api/v2/validate), gating onerror_count.Notes
GET https://schema.ocsf.io/{version}/export/v2/schema(the legacy/export/schemareturns 400 for 1.8.0). Pinned to OCSF 1.8.0.ocsf-checksCI job and skips locally only on genuine connectivity failure (a 200 with errors is a failure, never a skip).metadata.profilesfor profile-gated attributes (cloud,disposition_id,authorizations); the conformance events do this.Test
go test ./...green across all six packages;go vetclean;buf lint/buf buildpass on the generated proto; each commit builds independently.🤖 Generated with Claude Code