Add a hidden `_prov` attribute for extrinsic provenance at pipeline boundaries

[dimitri-yatsenko]

The gap

Intrinsic provenance is structural: a Compute table's row cannot exist unless its declared upstream exists and is correct, so the foreign-key graph is the lineage. Nothing needs to be recorded for that to hold.

At the boundary, the structure runs out. Rows arrive from outside the pipeline — an Entry table filled by a person or a feed, an Ingest table's make() reading a file the workflow does not track, or a fan-out write into Entry tables that carry no foreign key back to the writer. For those rows the framework has no way to say where they came from, and today each pipeline invents its own: a source_file column here, a notes varchar there, an ingestion log somewhere else, or nothing at all.

That is the one place where DataJoint's provenance story depends entirely on the workflow author's diligence, with no shape to conform to.

The precedent this should follow

The mechanism already exists in the codebase. config.jobs.add_job_metadata adds hidden attributes to Computed/Imported tables at declaration:

_job_start_time  datetime(3)
_job_duration    float
_job_version     varchar(64)

(src/datajoint/adapters/base.py, and per-adapter in mysql.py / the PostgreSQL path.) These are per-row, hidden from heading, written by populate, and durable on the table itself rather than in the job queue. That is the right shape and the right place — it just covers the automated side, where provenance is already intrinsic, and not the boundary, where it is not.

Proposal

A hidden _prov attribute, JSON-typed, on tables that take rows from outside the pipeline.

_prov  json  DEFAULT NULL   # extrinsic provenance for a row that entered from outside

• Where: available on Entry and Ingest tables. A Compute table has no use for it, since its provenance is entailed.
• Who writes it: the workflow author, at the point of entry — insert on an Entry table, or a fan-out write from inside a make(). The framework provides the slot and the shape; it cannot infer content it did not produce.
• What it holds: at minimum the agent, the external source and record identifier, the time, and by what means. A conventional key set matters more than a rigid schema — the point is that two pipelines answering "where did this row come from" answer it in the same shape.
• Master carries it; parts inherit. Consistent with how the master/part relationship works elsewhere.
• Fan-out: inside make(), the intrinsic record of the ingesting table (its key, its _job_version) is available to propagate into each extrinsic destination, which is what makes a fanned-out row traceable without a foreign key.

Even when the source offers nothing useful — a nightly sync against a colony-management API — the author still records "received from PyRat at 02:15". A slot with a weak value beats no slot.

Why in the framework rather than left to each pipeline

Three things follow from having one shape:

1. Export becomes mechanical. W3C PROV wants wasAttributedTo and wasDerivedFrom on exactly these rows; OpenLineage wants the same content run-centric. With per-pipeline conventions, every export is bespoke.
2. The boundary becomes inspectable. "Which Entry rows have no recorded origin" turns into a query rather than an audit.
3. ALCOA+ attributability lands where it belongs. Deployments that must answer attributable and contemporaneous for externally-sourced data currently have nowhere standard to put the answer.

Open questions

• Default on or off? add_job_metadata defaults to False, and tables declared without it never get the columns — a migration edge worth not repeating. A _prov slot that is absent on most tables is a slot nobody codes against.
• Validation. Enforce a minimal key set at insert, or accept any JSON and let deployments constrain it? Leaning toward the latter in the framework, with strictness as a deployment concern.
• Interaction with allow_direct_insert. A direct insert into an Ingest table is already a modeling smell (datajoint-docs#267); should it require _prov?
• Naming. _prov is short and matches the _job_* convention. _source or _origin would read more plainly to someone who has not met the term.

Related: datajoint-docs#267 (tier names on the entry/ingest axis), and the fan-out ingestion explanation, which currently tells authors to record source identity without giving them a place to record it.