`run` silently skips regeneration when only inline transformation logic changes, re-serving stale output

[mark124]

First off — thanks for earthmover. The lineage/manifest design and the "report unmapped fields rather than absorb them" posture are genuinely good. This is a data-correctness report with a minimal repro and a suggested fix.

Summary

When a project uses a state_file, earthmover's change-detection hashes the config: block, source files, destination template files, and map_values map_files — but not the inline transformation logic in the YAML body. So editing an inline add_columns Jinja expression, an inline map_values mapping:, a filter_rows query, or a join's join_type changes none of the hashes. earthmover reports "skipping (no changes since the last run)", exits with code 99, and re-serves the previous run's output — with no regeneration. A common analyst fix (correcting a mapping or expression in the config) is silently ignored.

Impact

This is silent and sits on the correctness path. The 99 exit code is earthmover's sentinel for a deliberate no-op skip — the same code returned when a run is legitimately up-to-date — so nothing distinguishes a wrongly-skipped run from a correctly-skipped one: not the log (only the normal "skipping (no changes)" line), not the exit code, not the output (unchanged on disk). You fix the transformation, re-run, and the old wrong data ships.

Version

earthmover 0.4.10 (repro also present on main @ 0c73e2b).

Minimal reproduction

mkdir em-stale && cd em-stale
printf 'id,value\ns1,10\ns2,21\n' > data.csv
printf '{"id":"{{ id }}","result":{{ result }}}\n' > tmpl.jinja
cat > earthmover.yaml <<'YAML'
version: 2
config:
  output_dir: ./output
  state_file: ./runs.csv
sources:
  input:
    file: ./data.csv
    header_rows: 1
transformations:
  doubled:
    source: $sources.input
    operations:
      - operation: add_columns
        columns:
          result: "{% raw %}{{ value|int * 2 }}{% endraw %}"
destinations:
  out:
    source: $transformations.doubled
    template: ./tmpl.jinja
    extension: jsonl
    linearize: True
YAML

earthmover run -c earthmover.yaml
cat output/out.jsonl          # result = 20, 42   (correct)

Now fix the transformation logic — change * 2 to * 3 in the add_columns line — touching no source file, template file, or map_file, and re-run:

earthmover run -c earthmover.yaml
# logs: "skipping (no changes since the last run ...)"  and exits 99
cat output/out.jsonl          # STILL result = 20, 42   <-- the *3 edit was silently ignored

Expected: the transformation change is detected and out.jsonl is regenerated with result = 30, 63.
Actual: earthmover treats the run as unchanged, skips generation, and re-serves the old *2 output.

(Passing --skip-hashing or --force, or deleting runs.csv, regenerates correctly — so a workaround exists once you know the cause, but the default is silent.)

Root cause

earthmover/runs_file.py :: RunsFile._build_hashes (runs_file.py:120; config hash at line 139) builds the run signature from:

• config_hash = get_string_hash(json.dumps(self.earthmover.state_configs)) — only the config: block, not the source/transformation/destination node definitions
• sources_hash — source file contents
• templates_hash — destination template file contents
• mappings_hash — only map_values operations that use an external map_file

Inline transformation/source/destination definitions in the YAML are never hashed, so get_newest_compatible_run() returns the prior run (its config_hash matches), find_hash_differences() on it returns [], do_generate is set False, and generate() exits 99 before execute() runs.

Suggested fix

Fold the parsed node definitions into the signature — e.g. add a hash of the normalized transformation, source, and destination configs (the YAML bodies, not just the referenced files) to _build_hashes. Then any edit to inline logic invalidates the prior run the same way a source-file edit does.

Note

I hit this while auditing earthmover for silent-failure modes. I have a small test that pins this exact behavior (and confirms it fails once the hash includes the transformation logic) — happy to open a PR with the fix + regression test if that's welcome.

[mark124]

Happy to put up a PR for this, there's just a judgment call in the fix that felt like yours to make.

Simplest version is to fold the transformation/source/destination node definitions into the run hash, so editing an inline mapping or expression invalidates the last run the same way editing a source file does. The one catch is it'd invalidate everyone's existing state files on the first run after the change, so every project regenerates once. If you'd rather not do that to people it could go behind a flag or config opt-in instead.

Let me know which way you want it and I'll write it up. Appreciate the tool by the way, the lineage and manifest stuff has been really solid to work with.