Skip to content

fix: resolve silent NaN load data when fixed_year differs from snapshot year#2209

Open
stan-buren wants to merge 4 commits into
PyPSA:masterfrom
stan-buren:fix/silent-nan-fixed-year
Open

fix: resolve silent NaN load data when fixed_year differs from snapshot year#2209
stan-buren wants to merge 4 commits into
PyPSA:masterfrom
stan-buren:fix/silent-nan-fixed-year

Conversation

@stan-buren

Copy link
Copy Markdown
Contributor

Fixes #2187. When fixed_year differs from the snapshot year, load data was silently filled with NaN due to reindexing before year mapping.

  • Reverse mapping direction: map snapshots to fixed_year first, then index into load data (approach by @gianvicolux)
  • Add leap year validation guard (analysis by @luciboon)
  • Add regression tests covering normal, leap year, and edge cases
  • Move assert after reindexing to catch mapping failures

Closes #2187.

Changes proposed in this Pull Request

This PR resolves a critical issue where setting load.fixed_year different from the snapshots' year results in a DataFrame containing only NaN values. The problem stems from performing .reindex(index=snapshots) on load data that has not yet had its year replaced.

To address this, we implemented the following changes:

  1. Added import calendar at the top of the scripts/build_electricity_demand.py file to enable leap year verification.
  2. Reversed mapping order: Instead of slicing the year and reindexing immediately, we now map the snapshots' timestamps to the target fixed_year using snapshots.map(lambda t: t.replace(year=fixed_year)) first, index into the loaded dataset using this new index, and then restore the original snapshots index on the result.
  3. Leap year guard: Added a check that raises a descriptive ValueError if the snapshots contain February 29th but the target fixed_year is not a leap year, prompting the user to drop the leap day or select a leap year.
  4. Moved asset checks: Relocated the isna() assert statement to the end of the script (after reindexing) to catch any potential alignment failures.
  5. Added Regression Tests: Created test/test_build_electricity_demand.py to cover same-year mappings, different-year mappings (regression test for Silent NaN load data when using fixed_year with snapshots from a different year #2187), leap-to-non-leap mappings, non-leap-to-leap error raising, default behavior when no fixed_year is set, and correctness of mapped values.
  6. Added Release Notes: Documented this bugfix under the "Upcoming Release" section in doc/release_notes.md.

Checklist

Required:

  • Changes are tested locally and behave as expected.
  • Code and workflow changes are documented.
  • A release note entry is added to doc/release_notes.md.

If applicable:

  • Changes in configuration options are reflected in scripts/lib/validation.
  • For new data sources or versions, these instructions have been followed.
  • New rules are documented in the appropriate doc/*.md files.

stan-buren and others added 3 commits June 18, 2026 16:14
…ot year

Fixes PyPSA#2187. When fixed_year differs from the snapshot year, load data
was silently filled with NaN due to reindexing before year mapping.

- Reverse mapping direction: map snapshots to fixed_year first,
  then index into load data (approach by @gianvicolux)
- Add leap year validation guard (analysis by @luciboon)
- Add regression tests covering normal, leap year, and edge cases
- Move assert after reindexing to catch mapping failures
@stan-buren

Copy link
Copy Markdown
Contributor Author

I've refactored the fixed year reindexing logic into a standalone helper function reindex_load_fixed_year in build_electricity_demand.py and imported it directly in test_build_electricity_demand.py.

This removes the code duplication in the test suite and ensures we are testing the actual production code.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Silent NaN load data when using fixed_year with snapshots from a different year

1 participant