fix: resolve silent NaN load data when fixed_year differs from snapshot year#2209
Open
stan-buren wants to merge 4 commits into
Open
fix: resolve silent NaN load data when fixed_year differs from snapshot year#2209stan-buren wants to merge 4 commits into
stan-buren wants to merge 4 commits into
Conversation
…ot year Fixes PyPSA#2187. When fixed_year differs from the snapshot year, load data was silently filled with NaN due to reindexing before year mapping. - Reverse mapping direction: map snapshots to fixed_year first, then index into load data (approach by @gianvicolux) - Add leap year validation guard (analysis by @luciboon) - Add regression tests covering normal, leap year, and edge cases - Move assert after reindexing to catch mapping failures
for more information, see https://pre-commit.ci
Contributor
Author
|
I've refactored the fixed year reindexing logic into a standalone helper function This removes the code duplication in the test suite and ensures we are testing the actual production code. |
…e reindex to handle missing hours
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #2187. When fixed_year differs from the snapshot year, load data was silently filled with NaN due to reindexing before year mapping.
Closes #2187.
Changes proposed in this Pull Request
This PR resolves a critical issue where setting
load.fixed_yeardifferent from the snapshots' year results in a DataFrame containing onlyNaNvalues. The problem stems from performing.reindex(index=snapshots)on load data that has not yet had its year replaced.To address this, we implemented the following changes:
import calendarat the top of thescripts/build_electricity_demand.pyfile to enable leap year verification.fixed_yearusingsnapshots.map(lambda t: t.replace(year=fixed_year))first, index into the loaded dataset using this new index, and then restore the original snapshots index on the result.ValueErrorif the snapshots contain February 29th but the targetfixed_yearis not a leap year, prompting the user to drop the leap day or select a leap year.isna()assert statement to the end of the script (after reindexing) to catch any potential alignment failures.test/test_build_electricity_demand.pyto cover same-year mappings, different-year mappings (regression test for Silent NaN load data when using fixed_year with snapshots from a different year #2187), leap-to-non-leap mappings, non-leap-to-leap error raising, default behavior when nofixed_yearis set, and correctness of mapped values.doc/release_notes.md.Checklist
Required:
doc/release_notes.md.If applicable:
scripts/lib/validation.doc/*.mdfiles.