Preserve time and timezone for regex-extracted dates (#174)#194
Open
apoorvdarshan wants to merge 1 commit into
Open
Preserve time and timezone for regex-extracted dates (#174)#194apoorvdarshan wants to merge 1 commit into
apoorvdarshan wants to merge 1 commit into
Conversation
Dates extracted from free text and JSON-LD go through regex patterns (JSON_PUBLISHED, JSON_MODIFIED, TIMESTAMP_PATTERN) that only captured the YYYY-MM-DD part, so find_date dropped the time of day and timezone even when the requested outputformat asked for them (e.g. '%Y-%m-%d %H:%M:%S%z'). Capture the optional ISO-8601 time/timezone suffix in these patterns and, when the output format contains time or timezone directives, parse the full timestamp through custom_parse (the same routine the HTML-markup path already uses). Date-only output is unchanged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #174
Root cause
When a date is found in HTML markup (e.g. a
<meta property="article:published_time">tag), the full ISO-8601 string is handed tocustom_parse, which usesdatetime.fromisoformat/dateutiland keeps the time of day and time zone.When a date is instead extracted from free text or a JSON-LD block, it goes through regexes.
JSON_PUBLISHED,JSON_MODIFIEDandTIMESTAMP_PATTERNonly captured theYYYY-MM-DDpart in group 1, andpattern_searchconverted just that substring withconvert_date(match[1], "%Y-%m-%d", ...). Any trailingThh:mm:ss±tzwas discarded, sofind_datereturned midnight with no timezone even when the requestedoutputformatexplicitly asked for time/tz.Reproduction
Observed (on
master), time and timezone dropped:The same happens for a timestamp sitting in the body text (handled by
TIMESTAMP_PATTERN):After the fix, both return:
Fix
TIME_TZ_REsub-pattern for the optional ISO-8601 time/timezone suffix and letJSON_PUBLISHED,JSON_MODIFIEDandTIMESTAMP_PATTERNcapture it as an optional second group.pattern_search, when that suffix is present and the output format contains a time/timezone directive, the fulldate + suffixstring is parsed through the existingcustom_parse(the same routine the HTML-markup path already uses) instead of the date-only substring.The change is localized to
pattern_searchand those three patterns. When the output format is date-only (the default%Y-%m-%d), or when no time suffix is present, behaviour is byte-for-byte unchanged — the code falls back to the previousconvert_date(match[1], "%Y-%m-%d", ...). TheZsuffix is handled viacustom_parse's existingdateutilfallback on Python 3.10 wherefromisoformatdoes not accept it.Tests
Added
test_free_text_timezoneintests/unit_tests.pycovering the JSON-LDdatePublished/dateModifiedand free-text-body cases with a%Y-%m-%d %H:%M:%S%zformat, and asserting the default date-only output is unchanged. The test fails on the unmodified code (2024-11-06 00:00:00) and passes with the fix.pytest tests/unit_tests.py— 25 passedruff check .— clean;ruff format— cleanmypy htmldate/— no new errors (pre-existing stub-related errors only, count unchanged)Disclosure: prepared with AI assistance; reviewed and verified locally.