feat: script for detecting unstable producer urls

[ianktc]

Summary

is_producer_url_unstable is used to identify feeds whose urls may change over time due to a date or a version baked into it. The feed url therefore stops being accessible the moment the agency publishes again with a different version or date. Only active and inactive feeds are scanned, and deprecated feeds can be scanned if a flag is added.

This PR adds scripts/identify_unstable_urls.py, which scans producer URLs for that
evidence and applies the field to what it finds. It also adds the is_producer_url_unstable field to the realtime feed schema in case of future adoption.

The actual feed update: #1627
csv file detailing the motivation for each decision:
unstable_producer_urls.csv

Usage

The script never writes on a plain run. The intended workflow is scan, read the CSV, then
apply:

# 1. Dry run. Prints per-rule hit counts and one line per feed, writes nothing.
python scripts/identify_unstable_urls.py

# 2. Same scan, but dump the reviewable CSV.
python scripts/identify_unstable_urls.py --report unstable_producer_urls.csv

# 3. Read the CSV. Then, once satisfied, write the field.
python scripts/identify_unstable_urls.py --apply

Other options: --data-type {gtfs,gtfs-rt,all} to narrow the catalog, --include-deprecated
to also scan deprecated feeds (raises the count from 86 to 126), and --weak to print the
signals that are deliberately never flagged, for manual review.

Redaction rules — Tokens to ignore (allow-list)

Blanked to a sentinel before any matching runs, so they cannot contribute a hit.

Rule	Matches	Why it's stable
uuid	8-4-4-4-12 hex	CKAN dataset/resource keys
hex_id	exactly 32 hex	ArcGIS item ids, Wix/hibu site keys, opendatasoft file ids. Deliberately not 40 — that's a git SHA
azure_service_version	sv=YYYY-MM-DD	Azure storage API version. The freshest-looking date in a SAS URL and it never moves
azure_signature	sig=…	SAS signature blob
credential	api_key/token/key/subscription-key=…	A 40-hex Mecatran apiKey is indistinguishable from a pinned commit; 8 feeds were mis-flagged before this
api_version_segment	/vN/ as a whole segment	API version, not feed version
api_version_segment_dotted	/vN.0/ as a whole segment	A zero minor is decorative — data.waltti.fi serves 11 feeds from /v1.0/
transitfeeds_id	/p/<agency>/<id>/latest	TransitFeeds numeric feed key
drive_file_id	?id=<25+ chars>	Google Drive file id

Detection rules — Token to trigger hit

Setis_producer_url_unstable to "True"

Rule	Catches
iso_date	YYYYMMDD / - / _ / . separated, plus optional HHMM/HHMMSS
delimited_year	standalone year 2010–2029, delimiter-anchored both sides
wordpress_upload_folder	wp-content/uploads/[sites/N/]YYYY/MM/
dated_version_tag	2020.0401.1 style release tags
dotted_version	v2.8, ?v=0.16.0, v1.0.0
locale_date	day-first 03.10.2025, 06-07-2021, 5.12.21
month_abbreviation_with_digits	3-letter month only fused to digits: coquimbo10feb16
version_query_parameter	&version=20250220
epoch_timestamp	10-digit Unix seconds
month_name	full month names, 6 languages
dotnet_ticks	18-digit DateTime.Ticks
labelled_year	2025Winter, Spring2025, Q1-CY2025, CY2026
shared_access_signature_window	SAS st=/se= expiry
cache_buster	refresh=/nocache=/timestamp= + 8+ hex
pinned_git_sha	40 hex as a path segment
large_version_counter	vNN+ — GTFS-V126
year_month	bare YYYYMM — gtff-202311

[emmambd]

Someone else on the team is probably better positioned to review this one than me. Are there specific requirements you wanted to discuss?

[ianktc]

Someone else on the team is probably better positioned to review this one than me. Are there specific requirements you wanted to discuss?

I saw comments on the other pr #1629 so I'll refine the detection rules so we don't catch false positives. I'll bring it up to the team after I can finalize them with you

[emmambd]

@ianktc Sounds good - looking at the "Detection rules" here, there may be some that I mistook as a false positive

[ianktc]

Update

Detection rules are now constrained to scan just for date-related tokens. The following rules were removed:

dotted_version, dotnet_ticks, shared_access_signature_window, cache_buster, pinned_git_sha, large_version_counter

The sentinels that were previously required to safeguard against false positives are no longer required, because the above rules were removed. (api key, hash, version numbers are no longer mistaken as date/time tokens).

Updated csv of marked feeds

unstable_producer_urls.csv

Detection rules — any match sets is_producer_url_unstable to "True"

Counts are hits across the 2,857 non-deprecated feeds scanned.

Rule	Catches
iso_date	YYYYMMDD / - / _ / . separated, plus optional HHMM/HHMMSS
delimited_year	standalone year 2010–2029, delimiter-anchored both sides
wordpress_upload_folder	wp-content/uploads/[sites/N/]YYYY/MM/
dated_version_tag	2020.0401.1 style release tags
locale_date	day-first 03.10.2025, 06-07-2021, 5.12.21
month_abbreviation_with_digits	3-letter month only fused to digits: coquimbo10feb16
version_query_parameter	&version=20250220
epoch_timestamp	10-digit Unix seconds
month_name	full month names, 6 languages
labelled_year	2025Winter, Spring2025, Q1-CY2025, CY2026
year_month	bare YYYYMM — gtff-202311

[emmambd]

@ianktc These make sense to me, upon reviewing the CSV

[ianktc]

@emmambd Ok, will be re-running the script and updating the other pr with the results