Skip to content

fix(release-controller): refresh stale changelog on release notes branch - #2112

Open
pietrodimarco-dfinity wants to merge 1 commit into
mainfrom
pmarco/fix-stale-release-notes-pr
Open

fix(release-controller): refresh stale changelog on release notes branch#2112
pietrodimarco-dfinity wants to merge 1 commit into
mainfrom
pmarco/fix-stale-release-notes-pr

Conversation

@pietrodimarco-dfinity

Copy link
Copy Markdown
Contributor

Problem

ensure_published() writes the changelog with Repository.create_file() — the GitHub create endpoint, which returns 422 when the path already exists. The failure is swallowed by a bare except, and create_pull() then runs anyway:

if not [b for b in self.repo.get_branches() if b.name == branch_name]:
    self.repo.create_git_ref(...)          # branch already existed -> skipped
try:
    self.repo.create_file(path=version_path, ..., content=changelog, branch=branch_name)
except:                                     # <- bare except
    logger.warning("Failed to create file on branch %s", branch_name)
self.repo.create_pull(...)                  # <- runs anyway, off the stale blob

So a branch left over from an earlier reconciler pass keeps its original changelog forever, and a PR gets opened advertising it.

Observed in production

rc--2026-07-23_04-21 security hotfix notes were committed to their branches at 16:11 and 16:14, before #2105 pinned changelog_base (merged 16:44). The Google Docs were then correctly regenerated against the new bases — but the branches could not be:

II:publish_notes.0c121276… — Creating file on branch replica-release-notes-0c121276…
WW:publish_notes.0c121276… — Failed to create file on branch replica-release-notes-0c121276…
II:publish_notes.0c121276… — Creating pull request for branch replica-release-notes-0c121276…

Net effect — doc and PR disagreed about which commits the release contained:

doc PR
security-hotfix release-2026-07-17_04-19-base release-2026-07-23_04-21-base
deterministic-tracker-security-hotfix release-2026-07-23_04-21-security-hotfix release-2026-07-23_04-21-deterministic-tracker

Recovery required closing both PRs (#2108, #2109) and deleting both branches by hand. The reconciler cannot heal this on its own — it logs the same warning every 30s forever.

Fix

_write_changelog():

  • update_file when the path exists, so a stale branch gets refreshed instead of silently keeping old content.
  • Skip the write when content already matches, so a reconciler running every 30s doesn't push an identical commit each pass.
  • Do not open a pull request when the write failed. A PR on top of a failed write advertises content that was never committed. The exception is still caught (no crash, transient GitHub errors stay survivable) but now returns early and retries next pass.

Tests

ensure_published() previously had no coverage — every existing test mocks it out. Four added:

test asserts
…creates_file_when_branch_is_fresh create_file, no update_file
…updates_stale_changelog_on_existing_branch update_file with the new content + blob sha, no create_file
…does_not_recommit_identical_changelog neither write is called
…opens_no_pull_request_when_the_write_fails create_pull not called

Verified they actually catch the bug — with the fix reverted:

3 failed, 6 passed
FAILED …test_ensure_published_updates_stale_changelog_on_existing_branch
FAILED …test_ensure_published_does_not_recommit_identical_changelog
FAILED …test_ensure_published_opens_no_pull_request_when_the_write_fails

and with it applied, 9 passed. mypy --strict clean on both files.

Note

This is the third existence-only guard in this path today, after markdown_file() and google_docs.ensure() — neither compares content either. Those two are working as intended given a manual doc deletion; this one had no recovery path at all.

🤖 Generated with Claude Code

ensure_published() wrote the changelog with Repository.create_file(), which
is the GitHub *create* endpoint and returns 422 when the path already exists.
The failure was swallowed by a bare `except`, and create_pull() then ran
anyway -- so a branch left over from an earlier reconciler pass kept its
original changelog while a pull request was opened advertising it.

Observed in production today.  rc--2026-07-23_04-21 security hotfix notes
were committed to their branches at 16:11 and 16:14, before #2105 pinned
changelog_base.  After that merged, the Google Docs were regenerated against
the new bases but the branches could not be updated:

    II:publish_notes.0c121276... - Creating file on branch replica-release-notes-0c121276...
    WW:publish_notes.0c121276... - Failed to create file on branch replica-release-notes-0c121276...
    II:publish_notes.0c121276... - Creating pull request for branch replica-release-notes-0c121276...

The resulting PRs cited release-2026-07-23_04-21-base while the docs cited
release-2026-07-17_04-19-base.  Recovery required closing both PRs and
deleting both branches by hand; the reconciler could not heal itself.

Update the file in place when it already exists, skip the write entirely when
the committed content already matches (the reconciler runs every 30s), and do
not open a pull request when the write failed -- a PR on top of a failed write
advertises content that was never committed.

Adds the first tests for ensure_published(), which had none.  All three new
tests fail without this change.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@pietrodimarco-dfinity
pietrodimarco-dfinity requested a review from a team as a code owner July 27, 2026 17:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant