Skip to content

Write formatted files atomically to avoid corruption on write failure#5207

Open
ParamChordiya wants to merge 3 commits into
psf:mainfrom
ParamChordiya:fix/atomic-write-in-place
Open

Write formatted files atomically to avoid corruption on write failure#5207
ParamChordiya wants to merge 3 commits into
psf:mainfrom
ParamChordiya:fix/atomic-write-in-place

Conversation

@ParamChordiya

@ParamChordiya ParamChordiya commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

Description

Fixes #2479.

Right now format_file_in_place opens the target file directly with open(src, "w") before writing the reformatted contents. That call truncates the file immediately, so if the write fails partway through for any reason (disk fills up, permissions get revoked mid-run, process gets killed, etc.) the original file is left wiped or half-written with no way to get it back. Since most people run Black on files that aren't always safely committed yet, that's a pretty rough way to lose code.

This was actually discussed in the issue back in 2021 - the suggestion from qlyoung (with JelleZijlstra agreeing) was to write to a temp file and swap it in, so that's what this PR does:

  • Write the reformatted contents to a temp file created in the same directory as the target (so it's guaranteed to be on the same filesystem, which is what makes the swap atomic).
  • flush() + fsync() the temp file before doing anything else, so we know the data actually made it to disk.
  • Copy the original file's permission bits onto the temp file, then os.replace() it over the original. os.replace is atomic on both POSIX and Windows, so there's never a window where the file is half-written.
  • If anything raises along the way, the temp file gets cleaned up and the original file is never touched.

One thing that took a bit of extra care: if the path being formatted is a symlink, os.replace operates on the symlink itself rather than what it points to, which would silently turn the symlink into a regular file. So there's a check for that - if src is a symlink, we resolve it first and write through to the real target, leaving the symlink itself alone.

Worth calling out (as was already flagged in the issue thread): this does mean Black now needs write permission on the containing directory, not just the file, since it has to create the temp file there. That seems like a reasonable trade-off for not losing people's work, but flagging it in case it's a concern for some setups.

Since JelleZijlstra asked in the thread whether other formatters handle this the same way, I went and checked a few:

  • clang-format actually does exactly this - it writes to a temp file through LLVM's TempFile utility and renames it over the target (llvm/lib/Support/Path.cpp).
  • gofmt doesn't rename a temp file over the target, but it does something in the same spirit: it copies the original file to a backup path before writing, and restores from that backup if the real write fails.
  • rustfmt, Prettier, yapf, and autopep8 all just open the target file and write over it directly, as far as I could tell - I didn't find a fallback for this failure mode in any of them.

So it's a mixed bag rather than an industry standard - but there's real precedent for it in at least one widely-used formatter, and gofmt clearly cares about the same failure mode even though it solves it differently. Given Black is often run against files that aren't committed yet, I think it's worth fixing here regardless of what the others do.

On cooperlees' question about IO cost on large repos: I wasn't able to put together a reliable large-scale benchmark on my end, so take this as reasoning rather than a measured number. The extra work per file is one additional file creation, one fsync, and one rename, all constant-time metadata operations against the same directory regardless of the file's size. Parsing and re-rendering the file - which already happens today - should dominate the per-file cost by a good margin for anything but a near-empty file. Happy to help put together a proper before/after benchmark against a large real-world codebase if that would help move this along, or run whatever perf harness the project already uses if there is one.

Checklist - did you ...

  • Implement any code style changes under the --preview style, following the stability policy? (N/A - this doesn't change any formatting output, just how the result gets written to disk)
  • Add an entry in CHANGES.md if necessary?
  • Add / update tests if necessary?
  • Add new / update outdated documentation? (N/A - no user-facing behavior or docs to update)

ParamChordiya and others added 2 commits June 30, 2026 22:01
format_file_in_place() previously opened the target file for writing
directly, which truncates it immediately. If the write then failed
partway through (e.g. disk full), the original file was left wiped or
corrupted with no way to recover it.

Now the reformatted contents are written to a temporary file in the
same directory, fsync'd, and atomically swapped in with os.replace().
On any failure the temporary file is discarded and the original is
left completely untouched. Symlinked targets are written through to
their resolved real path so the symlink itself is preserved rather
than being replaced by a regular file.

Fixes psf#2479
@github-actions

github-actions Bot commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

diff-shades results comparing this PR (bf04569) to main (c4c9a93):

--preview style: no changes

--stable style: no changes


What is this? | Workflow run | diff-shades documentation

os.readlink() on Windows returns the extended-length \\?\ path form,
which doesn't match the plain path string passed to symlink_to(),
failing the assertion on every Windows CI job even though the actual
fix worked correctly (the symlink was preserved and pointed at the
right file). Compare resolved paths instead, which normalizes both
sides the same way on every platform.
@JelleZijlstra

Copy link
Copy Markdown
Collaborator

Given that most other leading formatters don't do this, I'm not enthusiastic about the extra complexity here, and I suspect there's some risk of losing metadata associated with the original file.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

black in-place reformat wipes or corrupts target when disk is full

2 participants