config: add O_DIRECT hint to config read failures#31053
Open
nvartolomei wants to merge 1 commit into
Open
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR improves startup diagnostics when reading Redpanda config files via Seastar open_file_dma fails (notably with EINVAL on filesystems that don’t support O_DIRECT), and it hardens bootstrap config handling so unreadable-but-present bootstrap config fails startup instead of being silently ignored.
Changes:
- Add
utils::format_file_io_error(...)to include file path context and anO_DIRECThint onEINVAL. - Wrap node config (
redpanda.yaml) reads to throw clearer errors on read failures. - Update
config_manager::load_bootstrapto ignore onlyENOENT(missing file) and to fail startup on other read errors.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| src/v/utils/file_io.h | Declares new helper to format file I/O errors with an O_DIRECT hint on EINVAL. |
| src/v/utils/file_io.cc | Implements the formatted error helper using fmt. |
| src/v/utils/BUILD | Adds @fmt dependency to the file_io library. |
| src/v/redpanda/application.cc | Wraps node-config read failures to throw enriched error messages (including path + O_DIRECT hint on EINVAL). |
| src/v/cluster/config_manager.cc | Stops swallowing non-ENOENT bootstrap read errors; improves error reporting for bootstrap read/parse failures. |
Wrap failures reading the node and bootstrap config files with the file path and, when the underlying syscall returns EINVAL, a hint that the filesystem may not support O_DIRECT (the read helpers use open_file_dma). Eases diagnosing startup failures on filesystems without O_DIRECT support. This surfaces in some environments that bind-mount the bootstrap config from an overlayfs whose lower layer is squashfs: reads through such an overlay fail the O_DIRECT open with EINVAL, and the bare "Invalid argument" gave no indication of the cause. Behavior change: on some overlayfs-based configs the failure instead arrives as a std::filesystem::filesystem_error, which load_bootstrap previously treated as "bootstrap file absent", silently starting without applying the bootstrap config. It now ignores only a genuinely missing file (ENOENT) and surfaces any other error, so a node that previously started despite an unreadable bootstrap file will now fail startup. See: scylladb/seastar#3518 redpanda-data#14473
63c844d to
a8d00fa
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Config files are read with O_DIRECT (via Seastar's
open_file_dma). On afilesystem that doesn't support direct I/O the read fails with EINVAL, which
previously surfaced as an opaque
Failure during startup: system_error (error system:22, Invalid argument)with no indication of the cause. This happens insome environments that bind-mount the bootstrap config from an overlayfs whose
lower layer is squashfs.
This wraps the node- and bootstrap-config reads to report the file path and, on
EINVAL, a hint that the filesystem may not support O_DIRECT. The hint is scoped
to config files (often bind-mounted onto arbitrary filesystems); data
directories are validated separately by
syschecks::disk, which does around-trip O_DIRECT check.
It also stops
config_manager::load_bootstrapfrom swallowing non-ENOENTerrors: previously any
filesystem_error(including the O_DIRECT EINVAL) wastreated as "bootstrap file absent", so the node started silently without
applying the bootstrap config. Now only a genuinely missing file (ENOENT) is
ignored; any other error fails startup.
See:
Backports Required
Release Notes
Improvements
Startup failures caused by reading config files on filesystems without
O_DIRECT support (e.g. an overlayfs over squashfs, common when the config is
bind-mounted) now report the file path and an O_DIRECT hint instead of an
opaque "Invalid argument".
Behavior change: if the bootstrap config file (
.bootstrap.yaml) ispresent but unreadable, the node now fails to start instead of silently
starting as though no bootstrap config existed. A missing bootstrap file is
still ignored, as before.