arm64: Batch PSTATE.TCO handling in kernel nofault loops - #13439
arm64: Batch PSTATE.TCO handling in kernel nofault loops#13439kernel-patches-daemon-bpf[bot] wants to merge 7 commits into
Conversation
|
Upstream branch: 5e289c5 |
AI reviewed your patch. Please fix the bug or email reply why it's not a bug. In-Reply-To-Subject: |
AI reviewed your patch. Please fix the bug or email reply why it's not a bug. In-Reply-To-Subject: |
AI reviewed your patch. Please fix the bug or email reply why it's not a bug. In-Reply-To-Subject: |
AI reviewed your patch. Please fix the bug or email reply why it's not a bug. In-Reply-To-Subject: |
AI reviewed your patch. Please fix the bug or email reply why it's not a bug. In-Reply-To-Subject: |
AI reviewed your patch. Please fix the bug or email reply why it's not a bug. In-Reply-To-Subject: |
|
Forwarding comment 5398481698 via email |
|
Forwarding comment 5398513279 via email |
|
Forwarding comment 5398528209 via email |
|
Forwarding comment 5398545357 via email |
|
Forwarding comment 5398552308 via email |
|
Forwarding comment 5398567921 via email |
71e031f to
909ca3a
Compare
|
Upstream branch: d83fba2 |
582a0a0 to
20cb4da
Compare
909ca3a to
b010507
Compare
|
Upstream branch: ce36e38 |
20cb4da to
6979f98
Compare
b010507 to
975b11a
Compare
|
Upstream branch: 05ea1b6 |
6979f98 to
c0e08bb
Compare
975b11a to
f8c8078
Compare
|
Upstream branch: 1555de3 |
With Hardware Tag-Based KASAN in asynchronous or asymmetric mode, arm64 sets and clears PSTATE.TCO around every kernel nofault load or store. A loop pays that cost for every access even though tag checking can stay disabled until the operation finishes. Separate TCO management from the fault-tolerant access and add begin and end hooks for callers that want to batch several accesses. Keep the existing accessors self-contained, and provide aliases and no-op hooks for architectures that do not need special handling. A context switch re-enables tag checking, so a batched region must not schedule. Continue to evaluate accessor arguments before overriding TCO, as those expressions may block. Signed-off-by: Muhammad Usama Anjum <usama.anjum@arm.com>
A batched nofault region must run its end hook on every exit. Pairing the hooks by hand makes early returns and error paths easy to get wrong. Add a scope guard that starts the region on entry and ends it when the scope is left. Callers can keep their natural control flow while the architecture state remains balanced on success and failure. No functional change until a caller uses the guard. Signed-off-by: Muhammad Usama Anjum <usama.anjum@arm.com>
A zero-sized kernel nofault copy does not enter an access loop, but it still disables and re-enables page faults. Zero sizes are valid. BPF probe-read helpers accept them, and KGDB memory packets may carry a zero length. Return before changing page-fault state when there is nothing to copy. For reads, keep architecture-specific address validation before the fast path so its behavior is unchanged. Signed-off-by: Muhammad Usama Anjum <usama.anjum@arm.com>
Kernel nofault copy and string paths open-code page-fault disable and enable around label-based loops, duplicating cleanup on success and failure. Use a page-fault scope guard instead. Leaving the scope now re-enables page faults on both paths without separate cleanup at the fault label. No functional change. Signed-off-by: Muhammad Usama Anjum <usama.anjum@arm.com>
With Hardware Tag-Based KASAN in asynchronous or asymmetric mode, every
arm64 kernel nofault access sets and clears PSTATE.TCO. Copy and string
loops repeat that pair even though tag checking can stay disabled for the
whole operation.
Cover each non-empty operation with one bare nofault region and use bare
accessors in the loop. Leaving the region restores TCO before page faults
are enabled again, including after an access fault. Existing empty-work
checks ensure that every new region performs at least one access.
The number of dynamic MSR TCO executions therefore changes as follows:
Work Before After
N nofault accesses 2N 2
4 KiB nofault copy 1,024 2
N-byte strncpy 2N 2
The 4 KiB case assumes 512 64-bit accesses. These figures come from the
control flow rather than a runtime measurement, so the time saved depends
on the CPU and workload. Generic fallbacks leave other architectures
unchanged.
Signed-off-by: Muhammad Usama Anjum <usama.anjum@arm.com>
A zero limit is valid for several length-bounded BPF string operations. Their loops perform no load in that case, but they still enter and leave a page-fault-disabled region. Return the existing empty result before changing page-fault state. Keep address validation first so an invalid pointer continues to return -ERANGE. Signed-off-by: Muhammad Usama Anjum <usama.anjum@arm.com>
BPF string kfuncs cannot rely on NUL termination, so they scan memory with kernel nofault loads. With Hardware Tag-Based KASAN in asynchronous or asymmetric mode, every arm64 load sets and clears PSTATE.TCO. Use bare loads and hold one nofault region across each string operation. The scope guard restores TCO before page faults are enabled again on every exit, including an access fault. A character comparison performs two nofault loads. For N compared characters, the number of dynamic MSR TCO executions therefore falls from 4N to 2. Signed-off-by: Muhammad Usama Anjum <usama.anjum@arm.com>
c0e08bb to
8c4c3a2
Compare
|
At least one diff in series https://patchwork.kernel.org/project/netdevbpf/list/?series=1151050 irrelevant now. Closing PR. |
Pull request for series with
subject: arm64: Batch PSTATE.TCO handling in kernel nofault loops
version: 1
url: https://patchwork.kernel.org/project/netdevbpf/list/?series=1151050