diff --git a/docs/impl-notes/2026-07-24-uprobe-support.md b/docs/impl-notes/2026-07-24-uprobe-support.md new file mode 100644 index 0000000000..6e61c146d3 --- /dev/null +++ b/docs/impl-notes/2026-07-24-uprobe-support.md @@ -0,0 +1,581 @@ +# 实现笔记:uprobe 支持(issue #2150 阶段一) + +> 对应计划:`docs/plans/2026-07-24-uprobe-support.md`(步骤 1+2)。 +> 本文件为工作产物,不进 git。本批只完成 **crate 骨架 + x86 指令分析**,不动 kernel/src/mm、perf、exception。 + +## 本批交付物 + +新建 `kernel/crates/uprobe/`: +- `Cargo.toml`:依赖 `yaxpeax-x86=2` + `yaxpeax-arch=0`(照 kprobe,仅 x86_64 target)。 +- `src/lib.rs`:crate 根,导出 `core` 与 `arch`。 +- `src/core.rs`(架构无关):`UprobePoint` / `UprobeBasic` / `UprobeBuilder`。 +- `src/arch/mod.rs`:`ProbeArgs` / `UprobeOps` / `CallBackFunc` trait + `impl UprobeOps for UprobePoint` + `pub(crate) ProbeHandler`。 +- `src/arch/x86/mod.rs`:`analyze_insn` / `build_xol_slot` / `InsnAnalysis` / `RipReloc` / `UprobeInsnError` + 7 个 `#[cfg(test)]` 单测。 +- `kernel/Cargo.toml`:新增 `uprobe = { path = "crates/uprobe" }`。 + +验证: +- `cargo test -p uprobe` → 7 passed。 +- `cargo clippy -p uprobe --all-features` → clean。 +- `cargo fmt -p uprobe --check` → clean。 +- `cargo build --release -p dragonos_kernel --target arch/x86_64/x86_64-unknown-none.json`(即 `make kernel` 的 kernel_rust 步骤)→ 无 error / 无 warning。 + +--- + +## 计划偏离(与计划的差异 + 理由) + +1. **模块名 `core` 与 `core` crate 同名** → 仍按计划命名为 `pub mod core`,但 crate 内所有对*外部* `core`/`alloc` crate 的引用一律用前导冒号 `::core::` / `::alloc::`,避免被本地 `core` 模块遮蔽(实测:`pub mod core;` + `use ::core::fmt::Debug` 可正常编译)。这是 no_std 下保留语义清晰模块名的标准做法。 + +2. **`UprobePoint.old_instruction: [u8; 16]`(非 kprobe 的 15)** → 按任务规格采用 16(`UPROBE_INSN_COPY_SIZE = 16`)。理由:2 的幂、对齐友好,且与典型 XOL slot 宽度一致(slot 需容纳 ≤15 字节指令副本,并保证其后字节可安全执行)。x86_64 单指令上限仍按 15 校验。 + +3. **`UprobeBuilder` 只接收 `probe_vaddr`,不接收 `path`/`offset`** → 计划步骤文字写“字段含 path/offset 或 probe_vaddr”。本 crate 选 `probe_vaddr`(已解析的用户虚拟地址):`path`+`offset`→`probe_vaddr` 的解析依赖 VMA/ELF,属 perf 层(步骤 7)职责,引入会耦合 mm。保持 crate 纯粹。 + +4. **`event_callback: Arc`(kprobe 用 `Box`)** → 按任务要求用 `Arc`:同一 eBPF 回调需在多个 per-mm 探测点间共享。 + +5. **`UprobeBasic` 直接持有 `Arc`**(不做 kprobe 那样的 `Kprobe{basic,point}` 分层包装)→ 更简单,且与任务“UprobeBasic 含 probe_point”一致。 + +6. **`ProbeArgs` / `CallBackFunc` 在 uprobe crate 内独立定义,不 `use`/依赖 kprobe crate** → 低耦合。签名与 kprobe 对齐,以便复用同一套 TrapFrame 适配模式(后续步骤各写各的 args 适配器)。 + +--- + +## 边界情况处理 + +### RIP-relative 检测必须覆盖两种操作数呈现 +- yaxpeax 对 `[rip+disp]`(disp≠0)给出 `Operand::Disp { base: RegSpec::RIP, disp }`; +- 但对 `[rip]`(disp==0)给出 `Operand::MemDeref { base: RegSpec::RIP }`(**实测发现,单测失败暴露**)。 +- 二者都是 RIP-relative,都需重定位。`operand_rip_disp()` 同时处理这两种。 +- **后果严重性**:漏判任一形式 → XOL slot 用原始 disp 执行 → 指向错误地址 → 静默损坏/崩溃。故对 base==RIP 的*所有*操作数变体(含掩码 / 带 index 的 EVEX 形式)显式处理:可重定位的两种返回 disp,其余返回 `Err`(fail-fast),绝不静默放过。 + +### disp_offset ≠ insn_len − 4 +- RIP-relative 编码位移恒为 4 字节(disp32),但当指令**同时**带尾随立即数时(如 `mov dword [rip+x], imm32`、`add [rip+x], imm8`),位移位于立即数**之前**。 +- 故 `disp_offset = insn_len − 4 − imm_size`,`imm_size` 由遍历操作数的立即数变体得出(含 `[rip+disp32]` 内存操作数的指令至多一个立即数)。 +- 单测 `rip_relative_with_immediate`(`c7 05 ...`,len=10)验证 disp_offset=2。 + +### 重定位位移溢出 +- `new_disp = disp + (probe_vaddr − slot_vaddr)`,若超出 i32 范围 → `UprobeInsnError::DisplacementOverflow`(uprobe 装不下,放弃该探测点)。 +- 单测 `build_slot_displacement_overflow` 验证。 + +### 错误分类 +- 空输入 → `Truncated`。 +- 解码耗尽输入(如 `0xe8` 缺 imm32)→ 经 `DecodeError::data_exhausted()` 判定为 `Truncated`。 +- 非法操作码(如 `0x06` push es 在 64 位无效)→ `DecodeFailed`。 +- 解码长度 > 15 → `TooLong`(`==0` 理论不可达,一并归 TooLong)。 + +--- + +## 保守决策(做了哪些简化 + 为什么) + +1. **本 crate 不读用户内存**。uprobe 探测对象在用户地址空间,读取需目标 mm 的页表上下文(CPL=0 切换 / page table walk),属 mm 层职责。故 `UprobeBuilder` 只持 `probe_vaddr`;原指令字节由调用方(步骤 4 断点安装)读取后交给 `analyze_insn`。 + +2. **`InsnAnalysis` 不内嵌进 `UprobePoint`**。`UprobePoint` 保持 4 字段的架构无关 DTO(按任务规格)。`InsnAnalysis`/`RipReloc` 是 x86 专属,放 `arch/x86`。二者由 mm 层在 per-mm 探测项中并列存储(见“开放问题”)。这样 `core` 不反向依赖 x86,耦合最低。 + +3. **`UprobeOps` 不含 `single_step_address`**(计划决策 1)。uprobe 单步地址 = per-mm XOL slot 用户地址(`xol_page_base + xol_slot_offset`),需 mm 上下文运行时算,不在本 crate。handler 通过 trait 取得 `old_instruction / insn_len / xol_slot_offset`,slot 真实地址由 mm 层另给。 + +4. **仅 x86_64 实现指令分析**。`analyze_insn`/`build_xol_slot` 在 `cfg(target_arch = "x86_64")` 下;`core` 与 arch trait 在所有架构可用。DragonOS uprobe 首期只针对 x86_64。 + +--- + +## 开放问题(留给后续步骤的接口契约) + +### 步骤 3(per-mm uprobe 管理 + XOL 区) +- per-mm 探测项须**并列存储** `Arc` 与 `InsnAnalysis`(本 crate 提供二者,组合由消费方完成)。 +- 负责填充 `UprobePoint.old_instruction` / `insn_len` / `xol_slot_offset`。 +- XOL slot 页预分配;slot 分配/回收;维护 `xol_page_base` 以便命中时算 slot 用户地址。 + +### 步骤 4(断点页安装) +- 读取目标 mm 在 `probe_vaddr` 处的用户指令字节(≥16 字节)。 +- 调 `analyze_insn(bytes)` 校验 + 取 `insn_len`;失败则放弃该探测点(fail-fast)。 +- 将 `old_instruction[..insn_len]` 与 `insn_len` 回填 `UprobePoint`。 +- 私有 COW 副本上 patch 0xcc(计划决策 3)。 + +### 步骤 5(异常分发) +- `do_int3`(用户态 #BP):经 `UprobeOps` 取 `break_address`(= BPF 看到的 rip,**绝不暴露 XOL slot 地址给 BPF**,计划 F5)、`old_instruction`/`insn_len`/`xol_slot_offset`; + 算 `slot_vaddr = xol_page_base + xol_slot_offset`;调 `build_xol_slot(&analysis, probe_vaddr, slot_vaddr, old_instruction, slot_buf)` 填 XOL slot;rip→slot;设 TF;`pre_handler`/event_callback。 +- `do_debug`(用户态 #DB,XOL 完成):rip 回 `return_address()`(= break_address + insn_len),清 TF,`post_handler`。 +- **`NEED_UPROBE` 位**(计划步骤 8)用于在 `do_debug` 区分“XOL 单步完成的 #DB”与 ptrace/硬件断点 #DB。 + +### TrapFrame 适配(ProbeArgs) +- 内核需提供一个 `TrapFrame` 适配器 `impl uprobe::ProbeArgs`(照 `kernel/src/debug/kprobe/args.rs`): + - `break_address()` → 探针址(probe_vaddr); + - `debug_address()` → XOL slot 中原指令执行后的下一条(slot_vaddr + insn_len); + - `as_any()` → 供回调 downcast 到具体 TrapFrame。 + +### CallBackFunc +- 步骤 7(perf 接入)提供 `Arc`(eBPF 入口),经 `UprobeBuilder::with_event_callback` 或 `UprobeBasic::update_event_callback` 注入。 + +--- + +## XOL RIP-relative 重定位公式(备忘) + +原指令在 `probe_vaddr` 执行:`[rip+disp]` 有效地址 = `probe_vaddr + insn_len + disp`(rip 指向下一条指令)。 +副本在 `slot_vaddr` 执行,欲保持同一有效地址: + +``` +slot_vaddr + insn_len + new_disp = probe_vaddr + insn_len + disp +→ new_disp = disp + (probe_vaddr − slot_vaddr) +``` + +`new_disp` 以小端 i32 写入 `slot[disp_offset .. disp_offset+4]`,`disp_offset = insn_len − 4 − imm_size`。 + +--- + +# 批次 2:mm 集成(per-mm uprobe 表 + XOL 区 + 断点页安装) + +> 计划步骤 3+4。本节为工作产物,不进 git。 + +## 本批交付物 + +- **新文件** `kernel/src/mm/ucontext/uprobe.rs`(~670 行): + - `XolArea`:per-mm XOL 页(用户态 R-X 匿名页,256 个 16 字节 slot,位图分配/回收)。 + - `UprobeInstance`:per-mm 实例(`UprobeBasic` + `InsnAnalysis`)。 + - `UprobePageState`:per-page 断点追踪(原始页 + COW 副本 + refcount)。 + - `UprobeHandle`:注册句柄(Drop 自动注销,镜像 `KprobePerfEvent::drop`)。 + - `uprobe_register` / `uprobe_unregister`:公开 API。 + - `install_breakpoint_page` / `restore_breakpoint_page`:断点安装/恢复(复刻 do_wp_page)。 +- **修改** `kernel/src/mm/ucontext/address_space.rs`: + - `AddressSpace` 新增 3 个 irqsave SpinLock 字段(位于 `inner` 之外,F8)。 + - `AddressSpace::new()` 初始化为空。 +- **修改** `kernel/src/mm/ucontext/inner.rs`: + - `try_clone`:fork 时子进程映射原页(非 COW 副本),避免继承 0xcc。 +- **修改** `kernel/src/mm/ucontext/mod.rs`: + - 新增 `pub(crate) mod uprobe;`,导出公开 API。 + +验证:`make kernel` → 0 error / 0 warning。 + +--- + +## 计划偏离(与计划的差异 + 理由) + +1. **uprobe 字段挂在外层 `AddressSpace`(非 `InnerAddressSpace`)** → 计划写"挂在 `AddressSpace`/`InnerAddressSpace` 上"。实际选择外层 `AddressSpace`:inner 是 `RwSem`(睡眠锁),命中路径关中断不能取它。将 `uprobe_list` / `xol_area` / `uprobe_page_state` 直接放在 `AddressSpace` 上(与 `active_cpus` / `tlb_gen` 同级),由独立 `SpinLock` 保护,命中路径仅 `lock_irqsave` + 查表。 + +2. **XOL 页物理地址预存** → 计划未提。实际在 `XolArea` 中存 `page_paddr`,注册时 translate 一次存入。理由:batch3 #BP handler 关中断,不能取 `inner` 的 RwSem 拿 mapper 做 translate;有了 `page_paddr`,batch3 直接 `phys_2_virt(page_paddr)` + 偏移写 slot 内容。 + +3. **per-page refcount 而非每 uprobe 独立副本** → 计划写"阶段一可简化:每 uprobe 独立副本"。实际选择 per-page refcount:因为同一 PTE 只有一个物理页帧,第二个 uprobe COW 会替换第一个的副本,丢失其 0xcc。正确做法是共享 COW 副本 + refcount,与 Linux 一致。 + +4. **注销时 unpatch 0xcc(恢复原指令首字节)** → 计划写"恢复原指令页"。实际在 refcount > 0 时先在 COW 副本中恢复该 uprobe 的原指令首字节(`old_instruction[0]`),不影响同页其他 uprobe 的 0xcc。refcount == 0 时才恢复整页(set_entry 回原 paddr)。 + +5. **inode rmap 全量(pid==-1)未实现** → 计划步骤 4。本批仅实现单 mm(pid>=0)。`uprobe_register` 接受 `&Arc` + `probe_vaddr`(已解析)。inode rmap 全量遍历由 batch4 在 perf 层做(`collect_file_vmas` → 对每个 mm 调 `uprobe_register`)。 + +--- + +## 边界情况处理 + +### 同页多 uprobe +- 支持:第一个 uprobe COW 出私有副本 + patch 0xcc,后续 uprobe 在同一副本 patch 额外 0xcc + refcount++。 +- 注销时 refcount > 0:仅恢复该偏移的原指令字节(从 `UprobePoint.old_instruction[0]` 读)。 +- refcount == 0:恢复整页(set_entry 回原 paddr + rmap + flush_tlb)。 + +### fork(try_clone) +- 子进程**不继承** uprobe(uprobe_list / xol_area / uprobe_page_state 均为空——来自 `AddressSpace::new()`)。 +- 子进程**不继承 0xcc**:try_clone 检查 `parent_mm.uprobe_page_state`,对断点页映射原始物理页(非 COW 副本)。 +- 已知限制(stage 1):子进程不继承 uprobe handler。若需继承,需在 try_clone 中克隆 uprobe_list 并重新安装断点(留 stage 2)。 + +### mmap/unmap 并发 +- 若被探测页在注销前被 munmap:`restore_breakpoint_page` 检测 PTE 缺失(`get_table` 返回 None),跳过 PTE 恢复,仅清理 `uprobe_page_state`。COW 副本 Arc drop 后回收。 +- 若 VMA 在注销前被 munmap 但 PTE 仍在(罕见):rmap attach/detach 跳过(VMA 不存在),仅做 PTE 恢复。 + +### 指令跨页边界 +- fail-fast:`read_user_insn_bytes` 只读当前页内剩余字节(`min(16, PAGE_SIZE - page_offset)`)。若指令长度超过可读字节,`analyze_insn` 返回 `Truncated`,注册失败。 +- 影响:page_offset > 4081 时指令可能跨页。函数入口通常对齐,实际影响极小。 + +### VM_SHARED 可写映射 +- 已知限制:uprobe 安装在 VM_SHARED 可写映射上时,COW 副本打破了共享语义(后续 write 会修改副本而非共享页)。agentsight 目标(.text 只读段)不受影响。 + +--- + +## 保守决策与简化 + +1. **XOL 仅 1 页(256 slot)** → 足够 256 个并发 uprobe/进程。超出返回 ENOMEM。Linux 用多页+树。 +2. **注销时不 unmap XOL 页** → XOL 页在 mm 生命周期内常驻。slot 回收到位图但不释放物理页。简化生命周期管理。 +3. **无写保护冲突处理** → 若安装 0xcc 后用户对同页做 write fault,do_wp_page 会再次 COW(丢失 0xcc)。uprobe 仍留在表中但不再触发 #BP。stage 1 接受此限制(.text 段不应被写)。 +4. **pre/post handler 是函数指针(非 trait 对象)** → 镜像 kprobe 的 `KprobeBuilder::new(probe_addr, pre, post, enable)`。batch3 提供实际 handler 函数;batch4 通过 `event_callback`(`Arc`)注入 BPF。 + +--- + +## 留给 batch3/batch4 的 API 契约 + +### batch3(异常分发 #BP/#DB) + +**命中路径(关中断,仅 SpinLock)**: +```ignore +let mm = ProcessManager::current_pcb().basic().user_vm()?; + +// 1. 查 uprobe 表 +let list = mm.uprobe_list.lock_irqsave(); // irqsave SpinLock +let Some(entries) = list.get(&trapframe.rip) else { return; }; +drop(list); // 缩短锁持有(或持有期间调 handler) + +for entry in entries { + let inst = entry.read(); // RwLock (spinlock-based, 安全关中断) + if !inst.basic.is_enabled() { continue; } + + // 2. 调 pre_handler + event_callback(BPF 入口) + inst.basic.call_pre_handler(&trapframe_adapter); + inst.basic.call_event_callback(&trapframe_adapter); + + // 3. 算 XOL slot 地址 + 填 slot 内容 + let point = inst.basic.probe_point().unwrap(); + let offset = point.xol_slot_offset; + let xol = mm.xol_area.lock_irqsave(); + let area = xol.as_ref().unwrap(); + let slot_vaddr = area.slot_vaddr(offset); + let slot_paddr = area.page_paddr(); + drop(xol); + + // 通过 phys_2_virt 写 slot 内容(无需 mapper / RwSem) + let slot_kva = unsafe { MMArch::phys_2_virt(slot_paddr) }.unwrap(); + let slot_buf = unsafe { + core::slice::from_raw_parts_mut( + (slot_kva.data() + offset) as *mut u8, + UPROBE_INSN_COPY_SIZE, + ) + }; + uprobe::build_xol_slot( + &inst.insn_analysis, + point.probe_vaddr, + slot_vaddr.data(), + &point.old_instruction, + slot_buf, + ).unwrap(); + + // 4. rip → slot, 设 TF(XOL 单步) + trapframe.rip = slot_vaddr.data(); + trapframe.set_tf(); +} +``` + +**#DB 完成(XOL 单步后)**: +- 检查 NEED_UPROBE(计划步骤 8)判别 XOL 完成的 #DB; +- rip 回 `return_address()`(= probe_vaddr + insn_len); +- 清 TF; +- 调 `post_handler`。 + +### batch4(perf 接入) +```ignore +// 注册 uprobe +let handle = uprobe_register(&mm, probe_vaddr, noop_handler, noop_handler)?; + +// 注入 BPF 回调 +if let Some(instance) = handle.instance() { + instance.write().basic.update_event_callback(bpf_callback_arc); +} + +// Drop handle → 自动注销(镜像 KprobePerfEvent::drop) +``` + +### TrapFrame 适配(ProbeArgs) +- 照 `kernel/src/debug/kprobe/args.rs` 实现 `impl ProbeArgs for UprobeTrapFrame`。 + - `break_address()` → `trapframe.rip`(#BP 时 = probe_vaddr); + - `debug_address()` → `trapframe.rip`(#DB 时 = slot_vaddr + insn_len); + - `as_any()` → `&self as &dyn Any`(供回调 downcast)。 + +### 字段访问 +- `mm.uprobe_list` — `pub SpinLock>>>>` +- `mm.xol_area` — `pub SpinLock>>` +- `mm.uprobe_page_state` — `pub(crate) SpinLock>`(仅内部用) + +--- + +# 批次 4:perf 接入(完成记录,2026-07-24) + +> 仅改 `kernel/src/perf/`(新增 `uprobe.rs` + `mod.rs` 分发臂)。不碰 exception/trap(batch3)。 + +## 交付物 + +- **`kernel/src/perf/uprobe.rs`**(新):`UprobePerfEvent` + `UprobePerfCallBack` + + `perf_event_open_uprobe(args)` + `resolve_target` + `parse_path_and_offset`。 +- **`kernel/src/perf/mod.rs`**:`mod uprobe;` + `perf_event_open` 的 `PERF_TYPE_MAX` 分发臂 + 按 `args.name.contains('/')` 二分(F9):含 `/` → uprobe(path:offset),否则 → kprobe(现有)。 + +## A. F9 分发臂 + +`PERF_TYPE_MAX(=6)` 处: +```ignore +perf_type_id::PERF_TYPE_MAX => { + if args.name.contains('/') { + let uprobe_event = uprobe::perf_event_open_uprobe(args)?; // config2 = 文件偏移 + Box::new(uprobe_event) + } else { + let kprobe_event = kprobe::perf_event_open_kprobe(args); // 现有行为不变 + Box::new(kprobe_event) + } +} +``` +判定依据:kprobe 的 config1 是内核符号名(不含 `/`);uprobe 的 config1 是二进制路径 +(必含 `/`)。无 sysfs event-source 设备(后续阶段补)。 + +## B. path → vaddr 解析(设计偏离说明) + +**计划原文**:"解析 path(name 中 `:` 前的部分)+ offset(config2)"。该描述自相矛盾 +(若 offset 来自 config2,则 name 不应再含 `:offset`)。实际采用 **Linux 原生约定**: + +- `config1`(name) = 二进制路径; +- `config2`(args.offset) = 文件偏移(权威)。 + +`parse_path_and_offset` 做**防御性兼容**:若工具把 `"path:0xOFFSET"` 编码进 config1 且 +config2==0,则从 config1 的 `:` 后解析十六进制偏移;config2 非零时一律以 config2 为准。 + +**path → inode → VMA → probe_vaddr** 链路: +1. `ProcessManager::current_mntns().root_inode().lookup(&path)` → `Arc`; +2. `inode.page_cache()` → `Arc`(inode rmap 入口);无 page_cache → EINVAL + (目标不是已映射的常规文件); +3. `page_cache.collect_file_vmas()` → `Vec>`(所有映射该 inode 的 VMA); +4. 对每个 VMA:`probe_vaddr = region.start() + (offset - backing_pgoff*PAGE_SIZE)`, + 前提 `backing_pgoff*PAGE_SIZE <= offset < +region.size()`(`resolve_target`)。 + +`region.end()` 是 exclusive(`= start + size`);覆盖判断用字节区间半开 `[start, +size)`。 + +## C. pid 语义(B8) + +- `pid > 0`:`ProcessManager::find(RawPid::from(pid))` → `pcb.basic().user_vm()`,仅接受 + `Arc::ptr_eq` 的 VMA(单 mm)。 +- `pid == 0`:当前进程(`ProcessManager::current_pcb().raw_pid()`),其余同上。 +- `pid == -1`:不设 target_mm,接受 inode rmap 返回的**全部** VMA(跨所有 mm)。 + +对每个命中的 VMA 调 `uprobe_register(&mm, probe_vaddr, noop_handler, noop_handler)`。 +非可执行映射(如只读数据段)返回 `EACCES` → 静默跳过;其余错误上抛。一条都没注册成 → EINVAL。 + +### pid==-1 全量的实现程度 + +**API 路径完整**:`collect_file_vmas` → 逐 VMA 解析 mm + probe_vaddr → 逐 mm `uprobe_register`。 +即每个当前映射该文件的 mm 都装上 0xcc。**已知限制(阶段一可接受)**: +- `collect_file_vmas` 在返回时即释放 `i_mmap_read`;注册循环期间映射若并发变化,`uprobe_register` + 内部会校验 VMA 存在性(`mappings.contains` 失败 → EINVAL),不会写坏地址,但可能漏装/竞争。 + Linux 在此类操作期间持 `i_mmap_rwsem`;本批未持锁以避免与 mm 内部锁序耦合(batch2 边界)。 +- 仅装"当前已映射"的 mm;后续新 mmap 该文件的进程不会自动获得探针(Linux 的 inode 级 + registration 留后续阶段)。 + +## D. UprobePerfEvent 多 handle 生命周期 + +```ignore +pub struct UprobePerfEvent { + _args: PerfProbeArgs, + handles: Vec, // pid>=0: 该 mm 的所有覆盖 VMA;pid==-1: 所有 mm +} +``` + +- **注册**:`perf_event_open_uprobe` 一次性建好全部 `UprobeHandle`(每个 = 一个 per-mm 探针 + + 0xcc 页 + XOL slot)。 +- **BPF 注入**:`do_set_bpf_prog` JIT 出**一份** `Arc`,`clone` 进每个 + handle 的 `instance().write().basic.update_event_callback(..)`(多 mm 共用同一 JIT 产物)。 +- **注销**:`UprobePerfEvent::Drop` 不写手动 unregister —— `Vec` 析构时逐个 + drop,`UprobeHandle::Drop`(batch2)执行 `uprobe_unregister_internal`(恢复原页 → 移除表项 + → 回收 slot)。`PerfEventInode` 持 `Box`,fd 关闭 → inode drop → event + drop → handles drop。卸载顺序由 Vec 决定,确定性可预测。 + +## E. BPF attach 透明性(F5) + +`UprobePerfCallBack::call(&self, trap_frame: &dyn uprobe::ProbeArgs)`: +```ignore +let probe_addr = trap_frame.break_address(); // 原探针址 +let tf = trap_frame.as_any().downcast_ref::()?; +let mut pt_regs = KProbeContext::from(tf); // 复用 kprobe 的 pt_regs 布局 +pt_regs.rip = probe_addr as u64; // 强制 F5:BPF 见到 rip = 原探针址 +self.0.call(pt_regs_slice); // BasicPerfEbpfCallBack JIT 执行 +``` +**关键**:用 `break_address()` 覆写 `pt_regs.rip`,**不**暴露 XOL slot 地址、也不暴露 int3 +故障点 `rip+1`。即便 batch3 传入的 TrapFrame.rip 仍是 raw `probe_vaddr+1`,BPF 也只见到 +原探针址。`KProbeContext` 复用为 pt_regs 布局(F10:BPF_PROG_TYPE_KPROBE,无新枚举)。 + +`Box::leak` + `BasicPerfEbpfCallBack::drop`(`Box::from_raw`) 是 kprobe 模板的成对 JIT 内存 +管理,本批原样照搬(非真泄漏,event drop 时回收)。 + +## F. 与 batch3 的接口契约(关键) + +本批 perf 代码 **编译期不依赖** `impl uprobe::ProbeArgs for TrapFrame`(`&dyn uprobe::ProbeArgs` + 的 `as_any()` 对任意 `&dyn Any` 都能 downcast),**运行期依赖**:batch3 在 `#BP` handler 调 +`inst.basic.call_event_callback(args)` 时,`args` 必须是 `&dyn uprobe::ProbeArgs` 且其 +`as_any()` 能 downcast 出 `TrapFrame`。 + +**batch3 需补**(arch/x86_64/interrupt/mod.rs,紧挨现有 `impl kprobe::ProbeArgs for TrapFrame`): +```ignore +impl uprobe::ProbeArgs for TrapFrame { + fn as_any(&self) -> &dyn Any { self } + fn break_address(&self) -> usize { (self.rip - 1) as usize } // #BP: rip=probe_vaddr+1 → -1 + fn debug_address(&self) -> usize { self.rip as usize } +} +``` +本批 `UprobePerfCallBack` 已用 `break_address()` 覆写 rip,故只要 #BP 路径调 +`call_event_callback` 时 `break_address()`=probe_vaddr,F5 即成立(与 raw rip 是否已调整无关)。 + +## G. 编译验证状态 + +- `perf/uprobe.rs` + `mod.rs` 分发臂:**0 error / 0 warning**(cargo check 已确认本文件干净)。 +- 全 crate 当前剩余 4 个编译错误,**全部在 `src/exception/uprobe.rs`**(batch3 文件): + `TrapFrame: uprobe::ProbeArgs is not satisfied`(行 86/87/192/193)—— 即上述 F 节契约。 + batch3 的其余 3 个错(Vec 未导入 / phys_2_virt / interrupt_enable unsafe)已由 ExcDispatcher + 在并行中修复。已通过 hub 向 ExcDispatcher 同步契约,待其补 `impl uprobe::ProbeArgs for TrapFrame`。 +- 本批未碰 exception/trap/mm 内部,仅调 batch2 公开 API(`uprobe_register`/`UprobeHandle`/ + `noop_handler`)与 VFS/ProcessManager 公开 API。 + +--- + +# 批次 3:异常分发(#BP/#DB 用户态分发 + XOL 单步 + SIGTRAP + NEED_UPROBE) + +> 计划步骤 5(异常分发)+ 6(ptrace 协调)+ 8(NEED_UPROBE 判别位)。工作产物,不进 git。 +> 仅改 exception/trap/state,不碰 mm(batch2 已定)/perf(batch4)。 + +## 本批交付物 + +- **新文件** `kernel/src/exception/uprobe.rs`:用户态 #BP/#DB 分发(`uprobe_breakpoint_handler` + / `uprobe_debug_handler`)+ XOL slot 填充 + SIGTRAP 投递 + slot 反查辅助。 +- `kernel/src/exception/mod.rs`:`pub mod uprobe;`。 +- `kernel/src/arch/x86_64/interrupt/trap.rs`:`do_int3`/`do_debug` 加 `is_from_user()` 二分。 +- `kernel/src/arch/x86_64/interrupt/mod.rs`:新增 `impl uprobe::ProbeArgs for TrapFrame` + (紧挨现有 kprobe 版本,照 batch4 F 节契约)。 +- `kernel/src/process/state.rs`:`ProcessFlags` 加 `NEED_UPROBE = 1 << 14`(位 14,NEED_RSEQ/ + IN_IOWAIT 之后、PID_UNHASHED 之前)。 + +验证:`make kernel` → **0 error / 0 warning**(含 batch4 perf/uprobe.rs 全通过)。 + +## 计划偏离(与计划的差异 + 理由) + +### 1. F5 rip 透明性——保留 raw rip,由 BPF 回调归一化(非分发器预设) + +计划 F5 原文「call_pre_handler/event_callback 入口 rip = break_address()」字面读像是「分发器 +把 rip 预设成 probe_vaddr」。**实际不这样做**,理由:`break_address() = rip - 1`,若分发器先把 +rip 设成 probe_vaddr,回调内 `break_address()` 会得到 `probe_vaddr - 1`(错)。 + +**实际契约(与 batch4 对齐,经 hub 确认)**:#BP handler 调 callback 时 **rip 保持 raw** +(= probe_vaddr + 1)。BPF 回调(`UprobePerfCallBack`)自己读 `break_address()` = rip-1 = +probe_vaddr 并覆写 rip,从而 BPF 观察到 probe_vaddr。XOL slot 用户址只在**所有回调返回后** +(Phase 4)才写入 rip,绝不暴露给 BPF。这样 `break_address()` 语义(= rip-1 = 原探针址)始终成立。 + +### 2. #DB 不经 NEED_UPROBE 之外的 per-task 状态反查 probe_vaddr——改用 slot 偏移反算 + +计划步骤 8 把 NEED_UPROBE 定为 1 位判别位(不携带 probe_vaddr)。#DB 时需恢复 probe_vaddr + +insn_len。Linux 用 per-task `uprobe_task{active_uprobe}` 存活动探针。**本批为不扩 PCB**, +改用 XOL slot 几何反算: +- NEED_UPROBE 置位 ⇒ frame.rip 落在 XOL 页内; +- slot 16 字节对齐 ⇒ `slot_offset = (rip − page_base) & !0xF`(无需 insn_len); +- 遍历 `uprobe_list` 找 `xol_slot_offset == slot_offset` 的实例 → probe_vaddr。 + +代价:#DB 路径 O(活动探针数) 遍历。stage 1 探针少,可接受。若日后探针多,可加 per-task +`active_uprobe` 字段(O(1))——属 process 模块改动,超出本批「exception + ProcessFlags」范围。 + +### 3. event_callback 单次触发(#BP),#DB 仅 post_handler + +kprobe 在 #BP(pre) 与 #DB(post) 都调 `call_event_callback`。uprobe **只在 #BP 触发一次** +(计划步骤5:#BP=pre/BPF,#DB=post),#DB 仅调 `call_post_handler`,避免 perf event 双触发 +(双触发会污染采样计数)。post_handler 入口 rip 已设为 return_address(probe_vaddr+insn_len), +F5 同样不暴露 XOL slot。 + +### 4. NEED_UPROBE 不进 `exit_to_user_mode_work` + +明确:NEED_UPROBE **只**作 #DB 分发判别位(#BP 设、#DB 清),不加入 +`ProcessFlags::exit_to_user_mode_work` 的掩码(NEED_SCHEDULE|HAS_PENDING_SIGNAL|NEED_RSEQ), +即不是 exit-to-user 延迟工作。`fork_inherited` 也不继承(子进程无活动单步)。 + +## XOL slot 填充的关中断写法 + +命中路径关中断(entry.S cli),**不能**取 `PageMapper`/`RwSem`。复刻 batch2 +`patch_byte_in_phys`:`XolArea::page_paddr()` 给物理址 → `MMArch::phys_2_virt(paddr)` 得内核 +direct-map 虚拟址 → `copy_nonoverlapping` 写整 16 字节 slot(`build_xol_slot` 产出重定位指令副本 ++ 零填充,覆盖 slot 复用时残留字节)。XOL 页是普通 RAM,非 MMIO,非 volatile 拷贝即可。 + +锁顺序:uprobe_list 与 xol_area 是两个独立 irqsave SpinLock,**不嵌套**——Phase 1 取 +uprobe_list(跑 callback + 取 probe_point/analysis),释放;Phase 2 取 xol_area(取 +slot_vaddr/page_paddr),释放;Phase 3-4 无锁填 slot + 改 trapframe。 + +## ptrace 协调(步骤 6,文档化) + +处理顺序:**kprobe > uprobe > ptrace**(#BP/#DB)。do_int3/do_debug 按 is_from_user 二分: +内核态走 kprobe(EBreak/DebugException),用户态走 uprobe 分支。 + +TF 拥有权:uprobe 单步窗口(#BP 设 TF 到 #DB 清 TF)TF 归 uprobe。DragonOS `process/ptrace.rs` +**当前未实现 PTRACE_SINGLESTEP**(无 TF/0x100/SINGLESTEP 引用),故现阶段无 uprobe↔ptrace TF +冲突。NEED_UPROBE 已为未来 ptrace #DB 留判别位:用户态 #DB 若 NEED_UPROBE 未置位 → ptrace/ +硬件断点,阶段一 return Ok(不投信号、不动 TF,留给 ptrace 自有路径)。 + +已知限制(stage 1):ptrace 单步(PTRACE_SINGLESTEP)的 SIGTRAP 投递未实现;硬件断点 #DB +未处理。完整 uprobe+ptrace 同进程共存留后续。 + +## 留给 batch4 的接口(已确认对齐) + +- `impl uprobe::ProbeArgs for TrapFrame`:as_any→self / break_address→(rip-1) / + debug_address→rip。batch4 回调经 as_any() downcast TrapFrame、break_address() 取 probe_vaddr。 +- **运行时契约**:#BP 调 call_event_callback 时 rip 保持 raw(probe_vaddr+1);XOL slot 址 + 仅 Phase 4 写入。 +- event_callback 单次(#BP);若 batch4 需要 post 期回调,用 call_post_handler(#DB 触发)。 + +## 编译验证状态(更新) + +- `make kernel`:**0 error / 0 warning**。 +- batch4 perf/uprobe.rs 的 4 个「TrapFrame: uprobe::ProbeArgs」错误已由本批补的 + `impl uprobe::ProbeArgs for TrapFrame` 解决(即 batch4 G 节所述,现已闭环)。 + +--- + +# 局部 bug 修复(独立验证 P1/P2) + +> 阶段一独立验证发现 2 个局部 bug,架构正确无需回退,本节记录修复方案与设计决策。 + +## P1:重复注册同一 probe_vaddr 读到 0xcc + +**现象**:`uprobe_register` 先 `read_user_insn_bytes` 读原指令、后 `install_breakpoint_page` +装 0xcc。第二个 consumer 注册同一 `probe_vaddr` 时,PTE 已指向含 0xcc 的 COW 副本, +`read_user_insn_bytes` 读到 `0xcc` 当原指令首字节 → `old_instruction[0]=0xcc`。 +若该条目成 `entries[0]`(#BP handler 用 `entries[0]` 填 XOL slot),slot 填 0xcc → 无限 #BP; +注销也写回 0xcc(永久断点)。 + +**修复**:在读指令**之前**先 `lock_irqsave(uprobe_list)` 查是否已有该 `probe_vaddr` 条目: +- **有**:复用其 `probe_point.old_instruction` + `insn_analysis`(二者对所有同址实例一致, + 均为 `Copy` 类型,直接拷出,跳过 `read_user_insn_bytes` + `analyze_insn`)。 +- **无**:正常读取 + 分析。 + +**设计决策**: +- 复用的是指令**信息**(字节 + 分析结果),不是 `Arc` 本身——每个 consumer + 仍分配自己的 XOL slot、持有自己的 `UprobePoint`(含自己的 `xol_slot_offset`)。共享 Arc + 会使 slot 偏移串用,破坏注销时的 per-instance slot 回收。 +- 复用的 `old_instruction` 是首个 consumer 注册时(0xcc 发布**之前**)捕获的真原指令, + 绝不可能是 0xcc。这同时修正了注销恢复路径(`restore_breakpoint_page` 写回的是真原字节, + 而非原先错误读到的 0xcc)。 +- `probe_point()` 理论恒为 `Some`(`UprobeBuilder::build` 总设 `Some`);若防御性为 `None`, + `and_then` 链回退到正常读取路径(不劣于原行为)。 + +## P2:RIP-relative 位移溢出在 #BP 命中时 panic + +**现象**:`build_xol_slot` 只在 `exception/uprobe.rs` 的 `fill_xol_slot`(命中时)调用, +注册时从不验证。若 `|probe_vaddr - slot_vaddr|` 超 i32(64 位地址空间常见),返回 +`DisplacementOverflow` → handler `Err` → `do_int3` unwrap panic。 + +**修复(注册时预填 XOL slot)**:在分配 XOL slot **之后**(`slot_vaddr = xol_page_base + +slot_offset` 已知),立即用真实 `slot_vaddr` 调 `build_xol_slot`: +- **溢出**(或任何 `UprobeInsnError`)→ `free_xol_slot` + 返回 `EINVAL`(注册失败,fail-fast, + 绝不留下命中时 panic 的探针;此时尚未插入 `uprobe_list`,无需回滚表项)。 +- **成功** → slot 内容(重定位后指令副本 + 零填充)经 `XolArea::page_paddr` + `phys_2_virt` + 写入 slot 物理页对应偏移(复刻 `fill_xol_slot`/`patch_byte_in_phys` 写法)。 + +**命中路径简化**:`uprobe_breakpoint_handler` 移除 `fill_xol_slot` 调用(连同函数本身删除)—— +slot 已在注册时预填,命中时只需从 `entries[0]` 取 `xol_slot_offset`、算 `slot_vaddr`、 +`rip→slot`。关中断路径不再调 `build_xol_slot`,位移溢出无从发生。`slot_vaddr` 计算保留。 + +**清理**:`exception/uprobe.rs` 因移除 `fill_xol_slot` 而不再使用的导入一并删除 +(`build_xol_slot` / `PhysAddr` / `MMArch` / `MemoryManagementArch`——后者原在作用域内为 +`MMArch::phys_2_virt` trait 方法解析所需,移除调用后即多余)。`UprobeOps` 保留(#DB handler +的 `.return_address()` 依赖)。 + +## P1/P2 协同与不变量 + +两个修复均在 `uprobe_register` 注册流程,新顺序(保持 F6 装弹不变量): +1. 查 `uprobe_list` 同址条目(P1)→ 复用 old_instruction/insn_analysis 或新读; +2. 分配 XOL slot; +3. `build_xol_slot` 预填 slot + 验证位移(P2)→ 溢出 EINVAL; +4. 构造实体; +5. 插入 `uprobe_list`(表项在 0xcc 发布前就绪); +6. `install_breakpoint_page`(0xcc)。 + +- **F6 不变量保持**:预填(步骤 3)在表项插入(步骤 5)之前,0xcc 发布(步骤 6)之前任何 + 查表都能找到 slot 已就绪的表项。 +- **同页多 uprobe 的 `existing_cow` 逻辑不变**:仍 refcount + patch 额外字节;对同一 + `probe_vaddr`(同 page_offset)patch 0xcc 是幂等写(字节已是 0xcc),无需额外跳过判断。 +- **多 consumer 同址**:每个 consumer 分配独立 slot 并各自预填;命中用 `entries[0]` 的 slot + (该实例注册时已预填)。`entries[0]` 因注销变动时,新首项的 slot 亦已预填。 +- **无 regression**:kprobe 路径、fork(`try_clone` 用 `original_paddr`)、#DB handler + (`find_probe_vaddr_by_slot` + `return_address`)均未触及。 + +## 验证(P1/P2 修复后) + +- `make kernel`:**0 error / 0 warning**(dragonos_kernel 全量重编)。 +- `cargo test -p uprobe`:**7 passed / 0 failed**。 +- P1:同址二次注册复用已有 `old_instruction`(真原指令,非 0xcc),跳过 `read_user_insn_bytes`。 +- P2:位移溢出在注册时返回 `EINVAL`;命中路径无 `build_xol_slot`、slot 已预填。 diff --git a/docs/plans/2026-07-24-uprobe-support.md b/docs/plans/2026-07-24-uprobe-support.md new file mode 100644 index 0000000000..81e4c86afb --- /dev/null +++ b/docs/plans/2026-07-24-uprobe-support.md @@ -0,0 +1,96 @@ +# Plan: uprobe 断点探针支持(issue #2150 阶段一) + +> YOLO-dev 模式:agent 自主推进,subagent 对抗门控。本文档不进 git(见 Git 纪律)。 +> 盲区扫描:glm-5.2 scout(BlindSpotScanner-2)。对抗评审:glm-5.2 reviewer(PlanReviewer),10 条 findings 全采纳。 + +## 任务 +为 DragonOS 实现 uprobe 断点探针,使 agentsight 能在用户态函数 `SSL_read`/`SSL_write` 入口挂探针捕获参数。对应 issue #2150 **阶段一**(issue 明确建议"阶段一完成后验证再进入阶段二,不要一口吞")。**uretprobe(阶段二)不在本计划范围**——它是独立重型机制(栈返回地址改写 + trampoline 页)。 + +## kprobe 复用基础(已验证) +- 断点机制:`kernel/crates/kprobe/src/arch/x86/mod.rs` `KprobeBuilder::replace_inst()` 写 0xcc;`KprobeOps::single_step_address()` 返回**内核缓冲区**指针(对 uprobe 不可用) +- 异常分发:`#BP`→`ebreak.rs` `EBreak::handle()`(持 `KPROBE_MANAGER` 全局锁跨 callback);`#DB`→`debug.rs` +- eBPF attach:`perf/kprobe.rs` `perf_event_open_kprobe`→`KprobePerfEvent`→`do_set_bpf_prog` JIT +- **指令解码已就绪**:`kernel/crates/kprobe/Cargo.toml` 依赖 `yaxpeax-x86="2"`+`yaxpeax-arch="0"`,`arch/x86/mod.rs` 用 `InstDecoder::default().decode_slice()` 算指令长度——**uprobe 直接复用** + +## 盲区扫描结论(BlindSpotScanner-2) + +### 架构级盲区(已处理,见决策) +- **B1[高]** kprobe 单步把 rip 指向内核缓冲区执行——CPL=3 时内核页 supervisor-only + NX,绝无可能。→ XOL。 +- **B2[高]** `#BP`/`#DB` handler 关中断(entry.S `cli`),`page_table_edit()` `debug_assert!(is_irq_enabled())`。→ 命中路径零页表改动,XOL/断点页注册时预建。 +- **B3[高]** EBreak 持全局 `KPROBE_MANAGER` 锁跑 BPF。→ 独立 per-mm 分发。 +- **B4[高]** `do_int3`/`do_debug` 无 `is_from_user()` 分支;未匹配 #BP 静默吞,无 SIGTRAP。→ 新增用户态分发 + SIGTRAP。 + +### 设计接入点盲区(已处理) +- **B5[中]** 与 ptrace/SIGTRAP/TF 单步冲突(ptrace 活跃,`PTRACE_SINGLESTEP`→TF)。→ 定义处理顺序。 +- **B6[中·正向]** inode→VMA 反向映射已存在:`page_cache.rs` `i_mmap_rwsem`+`file_vmas`+`register_file_vma`/`collect_file_vmas`。→ 用于**定位**目标 VMA/mm。 +- **B7/B8/B9** perf type 约定;pid 语义;ProcessFlags 位。 + +### 已覆盖(别重造) +- TLB shootdown 成熟:`mm/tlb.rs`+`mmu_gather.rs`。直接用 `flush_tlb_range`。 +- per-mm 隔离:`AddressSpace`(user_vm)。 +- offset→vaddr:VMA `vm_file`+`backing_pgoff`+`address_space()`。 +- 内核内 ELF 符号解析不是依赖(offset 由用户态工具算,config2)。 + +## 高风险决策(评审修正后) + +1. **XOL 执行原指令(非复用 kprobe 内核缓冲区单步)** — [B1/B2] uprobe 不复用 `KprobeOps::single_step_address`。每个 mm 注册时预分配 XOL slot 页;命中时 rip→slot(原指令副本,RIP-relative 重定位),设 TF,iretq 后用户态执行,TF 触发 #DB,handler rip 回原址+insn_len。**指令解码直接复用 yaxpeax-x86**(kprobe 已依赖,完整 x86-64 解码器含 RIP-relative 操作数语义),非新工作(评审 F3)。 + +2. **独立 per-mm 分发(irqsave SpinLock,非 RwSem)** — [B3/B10, 评审 F8] uprobe 表 `uprobe_list: BTreeMap` 挂在 `AddressSpace`/`InnerAddressSpace` 上,由**独立 irqsave SpinLock** 保护(镜像 `KPROBE_MANAGER: SpinLock`,**不用** `RwSem`,因命中路径关中断不可睡眠)。`do_int3`/`do_debug` 按 `is_from_user()` 二分。命中路径仅 lock+查表+改 trapframe+跑 BPF。 + +3. **断点页安装复刻 do_wp_page 私有 COW 路径(非 unmap+map_phys)** — [评审 F1/F2/F7] 复刻 `fault.rs:957-979` 私有文件 COW:`copy_page_as_normal`(源 File 页→私有 Normal 副本)+ patch 0xcc + **单次** `get_table().set_entry(PageEntry::new(new_paddr, new_flags))` 原子帧替换(**绝不** unmap+map_phys 制造瞬时空 PTE)+ `detach_fault_mapped_page(old)`/`attach_fault_mapped_page(new)` rmap 账簿 + `flush_tlb_range`。**无论 pid>=0 还是 -1,都为每个目标 mm 生成私有 COW 副本,绝不修改共享 page-cache 页**(否则 writeback 回写 0xcc 损坏 .so / 双重释放)。 + +4. **inode rmap 仅用于定位目标 VMA/mm** — [B6] inode rmap(`file_vmas`+`i_mmap_rwsem`)用于**定位**哪些 mm/VMA 映射目标文件偏移;实际页替换按决策3。 + +5. **uretprobe 不在本阶段** — [B11] issue 明确分两阶段。uretprobe 的栈返回地址改写 + trampoline 留阶段二。 + +6. **pid 语义** — [B8] `pid>=0` 单 mm;`pid==-1` 经 inode rmap 全量 mm。**两者都为每个 mm 私有 COW**(见决策3)。 + +## 实现步骤(概要) + +1. **uprobe crate 骨架**(`kernel/crates/uprobe/`,新建) → 验证: `make kernel` 编译通过。`lib.rs`+core(`UprobeBuilder`/`UprobeBasic`/`UprobePoint`,含原指令副本 + XOL slot 偏移 + 回调);`arch/mod.rs` 定义 `UprobeOps`/独立 `CallBackFunc`。**不复用 kprobe 的 `single_step_address`**。 + +2. **x86 指令分析模块**(`kernel/crates/uprobe/src/arch/x86/`) → 验证: 算出指令长度 + 识别 RIP-relative。**直接复用 yaxpeax-x86**(kprobe 已依赖)算长度 + RIP-relative 检测;生成 XOL slot 副本(RIP-relative 重定位,其余 fail-fast)。 + +3. **per-mm uprobe 管理 + XOL 区**(`kernel/src/mm/ucontext/`) → 验证: 注册/注销 uprobe;每 mm 有 XOL VMA。`AddressSpace`/`InnerAddressSpace` 加 `uprobe_list`,由**独立 irqsave SpinLock** 保护(评审 F8);XOL slot 页注册时预分配、slot 分配/回收。 + +4. **断点页安装**(`kernel/src/mm/`) → 验证: 装 0xcc 后目标 CPU flush_tlb 后执行到该地址立即 #BP;fork/mmap/unmap 并发不崩;writeback 不回写 0xcc。**复刻 do_wp_page 私有 COW**(决策3):copy_page_as_normal + 0xcc + 单次 set_entry + rmap detach/attach + flush_tlb_range;每目标 mm 私有副本,不改共享 page-cache。 + +5. **异常分发分支**(`arch/x86_64/interrupt/trap.rs` + `exception/`) → 验证: 用户态执行到 uprobe 触发 #BP,handler 收正确 pt_regs;XOL 单步后正确返回原址继续;未消费 #BP 投递 SIGTRAP(TRAP_BRKPT)。`do_int3`/`do_debug` 加 `is_from_user()`:用户态 #BP→per-mm 查表→**pre_handler/BPF 入口 rip = break_address()(原探针址,XOL slot 绝不暴露给 BPF,评审 F5)**→BPF 返回后 rip→XOL slot→设 TF;用户态 #DB(XOL 完成)→rip 回原址+insn_len→清 TF→post_handler。 + +6. **ptrace 协调**(`process/ptrace.rs`) → 验证: 处理顺序文档化;被 ptrace 进程挂 uprobe 行为可预测(kprobe > uprobe > ptrace)。TF 拥有权在 uprobe 单步窗口归 uprobe。 + +7. **perf 接入**(`perf/`) → 验证: eBPF 经 `perf_event_open`+`PERF_EVENT_IOC_SET_BPF` attach 成功。**复用 PERF_TYPE_MAX(=6)**,按 config1 name 含 `/` 区分 uprobe/kprobe(评审 F9,最小 Linux 兼容路径);`UprobePerfEvent`(照 `KprobePerfEvent`);解析 path+config2(offset),**消费 pid**;无 sysfs event-source 设备(后续阶段)。 + +8. **NEED_UPROBE = #DB 分发判别位(非 exit-loop 延迟工作)**(`process/state.rs`+`exception/`) → 验证: do_debug 正确识别 XOL 完成 #DB。**用途**(评审 F4):#BP handler 设 NEED_UPROBE;`do_debug` 检查并清之以识别「XOL 单步完成的 #DB」(区别 ptrace/硬件断点 #DB),rip 改回 probe+insn_len、清 TF。 + +9. **bindings**(`include/bindings/linux_bpf.rs`) → 验证: 与 Linux 对齐。**确定** uprobe 复用 `BPF_PROG_TYPE_KPROBE(=2)`(评审 F10);`perf_type_id` 无需新增。 + +**装弹顺序不变量(评审 F6)**:perf_event_open 内严格按 XOL slot 分配 → 表项注册 → 0xcc 页发布 顺序;0xcc 发布前任何路径查该 vaddr 必须能找到就绪 uprobe 表项。 + +## 显式假设(替代 Interview) +- [假设A] ~~无解码器~~ → **已确认 kprobe 依赖 yaxpeax-x86,可直接复用**(评审 F3 核实)。 +- [假设B] `AddressSpace` 可挂 uprobe 表 + 加 XOL VMA — 不确定性: 低 — 已验证有 user_vm/page_table_edit。 +- [假设C] XOL 用 RIP-relative 重定位覆盖常见指令 — 不确定性: 中 — 不支持的指令 fail-fast,首期覆盖率受限 — 可接受(agentsight 探函数入口)。 + +## 已知风险 +- 指令解码风险**大幅降级**(yaxpeax-x86 已存在,评审 F3);主要剩余:RIP-relative 重定位正确性 + fail-fast 覆盖率。接受部分。 +- COW 页并发:复用 do_wp_page 模板 + i_mmap_rwsem + page_table_edit_lock + MmuGather shootdown-before-free。接受(基础成熟)。 +- ptrace + uprobe TF 冲突:定义处理顺序,阶段一不保证同进程 ptrace+uprobe 完美共存。接受(先正名)。 +- 共享 page-cache 绝不直接 patch(决策3 私有 COW 保证 writeback 不损坏)。 + +## 评审记录(PlanReviewer,glm-5.2,10 findings 全采纳) + +| # | finding | priority | 处理 | +|---|---|---|---| +| F1 | 断点页替换用错原语(unmap+map_phys 瞬时空 PTE) | 必须 | 决策3 改为复刻 do_wp_page 单次 set_entry 原子写。采纳。 | +| F2 | 断点页替换遗漏 page-type 转换 + rmap 账簿 | 必须 | 决策3 加 copy_page_as_normal + detach/attach。采纳。 | +| F3 | 假设A 错误,kprobe 已依赖 yaxpeax-x86 | 必须 | 步骤2 改复用 yaxpeax;假设A 删除;风险降级。采纳。 | +| F4 | 步骤8 误述 NEED_UPROBE(应是 #DB 判别位) | 必须 | 步骤8 重写为 #DB 分发判别位。采纳。 | +| F5 | 步骤5 未写死 BPF 透明性 | 建议 | 步骤5 补 pre_handler rip=break_address() 不变量。采纳。 | +| F6 | 装弹顺序不变量缺失 | 建议 | 加 XOL slot→表项→0xcc 发布顺序约束。采纳。 | +| F7 | pid 语义与页策略矛盾(共享页) | 建议 | 决策3/6 明确每 mm 私有 COW,不改共享 page-cache。采纳。 | +| F8 | per-mm 表锁类型未指定(IRQ-off 不可 RwSem) | 建议 | 决策2 改 irqsave SpinLock。采纳。 | +| F9 | perf type 约定未定 | 建议 | 步骤7 拍板复用 PERF_TYPE_MAX 按 name 含 / 区分。采纳。 | +| F10 | 步骤9 bindings 未确定 | 建议 | 步骤9 确定 BPF_PROG_TYPE_KPROBE,perf_type_id 无需新增。采纳。 | + +无"评审错了"项——10 条均为真实问题/合理建议。架构方向正确,未回退 Phase 0。 diff --git a/kernel/Cargo.lock b/kernel/Cargo.lock index d3cbe066f6..5783d8629f 100644 --- a/kernel/Cargo.lock +++ b/kernel/Cargo.lock @@ -515,6 +515,7 @@ dependencies = [ "uefi-raw", "unified-init", "unwinding", + "uprobe", "virtio-drivers", "wait_queue_macros", "x86", @@ -1871,6 +1872,15 @@ dependencies = [ "gimli", ] +[[package]] +name = "uprobe" +version = "0.1.0" +dependencies = [ + "log", + "yaxpeax-arch", + "yaxpeax-x86", +] + [[package]] name = "uuid" version = "0.8.2" diff --git a/kernel/Cargo.toml b/kernel/Cargo.toml index d3e8808bec..9e5ad7f770 100644 --- a/kernel/Cargo.toml +++ b/kernel/Cargo.toml @@ -92,6 +92,7 @@ paste = "=1.0.14" slabmalloc = { path = "crates/rust-slabmalloc" } log = "0.4.21" kprobe = { path = "crates/kprobe" } +uprobe = { path = "crates/uprobe" } lru = "0.12.3" rbpf = { git = "https://git.mirrors.dragonos.org.cn/DragonOS-Community/rbpf", rev = "f31e471a29", default-features = false } diff --git a/kernel/crates/uprobe/Cargo.toml b/kernel/crates/uprobe/Cargo.toml new file mode 100644 index 0000000000..b4e9e19990 --- /dev/null +++ b/kernel/crates/uprobe/Cargo.toml @@ -0,0 +1,11 @@ +[package] +name = "uprobe" +version = "0.1.0" +edition = "2021" + +[dependencies] +log = "0.4.21" + +[target.'cfg(target_arch = "x86_64")'.dependencies] +yaxpeax-x86 = { version = "2", default-features = false, features = ["fmt"] } +yaxpeax-arch = { version = "0", default-features = false } diff --git a/kernel/crates/uprobe/src/arch/mod.rs b/kernel/crates/uprobe/src/arch/mod.rs new file mode 100644 index 0000000000..dc40f9d187 --- /dev/null +++ b/kernel/crates/uprobe/src/arch/mod.rs @@ -0,0 +1,97 @@ +//! uprobe 的架构无关 trait 定义与架构分发。 +//! +//! - [`ProbeArgs`]:处理器寄存器/陷阱帧的架构无关视图(供回调使用)。 +//! - [`UprobeOps`]:探测点的访问接口(镜像 kprobe 的 `KprobeOps`,但面向用户态, +//! **不**提供返回内核缓冲区的 `single_step_address`)。 +//! - [`CallBackFunc`]:事件回调(典型为 eBPF 程序入口)。 + +use ::core::any::Any; + +#[cfg(target_arch = "x86_64")] +mod x86; + +#[cfg(target_arch = "x86_64")] +pub use x86::*; + +use crate::core::UprobePoint; + +/// 处理器寄存器/陷阱帧的架构无关视图(供回调使用)。 +/// +/// 与 kprobe 的 `ProbeArgs` 保持相同签名,以便复用同一套 TrapFrame 适配模式; +/// 但 uprobe 是独立 crate,不依赖 kprobe crate(低耦合),故单独定义。 +pub trait ProbeArgs: Send { + /// 供使用者转换为特定架构的 TrapFrame。 + fn as_any(&self) -> &dyn Any; + /// 触发断点异常(#BP)的指令地址(对 uprobe 即 probe_vaddr)。 + fn break_address(&self) -> usize; + /// 触发单步异常(#DB)的指令地址(XOL slot 中原指令执行后的下一条)。 + fn debug_address(&self) -> usize; +} + +/// uprobe 探测点的访问接口。 +/// +/// **关键差异(相对 kprobe 的 `KprobeOps`)**:不提供 `single_step_address`——uprobe +/// 的单步地址是 per-mm 的 XOL slot 用户地址,需 mm 上下文在运行时计算(计划步骤 +/// 3/5),不属于本 crate。分发器 / handler 通过本 trait 即可取得「原指令副本 + 长度 +/// + XOL slot 偏移」,XOL slot 的真实用户地址由 mm 层另行提供。 +pub trait UprobeOps: Send { + /// 0xcc 断点安装地址(即 probe_vaddr)。 + fn break_address(&self) -> usize; + /// 原指令执行完毕后应恢复执行的地址(= break_address + insn_len)。 + fn return_address(&self) -> usize; + /// 原指令副本(前 [`UprobeOps::insn_len`] 字节有效)。 + fn old_instruction(&self) -> &[u8]; + /// 原指令解码长度。 + fn insn_len(&self) -> usize; + /// XOL slot 在 per-mm XOL 页内的偏移(mm 层填充)。 + fn xol_slot_offset(&self) -> usize; +} + +impl UprobeOps for UprobePoint { + fn break_address(&self) -> usize { + self.probe_vaddr + } + fn return_address(&self) -> usize { + self.probe_vaddr + self.insn_len + } + fn old_instruction(&self) -> &[u8] { + &self.old_instruction[..self.insn_len] + } + fn insn_len(&self) -> usize { + self.insn_len + } + fn xol_slot_offset(&self) -> usize { + self.xol_slot_offset + } +} + +/// 处理器函数指针类型。 +pub type HandlerFn = fn(&dyn ProbeArgs); + +/// 函数指针形式的(pre/post)处理器包装。 +pub(crate) struct ProbeHandler { + func: fn(&dyn ProbeArgs), +} + +impl ProbeHandler { + pub fn new(func: fn(&dyn ProbeArgs)) -> Self { + ProbeHandler { func } + } + /// 调用处理器。 + pub fn call(&self, trap_frame: &dyn ProbeArgs) { + (self.func)(trap_frame); + } + + /// 包装的函数指针(供 fork 继承等场景读取)。 + pub fn func(&self) -> fn(&dyn ProbeArgs) { + self.func + } +} + +/// 事件回调(典型为 eBPF 程序入口)。 +/// +/// 与 kprobe 的 `CallBackFunc` 同签名。使用 `Arc` 以便在多个 per-mm 探测点间共享 +/// 同一回调实例。 +pub trait CallBackFunc: Send + Sync { + fn call(&self, trap_frame: &dyn ProbeArgs); +} diff --git a/kernel/crates/uprobe/src/arch/x86/mod.rs b/kernel/crates/uprobe/src/arch/x86/mod.rs new file mode 100644 index 0000000000..369779c3ea --- /dev/null +++ b/kernel/crates/uprobe/src/arch/x86/mod.rs @@ -0,0 +1,432 @@ +//! x86_64 指令分析与 XOL slot 副本生成。 +//! +//! 直接复用 yaxpeax-x86(kprobe 已依赖)解码器: +//! - 用 `InstDecoder::default().decode_slice(bytes)` 解码,取 `.len().to_const()` 得 +//! 指令长度(与 kprobe `arch/x86/mod.rs` 用法一致); +//! - 遍历操作数检测 RIP-relative(`[rip+disp]` 与 `[rip]` 两种呈现均处理); +//! - XOL slot 副本生成分两步:静态分析产出 [`InsnAnalysis`],运行时用真实 slot 地址 +//! 调用 [`build_xol_slot`] 做 RIP-relative 重定位(由 mm 层在命中时调用)。 + +use ::core::convert::TryFrom; + +use yaxpeax_arch::{DecodeError, LengthedInstruction}; +use yaxpeax_x86::amd64::{Instruction, Operand, RegSpec}; + +/// x86_64 单条指令最大长度(含前缀)。 +const MAX_INSN_SIZE: usize = 15; + +/// uprobe 指令分析错误。 +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum UprobeInsnError { + /// 输入字节数不足以解码一条完整指令。 + Truncated, + /// yaxpeax 解码失败(非法操作码 / 操作数 / 前缀等)。 + DecodeFailed, + /// 解码长度超过 x86_64 上限(15 字节)。 + TooLong, + /// RIP-relative 重定位后的位移超出 i32 范围(disp32 装不下)。 + DisplacementOverflow, + /// 控制流指令(call/jmp/ret/jcc/loop/int 等)——XOL 执行会跳出 slot, + /// 后续 #DB 无法反推探针址,且可能损坏栈/控制流。注册时拒绝。 + UnsupportedControlFlow, + /// 指令抑制 #DB(MOV SS/POP SS)、观察临时 TF(PUSHF*)或整体改写 + /// RFLAGS(POPF*)——XOL 单步会改变用户可见状态或丢失 #DB。注册时拒绝。 + UnsafeForXol, + /// REP/REPE/REPNE string instructions may report an intermediate #DB + /// with RIP still at the copied instruction. The phase-1 exact-end XOL + /// state machine cannot complete those iterations safely. + UnsupportedRepeatedString, +} + +/// RIP-relative 重定位信息(静态分析得出,运行时用真实 slot 地址套用)。 +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct RipReloc { + /// 4 字节有符号位移在指令内的字节偏移。 + pub disp_offset: usize, + /// 解码得到的原始有符号位移。 + pub disp: i32, +} + +/// 指令静态分析结果。 +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub struct InsnAnalysis { + /// 解码长度(1..=15)。 + pub insn_len: usize, + /// 若为 RIP-relative 指令,给出重定位所需信息;否则为 `None`。 + pub rip_relative: Option, +} + +/// 解码并静态分析一条 x86_64 指令。 +/// +/// `bytes` 至少应包含完整指令(多余字节被忽略)。返回指令长度与(若存在的) +/// RIP-relative 重定位信息。 +/// +/// # Fail-fast +/// 字节不足 / 解码失败 / 长度超限 → 返回对应错误,调用方据此放弃该探测点。 +pub fn analyze_insn(bytes: &[u8]) -> Result { + if bytes.is_empty() { + return Err(UprobeInsnError::Truncated); + } + let decoder = yaxpeax_x86::amd64::InstDecoder::default(); + let inst = decoder.decode_slice(bytes).map_err(|e| { + if e.data_exhausted() { + UprobeInsnError::Truncated + } else { + UprobeInsnError::DecodeFailed + } + })?; + let insn_len = inst.len().to_const() as usize; + if insn_len == 0 || insn_len > MAX_INSN_SIZE { + return Err(UprobeInsnError::TooLong); + } + if bytes.len() < insn_len { + return Err(UprobeInsnError::Truncated); + } + // 控制流/不安全指令从 XOL slot 执行会破坏单步窗口,注册时拒绝: + // - 控制流(跳转/调用/返回/循环/中断/系统调用):跳出 slot,#DB 无法 + // 在 slot 内捕获,call/ret 还会损坏用户栈(与 Linux uprobe 阶段一致: + // boost/add_on_return 不在本范围)。 + // - MOV SS / POP SS:Intel SDM 规定其后的指令边界抑制 #DB——XOL 单步 + // 完成的 #DB 会丢失(评审 R10)。 + // - PUSHF:会把 uprobe 临时设置的 TF 压入用户栈,改变用户可见结果。 + // - POPF:整体覆写 RFLAGS,清掉 uprobe 置的 TF,单步窗口断裂(评审 R10)。 + if is_control_flow(&inst) { + return Err(UprobeInsnError::UnsupportedControlFlow); + } + if suppresses_debug_or_rewrites_flags(&inst) { + return Err(UprobeInsnError::UnsafeForXol); + } + if is_repeated_string(&inst) { + return Err(UprobeInsnError::UnsupportedRepeatedString); + } + + let rip_relative = find_rip_relative(&inst, insn_len)?; + Ok(InsnAnalysis { + insn_len, + rip_relative, + }) +} + +fn is_repeated_string(inst: &Instruction) -> bool { + use yaxpeax_x86::amd64::Opcode; + + inst.prefixes.rep_any() + && matches!( + inst.opcode(), + Opcode::CMPS + | Opcode::SCAS + | Opcode::MOVS + | Opcode::LODS + | Opcode::STOS + | Opcode::INS + | Opcode::OUTS + ) +} + +/// 判断指令是否为控制流指令(不可从 XOL slot 安全执行)。 +/// +/// 覆盖:直接跳转/调用/返回、条件跳转(Jcc)、循环(LOOP*)、 +/// 中断(INT/INT3/IRET*)、系统调用/返回(SYSCALL/SYSRET)。 +/// 这些指令改变 RIP 的方式使 XOL 单步后的 #DB 无法在 slot 内捕获, +/// 或会向用户栈写入 XOL 地址损坏控制流。 +fn is_control_flow(inst: &Instruction) -> bool { + use yaxpeax_x86::amd64::Opcode; + matches!( + inst.opcode(), + Opcode::CALL + | Opcode::CALLF + | Opcode::JMP + | Opcode::JMPF + | Opcode::RETURN + | Opcode::RETF + | Opcode::LOOP + | Opcode::INT + | Opcode::IRET + | Opcode::IRETD + | Opcode::IRETQ + | Opcode::SYSCALL + | Opcode::SYSRET + ) || inst.opcode().is_jcc() +} + +fn suppresses_debug_or_rewrites_flags(inst: &Instruction) -> bool { + use yaxpeax_x86::amd64::{Opcode, RegSpec}; + if matches!(inst.opcode(), Opcode::PUSHF | Opcode::POPF) { + return true; + } + // `MOV SS, r/m16`(8e /r):装载 SS 后到下一条指令边界之间 #DB 被抑制。 + // yaxpeax 将其解码为 `Opcode::MOV` + 目标操作数为 ss 段寄存器。 + // (`POP SS` 0x17 在 64 位模式为非法编码,解码器直接报错,无需处理。) + if inst.opcode() == Opcode::MOV { + if let Operand::Register { reg } = inst.operand(0) { + return reg == RegSpec::ss(); + } + } + false +} + +/// 在已解码指令中查找 RIP-relative 内存操作数,返回重定位信息。 +/// +/// 必须覆盖**所有** base 为 RIP 的操作数呈现:yaxpeax 对 `[rip+disp]` 给出 +/// `Disp { base: RIP, disp }`,对 `[rip]`(disp 为 0)给出 `MemDeref { base: RIP }`。 +/// 漏判任一形式都会导致 XOL slot 用原始 disp 执行、指向错误地址(静默损坏),故对 +/// 无法安全重定位的 RIP 形式(掩码 / 带 index)一律 fail-fast。 +fn find_rip_relative( + inst: &Instruction, + insn_len: usize, +) -> Result, UprobeInsnError> { + for i in 0..inst.operand_count() { + if let Some(disp) = operand_rip_disp(&inst.operand(i))? { + // [rip+disp32] 编码:位移恒为 4 字节,且位于任何尾随立即数之前。 + // 故 disp_offset = insn_len - 4 - imm_size。 + let imm_size = trailing_immediate_size(inst); + if imm_size + 4 > insn_len { + // 结构异常(理论不应发生),保守失败。 + return Err(UprobeInsnError::DecodeFailed); + } + let disp_offset = insn_len - 4 - imm_size; + return Ok(Some(RipReloc { disp_offset, disp })); + } + } + Ok(None) +} + +/// 判定单个操作数是否为 RIP-relative: +/// - `Ok(Some(disp))`:是,给出有符号位移(`[rip]` 视为 disp=0); +/// - `Ok(None)`:否; +/// - `Err`:是 RIP-relative 但属掩码 / 带 index 的非常规形式,无法安全重定位。 +fn operand_rip_disp(op: &Operand) -> Result, UprobeInsnError> { + match op { + Operand::MemDeref { base } if *base == RegSpec::RIP => Ok(Some(0)), + Operand::Disp { base, disp } if *base == RegSpec::RIP => Ok(Some(*disp)), + // 标准 RIP-relative 不带 SIB index、不带掩码;命中这些形式即 fail-fast。 + Operand::DispMasked { base, .. } + | Operand::MemDerefMasked { base, .. } + | Operand::MemBaseIndexScale { base, .. } + | Operand::MemBaseIndexScaleDisp { base, .. } + | Operand::MemBaseIndexScaleMasked { base, .. } + | Operand::MemBaseIndexScaleDispMasked { base, .. } + if *base == RegSpec::RIP => + { + Err(UprobeInsnError::DecodeFailed) + } + _ => Ok(None), + } +} + +/// 计算指令尾随立即数的字节数(用于定位 [rip+disp32] 的位移偏移)。 +/// +/// x86 编码顺序固定为:前缀 / 操作码 / ModRM / [SIB] / [disp] / [imm], +/// 故 disp 紧邻 imm 之前。对含 [rip+disp32] 内存操作数的指令,至多一个立即数。 +fn trailing_immediate_size(inst: &Instruction) -> usize { + for i in 0..inst.operand_count() { + match inst.operand(i) { + Operand::ImmediateI8 { .. } | Operand::ImmediateU8 { .. } => return 1, + Operand::ImmediateI16 { .. } | Operand::ImmediateU16 { .. } => return 2, + Operand::ImmediateI32 { .. } | Operand::ImmediateU32 { .. } => return 4, + Operand::ImmediateI64 { .. } | Operand::ImmediateU64 { .. } => return 8, + _ => {} + } + } + 0 +} + +/// 生成 XOL slot 副本(复制原指令并对 RIP-relative 做重定位)。 +/// +/// # 参数 +/// - `analysis`:[`analyze_insn`] 的结果。 +/// - `probe_vaddr`:原探测点用户虚拟地址。 +/// - `slot_vaddr`:XOL slot 的真实用户虚拟地址(per-mm,运行时由 mm 层给出)。 +/// - `old_instruction`:原指令字节(前 `analysis.insn_len` 字节有效)。 +/// - `slot`:输出缓冲,长度须 >= `analysis.insn_len`。 +/// +/// # RIP-relative 重定位 +/// 原指令在 `probe_vaddr` 执行时,`[rip+disp]` 的有效地址为 +/// `probe_vaddr + insn_len + disp`(rip 指向下一条指令)。副本在 `slot_vaddr` +/// 执行时,欲保持同一有效地址,需满足 +/// `slot_vaddr + insn_len + new_disp = probe_vaddr + insn_len + disp`,即 +/// `new_disp = disp + (probe_vaddr - slot_vaddr)`。若 `new_disp` 超出 i32 范围则失败。 +pub fn build_xol_slot( + analysis: &InsnAnalysis, + probe_vaddr: usize, + slot_vaddr: usize, + old_instruction: &[u8], + slot: &mut [u8], +) -> Result<(), UprobeInsnError> { + let len = analysis.insn_len; + if old_instruction.len() < len || slot.len() < len { + return Err(UprobeInsnError::Truncated); + } + // 复制原指令。 + slot[..len].copy_from_slice(&old_instruction[..len]); + + // RIP-relative 重定位。 + if let Some(reloc) = analysis.rip_relative { + let delta = probe_vaddr as i64 - slot_vaddr as i64; + let new_disp = reloc.disp as i64 + delta; + let new_disp = + i32::try_from(new_disp).map_err(|_| UprobeInsnError::DisplacementOverflow)?; + slot[reloc.disp_offset..reloc.disp_offset + 4].copy_from_slice(&new_disp.to_le_bytes()); + } + + // 原指令之后的尾随字节填 int3(0xcc)。 + // + // 正常路径:TF 在原指令执行后立即触发 #DB,不会执行到尾随字节。 + // 竞态路径:若 #BP 后、#DB 前该 uprobe 被注销(slot 被释放),且 slot + // 被重分配给另一探针,#DB handler 无法反推 probe_vaddr。此时线程从 slot + // 继续执行会命中尾随 int3 → 再次触发 #BP → 正常 uprobe 分发或 SIGTRAP, + // 而非执行零填充(可能解码为 add [rax], al 等意外指令损坏内存)。 + for b in &mut slot[len..] { + *b = 0xcc; + } + Ok(()) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn insn_len_basic() { + // nop + assert_eq!(analyze_insn(&[0x90]).unwrap().insn_len, 1); + // push rbp + assert_eq!(analyze_insn(&[0x55]).unwrap().insn_len, 1); + // mov rbp, rsp (48 89 e5) + assert_eq!(analyze_insn(&[0x48, 0x89, 0xe5]).unwrap().insn_len, 3); + } + + #[test] + fn no_rip_relative() { + let a = analyze_insn(&[0x48, 0x89, 0xe5]).unwrap(); + assert_eq!(a.insn_len, 3); + assert!(a.rip_relative.is_none()); + } + + #[test] + fn rip_relative_lea() { + // lea rax, [rip+0x1234] -> 48 8d 05 34 12 00 00 + let a = analyze_insn(&[0x48, 0x8d, 0x05, 0x34, 0x12, 0x00, 0x00]).unwrap(); + assert_eq!(a.insn_len, 7); + let r = a.rip_relative.expect("lea rip-rel must be detected"); + // disp occupies the last 4 bytes, no trailing immediate. + assert_eq!(r.disp_offset, 3); + assert_eq!(r.disp, 0x1234); + } + + #[test] + fn rip_relative_with_immediate() { + // mov dword [rip+0x10], 5 -> c7 05 10 00 00 00 05 00 00 00 + // disp32 precedes the imm32; disp_offset = len(10) - 4 - imm(4) = 2. + let bytes = [0xc7, 0x05, 0x10, 0x00, 0x00, 0x00, 0x05, 0x00, 0x00, 0x00]; + let a = analyze_insn(&bytes).unwrap(); + assert_eq!(a.insn_len, 10); + let r = a.rip_relative.expect("mov [rip+disp],imm must be detected"); + assert_eq!(r.disp_offset, 2); + assert_eq!(r.disp, 0x10); + } + + #[test] + fn control_flow_rejected() { + // call rel32 (e8), jmp rel8 (eb), ret (c3), je rel8 (74), syscall (0f 05) + for bytes in [ + &[0xe8, 0x00, 0x00, 0x00, 0x00][..], + &[0xeb, 0xfe][..], + &[0xc3][..], + &[0x74, 0x02][..], + &[0x0f, 0x05][..], + ] { + assert_eq!( + analyze_insn(bytes).unwrap_err(), + UprobeInsnError::UnsupportedControlFlow, + "bytes={bytes:x?}" + ); + } + } + + #[test] + fn debug_suppressing_rejected() { + // pushfq (9c);popfq (9d);mov ss, rax (8e d0) + assert_eq!( + analyze_insn(&[0x9c]).unwrap_err(), + UprobeInsnError::UnsafeForXol + ); + assert_eq!( + analyze_insn(&[0x9d]).unwrap_err(), + UprobeInsnError::UnsafeForXol + ); + assert_eq!( + analyze_insn(&[0x8e, 0xd0]).unwrap_err(), + UprobeInsnError::UnsafeForXol + ); + // 对照:mov ds, eax(8e d8,段编码 3=DS 非 SS)不抑制 #DB,可接受 + assert!(analyze_insn(&[0x8e, 0xd8]).is_ok()); + // 对照:普通 mov(非 SS 目标)可接受 + assert!(analyze_insn(&[0x48, 0x89, 0xe5]).is_ok()); + } + + #[test] + fn repeated_string_instructions_are_rejected() { + for bytes in [ + &[0xf3, 0xa4][..], // rep movsb + &[0xf2, 0xa6][..], // repne cmpsb + &[0xf3, 0xae][..], // repe scasb + ] { + assert_eq!( + analyze_insn(bytes).unwrap_err(), + UprobeInsnError::UnsupportedRepeatedString, + "bytes={bytes:x?}" + ); + } + + // F3 is also a mandatory/semantic prefix for non-string instructions. + // Do not reject PAUSE merely because it shares the REP byte. + assert!(analyze_insn(&[0xf3, 0x90]).is_ok()); + } + #[test] + fn build_slot_relocates_disp() { + // lea rax, [rip+0] -> 48 8d 05 00 00 00 00 + let insn: [u8; 7] = [0x48, 0x8d, 0x05, 0x00, 0x00, 0x00, 0x00]; + let a = analyze_insn(&insn).unwrap(); + let probe_vaddr: usize = 0x1000; + let slot_vaddr: usize = 0x2000; + let mut slot = [0u8; 16]; + build_xol_slot(&a, probe_vaddr, slot_vaddr, &insn, &mut slot).unwrap(); + + // new_disp = 0 + (0x1000 - 0x2000) = -0x1000;opcode/ModRM 不变,仅 disp 被改写。 + assert_eq!(&slot[..3], &insn[..3]); + assert_eq!(&slot[3..7], &(-0x1000i32).to_le_bytes()); + + // 从 slot 执行后仍指向原有效地址:slot+7+new_disp == probe+7+0 + let new_disp = i32::from_le_bytes([slot[3], slot[4], slot[5], slot[6]]); + let eff = slot_vaddr as i64 + 7 + new_disp as i64; + assert_eq!(eff, probe_vaddr as i64 + 7); + } + + #[test] + fn build_slot_displacement_overflow() { + let insn: [u8; 7] = [0x48, 0x8d, 0x05, 0x00, 0x00, 0x00, 0x00]; + let a = analyze_insn(&insn).unwrap(); + let mut slot = [0u8; 16]; + // 跨度超过 i32 范围 -> 重定位失败。 + let huge = (i32::MAX as usize) + 2; + let err = build_xol_slot(&a, huge, 0, &insn, &mut slot).unwrap_err(); + assert_eq!(err, UprobeInsnError::DisplacementOverflow); + } + + #[test] + fn analyze_errors() { + // 空输入。 + assert_eq!(analyze_insn(&[]).unwrap_err(), UprobeInsnError::Truncated); + // 不完整指令(call rel32 只有 opcode、缺 4 字节 imm)-> 解码耗尽输入。 + assert_eq!( + analyze_insn(&[0xe8]).unwrap_err(), + UprobeInsnError::Truncated + ); + // 非法操作码(push es 在 64 位长模式下无效)。 + assert_eq!( + analyze_insn(&[0x06]).unwrap_err(), + UprobeInsnError::DecodeFailed + ); + } +} diff --git a/kernel/crates/uprobe/src/core.rs b/kernel/crates/uprobe/src/core.rs new file mode 100644 index 0000000000..156e4d3223 --- /dev/null +++ b/kernel/crates/uprobe/src/core.rs @@ -0,0 +1,174 @@ +//! uprobe 的架构无关核心数据结构:探测点信息、注册实体与 builder。 +//! +//! 与 kprobe 的关键差异:被探测指令位于用户地址空间,单步执行原指令必须借助 +//! XOL(在用户态 slot 页执行副本),因此探测点结构只保存原指令副本与 XOL slot +//! 偏移,不保存任何内核态“单步地址”。原指令的读取与分析由调用方(mm 层 / perf +//! 层)借助本 crate 导出的 [`crate::analyze_insn`] 完成后回填入 [`UprobePoint`]。 + +use ::alloc::sync::Arc; + +use crate::arch::{CallBackFunc, ProbeArgs, ProbeHandler}; + +/// 用户态指令副本的最大字节数。 +/// +/// x86_64 单条指令最长 15 字节;取 16 既是 2 的幂、便于对齐,也正好与一个 XOL slot +/// 的典型宽度一致(slot 需容纳指令副本并保证其后字节可安全执行/跳转)。 +pub const UPROBE_INSN_COPY_SIZE: usize = 16; + +/// 探测点信息(架构无关数据载体)。 +/// +/// `old_instruction` 的前 `insn_len` 字节为有效原指令副本;其余字节填充 0。 +/// `xol_slot_offset` 为该探测点在所属 per-mm XOL 页内的偏移,由 mm 层(计划步骤 3) +/// 在分配 slot 时填充,本 crate 仅占位(初始 0)。 +#[derive(Debug)] +pub struct UprobePoint { + /// 被探测的用户态虚拟地址(即 0xcc 断点安装地址)。 + pub probe_vaddr: usize, + /// 原指令副本(前 `insn_len` 字节有效)。 + pub old_instruction: [u8; UPROBE_INSN_COPY_SIZE], + /// 原指令解码长度(1..=15)。 + pub insn_len: usize, + /// XOL slot 在 per-mm XOL 页内的偏移(mm 层填充,初始为 0)。 + pub xol_slot_offset: usize, +} + +impl UprobePoint { + /// 以给定探测地址创建一个空白探测点:原指令副本与长度待指令分析后填充, + /// XOL slot 偏移待 mm 层分配 slot 时填充。 + pub fn new(probe_vaddr: usize) -> Self { + UprobePoint { + probe_vaddr, + old_instruction: [0u8; UPROBE_INSN_COPY_SIZE], + insn_len: 0, + xol_slot_offset: 0, + } + } +} + +/// 注册后的 uprobe 实体:探测点 + 回调 + 使能标志。 +pub struct UprobeBasic { + probe_vaddr: usize, + pre_handler: ProbeHandler, + post_handler: ProbeHandler, + event_callback: Option>, + probe_point: Option>, + enable: bool, +} + +impl UprobeBasic { + /// 调用前置处理器(#BP 命中、BPF 入口前)。 + pub fn call_pre_handler(&self, trap_frame: &dyn ProbeArgs) { + self.pre_handler.call(trap_frame); + } + + /// 调用后置处理器(XOL 单步完成、返回原址前)。 + pub fn call_post_handler(&self, trap_frame: &dyn ProbeArgs) { + self.post_handler.call(trap_frame); + } + + /// 调用事件回调(典型为 eBPF 程序入口)。 + pub fn call_event_callback(&self, trap_frame: &dyn ProbeArgs) { + if let Some(callback) = &self.event_callback { + callback.call(trap_frame); + } + } + + /// 更新事件回调。 + pub fn update_event_callback(&mut self, callback: Arc) { + self.event_callback = Some(callback); + } + + pub fn disable(&mut self) { + self.enable = false; + } + + pub fn enable(&mut self) { + self.enable = true; + } + + pub fn is_enabled(&self) -> bool { + self.enable + } + + /// (pre/post)处理器函数指针(供 fork 继承读取)。 + pub fn handlers(&self) -> (crate::arch::HandlerFn, crate::arch::HandlerFn) { + (self.pre_handler.func(), self.post_handler.func()) + } + + /// 事件回调克隆(供 fork 继承读取)。 + pub fn event_callback_arc(&self) -> Option> { + self.event_callback.clone() + } + + /// 被探测的用户态虚拟地址。 + pub fn probe_vaddr(&self) -> usize { + self.probe_vaddr + } + + /// 关联的探测点(若已设置)。 + pub fn probe_point(&self) -> Option<&Arc> { + self.probe_point.as_ref() + } +} + +/// uprobe 的 builder(镜像 kprobe 的 `KprobeBuilder`,但面向用户态)。 +/// +/// uprobe 探测对象是用户态地址,本 crate 无法直接读取用户内存(需目标 mm 的页表 +/// 上下文),因此 builder 只持有“已解析的用户虚拟地址”与回调: +/// - `path` + `offset` 到 `probe_vaddr` 的解析属于 perf 层(计划步骤 7)职责; +/// - 原指令的读取与分析由调用方借助 [`crate::analyze_insn`] 完成后回填入 +/// [`UprobePoint`](可通过 [`UprobeBuilder::with_probe_point`] 注入)。 +pub struct UprobeBuilder { + probe_vaddr: usize, + pre_handler: ProbeHandler, + post_handler: ProbeHandler, + event_callback: Option>, + probe_point: Option>, + enable: bool, +} + +impl UprobeBuilder { + pub fn new( + probe_vaddr: usize, + pre_handler: fn(&dyn ProbeArgs), + post_handler: fn(&dyn ProbeArgs), + enable: bool, + ) -> Self { + UprobeBuilder { + probe_vaddr, + pre_handler: ProbeHandler::new(pre_handler), + post_handler: ProbeHandler::new(post_handler), + event_callback: None, + probe_point: None, + enable, + } + } + + pub fn with_event_callback(mut self, event_callback: Arc) -> Self { + self.event_callback = Some(event_callback); + self + } + + pub fn with_probe_point(mut self, probe_point: Arc) -> Self { + self.probe_point = Some(probe_point); + self + } + + /// 消费 builder,构造注册实体。 + /// + /// 若未显式提供 `probe_point`,则以 `probe_vaddr` 创建一个空白探测点(原指令 + /// 副本 / insn_len / XOL slot 偏移待后续填充)。 + pub fn build(self) -> UprobeBasic { + let probe_point = self + .probe_point + .unwrap_or_else(|| Arc::new(UprobePoint::new(self.probe_vaddr))); + UprobeBasic { + probe_vaddr: self.probe_vaddr, + pre_handler: self.pre_handler, + post_handler: self.post_handler, + event_callback: self.event_callback, + probe_point: Some(probe_point), + enable: self.enable, + } + } +} diff --git a/kernel/crates/uprobe/src/lib.rs b/kernel/crates/uprobe/src/lib.rs new file mode 100755 index 0000000000..acc6c36890 --- /dev/null +++ b/kernel/crates/uprobe/src/lib.rs @@ -0,0 +1,19 @@ +#![no_std] +//! 用户态断点探针(uprobe)支持——架构无关核心与 x86_64 指令分析。 +//! +//! 与 kprobe 的关键差异:被探测指令位于**用户地址空间**,单步执行原指令必须借助 +//! XOL(eXecute Out of Line,在用户态 slot 页执行副本),不能像 kprobe 那样把 rip +//! 指向内核缓冲区(CPL=3 时内核页 supervisor-only + NX 不可执行)。因此本 crate: +//! - 只保存原指令副本与 XOL slot 偏移,**不**提供任何内核态“单步地址”; +//! - 指令分析直接复用 yaxpeax-x86(kprobe 已依赖)。 +//! +//! 本 crate 是 uprobe 整体实现的第一批(计划步骤 1+2),仅含 crate 内部数据结构、 +//! trait 与 x86 指令分析;mm 集成 / 异常分发 / perf 接入由后续步骤完成。 + +extern crate alloc; + +pub mod arch; +pub mod core; + +pub use crate::core::*; +pub use arch::*; diff --git a/kernel/src/arch/x86_64/interrupt/entry.rs b/kernel/src/arch/x86_64/interrupt/entry.rs index 933ea0d404..eb3a9adffc 100644 --- a/kernel/src/arch/x86_64/interrupt/entry.rs +++ b/kernel/src/arch/x86_64/interrupt/entry.rs @@ -575,6 +575,13 @@ pub unsafe fn set_system_trap_gate(irq: u32, ist: u8, vaddr: VirtAddr) { set_gate(idt_entry, 0xEF, ist, vaddr); } +/// 设置中断门(DPL=3) +#[allow(dead_code)] +pub unsafe fn set_system_intr_gate(irq: u32, ist: u8, vaddr: VirtAddr) { + let idt_entry = get_idt_entry(irq); + set_gate(idt_entry, 0xEE, ist, vaddr); +} + #[allow(static_mut_refs)] unsafe fn get_idt_entry(irq: u32) -> &'static mut [u64] { assert!(irq < 256); diff --git a/kernel/src/arch/x86_64/interrupt/mod.rs b/kernel/src/arch/x86_64/interrupt/mod.rs index 21335665fd..1e66e49ded 100644 --- a/kernel/src/arch/x86_64/interrupt/mod.rs +++ b/kernel/src/arch/x86_64/interrupt/mod.rs @@ -242,6 +242,21 @@ impl ProbeArgs for TrapFrame { } } +// uprobe 的 ProbeArgs 与 kprobe 同签名(独立 trait,低耦合);TrapFrame 两套都实现, +// 以便用户态 #BP/#DB 分发把同一个 trapframe 传给 uprobe 的 pre/post/event callback。 +impl uprobe::ProbeArgs for TrapFrame { + fn as_any(&self) -> &dyn Any { + self + } + fn break_address(&self) -> usize { + (self.rip - 1) as usize + } + + fn debug_address(&self) -> usize { + self.rip as usize + } +} + impl crate::process::rseq::RseqTrapFrame for TrapFrame { #[inline] fn rseq_ip(&self) -> usize { diff --git a/kernel/src/arch/x86_64/interrupt/trap.rs b/kernel/src/arch/x86_64/interrupt/trap.rs index 60c5b6faaf..8efe2e28d0 100644 --- a/kernel/src/arch/x86_64/interrupt/trap.rs +++ b/kernel/src/arch/x86_64/interrupt/trap.rs @@ -4,7 +4,7 @@ use log::{error, trace, warn}; use system_error::SystemError; use super::{ - entry::{set_intr_gate, set_system_trap_gate}, + entry::{set_intr_gate, set_system_intr_gate, set_system_trap_gate}, TrapFrame, }; use crate::exception::debug::DebugException; @@ -12,7 +12,7 @@ use crate::exception::ebreak::EBreak; use crate::{ arch::{ipc::signal::Signal, CurrentIrqArch, MMArch}, exception::InterruptArch, - ipc::signal::force_kernel_signal_to_current, + ipc::signal::{force_kernel_signal_to_current, force_sig_fault_to_current}, mm::VirtAddr, process::ProcessManager, smp::core::smp_get_processor_id, @@ -87,7 +87,12 @@ pub fn arch_trap_init() -> Result<(), SystemError> { set_intr_gate(0, 0, VirtAddr::new(trap_divide_error as usize)); set_intr_gate(1, 0, VirtAddr::new(trap_debug as usize)); set_intr_gate(2, 0, VirtAddr::new(trap_nmi as usize)); - set_system_trap_gate(3, 0, VirtAddr::new(trap_int3 as usize)); + // #BP must enter with maskable interrupts disabled. Uprobe teardown + // uses a synchronous IPI as the grace point between restoring the + // opcode and removing its hit-table entry; a trap gate could + // acknowledge that IPI before do_int3 acquires the entry. The user + // uprobe path reenables interrupts after capturing its XOL slot lease. + set_system_intr_gate(3, 0, VirtAddr::new(trap_int3 as usize)); set_system_trap_gate(4, 0, VirtAddr::new(trap_overflow as usize)); set_system_trap_gate(5, 0, VirtAddr::new(trap_bounds as usize)); set_intr_gate(6, 0, VirtAddr::new(trap_undefined_opcode as usize)); @@ -115,7 +120,21 @@ pub fn arch_trap_init() -> Result<(), SystemError> { /// 处理除法错误 0 #DE #[no_mangle] -unsafe extern "C" fn do_divide_error(regs: &'static TrapFrame, error_code: u64) { +unsafe extern "C" fn do_divide_error(regs: &'static mut TrapFrame, error_code: u64) { + if regs.is_from_user() { + // A divide fault from an XOL instruction is a synchronous user fault, + // not a kernel failure. Mark the transaction trapped so the signal + // gate restores the original probe context before building sigframe. + crate::exception::uprobe::mark_current_xol_trapped(); + CurrentIrqArch::interrupt_enable(); + const FPE_INTDIV: i32 = 1; + if let Err(err) = + force_sig_fault_to_current(Signal::SIGFPE, FPE_INTDIV, VirtAddr::new(regs.rip as usize)) + { + error!("failed to send SIGFPE for user divide error: {:?}", err); + } + return; + } error!( "do_divide_error(0), \tError code: {:#x},\trsp: {:#x},\trip: {:#x},\t CPU: {}, \tpid: {:?}", error_code, @@ -130,15 +149,49 @@ unsafe extern "C" fn do_divide_error(regs: &'static TrapFrame, error_code: u64) /// 处理调试异常 1 #DB #[no_mangle] unsafe extern "C" fn do_debug(regs: &'static mut TrapFrame, error_code: u64) { + // DR6 可同时报告 BS(single-step) 与 B0-B3(hardware breakpoint)。必须在 + // 任何 handler 前保存并复位,避免旧 cause 污染下一次 #DB。 + let dr6 = read_and_reset_dr6(); trace!( - "do_debug(1), \tError code: {:#x},\trsp: {:#x},\trip: {:#x},\t CPU: {}, \tpid: {:?}", + "do_debug(1), \tError code: {:#x},\tdr6: {:#x},\trsp: {:#x},\trip: {:#x},\t CPU: {}, \tpid: {:?}", error_code, + dr6, regs.rsp, regs.rip, smp_get_processor_id().data(), ProcessManager::current_pid() ); - DebugException::handle(regs).unwrap(); + if regs.is_from_user() { + // 用户态 #DB:uprobe XOL 单步完成优先(handler 返回是否消费,评审 R4); + // 未消费的用户态单步进入 SIGTRAP/ptrace 路径。DebugException 只查 + // 内核 kprobe 表,不能处理用户态 #DB,否则会静默吞掉调试异常。 + if crate::exception::uprobe::uprobe_debug_handler(regs, dr6).unwrap() { + // 已被 uprobe 消费(精确 XOL 完成)。 + } else { + crate::exception::uprobe::send_user_debug_sigtrap(regs.rip as usize, dr6).unwrap(); + } + } else { + // 内核态 #DB:kprobe 单步完成。 + DebugException::handle(regs).unwrap(); + } +} + +#[inline] +unsafe fn read_and_reset_dr6() -> u64 { + let dr6: u64; + core::arch::asm!( + "mov {}, dr6", + out(reg) dr6, + options(nomem, nostack, preserves_flags) + ); + // Intel/Linux 要求保留 DR6 的固定 1 位并清除可写状态位。 + const DR6_RESET: u64 = 0xffff_0ff0; + core::arch::asm!( + "mov dr6, {}", + in(reg) DR6_RESET, + options(nomem, nostack, preserves_flags) + ); + dr6 } /// 处理NMI中断 2 NMI @@ -166,7 +219,20 @@ unsafe extern "C" fn do_int3(regs: &'static mut TrapFrame, error_code: u64) { smp_get_processor_id().data(), ProcessManager::current_pid() ); - EBreak::handle(regs).unwrap(); + if regs.is_from_user() { + // 用户态 #BP:uprobe 命中或未消费 #BP(→ SIGTRAP)。 + crate::exception::uprobe::uprobe_breakpoint_handler(regs).unwrap(); + } else { + // Vector 3 is now an interrupt gate for the user-uprobe entry race. + // Preserve the old trap-gate behavior for kernel breakpoints: a + // context entered with IF set may run kprobe callbacks with interrupts + // enabled, while an irq-off context remains irq-off. + if regs.rflags & (1 << 9) != 0 { + CurrentIrqArch::interrupt_enable(); + } + // 内核态 #BP:kprobe 命中。 + EBreak::handle(regs).unwrap(); + } } /// 处理溢出异常 4 #OF @@ -199,8 +265,11 @@ unsafe extern "C" fn do_bounds(regs: &'static TrapFrame, error_code: u64) { /// 处理未定义操作码异常 6 #UD #[no_mangle] -unsafe extern "C" fn do_undefined_opcode(regs: &'static TrapFrame, error_code: u64) { +unsafe extern "C" fn do_undefined_opcode(regs: &'static mut TrapFrame, error_code: u64) { if regs.is_from_user() { + // #UD 不可恢复且下面确定投递 SIGILL;若发生在 XOL,signal frame 必须 + // 看到原探针址,而不是 slot 地址。 + crate::exception::uprobe::mark_current_xol_trapped(); CurrentIrqArch::interrupt_enable(); if let Err(err) = force_kernel_signal_to_current(Signal::SIGILL) { error!( @@ -333,8 +402,11 @@ unsafe extern "C" fn do_stack_segment_fault(regs: &'static TrapFrame, error_code /// 处理一般保护异常 13 #GP #[no_mangle] -unsafe extern "C" fn do_general_protection(regs: &'static TrapFrame, error_code: u64) { +unsafe extern "C" fn do_general_protection(regs: &'static mut TrapFrame, error_code: u64) { if regs.is_from_user() { + // 用户 #GP 在这里确定转换为 SIGSEGV;不要在异常入口无条件 abort, + // 只标记实际信号递送路径。 + crate::exception::uprobe::mark_current_xol_trapped(); CurrentIrqArch::interrupt_enable(); if let Err(err) = force_kernel_signal_to_current(Signal::SIGSEGV) { error!( diff --git a/kernel/src/arch/x86_64/ipc/signal.rs b/kernel/src/arch/x86_64/ipc/signal.rs index e1e6ce3575..07fc694bf7 100644 --- a/kernel/src/arch/x86_64/ipc/signal.rs +++ b/kernel/src/arch/x86_64/ipc/signal.rs @@ -749,6 +749,12 @@ struct SignalFrameLocation { unsafe fn do_signal(frame: &mut TrapFrame, got_signal: &mut bool) { let pcb = ProcessManager::current_pcb(); + // Linux uprobe_deny_signal(): 普通异步信号不能在 XOL 单条指令中间构造 + // signal frame;同步陷阱或 fatal signal 则必须先把 RIP 恢复到原探针址。 + if !crate::exception::uprobe::signal_gate(frame) { + return; + } + let siginfo = pcb.try_siginfo_irqsave(5); if unlikely(siginfo.is_none()) { diff --git a/kernel/src/arch/x86_64/mm/fault.rs b/kernel/src/arch/x86_64/mm/fault.rs index 3e3e001677..a18e225817 100644 --- a/kernel/src/arch/x86_64/mm/fault.rs +++ b/kernel/src/arch/x86_64/mm/fault.rs @@ -278,7 +278,13 @@ impl X86_64MMArch { flags |= FaultFlags::FAULT_FLAG_INSTRUCTION; } + let fault_from_user = regs.is_from_user(); let send_fault_signal = |sig: Signal, code: i32, addr: VirtAddr| { + // 只有走到最终 fault->signal 分支才标记 Trapped;可恢复的缺页、 + // retry 和 exception-table fixup 均保持 Running,继续完成同一 XOL。 + if fault_from_user { + crate::exception::uprobe::mark_current_xol_trapped(); + } if let Err(e) = force_sig_fault_to_current(sig, code, addr) { error!( "failed to force {:?} fault to current process: pid={:?}, code={}, addr={:#x}, err={:?}", diff --git a/kernel/src/bpf/prog/mod.rs b/kernel/src/bpf/prog/mod.rs index 3866c9dfa0..b5bbe29b7c 100644 --- a/kernel/src/bpf/prog/mod.rs +++ b/kernel/src/bpf/prog/mod.rs @@ -8,7 +8,7 @@ use crate::bpf::prog::verifier::BpfProgVerifier; use crate::filesystem::vfs::file::{File, FileFlags}; use crate::filesystem::vfs::InodeMode; use crate::filesystem::vfs::{FilePrivateData, FileSystem, FileType, IndexNode, Metadata}; -use crate::include::bindings::linux_bpf::bpf_attr; +use crate::include::bindings::linux_bpf::{bpf_attr, bpf_prog_type, BPF_F_SLEEPABLE}; use crate::libs::mutex::MutexGuard; use crate::process::ProcessManager; use alloc::string::String; @@ -39,6 +39,14 @@ impl BpfProg { &mut self.meta.insns } + pub fn prog_type(&self) -> bpf_prog_type { + self.meta.prog_type + } + + pub fn is_sleepable(&self) -> bool { + self.meta.prog_flags & BPF_F_SLEEPABLE != 0 + } + pub fn insert_map(&mut self, map_ptr: usize) { self.raw_file_ptr.push(map_ptr); } diff --git a/kernel/src/exception/mod.rs b/kernel/src/exception/mod.rs index 90dcd405d7..1195f8c5db 100644 --- a/kernel/src/exception/mod.rs +++ b/kernel/src/exception/mod.rs @@ -24,6 +24,8 @@ mod resend; pub mod softirq; pub mod sysfs; pub mod tasklet; +#[cfg(target_arch = "x86_64")] +pub mod uprobe; pub mod workqueue; pub(crate) use interrupt_context::enter_hardirq; diff --git a/kernel/src/exception/uprobe.rs b/kernel/src/exception/uprobe.rs new file mode 100644 index 0000000000..276838764b --- /dev/null +++ b/kernel/src/exception/uprobe.rs @@ -0,0 +1,430 @@ +//! uprobe 用户态异常分发(计划步骤 5/6/8)。 +//! +//! `do_int3`/`do_debug` 按 [`TrapFrame::is_from_user`] 二分:用户态走本模块, +//! 内核态走现有 kprobe 分发([`crate::exception::ebreak::EBreak`] / +//! [`crate::exception::debug::DebugException`])。 +//! +//! # 命中路径约束(关中断,entry.S `cli`) +//! +//! - 仅 `lock_irqsave` + 查表 + 改 trapframe + 跑 BPF(callback); +//! - 绝不改页表、绝不睡眠、绝不取 `AddressSpace` 的 `RwSem`(会睡眠); +//! - XOL slot 内容通过物理页的内核 direct-map 写入(`phys_2_virt`),无需 +//! `PageMapper`/`RwSem`——这是 batch2 `XolArea::page_paddr` 的设计意图。 +//! +//! # F4:NEED_UPROBE 是 #DB 分发判别位 +//! +//! #BP handler 在重定向 rip 到 XOL slot 前置 `NEED_UPROBE`;用户态 #DB handler +//! 检查并清之以识别「XOL 单步完成的 #DB」,区别于 ptrace/硬件断点 #DB。 +//! +//! # F5:BPF 透明性 +//! +//! `call_pre_handler`/`call_event_callback`/`call_post_handler` 入口 rip 必须是 +//! 原探针语境(`probe_vaddr` 或 `probe_vaddr + insn_len`),XOL slot 用户地址 +//! **绝不**暴露给 BPF。 + +use alloc::sync::Arc; +use alloc::vec::Vec; + +use crate::arch::interrupt::TrapFrame; +use crate::arch::ipc::signal::Signal; +use crate::arch::CurrentIrqArch; +use crate::exception::InterruptArch; +use crate::ipc::signal::force_sig_fault_to_current; +use crate::mm::ucontext::{UprobeConsumerRuntimeSnapshot, XolSlotLease}; +use crate::mm::VirtAddr; +use crate::process::{ProcessControlBlock, ProcessFlags, ProcessManager}; +use kprobe::ProbeArgs; +use log::{debug, warn}; +use system_error::SystemError; +use uprobe::UprobeOps; + +/// `SIGTRAP` 的 `si_code`:breakpoint trap(镜像 Linux `TRAP_BRKPT`,用于未消费的 +/// 用户态 #BP)。 +const TRAP_BRKPT: i32 = 1; + +/// `SIGTRAP` 的 `si_code`:single-step trap。 +/// +/// Linux 在用户态 TF 单步完成后使用该 code。DragonOS 已在 #DB 入口区分 +/// DR6.BS 与 B0-B3,但尚未实现完整 ptrace virtual_dr6。 +const TRAP_TRACE: i32 = 2; + +/// `SIGTRAP` 的 `si_code`:硬件断点/观察点。 +const TRAP_HWBKPT: i32 = 4; + +/// x86 DR6 cause bits。 +pub const DR6_TRAP_BITS: u64 = 0xf; +pub const DR6_SINGLE_STEP: u64 = 1 << 14; + +/// RFLAGS 的 TF(Trap Flag)位——置位后每条指令执行完触发 #DB,用于 XOL 单步。 +const RFLAGS_TF: u64 = 1 << 8; + +/// XOL 单步的生命周期状态,镜像 Linux 的 `UTASK_SSTEP` 与 +/// `UTASK_SSTEP_TRAPPED`。 +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum ActiveXolState { + Running, + Trapped, +} + +/// Per-thread 活跃 XOL 单步状态(评审 R2/R3/R5/R12)。 +/// 在 #BP 重定向 rip 到 XOL slot **之前**保存到执行线程的 PCB; +/// #DB 到达时取回——不依赖 uprobe_list/slot 反查,使「另一线程在 XOL +/// 窗口内注销探针并释放 slot」的竞态不影响本线程的恢复语义。 +pub struct ActiveXol { + /// 被探测的原地址(abort 路径重新执行处)。 + pub probe_vaddr: usize, + /// 原指令执行完毕后的返回地址(= probe_vaddr + insn_len)。 + pub return_addr: usize, + /// 进入 XOL 前 RFLAGS.TF 的原始值(程序自身/调试器可能已置单步)。 + pub orig_tf: bool, + /// 原指令在 XOL slot 中执行完毕后的精确 RIP。 + pub slot_end: usize, + /// Strongly hold the slot lease so concurrent close can only withdraw + /// future hits and cannot reuse this in-flight instruction slot. + pub xol_lease: Arc, + /// #BP 时 enabled consumer 的稳定快照。post handler 必须遍历它,不能从 + /// 可能已被 close 删除的 per-mm 命中表重新查找。 + pub participants: Arc>, + pub state: ActiveXolState, +} + +impl core::fmt::Debug for ActiveXol { + fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result { + f.debug_struct("ActiveXol") + .field("probe_vaddr", &self.probe_vaddr) + .field("return_addr", &self.return_addr) + .field("orig_tf", &self.orig_tf) + .field("slot_end", &self.slot_end) + .field("xol_slot_offset", &self.xol_lease.offset()) + .field("participant_count", &self.participants.len()) + .field("state", &self.state) + .finish_non_exhaustive() + } +} + +/// 用户态 #BP(int3)分发:uprobe 命中则 XOL 单步原指令,否则投递 +/// `SIGTRAP(TRAP_BRKPT)`。 +/// +/// 调用方(`do_int3`)已保证 `is_from_user()` 为真。 +pub fn uprobe_breakpoint_handler(frame: &mut TrapFrame) -> Result<(), SystemError> { + let break_addr = frame.break_address(); // = rip - 1 = probe_vaddr(raw rip 保留, + // 供 BPF 回调经 break_address() 取得原探针址——batch3↔batch4 运行时契约)。 + + // F5:回调执行期间 rip 必须保持 raw(probe_vaddr+1),绝不在此预设为原探针址, + // 否则回调内 `break_address()=rip-1` 会得到 probe_vaddr-1 而非 probe_vaddr。 + // XOL slot 用户址仅在所有回调返回后才写入 rip,绝不暴露给 BPF。 + + let pcb = ProcessManager::current_pcb(); + let mm = match pcb.basic().user_vm() { + Some(mm) => mm, + None => { + // 用户态 #BP 但无 user_vm(不应发生):防御性投递 SIGTRAP。 + return send_sigtrap_brkpt(frame, break_addr); + } + }; + + // ── Phase 1:uprobe_list 锁内——仅做查表与 Arc 收集(评审 R12)── + // 短临界区:收集实例 Arc + 首实例的 slot/返回址,回调在锁外执行, + // 避免持 per-mm 锁跑 BPF 造成长关中断。 + let (xol_lease, slot_vaddr, slot_end, return_addr, participants) = { + let list = mm.uprobe_list.lock_irqsave(); + let Some(entries) = list.get(&break_addr) else { + drop(list); + // 无匹配 uprobe → 未消费用户态 #BP → SIGTRAP(TRAP_BRKPT) + return send_sigtrap_brkpt(frame, break_addr); + }; + if entries.is_empty() { + drop(list); + return send_sigtrap_brkpt(frame, break_addr); + } + let first = entries[0].read(); + let pp = first.point.clone(); + let site = first.site.clone(); + let xol_lease = first.xol_lease.clone(); + let slot_vaddr = xol_lease.slot_vaddr().data(); + let slot_end = slot_vaddr + .checked_add(pp.insn_len()) + .ok_or(SystemError::EINVAL)?; + let return_addr = pp.return_address(); + drop(first); + + let participants = site.participants.read().clone(); + (xol_lease, slot_vaddr, slot_end, return_addr, participants) + }; // uprobe_list 释放 + + // #BP uses a DPL=3 interrupt gate so teardown cannot observe this CPU's + // shootdown acknowledgement before the hit-table lookup and slot lease + // capture are complete. The XOL VMA is immutable, so callbacks may run + // from this point with normal user-exception interrupt semantics. + unsafe { CurrentIrqArch::interrupt_enable() }; + + // ── Phase 1.5:锁外跑 pre_handler + event_callback(评审 R12)── + // rip 保持 raw(probe_vaddr+1):BPF 回调经 break_address()=rip-1 取得原探针址。 + for participant in participants.iter() { + if !participant.permits_task(&pcb) { + continue; + } + (participant.pre_handler)(frame); + if let Some(callback) = participant.event_callback.as_ref() { + callback.call(frame); + } + } + + // ── Phase 2:保存 per-thread 活跃状态(评审 R2/R5)→ 重定向 rip → 置 TF ── + let orig_tf = frame.rflags & RFLAGS_TF != 0; + { + let mut ss = pcb.uprobe_ss.lock_irqsave(); + *ss = Some(ActiveXol { + probe_vaddr: break_addr, + return_addr, + orig_tf, + slot_end, + xol_lease, + participants, + state: ActiveXolState::Running, + }); + } + frame.set_rip(slot_vaddr); + frame.rflags |= RFLAGS_TF; + pcb.flags().insert(ProcessFlags::NEED_UPROBE); + + Ok(()) +} + +/// 用户态 #DB 分发:`NEED_UPROBE` 置位 → XOL 单步完成(消费);否则本异常 +/// **不属于 uprobe**,返回 `false` 交由调用方路由到正常 debug 路径 +/// (ptrace 单步 / 硬件断点 / SIGTRAP,评审 R4)。 +/// +/// 调用方(`do_debug`)已保证 `is_from_user()` 为真。 +/// +/// 返回值:`true` = 精确完成本次 XOL;`false` = 非 uprobe #DB,或异常 #DB +/// 已 abort 但仍需走正常用户 SIGTRAP 路径。 +pub fn uprobe_debug_handler(frame: &mut TrapFrame, dr6: u64) -> Result { + let pcb = ProcessManager::current_pcb(); + + // ── F4:NEED_UPROBE 是 #DB 判别位 ── + if !pcb.flags().contains(ProcessFlags::NEED_UPROBE) { + // 非 uprobe 单步 #DB:不吞掉(评审 R4),交还 do_debug 走正常 + // DebugException 路径(ptrace / 硬件断点 / SIGTRAP)。 + return Ok(false); + } + + // 清 NEED_UPROBE + 取回 per-thread 活跃状态(评审 R2/R12:O(1), + // 不经 uprobe_list/slot 反查——注销竞态下本线程仍能正确恢复)。 + pcb.flags().remove(ProcessFlags::NEED_UPROBE); + let state = { pcb.uprobe_ss.lock_irqsave().take() }; + let Some(state) = state else { + // NEED_UPROBE 置位但无活跃状态(不应发生):防御性按未消费处理。 + warn!( + "uprobe #DB: NEED_UPROBE set but no active state @ rip {:#x}", + frame.rip + ); + return Ok(false); + }; + + // 只有原指令在本租约的 slot 内恰好执行完毕,才是本次 XOL 的完成 #DB。 + // 页范围判断会把硬件断点或异常改道后的 #DB 错认成完成。 + let rip = frame.rip as usize; + if state.state != ActiveXolState::Running || rip != state.slot_end || dr6 & DR6_SINGLE_STEP == 0 + { + warn!( + "uprobe #DB abort: rip {:#x}, expected {:#x}, dr6 {:#x}, state {:?}, re-execute probe {:#x}", + rip, state.slot_end, dr6, state.state, state.probe_vaddr + ); + restore_after_abort(frame, &state); + pcb.recalc_sigpending(); + // 当前 #DB 不是 XOL 完成事件,交给用户 debug/SIGTRAP 路径。 + return Ok(false); + } + + // ── XOL 完成:恢复 rip 到返回址 + 恢复原始 TF(评审 R5)── + frame.set_rip(state.return_addr); + if state.orig_tf { + // 这次 XOL 就是调试器要求单步的那一条指令。Linux + // arch_uprobe_post_xol() 在这里立即排队 SIGTRAP;若只保留 TF 等待 + // 下一次 #DB,会额外执行一条真实指令后才通知调试器。 + frame.rflags |= RFLAGS_TF; + } else { + frame.rflags &= !RFLAGS_TF; + } + + // post 使用 #BP 时的 participant 快照。并发 close 可从 mm 表移除实例, + // 但不能让已经执行过 pre 的 consumer 丢失配对的 post。 + for participant in state.participants.iter() { + if !participant.permits_task(&pcb) { + continue; + } + (participant.post_handler)(frame); + } + + debug!( + "uprobe XOL single-step done: resume {:#x} (orig_tf={})", + state.return_addr, state.orig_tf + ); + + // 重新发布 XOL 窗口内暂时延迟的普通 pending 信号。 + pcb.recalc_sigpending(); + + // 回调与所有 irqsave 临界区结束后再开中断并排队调试信号。 + if state.orig_tf { + send_sigtrap_trace(state.return_addr)?; + } else if dr6 & DR6_TRAP_BITS != 0 { + // uprobe 只消费自己置 TF 产生的 BS;同一 #DB 中并发出现的硬件断点 + // cause 仍必须对用户可见,不能随 XOL 完成一起吞掉。 + send_sigtrap_hwbkpt(state.return_addr)?; + } + + Ok(true) +} + +/// 向当前进程投递 `SIGTRAP(TRAP_BRKPT)`(未消费的用户态 #BP)。 +/// +/// 参照 `do_undefined_opcode` 发 `SIGILL` 的模式:先开中断再投递信号。 +/// `si_addr` 取断点地址。 +fn send_sigtrap_brkpt(frame: &mut TrapFrame, break_addr: usize) -> Result<(), SystemError> { + // 若这个 #BP 来自 XOL 指令本身,它是确定会投递信号的同步异常。 + mark_current_xol_trapped(); + // #Safety: `interrupt_enable` 仅置位 RFLAGS.IF;此处已脱离 uprobe 命中路径的 + // irqsave 临界区(uprobe_list/xol_area 均已释放),与 do_undefined_opcode 一致。 + unsafe { CurrentIrqArch::interrupt_enable() }; + if let Err(err) = + force_sig_fault_to_current(Signal::SIGTRAP, TRAP_BRKPT, VirtAddr::new(break_addr)) + { + abort_current_xol(frame); + warn!( + "failed to send SIGTRAP(TRAP_BRKPT) for user #BP, pid: {:?}, addr: {:#x}, err: {:?}", + ProcessManager::current_pid(), + break_addr, + err + ); + } + Ok(()) +} + +fn restore_after_abort(frame: &mut TrapFrame, state: &ActiveXol) { + frame.set_rip(state.probe_vaddr); + if state.orig_tf { + frame.rflags |= RFLAGS_TF; + } else { + frame.rflags &= !RFLAGS_TF; + } +} + +/// 把当前 XOL 标成已陷阱。幂等;只应在同步异常确定将投递信号后调用。 +pub fn mark_current_xol_trapped() -> bool { + let pcb = ProcessManager::current_pcb(); + let mut active = pcb.uprobe_ss.lock_irqsave(); + let Some(active) = active.as_mut() else { + return false; + }; + active.state = ActiveXolState::Trapped; + true +} + +/// 幂等中止当前 XOL,并把 trapframe 恢复为可重试原指令的状态。 +pub fn abort_current_xol(frame: &mut TrapFrame) -> bool { + let pcb = ProcessManager::current_pcb(); + pcb.flags().remove(ProcessFlags::NEED_UPROBE); + let state = pcb.uprobe_ss.lock_irqsave().take(); + let Some(state) = state else { + return false; + }; + restore_after_abort(frame, &state); + drop(state); + pcb.recalc_sigpending(); + true +} + +/// exec/exit 等不再返回旧用户上下文的路径可调用此函数释放 ActiveXol 的 +/// site/slot/consumer 强引用。无需也不应修改一个即将废弃的 trapframe。 +pub fn cleanup_task_active_xol(pcb: &ProcessControlBlock) { + pcb.flags().remove(ProcessFlags::NEED_UPROBE); + let state = pcb.uprobe_ss.lock_irqsave().take(); + drop(state); + pcb.recalc_sigpending(); +} + +/// 信号递送门:返回 `false` 表示普通异步信号需要延迟到本条 XOL 指令完成。 +/// fatal 或已标记 Trapped 的路径会先 abort,再允许信号构造用户 frame。 +pub fn signal_gate(frame: &mut TrapFrame) -> bool { + let pcb = ProcessManager::current_pcb(); + let state = pcb + .uprobe_ss + .lock_irqsave() + .as_ref() + .map(|active| active.state); + let Some(state) = state else { + return true; + }; + + // group-exit 与线程/共享 SIGKILL 都必须立刻终止 XOL。该 helper 虽因 OOM + // 命名,但语义正是这里需要的完整 fatal-pending 查询。 + let fatal = Signal::oom_fatal_signal_pending(&pcb); + if state == ActiveXolState::Running && !fatal { + // pending 队列保持不变;只暂时清 fast flag,避免 exit-to-user loop + // 原地重入。XOL 的紧邻 #DB 完成/abort 会 recalc 并重新置位。 + pcb.flags().remove(ProcessFlags::HAS_PENDING_SIGNAL); + // 与并发 signal sender 再校验一次:若 fatal 在首次查询和清 flag 之间 + // 到达,必须本次就 abort;若在本次查询之后到达,sender 会重新置 + // HAS_PENDING_SIGNAL,exit-to-user loop 会再次进入本 gate。 + if Signal::oom_fatal_signal_pending(&pcb) { + mark_current_xol_trapped(); + abort_current_xol(frame); + return true; + } + return false; + } + + if fatal { + mark_current_xol_trapped(); + } + abort_current_xol(frame); + true +} + +/// 投递用户态单步产生的 `SIGTRAP(TRAP_TRACE)`。 +/// +/// 该入口只负责用户调试异常,不得路由到仅处理内核 kprobe 的 +/// `DebugException`。 +pub fn send_sigtrap_trace(addr: usize) -> Result<(), SystemError> { + unsafe { CurrentIrqArch::interrupt_enable() }; + if let Err(err) = force_sig_fault_to_current(Signal::SIGTRAP, TRAP_TRACE, VirtAddr::new(addr)) { + warn!( + "failed to send SIGTRAP(TRAP_TRACE), pid: {:?}, addr: {:#x}, err: {:?}", + ProcessManager::current_pid(), + addr, + err + ); + } + Ok(()) +} + +fn send_sigtrap_hwbkpt(addr: usize) -> Result<(), SystemError> { + send_sigtrap_fault(TRAP_HWBKPT, addr, "TRAP_HWBKPT") +} + +/// 投递未被 uprobe 消费的用户 #DB。DragonOS 尚无完整 ptrace virtual_dr6, +/// 但至少按 Linux get_si_code() 的优先级保留 single-step 与 hardware cause。 +pub fn send_user_debug_sigtrap(addr: usize, dr6: u64) -> Result<(), SystemError> { + if dr6 & DR6_SINGLE_STEP != 0 { + send_sigtrap_trace(addr) + } else if dr6 & DR6_TRAP_BITS != 0 { + send_sigtrap_hwbkpt(addr) + } else { + send_sigtrap_fault(TRAP_BRKPT, addr, "TRAP_BRKPT") + } +} + +fn send_sigtrap_fault(code: i32, addr: usize, name: &str) -> Result<(), SystemError> { + unsafe { CurrentIrqArch::interrupt_enable() }; + if let Err(err) = force_sig_fault_to_current(Signal::SIGTRAP, code, VirtAddr::new(addr)) { + warn!( + "failed to send SIGTRAP({}), pid: {:?}, addr: {:#x}, err: {:?}", + name, + ProcessManager::current_pid(), + addr, + err + ); + } + Ok(()) +} diff --git a/kernel/src/libs/elf.rs b/kernel/src/libs/elf.rs index 03785b2c7b..0ccaed070f 100644 --- a/kernel/src/libs/elf.rs +++ b/kernel/src/libs/elf.rs @@ -250,7 +250,15 @@ impl ElfLoader { prot }; let start_page = user_vm_guard - .map_anonymous(addr_to_map, map_len, tmp_prot, map_flags, false, true) + .map_file_backed( + addr_to_map, + map_len, + tmp_prot, + map_flags, + false, + param.file(), + file_page_offset, + ) .map_err(map_err_handler)?; let mapped = start_page.virt_address(); @@ -1257,6 +1265,13 @@ impl BinaryLoader for ElfLoader { user_vm.end_data = end_data.unwrap_or(VirtAddr::new(0)); let result = BinaryLoaderResult::new(interp_load_addr.unwrap_or(program_entrypoint)); + drop(user_vm); + // ELF segments are installed through locked InnerAddressSpace helpers, + // bypassing the ordinary mmap post-commit hook. Reconcile definitions + // only after the final write guard is released so fault-in cannot + // recurse on the address-space lock. + #[cfg(target_arch = "x86_64")] + crate::mm::ucontext::uprobe::uprobe_apply_to_all_vmas(&binding); // kdebug!("elf load OK!!!"); return Ok(result); } diff --git a/kernel/src/misc/events/kprobe/device.rs b/kernel/src/misc/events/device.rs similarity index 78% rename from kernel/src/misc/events/kprobe/device.rs rename to kernel/src/misc/events/device.rs index a2f93172b8..476dc7dd5f 100644 --- a/kernel/src/misc/events/kprobe/device.rs +++ b/kernel/src/misc/events/device.rs @@ -1,3 +1,5 @@ +//! Shared sysfs device for DragonOS software probe PMUs. + use crate::driver::base::class::Class; use crate::driver::base::device::bus::Bus; use crate::driver::base::device::driver::Driver; @@ -18,38 +20,44 @@ use system_error::SystemError; #[derive(Debug)] #[cast_to([sync] Device)] -pub struct KprobeDevice { - inner: SpinLock, +pub struct ProbePmuDevice { + inner: SpinLock, kobj_state: LockedKObjectState, name: String, + pmu_type: u32, } #[derive(Debug)] -struct InnerKprobeDevice { +struct InnerProbePmuDevice { kobject_common: KObjectCommonData, device_common: DeviceCommonData, } -impl KprobeDevice { - pub fn new(parent: Option>) -> Arc { +impl ProbePmuDevice { + pub fn new(name: &str, pmu_type: u32, parent: Option>) -> Arc { let bus_device = Self { - inner: SpinLock::new(InnerKprobeDevice { + inner: SpinLock::new(InnerProbePmuDevice { kobject_common: KObjectCommonData::default(), device_common: DeviceCommonData::default(), }), kobj_state: LockedKObjectState::new(None), - name: "kprobe".to_string(), + name: name.to_string(), + pmu_type, }; bus_device.set_parent(parent); return Arc::new(bus_device); } - fn inner(&self) -> SpinLockGuard<'_, InnerKprobeDevice> { + fn inner(&self) -> SpinLockGuard<'_, InnerProbePmuDevice> { self.inner.lock() } + + fn pmu_type(&self) -> u32 { + self.pmu_type + } } -impl KObject for KprobeDevice { +impl KObject for ProbePmuDevice { fn as_any_ref(&self) -> &dyn core::any::Any { self } @@ -105,7 +113,7 @@ impl KObject for KprobeDevice { } } -impl Device for KprobeDevice { +impl Device for ProbePmuDevice { #[inline] #[allow(dead_code)] fn dev_type(&self) -> DeviceType { @@ -114,7 +122,7 @@ impl Device for KprobeDevice { #[inline] fn id_table(&self) -> IdTable { - IdTable::new("kprobe".to_string(), None) + IdTable::new(self.name.clone(), None) } fn bus(&self) -> Option> { @@ -164,9 +172,9 @@ impl Device for KprobeDevice { } #[derive(Debug)] -pub struct KprobeAttr; +pub struct ProbeTypeAttr; -impl Attribute for KprobeAttr { +impl Attribute for ProbeTypeAttr { fn name(&self) -> &str { "type" } @@ -178,12 +186,18 @@ impl Attribute for KprobeAttr { fn support(&self) -> SysFSOpsSupport { SysFSOpsSupport::ATTR_SHOW } - fn show(&self, _kobj: Arc, buf: &mut [u8]) -> Result { - if buf.is_empty() { + fn show(&self, kobj: Arc, buf: &mut [u8]) -> Result { + let device = kobj + .as_any_ref() + .downcast_ref::() + .ok_or(SystemError::EINVAL)?; + let value = alloc::format!("{}\n", device.pmu_type()); + if buf.len() < value.len() { return Err(SystemError::EINVAL); } - // perf_type_id::PERF_TYPE_MAX - buf[0] = b'6'; - Ok(1) + buf[..value.len()].copy_from_slice(value.as_bytes()); + Ok(value.len()) } } + +pub static PROBE_TYPE_ATTR: ProbeTypeAttr = ProbeTypeAttr; diff --git a/kernel/src/misc/events/kprobe/mod.rs b/kernel/src/misc/events/kprobe/mod.rs index 90bf874cf3..ce65bebd05 100644 --- a/kernel/src/misc/events/kprobe/mod.rs +++ b/kernel/src/misc/events/kprobe/mod.rs @@ -2,20 +2,22 @@ use crate::driver::base::device::bus::Bus; use crate::driver::base::device::{device_manager, device_register, sys_devices_kset, Device}; use crate::driver::base::kobject::KObject; use crate::init::initcall::INITCALL_DEVICE; +use crate::misc::events::device::{ProbePmuDevice, PROBE_TYPE_ATTR}; use crate::misc::events::get_event_source_bus; -use crate::misc::events::kprobe::device::{KprobeAttr, KprobeDevice}; +use crate::perf::PERF_TYPE_KPROBE; use alloc::sync::Arc; use system_error::SystemError; use unified_init::macros::unified_init; -pub mod device; -static mut KPROBE_DEVICE: Option> = None; +static mut KPROBE_DEVICE: Option> = None; #[unified_init(INITCALL_DEVICE)] pub fn kprobe_subsys_init() -> Result<(), SystemError> { - let kprobe_device = KprobeDevice::new(Some(Arc::downgrade( - &(sys_devices_kset() as Arc), - ))); + let kprobe_device = ProbePmuDevice::new( + "kprobe", + PERF_TYPE_KPROBE, + Some(Arc::downgrade(&(sys_devices_kset() as Arc))), + ); let event_source_bus = get_event_source_bus().ok_or(SystemError::EINVAL)?; kprobe_device.set_bus(Some(Arc::downgrade(&(event_source_bus as Arc)))); @@ -26,6 +28,6 @@ pub fn kprobe_subsys_init() -> Result<(), SystemError> { KPROBE_DEVICE = Some(kprobe_device.clone()); } - device_manager().create_file(&(kprobe_device as Arc), &KprobeAttr)?; + device_manager().create_file(&(kprobe_device as Arc), &PROBE_TYPE_ATTR)?; Ok(()) } diff --git a/kernel/src/misc/events/mod.rs b/kernel/src/misc/events/mod.rs index 8590e32edf..c846709815 100644 --- a/kernel/src/misc/events/mod.rs +++ b/kernel/src/misc/events/mod.rs @@ -5,8 +5,11 @@ use alloc::sync::Arc; use system_error::SystemError; use unified_init::macros::unified_init; +mod device; mod kprobe; mod subsys; +#[cfg(target_arch = "x86_64")] +mod uprobe; static mut EVENT_SOURCE_BUS: Option> = None; diff --git a/kernel/src/misc/events/uprobe/mod.rs b/kernel/src/misc/events/uprobe/mod.rs new file mode 100644 index 0000000000..6dc713f028 --- /dev/null +++ b/kernel/src/misc/events/uprobe/mod.rs @@ -0,0 +1,32 @@ +use crate::driver::base::device::bus::Bus; +use crate::driver::base::device::{device_manager, device_register, sys_devices_kset, Device}; +use crate::driver::base::kobject::KObject; +use crate::init::initcall::INITCALL_DEVICE; +use crate::misc::events::device::{ProbePmuDevice, PROBE_TYPE_ATTR}; +use crate::misc::events::get_event_source_bus; +use crate::perf::PERF_TYPE_UPROBE; +use alloc::sync::Arc; +use system_error::SystemError; +use unified_init::macros::unified_init; + +static mut UPROBE_DEVICE: Option> = None; + +#[unified_init(INITCALL_DEVICE)] +pub fn uprobe_subsys_init() -> Result<(), SystemError> { + let uprobe_device = ProbePmuDevice::new( + "uprobe", + PERF_TYPE_UPROBE, + Some(Arc::downgrade(&(sys_devices_kset() as Arc))), + ); + + let event_source_bus = get_event_source_bus().ok_or(SystemError::EINVAL)?; + uprobe_device.set_bus(Some(Arc::downgrade(&(event_source_bus as Arc)))); + + device_register(uprobe_device.clone())?; + unsafe { + UPROBE_DEVICE = Some(uprobe_device.clone()); + } + + device_manager().create_file(&(uprobe_device as Arc), &PROBE_TYPE_ATTR)?; + Ok(()) +} diff --git a/kernel/src/mm/syscall/mod.rs b/kernel/src/mm/syscall/mod.rs index 89b32ec84f..5f97aa7d22 100644 --- a/kernel/src/mm/syscall/mod.rs +++ b/kernel/src/mm/syscall/mod.rs @@ -21,7 +21,7 @@ mod sys_msync; mod sys_munlock; mod sys_munlockall; mod sys_munmap; -mod sys_process_vm; +pub mod sys_process_vm; pub mod sys_sbrk; bitflags! { diff --git a/kernel/src/mm/syscall/sys_mremap.rs b/kernel/src/mm/syscall/sys_mremap.rs index 8c80d53fd2..7ffede45f9 100644 --- a/kernel/src/mm/syscall/sys_mremap.rs +++ b/kernel/src/mm/syscall/sys_mremap.rs @@ -106,6 +106,13 @@ impl Syscall for SysMremapHandle { (*g.vm_flags(), *g.region()) }; + // Linux vma_to_resize() rejects special mappings before + // MREMAP_FIXED destroys the destination. The uprobe XOL trampoline is + // VM_DONTEXPAND and must remain kernel-owned. + if vm_flags.intersects(VmFlags::VM_DONTEXPAND | VmFlags::VM_PFNMAP) { + return Err(SystemError::EINVAL); + } + // Linux vma_to_resize() semantics: // With MREMAP_FIXED, the *source span being remapped* must be within a single VMA. // - For shrinking, Linux unmaps the tail first and then checks the shrunken length. diff --git a/kernel/src/mm/syscall/sys_process_vm.rs b/kernel/src/mm/syscall/sys_process_vm.rs index 794f7c7c38..52b6c93178 100644 --- a/kernel/src/mm/syscall/sys_process_vm.rs +++ b/kernel/src/mm/syscall/sys_process_vm.rs @@ -149,7 +149,7 @@ fn find_target_process(pid: usize) -> Result, SystemErr /// 3. Current process's uid/gid match target's euid/suid/uid and egid/sgid/gid /// /// See Linux kernel: kernel/ptrace.c __ptrace_may_access() -fn check_process_vm_access(target_pcb: &Arc) -> Result<(), SystemError> { +pub fn check_process_vm_access(target_pcb: &Arc) -> Result<(), SystemError> { let current_pcb = ProcessManager::current_pcb(); // Self-access is always allowed diff --git a/kernel/src/mm/ucontext/address_space.rs b/kernel/src/mm/ucontext/address_space.rs index 2fe8fd9eb1..70e9945e80 100644 --- a/kernel/src/mm/ucontext/address_space.rs +++ b/kernel/src/mm/ucontext/address_space.rs @@ -52,6 +52,26 @@ pub struct AddressSpace { inner: RwSem, /// Wait for pending mmap reservations to be committed or cancelled. reservation_wait: WaitQueue, + // ── uprobe 子系统字段(计划步骤 3)── + // + // 这些字段位于 `inner` **之外**,由独立 irqsave `SpinLock` 保护(评审 F8)。 + // 命中路径(#BP/#DB 关中断)仅 `lock_irqsave` + 查表,绝不取 `inner` 的 RwSem(会睡眠)。 + // + /// Per-mm uprobe 表:`probe_vaddr → 已注册实例列表`。 + /// 镜像 kprobe 的 `KPROBE_MANAGER: SpinLock`。 + #[cfg(target_arch = "x86_64")] + pub uprobe_list: SpinLock>>>>, + /// Per-mm XOL 区(懒初始化,首次注册 uprobe 时创建)。 + #[cfg(target_arch = "x86_64")] + pub xol_area: SpinLock>>, + /// Per-page 断点状态(追踪 COW 副本 + refcount,供注销恢复原页)。 + #[cfg(target_arch = "x86_64")] + pub(crate) uprobe_page_state: SpinLock>, + /// A lifecycle operation removed the XOL VMA and therefore all sites; + /// the post-commit path must reconcile every file VMA, not only the user + /// supplied range (which may contain only the anonymous XOL mapping). + #[cfg(target_arch = "x86_64")] + pub(crate) uprobe_needs_full_reapply: AtomicBool, } impl AddressSpace { @@ -63,7 +83,7 @@ impl AddressSpace { /// retry. Do not move this loop back into `InnerAddressSpace`: doing so /// would turn a normal PageCache invalidation conflict into a false /// population failure or recreate the mmap/writeback lock cycle. - fn populate_range_post_commit( + pub(crate) fn populate_range_post_commit( self: &Arc, start: VirtAddr, len: usize, @@ -251,6 +271,14 @@ impl AddressSpace { oom_reclaim_generation: AtomicU64::new(0), inner: RwSem::new(inner), reservation_wait: WaitQueue::default(), + #[cfg(target_arch = "x86_64")] + uprobe_list: SpinLock::new(BTreeMap::new()), + #[cfg(target_arch = "x86_64")] + xol_area: SpinLock::new(None), + #[cfg(target_arch = "x86_64")] + uprobe_page_state: SpinLock::new(BTreeMap::new()), + #[cfg(target_arch = "x86_64")] + uprobe_needs_full_reapply: AtomicBool::new(false), }); // Back-fill the Weak so that InnerAddressSpace methods can obtain // the outer Arc to construct MmuGather / initiate TLB shootdown. @@ -613,6 +641,8 @@ impl AddressSpace { Some(expected_vma), ); } + #[cfg(target_arch = "x86_64")] + super::uprobe::uprobe_apply_to_range(self, VirtRegion::new(page.virt_address(), len)); return Ok(page); } } @@ -1062,6 +1092,11 @@ impl AddressSpace { Some(expected_vma), ); } + // uprobe:新文件映射提交后,迟到应用注册表中的探针(评审 R9: + // dlopen / 后续 mmap 的文件获得已注册的 uprobe)。写锁已释放, + // 函数内部自取锁;注册表为空时为一次快速查表。 + #[cfg(target_arch = "x86_64")] + super::uprobe::uprobe_apply_to_range(self, region); return Ok(page); } } @@ -1083,11 +1118,15 @@ impl AddressSpace { Ok(notifications) => { drop(guard); InnerAddressSpace::notify_close_notifications(notifications); + #[cfg(target_arch = "x86_64")] + super::uprobe::uprobe_apply_to_range(self, region); return Ok(()); } Err(failure) => { drop(guard); InnerAddressSpace::notify_close_notifications(failure.notifications); + #[cfg(target_arch = "x86_64")] + super::uprobe::uprobe_apply_to_range(self, region); return Err(failure.err); } } @@ -1118,10 +1157,17 @@ impl AddressSpace { continue; } match guard.mprotect_collect(start_page, page_count, prot_flags) { - Ok(()) => return Ok(()), + Ok(()) => { + drop(guard); + #[cfg(target_arch = "x86_64")] + super::uprobe::uprobe_apply_to_range(self, region); + return Ok(()); + } Err(failure) => { drop(guard); InnerAddressSpace::notify_close_notifications(failure.notifications); + #[cfg(target_arch = "x86_64")] + super::uprobe::uprobe_apply_to_range(self, region); return Err(failure.err); } } @@ -1143,10 +1189,28 @@ impl AddressSpace { continue; } match guard.madvise_collect(start_page, page_count, behavior) { - Ok(()) => return Ok(()), + Ok(()) => { + drop(guard); + #[cfg(target_arch = "x86_64")] + if behavior == MadvFlags::MADV_DONTNEED + || behavior == MadvFlags::MADV_DONTNEED_LOCKED + { + // MADV_DONTNEED is advice: refault only pages that host + // persistent probes, so the VMA remains observable + // without retaining unrelated pages. + super::uprobe::uprobe_apply_to_range(self, region); + } + return Ok(()); + } Err(failure) => { drop(guard); InnerAddressSpace::notify_close_notifications(failure.notifications); + #[cfg(target_arch = "x86_64")] + if behavior == MadvFlags::MADV_DONTNEED + || behavior == MadvFlags::MADV_DONTNEED_LOCKED + { + super::uprobe::uprobe_apply_to_range(self, region); + } return Err(failure.err); } } @@ -1261,6 +1325,14 @@ impl AddressSpace { Some(expected_vma), ); } + #[cfg(target_arch = "x86_64")] + super::uprobe::uprobe_apply_to_mremap_ranges( + self, + old_vaddr, + old_len, + outcome.addr, + new_len, + ); return Ok(outcome.addr); } Err(failure) if failure.err == SystemError::EAGAIN_OR_EWOULDBLOCK => { @@ -1285,16 +1357,28 @@ impl AddressSpace { { drop(guard); InnerAddressSpace::notify_close_notifications(failure.notifications); + #[cfg(target_arch = "x86_64")] + super::uprobe::uprobe_apply_to_mremap_ranges( + self, old_vaddr, old_len, new_vaddr, new_len, + ); self.wait_for_no_reservation_conflict(retry_region); continue; } drop(guard); InnerAddressSpace::notify_close_notifications(failure.notifications); + #[cfg(target_arch = "x86_64")] + super::uprobe::uprobe_apply_to_mremap_ranges( + self, old_vaddr, old_len, new_vaddr, new_len, + ); return Err(SystemError::EAGAIN_OR_EWOULDBLOCK); } Err(failure) => { drop(guard); InnerAddressSpace::notify_close_notifications(failure.notifications); + #[cfg(target_arch = "x86_64")] + super::uprobe::uprobe_apply_to_mremap_ranges( + self, old_vaddr, old_len, new_vaddr, new_len, + ); return Err(failure.err); } } diff --git a/kernel/src/mm/ucontext/inner.rs b/kernel/src/mm/ucontext/inner.rs index ddaa65e04e..262b1fb747 100644 --- a/kernel/src/mm/ucontext/inner.rs +++ b/kernel/src/mm/ucontext/inner.rs @@ -117,6 +117,7 @@ impl InnerAddressSpace { let mut parent_cow_remaps: Vec<(VirtAddr, EntryFlags)> = Vec::new(); let mut child_present_pages = 0usize; + let clone_result: Result<(), SystemError> = (|| { // Iterate over each VMA of the parent process and perform appropriate copying based on VMA attributes // Reference Linux: https://code.dragonos.org.cn/xref/linux-6.6.21/mm/memory.c#copy_page_range @@ -255,6 +256,12 @@ impl InnerAddressSpace { // Complete the parent mm's mm-aware shootdown: INV-3 requires TLB completion before continuing with subsequent logic; // since no pages enter pending_pages here, this actually only triggers flush_tlb_mm_range. parent_tlb.finish(); + + // uprobe:把父 mm 的探针继承到子 mm(评审 R9——fork 后探针存活; + // 子页经上面的正常 fork 拷贝已含 0xcc,这里私有化并重建 per-mm 实例)。 + #[cfg(target_arch = "x86_64")] + super::uprobe::fork_inherit_uprobes(&parent_mm, &new_addr_space)?; + return Ok(new_addr_space); } diff --git a/kernel/src/mm/ucontext/mmap.rs b/kernel/src/mm/ucontext/mmap.rs index a413494baf..06cd8823f7 100644 --- a/kernel/src/mm/ucontext/mmap.rs +++ b/kernel/src/mm/ucontext/mmap.rs @@ -118,6 +118,99 @@ impl InnerAddressSpace { return Ok((start_page, notifications)); } + /// 创建**文件关联的立即映射**(file-backed eager mapping)。 + /// + /// 与 [`Self::map_anonymous`] 的区别:创建的 VMA 绑定 `vm_file` + + /// `backing_pgoff`,使 `attach_vma` 将其注册到 inode 的 file-rmap + /// (`page_cache::register_file_vma`)。物理页仍由内核立即分配并零填充 + /// (eager),调用方随后用字节拷贝(如 ELF loader 的 `do_load_file`) + /// 覆盖为真实文件内容。 + /// + /// 这样设计的原因:ELF loader 在持有 `InnerAddressSpace` 写锁的进程上下文中 + /// 加载段,不能触发缺页(缺页需取地址空间锁)。立即映射 + 拷贝既满足 + /// 「页已驻留」(uprobe 注册等需要 translate 目标页)又满足「VMA 在 inode + /// rmap 中可被发现」(uprobe 经 `collect_file_vmas` 定位)。 + /// + /// ## 参数 + /// - `file`:关联的文件(VMA 的 `vm_file`)。 + /// - `file_offset`:段在文件中的**页对齐**偏移(字节),用于计算 + /// `backing_pgoff = file_offset >> PAGE_SHIFT`。 + #[allow(clippy::too_many_arguments)] + pub(crate) fn map_file_backed( + &mut self, + start_vaddr: VirtAddr, + len: usize, + prot_flags: ProtFlags, + map_flags: MapFlags, + round_to_min: bool, + file: Arc, + file_offset: usize, + ) -> Result { + let (page, notifications) = match self.map_file_backed_collect( + start_vaddr, + len, + prot_flags, + map_flags, + round_to_min, + file, + file_offset, + ) { + Ok(outcome) => outcome, + Err(failure) => { + debug_assert!( + failure.notifications.is_empty(), + "locked map_file_backed caller must not replace existing VMAs" + ); + return Err(failure.err); + } + }; + debug_assert!( + notifications.is_empty(), + "locked map_file_backed caller must not replace existing VMAs" + ); + Ok(page) + } + + #[allow(clippy::too_many_arguments)] + fn map_file_backed_collect( + &mut self, + start_vaddr: VirtAddr, + len: usize, + prot_flags: ProtFlags, + map_flags: MapFlags, + round_to_min: bool, + file: Arc, + file_offset: usize, + ) -> Result<(VirtPageFrame, VmaCloseNotifications), MmapFailure> { + let pgoff = file_offset >> MMArch::PAGE_SHIFT; + let fixed_hint = map_flags.intersects(MapFlags::MAP_FIXED | MapFlags::MAP_FIXED_NOREPLACE); + let (start_page, notifications) = self.mmap_collect( + AddressSpace::round_mmap_hint(start_vaddr, round_to_min, fixed_hint), + PageFrameCount::from_bytes(page_align_up(len)).unwrap(), + prot_flags, + map_flags, + move |page, count, vm_flags, flags, mapper, flusher| { + let vma_file = Some(file.clone()); + let vma_pgoff = Some(pgoff); + if !MMArch::PAGE_FAULT_ENABLED { + // 无按需分页:立即映射物理页(零填充),内容由调用方覆盖。 + Ok(VMA::zeroed( + page, count, vm_flags, flags, mapper, flusher, vma_file, vma_pgoff, + )?) + } else { + // 按需分页可用:创建 lazy 文件映射 VMA。 + // 注意:ELF loader 当前对只读段仍需 eager(uprobe 要求页驻留), + // 走 eager 路径时 PAGE_FAULT_ENABLED 为真但调用方仍期望立即映射。 + // 这里与 map_anonymous 一致:无 allocate_at_once 参数时一律 eager。 + Ok(VMA::zeroed( + page, count, vm_flags, flags, mapper, flusher, vma_file, vma_pgoff, + )?) + } + }, + )?; + Ok((start_page, notifications)) + } + /// Map pages into the process's address space /// /// # Parameters diff --git a/kernel/src/mm/ucontext/mod.rs b/kernel/src/mm/ucontext/mod.rs index 8021637d49..1a155df554 100644 --- a/kernel/src/mm/ucontext/mod.rs +++ b/kernel/src/mm/ucontext/mod.rs @@ -5,7 +5,7 @@ use core::{ hash::Hasher, intrinsics::unlikely, ops::Add, - sync::atomic::{compiler_fence, AtomicU64, AtomicUsize, Ordering}, + sync::atomic::{compiler_fence, AtomicBool, AtomicU64, AtomicUsize, Ordering}, }; use alloc::{ @@ -35,6 +35,7 @@ use crate::{ align::page_align_up, cpumask::CpuMask, mutex::{Mutex, MutexGuard}, + rwlock::RwLock, rwsem::{RwSem, RwSemReadGuard, RwSemWriteGuard}, spinlock::SpinLock, wait_queue::WaitQueue, @@ -107,15 +108,29 @@ mod mmap; mod mremap; mod notifications; mod stack; +#[cfg(target_arch = "x86_64")] +pub(crate) mod uprobe; mod vma; mod vma_ops; - +#[cfg(target_arch = "x86_64")] +use self::{mappings::UserMappings, notifications::*, uprobe::UprobePageState, vma::VmaSplitSides}; +#[cfg(not(target_arch = "x86_64"))] use self::{mappings::UserMappings, notifications::*, vma::VmaSplitSides}; pub use address_space::{AddressSpace, FileMappingWithFileArgs}; pub use inner::InnerAddressSpace; pub use mapper::UserMapper; pub use stack::UserStack; +#[cfg(target_arch = "x86_64")] +#[allow(unused_imports)] +pub use uprobe::{ + fork_inherit_uprobes, noop_handler, uprobe_apply_to_existing_vma, uprobe_apply_to_new_vma, + uprobe_new_consumer_id, uprobe_registry_add, uprobe_registry_remove_consumer, + uprobe_registry_set_callback, uprobe_registry_set_enabled, UprobeConsumer, UprobeConsumerReg, + UprobeConsumerRuntime, UprobeConsumerRuntimeSnapshot, UprobeConsumerScope, UprobeDefinition, + UprobeHandle, UprobeInstance, UprobeSite, UprobeSiteState, UprobeTaskScope, XolArea, + XolSlotLease, +}; #[allow(unused_imports)] pub use vma::{ AnonSharedMapping, LockedVMA, PhysmapParams, PresentPfn, Provider, VMASplitResult, VMA, diff --git a/kernel/src/mm/ucontext/mremap.rs b/kernel/src/mm/ucontext/mremap.rs index a9d1fc3981..12f46c73bb 100644 --- a/kernel/src/mm/ucontext/mremap.rs +++ b/kernel/src/mm/ucontext/mremap.rs @@ -385,6 +385,33 @@ impl InnerAddressSpace { self.mappings.insert_vma(new_vma.clone()); let move_len = core::cmp::min(source_len, new_len); + // The site identity includes its virtual address. Detach source sites + // before moving PTEs; after commit the outer AddressSpace owner reapplies + // matching definitions at the destination file offset. On rollback it + // reapplies the unchanged source mapping instead. + #[cfg(target_arch = "x86_64")] + if old_len != 0 { + if let Err(err) = super::uprobe::uprobe_disarm_range_locked(&mm, self, source_region) { + self.mappings.remove_vma(&new_region); + if let (Some(file), Some(VmaOpenRollback::Close)) = + (vm_file.as_ref(), target_vma_open_rollback) + { + notifications.vma.push(VmaCloseNotification { + file: file.clone(), + region: new_region, + vm_flags, + }); + } + if let Some(sysv_shm) = sysv_shm.as_ref() { + notifications.sysv.push(sysv_shm.clone()); + } + if let Some(lifecycle) = source_split_lifecycle.take() { + lifecycle.rollback_into(&mut notifications); + } + mremap_fail!(err); + } + } + // mremap does not free physical pages; old PTEs are migrated to the new VMA, while // old_len==0 keeps the legacy duplicate-mapping behavior. // using MmuGather here is solely for a unified cross-core TLB shootdown at the end. diff --git a/kernel/src/mm/ucontext/uprobe.rs b/kernel/src/mm/ucontext/uprobe.rs new file mode 100644 index 0000000000..9492f5c77e --- /dev/null +++ b/kernel/src/mm/ucontext/uprobe.rs @@ -0,0 +1,1920 @@ +//! Per-mm uprobe 管理 + XOL 区 + 断点页安装(计划步骤 3+4)。 +//! +//! 本模块提供 uprobe 注册/注销的基础设施,供 batch3(异常分发)与 batch4(perf 接入)调用。 +//! +//! # 关键设计(评审 findings) +//! +//! - **F8**:`uprobe_list` / `xol_area` / `uprobe_page_state` 挂在 `AddressSpace` 上,由 +//! **独立 irqsave `SpinLock`** 保护(**不**走 `inner: RwSem`),命中路径(#BP/#DB 关中断) +//! 仅 `lock_irqsave` + 查表,绝不睡眠。 +//! - **F1/F2**:断点页安装复刻 `do_wp_page` 私有文件 COW——`copy_page_as_normal` + patch +//! 0xcc + **单次** `set_entry` 原子帧替换(**绝不** unmap+map_phys 制造瞬时空 PTE)+ +//! `insert_vma`/`remove_vma` rmap 账簿 + `flush_tlb_range`。 +//! - **F7**:每目标 mm 私有 COW 副本(type `Normal`),**绝不**修改共享 page-cache 页 +//! (否则 writeback 回写 0xcc 损坏 .so)。 +//! - **F6 装弹顺序不变量**:注册时严格按 XOL slot 分配 → uprobe 表项插入 → 0xcc 页发布; +//! 0xcc 发布前任何路径查该 vaddr 必须能找到就绪 uprobe 表项。 +use crate::libs::{spinlock::SpinLock, wait_queue::WaitQueue}; +use alloc::{ + collections::BTreeMap, + sync::{Arc, Weak}, + vec::Vec, +}; +use core::sync::atomic::{AtomicBool, AtomicU64, AtomicU8, AtomicUsize, Ordering}; + +use system_error::SystemError; + +use crate::{ + arch::{mm::PageMapper, MMArch}, + filesystem::{page_cache::PageCache, vfs::IndexNode}, + libs::{mutex::Mutex, rwlock::RwLock}, + mm::{ + page::{page_manager_lock, Page, PageEntry}, + syscall::{MapFlags, ProtFlags}, + MemoryManagementArch, PhysAddr, VirtAddr, VirtRegion, VmFlags, + }, + process::ProcessControlBlock, +}; + +use super::RwSemWriteGuard; +use super::{AddressSpace, InnerAddressSpace, LockedVMA}; + +use uprobe::{ + analyze_insn, build_xol_slot, InsnAnalysis, ProbeArgs, UprobePoint, UPROBE_INSN_COPY_SIZE, +}; + +// ──────────────────────────── XOL 区 ──────────────────────────── + +/// 每个 slot 的宽度(= `UPROBE_INSN_COPY_SIZE` = 16 字节)。 +const XOL_SLOT_SIZE: usize = UPROBE_INSN_COPY_SIZE; + +/// 每页 slot 数量(4096 / 16 = 256)。 +const XOL_SLOTS_PER_PAGE: usize = MMArch::PAGE_SIZE / XOL_SLOT_SIZE; + +/// slot 位图需要的 u64 字数(256 bits → 4 words)。 +const XOL_BITMAP_WORDS: usize = XOL_SLOTS_PER_PAGE.div_ceil(64); + +/// Per-mm XOL(eXecute Out of Line)区。 +/// +/// 在用户地址空间映射一个可读可执行页,分成 16 字节对齐的 slot。每个 uprobe 分配一个 slot, +/// 命中时 batch3 在 slot 中写入原指令副本(RIP-relative 重定位后),rip 指向 slot 执行。 +/// +/// XOL 页在**注册时**(进程上下文、开中断)创建,**不能**在命中路径(关中断)创建。 +pub struct XolArea { + /// XOL 页在用户空间的基地址。 + page_base: VirtAddr, + /// XOL 页的物理地址(供 batch3 在关中断路径下通过 `phys_2_virt` 直接写 slot 内容, + /// 无需 mapper / RwSem)。 + page_paddr: PhysAddr, + /// 保证 XOL 物理页覆盖整个租约生命周期;不能只保存裸物理地址。 + _page: Arc, + /// 区域代次,用于阻止旧租约释放新区域的同号 slot。 + generation: u64, + /// slot 分配位图(bit=1 表示已占用)。 + slot_bitmap: SpinLock<[u64; XOL_BITMAP_WORDS]>, +} + +impl XolArea { + fn alloc_slot(self: &Arc) -> Option { + let mut bitmap = self.slot_bitmap.lock_irqsave(); + for (word_idx, word) in bitmap.iter_mut().enumerate() { + if *word != u64::MAX { + let bit = (!*word).trailing_zeros() as usize; + let slot = word_idx * 64 + bit; + if slot >= XOL_SLOTS_PER_PAGE { + break; + } + *word |= 1u64 << bit; + return Some(XolSlotLease { + area: self.clone(), + offset: slot * XOL_SLOT_SIZE, + generation: self.generation, + }); + } + } + None + } + + fn free_slot(&self, offset: usize, generation: u64) { + if generation != self.generation { + return; + } + let slot = offset / XOL_SLOT_SIZE; + if slot < XOL_SLOTS_PER_PAGE { + self.slot_bitmap.lock_irqsave()[slot / 64] &= !(1u64 << (slot % 64)); + } + } + + /// 计算 slot 对应的用户虚拟地址(供 batch3 使用)。 + pub fn slot_vaddr(&self, offset: usize) -> VirtAddr { + VirtAddr::new(self.page_base.data() + offset) + } + + /// XOL 页基地址(供 batch3 计算 slot 地址)。 + pub fn page_base(&self) -> VirtAddr { + self.page_base + } + + /// XOL 页物理地址(供 batch3 在关中断路径下通过 `phys_2_virt` 写 slot 内容)。 + pub fn page_paddr(&self) -> PhysAddr { + self.page_paddr + } +} + +/// 一个 XOL slot 的唯一所有权租约。命中路径应把 `Arc` 放入 +/// `ActiveXol`,从而让注销只撤销后续命中,不能复用仍在执行的 slot。 +pub struct XolSlotLease { + area: Arc, + offset: usize, + generation: u64, +} + +impl XolSlotLease { + pub fn offset(&self) -> usize { + self.offset + } + + pub fn slot_vaddr(&self) -> VirtAddr { + self.area.slot_vaddr(self.offset) + } + + pub fn page_paddr(&self) -> PhysAddr { + self.area.page_paddr() + } + + pub fn area(&self) -> &Arc { + &self.area + } +} + +impl Drop for XolSlotLease { + fn drop(&mut self) { + self.area.free_slot(self.offset, self.generation); + } +} + +impl core::fmt::Debug for XolSlotLease { + fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result { + f.debug_struct("XolSlotLease") + .field("offset", &self.offset) + .field("generation", &self.generation) + .finish_non_exhaustive() + } +} + +static NEXT_XOL_GENERATION: AtomicU64 = AtomicU64::new(1); + +/// Linux `struct uprobe` 的定义域:文件对象与文件偏移唯一确定探针,指令只分析一次。 +pub struct UprobeDefinition { + inode: Arc, + page_cache: Arc, + inode_id: usize, + inode_key: usize, + offset: usize, + old_instruction: [u8; UPROBE_INSN_COPY_SIZE], + analysis: InsnAnalysis, +} + +impl UprobeDefinition { + pub fn new(inode: Arc, offset: usize) -> Result, SystemError> { + let metadata = inode.metadata()?; + let file_size = usize::try_from(metadata.size).map_err(|_| SystemError::EINVAL)?; + if offset >= file_size { + return Err(SystemError::EINVAL); + } + let inode_id = metadata.inode_id.data(); + let page_cache = inode.page_cache().ok_or(SystemError::EINVAL)?; + // Mount wrappers and hardlink dentries may expose different IndexNode + // Arcs for the same underlying inode. The shared page cache is the + // canonical file-mapping identity used by every mmap/rmap path. + let inode_key = Arc::as_ptr(&page_cache) as usize; + { + let definitions = UPROBE_DEFINITIONS.lock_irqsave(); + if let Some(existing) = definitions + .get(&(inode_key, offset)) + .and_then(Weak::upgrade) + { + return Ok(existing); + } + } + + // Linux copies the definition instruction from the file mapping, not + // from a particular process's possibly private/COW mapping. This also + // allows a valid instruction to straddle a page or adjacent VMAs. + let available = (file_size - offset).min(UPROBE_INSN_COPY_SIZE); + let mut bytes = [0u8; UPROBE_INSN_COPY_SIZE]; + let read = page_cache.read(offset, &mut bytes[..available])?; + if read == 0 { + return Err(SystemError::EIO); + } + let analysis = analyze_insn(&bytes).map_err(|_| SystemError::EINVAL)?; + if analysis.insn_len > read { + return Err(SystemError::EINVAL); + } + let mut old_instruction = [0; UPROBE_INSN_COPY_SIZE]; + old_instruction[..analysis.insn_len].copy_from_slice(&bytes[..analysis.insn_len]); + + let definition = Arc::new(Self { + inode, + page_cache, + inode_id, + inode_key, + offset, + old_instruction, + analysis, + }); + let mut definitions = UPROBE_DEFINITIONS.lock_irqsave(); + if let Some(existing) = definitions + .get(&(inode_key, offset)) + .and_then(Weak::upgrade) + { + return Ok(existing); + } + definitions.insert((inode_key, offset), Arc::downgrade(&definition)); + Ok(definition) + } + + pub fn inode(&self) -> &Arc { + &self.inode + } + + fn matches_inode(&self, inode: &Arc) -> bool { + inode + .page_cache() + .is_some_and(|page_cache| Arc::ptr_eq(&page_cache, &self.page_cache)) + } + + pub fn inode_id(&self) -> usize { + self.inode_id + } + + pub fn offset(&self) -> usize { + self.offset + } + + fn instruction(&self) -> ([u8; UPROBE_INSN_COPY_SIZE], InsnAnalysis) { + (self.old_instruction, self.analysis) + } +} + +impl Drop for UprobeDefinition { + fn drop(&mut self) { + let key = (self.inode_key, self.offset); + let self_ptr = core::ptr::from_ref(self); + let mut definitions = UPROBE_DEFINITIONS.lock_irqsave(); + if definitions + .get(&key) + .is_some_and(|weak| core::ptr::eq(weak.as_ptr(), self_ptr)) + { + definitions.remove(&key); + } + } +} + +#[derive(Clone)] +pub struct UprobeTaskScope(Arc); + +/// The global weak reference keeps the PCB allocation from being reused while +/// a scope exists. The pointer cookie can therefore be compared on the hit +/// path without taking the global scope lock. +struct UprobeTaskScopeToken { + id: u64, + target_ptr: usize, +} + +impl Drop for UprobeTaskScopeToken { + fn drop(&mut self) { + UPROBE_TASK_SCOPES.lock_irqsave().remove(&self.id); + } +} + +impl UprobeTaskScope { + pub fn new(target: &Arc) -> Self { + let id = NEXT_TASK_SCOPE_ID.fetch_add(1, Ordering::Relaxed); + UPROBE_TASK_SCOPES + .lock_irqsave() + .insert(id, Arc::downgrade(target)); + Self(Arc::new(UprobeTaskScopeToken { + id, + target_ptr: Arc::as_ptr(target) as usize, + })) + } + + fn permits_task(&self, current: &Arc) -> bool { + Arc::as_ptr(current) as usize == self.0.target_ptr + } +} + +pub enum UprobeConsumerScope { + Task(UprobeTaskScope), + SystemWideAuthorized, +} + +impl UprobeConsumerScope { + fn permits(&self, mm: &Arc) -> bool { + match self { + Self::Task(target) => { + let target = { UPROBE_TASK_SCOPES.lock_irqsave().get(&target.0.id).cloned() }; + target + .and_then(|target| target.upgrade()) + .and_then(|task| task.basic().user_vm()) + .is_some_and(|target_mm| Arc::ptr_eq(&target_mm, mm)) + } + Self::SystemWideAuthorized => true, + } + } + + fn task_scope(&self) -> Option { + match self { + Self::Task(scope) => Some(scope.clone()), + Self::SystemWideAuthorized => None, + } + } +} + +pub struct UprobeConsumerRuntime { + pub pre_handler: fn(&dyn ProbeArgs), + pub post_handler: fn(&dyn ProbeArgs), + pub event_callback: Option>, + pub enabled: bool, +} + +#[derive(Clone)] +pub struct UprobeConsumerRuntimeSnapshot { + pub pre_handler: fn(&dyn ProbeArgs), + pub post_handler: fn(&dyn ProbeArgs), + pub event_callback: Option>, + task_scope: Option, +} + +impl UprobeConsumerRuntimeSnapshot { + pub fn permits_task(&self, current: &Arc) -> bool { + self.task_scope + .as_ref() + .is_none_or(|scope| scope.permits_task(current)) + } +} + +struct InstalledSiteRef { + mm: Weak, + vaddr: usize, + site: Weak, +} + +pub struct UprobeConsumer { + id: u64, + definition: Arc, + scope: UprobeConsumerScope, + runtime: RwLock, + enabled: AtomicBool, + lifecycle: Mutex<()>, + closing: AtomicBool, + inflight: AtomicUsize, + inflight_wait: WaitQueue, + sites: SpinLock>, +} + +struct ConsumerInstallGuard<'a>(&'a UprobeConsumer); + +impl Drop for ConsumerInstallGuard<'_> { + fn drop(&mut self) { + if self.0.inflight.fetch_sub(1, Ordering::Release) == 1 { + self.0.inflight_wait.wakeup_all(None); + } + } +} + +impl UprobeConsumer { + pub fn new( + id: u64, + definition: Arc, + scope: UprobeConsumerScope, + runtime: UprobeConsumerRuntime, + ) -> Arc { + let enabled = runtime.enabled; + Arc::new(Self { + id, + definition, + scope, + runtime: RwLock::new(runtime), + enabled: AtomicBool::new(enabled), + lifecycle: Mutex::new(()), + closing: AtomicBool::new(false), + inflight: AtomicUsize::new(0), + inflight_wait: WaitQueue::default(), + sites: SpinLock::new(Vec::new()), + }) + } + + fn begin_install(&self, mm: &Arc) -> Option> { + if !self.scope.permits(mm) + || self.closing.load(Ordering::Acquire) + || !self.enabled.load(Ordering::Acquire) + { + return None; + } + self.inflight.fetch_add(1, Ordering::AcqRel); + if self.closing.load(Ordering::Acquire) || !self.enabled.load(Ordering::Acquire) { + if self.inflight.fetch_sub(1, Ordering::Release) == 1 { + self.inflight_wait.wakeup_all(None); + } + return None; + } + Some(ConsumerInstallGuard(self)) + } + + fn remember_site(&self, mm: &Arc, vaddr: usize, site: &Arc) { + let mut sites = self.sites.lock_irqsave(); + sites.retain(|installed| { + installed.mm.upgrade().is_some() + && installed + .site + .upgrade() + .is_some_and(|site| site.state() != UprobeSiteState::Dead) + }); + sites.push(InstalledSiteRef { + mm: Arc::downgrade(mm), + vaddr, + site: Arc::downgrade(site), + }); + } + + /// Runtime state belongs to the perf consumer, not to one particular VMA + /// instance. Current and future sites therefore observe the same + /// enable/callback state. + pub fn runtime_snapshot(&self) -> Option { + let runtime = self.runtime.read(); + if self.closing.load(Ordering::Acquire) || !self.enabled.load(Ordering::Acquire) { + return None; + } + Some(UprobeConsumerRuntimeSnapshot { + pre_handler: runtime.pre_handler, + post_handler: runtime.post_handler, + event_callback: runtime.event_callback.clone(), + task_scope: self.scope.task_scope(), + }) + } +} + +#[repr(u8)] +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +pub enum UprobeSiteState { + Prepared = 0, + Armed = 1, + Disarming = 2, + Dead = 3, +} + +pub struct UprobeSite { + pub definition: Arc, + pub probe_vaddr: usize, + mapping: Weak, + mapping_state_seq: u64, + pub xol_lease: Arc, + /// Immutable hit-path snapshot. Writers rebuild it in process context; + /// #BP only clones the Arc and never allocates. + pub participants: RwLock>>, + state: AtomicU8, +} + +impl UprobeSite { + pub fn state(&self) -> UprobeSiteState { + match self.state.load(Ordering::Acquire) { + 0 => UprobeSiteState::Prepared, + 1 => UprobeSiteState::Armed, + 2 => UprobeSiteState::Disarming, + _ => UprobeSiteState::Dead, + } + } +} + +impl core::fmt::Debug for XolArea { + fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result { + f.debug_struct("XolArea") + .field("page_base", &self.page_base) + .finish_non_exhaustive() + } +} + +// ──────────────────────── per-mm uprobe 实例 ──────────────────────── + +/// Per-mm uprobe 实例:注册实体 + 指令分析结果。 +/// +/// 存储在 `AddressSpace::uprobe_list` 中,以 `probe_vaddr` 为键。同一地址可有多个实例 +/// (`Vec>>`,镜像 kprobe 的 `break_list`)。 +/// +/// batch3 命中路径用法: +/// ```ignore +/// let list = mm.uprobe_list.lock_irqsave(); +/// if let Some(entries) = list.get(&probe_vaddr) { +/// for entry in entries { +/// let inst = entry.read(); +/// if inst.basic.is_enabled() { +/// inst.basic.call_pre_handler(args); +/// inst.basic.call_event_callback(args); +/// // 取 xol_slot_offset、insn_analysis 做 XOL slot 填充 … +/// } +/// } +/// } +/// ``` +pub struct UprobeInstance { + pub point: Arc, + /// x86 指令静态分析(命中时供 `build_xol_slot` 用)。 + pub insn_analysis: InsnAnalysis, + /// 拥有此实例的消费者(perf event fd)id(评审 R9)。 + /// fork 继承的子实例沿用父实例的 id,使消费者 close 时一并注销。 + pub consumer_id: u64, + /// 保持 site 的 XOL slot,后续异常路径迁移后还会把它克隆进 ActiveXol。 + pub xol_lease: Arc, + pub site: Arc, + pub consumer: Arc, +} + +impl core::fmt::Debug for UprobeInstance { + fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result { + f.debug_struct("UprobeInstance") + .field("probe_vaddr", &self.point.probe_vaddr) + .field("insn_analysis", &self.insn_analysis) + .finish() + } +} +/// 某个页上已安装断点的状态标记。 +/// +/// 以页基地址(`probe_vaddr & !(PAGE_SIZE-1)`)为键。多个 uprobe 命中同一页时共享 +/// 一个 COW 副本,`refcount` 记录活跃断点数。注销在**当前映射页**上恢复字节、 +/// 不换页(评审 R8),故此处仅保留计数标记(供安装路径判定「页已私有化」)。 +pub(crate) struct UprobePageState { + /// 活跃断点数。 + refcount: usize, +} + +impl core::fmt::Debug for UprobePageState { + fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result { + f.debug_struct("UprobePageState") + .field("refcount", &self.refcount) + .finish_non_exhaustive() + } +} +// ──────────────────────── 注册句柄 ──────────────────────── + +/// 已注册 uprobe 的句柄。 +/// +/// Drop 时自动撤销尚未转交给 mm 命中表的安装。 +pub struct UprobeHandle { + mm: Weak, + probe_vaddr: usize, + entry: Option>>, +} + +impl UprobeHandle { + /// 把所有权转交给 consumer 的弱 site 索引;用于 mmap/fork 的持久安装。 + fn persist(mut self) { + self.entry.take(); + } +} + +impl Drop for UprobeHandle { + fn drop(&mut self) { + if let Some(entry) = self.entry.take() { + if let Some(mm) = self.mm.upgrade() { + uprobe_unregister_internal(&mm, self.probe_vaddr, &entry); + } + } + } +} + +struct ExpectedProbeVma { + vma: Weak, + state_seq: u64, +} + +// ──────────────────────── 公开 API ──────────────────────── + +/// # 注册一个 uprobe +/// +/// 在目标 mm 的 `probe_vaddr` 处安装 0xcc 断点。注册流程(装弹顺序 F6): +/// 1. 查 `uprobe_list` 是否已有同址条目:有则复用其 old_instruction + insn_analysis +/// (避免读到 COW 副本里的 0xcc);无则读原指令 + `analyze_insn` 校验; +/// 2. 分配 XOL slot(填 `xol_slot_offset`); +/// 3. 用真实 slot_vaddr 调 `build_xol_slot` 预填 slot + 校验 RIP-relative 位移 +/// (溢出→EINVAL fail-fast,绝不留下命中时 panic 的探针); +/// 4. 插入 `uprobe_list` 表项; +/// 5. 安装 0xcc(私有 COW,复刻 `do_wp_page`)。 +/// +/// ## 参数 +/// - `mm`:目标地址空间。 +/// - `probe_vaddr`:被探测的用户虚拟地址(必须在已映射的可执行 VMA 内且页已 present)。 +/// - `pre_handler`:#BP 命中前置处理器(batch3 提供,或用 [`noop_handler`] 占位)。 +/// - `post_handler`:XOL 单步完成后置处理器。 +/// +/// ## 返回 +/// `Ok(UprobeHandle)` 或错误码(`EINVAL`=地址非法/指令不支持,`EFAULT`=页未映射, +/// `ENOMEM`=内存不足/XOL 区满,`EACCES`=VMA 不可执行)。 +/// +fn uprobe_register( + mm: &Arc, + probe_vaddr: usize, + _pre_handler: fn(&dyn ProbeArgs), + _post_handler: fn(&dyn ProbeArgs), + consumer_id: u64, + expected_vma: &ExpectedProbeVma, +) -> Result, SystemError> { + // registry 锁只用于取得稳定 Arc;后续 mm 锁内绝不再访问 registry。 + let consumer = registry_consumer(consumer_id).ok_or(SystemError::ENOENT)?; + let _install = consumer.begin_install(mm).ok_or(SystemError::ENOENT)?; + // ── 持有 inner.write() 整个注册过程 ── + let mut inner = mm.write(); + + // ── Step 1: 定位 VMA + 读原指令 + 分析 ── + let vaddr = VirtAddr::new(probe_vaddr); + let page_base_addr = probe_vaddr & !(MMArch::PAGE_SIZE - 1); + let page_offset = probe_vaddr & (MMArch::PAGE_SIZE - 1); + + // VMA 必须存在且可执行 + let Some(vma) = inner.mappings.contains(vaddr) else { + return Ok(None); + }; + let Some(expected) = expected_vma.vma.upgrade() else { + return Ok(None); + }; + if !Arc::ptr_eq(&vma, &expected) || vma.state_seq() != expected_vma.state_seq { + return Ok(None); + } + { + let vma_guard = vma.lock(); + let vm_flags = *vma_guard.vm_flags(); + let valid_mask = + VmFlags::VM_HUGETLB | VmFlags::VM_MAYEXEC | VmFlags::VM_MAYSHARE | VmFlags::VM_WRITE; + if (vm_flags & valid_mask) != VmFlags::VM_MAYEXEC { + return Ok(None); + } + let file = vma_guard.vm_file().ok_or(SystemError::EINVAL)?; + let inode = file.inode(); + let pgoff = vma_guard.backing_page_offset().ok_or(SystemError::EINVAL)?; + let mapped_offset = pgoff + .checked_mul(MMArch::PAGE_SIZE) + .and_then(|base| base.checked_add(probe_vaddr - vma_guard.region().start().data())) + .ok_or(SystemError::EINVAL)?; + if !consumer.definition.matches_inode(&inode) + || mapped_offset != consumer.definition.offset() + { + return Ok(None); + } + } + + // ── P1:重复注册同一 probe_vaddr 时复用已有指令信息(避免读到 0xcc)── + // 第二个 consumer 注册同一地址时 PTE 已指向含 0xcc 的 COW 副本, + // read_user_insn_bytes 会把 0xcc 当原指令。故先查 uprobe_list:若有同址条目, + // 复用其 old_instruction + insn_analysis(二者对所有同址实例一致),跳过读取。 + let reused = { + let list = mm.uprobe_list.lock_irqsave(); + list.get(&probe_vaddr) + .and_then(|entries| entries.first()) + .map(|entry| { + let inst = entry.read(); + ( + inst.point.old_instruction, + inst.insn_analysis, + inst.site.clone(), + ) + }) + }; + + let (old_instruction, analysis, existing_site) = if let Some((oi, an, site)) = reused { + if !Arc::ptr_eq(&site.definition, &consumer.definition) { + return Err(SystemError::EINVAL); + } + (oi, an, Some(site)) + } else { + // The definition comes from the file. As Linux verify_opcode() does, + // verify the target mapping still has the opcode we are about to + // replace so a private/COW modification is never overwritten. + let opcode = read_user_opcode(&inner.user_mapper.utable, probe_vaddr)?; + let (old_instruction, analysis) = consumer.definition.instruction(); + if opcode != old_instruction[0] { + return Err(SystemError::EINVAL); + } + (old_instruction, analysis, None) + }; + + // ── Step 2: 确保 XOL 区存在 + 分配 slot ── + let xol_lease = if let Some(site) = existing_site.as_ref() { + site.xol_lease.clone() + } else { + ensure_xol_and_alloc_slot(mm, &mut inner)? + }; + let xol_slot_offset = xol_lease.offset(); + + // ── P2:注册时预填 XOL slot + 验证 RIP-relative 位移(fail-fast)── + // slot_vaddr = xol_page_base + slot_offset 此时已知,立即用真实地址调 + // build_xol_slot:位移溢出→EINVAL(注册失败,不引入会在命中时 panic 的探针); + // 成功→slot 内容写入物理页,命中时(#BP handler)slot 已就绪、直接 rip→slot。 + if existing_site.is_none() { + let (slot_vaddr, page_paddr) = { (xol_lease.slot_vaddr(), xol_lease.page_paddr()) }; + + let mut slot_buf = [0u8; UPROBE_INSN_COPY_SIZE]; + if let Err(e) = build_xol_slot( + &analysis, + probe_vaddr, + slot_vaddr.data(), + &old_instruction, + &mut slot_buf, + ) { + log::warn!( + "uprobe_register: build_xol_slot failed at {:#x} (slot {:#x}): {:?}", + probe_vaddr, + slot_vaddr.data(), + e + ); + return Err(SystemError::EINVAL); + } + + // 写入 XOL slot 物理页(复刻 batch3 fill_xol_slot / patch_byte_in_phys 写法)。 + let kva = unsafe { MMArch::phys_2_virt(page_paddr) }.ok_or(SystemError::EFAULT)?; + unsafe { + let dst = (kva.data() + xol_slot_offset) as *mut u8; + core::ptr::copy_nonoverlapping(slot_buf.as_ptr(), dst, UPROBE_INSN_COPY_SIZE); + } + } + + // ── Step 3: 创建 uprobe 实体 ── + let mut point = UprobePoint::new(probe_vaddr); + point.old_instruction = old_instruction; + point.insn_len = analysis.insn_len; + point.xol_slot_offset = xol_slot_offset; + + let point = Arc::new(point); + + let site = existing_site.unwrap_or_else(|| { + Arc::new(UprobeSite { + definition: consumer.definition.clone(), + probe_vaddr, + mapping: Arc::downgrade(&vma), + mapping_state_seq: vma.state_seq(), + xol_lease: xol_lease.clone(), + participants: RwLock::new(Arc::new(Vec::new())), + state: AtomicU8::new(UprobeSiteState::Prepared as u8), + }) + }); + let entry = Arc::new(RwLock::new(UprobeInstance { + point, + insn_analysis: analysis, + consumer_id, + xol_lease: xol_lease.clone(), + site: site.clone(), + consumer: consumer.clone(), + })); + + // ── Step 4: 插入 uprobe_list(表项在 0xcc 发布前就绪 — F6)── + { + let mut list = mm.uprobe_list.lock_irqsave(); + list.entry(probe_vaddr).or_default().push(entry.clone()); + } + // Publish the weak reverse index before building the runtime snapshot so + // a concurrent enable/disable/SET_BPF cannot miss this in-flight site. + consumer.remember_site(mm, probe_vaddr, &site); + rebuild_site_participants(mm, probe_vaddr, &site); + + // ── Step 5: 安装 0xcc 断点页 ── + let first_site = site.state() == UprobeSiteState::Prepared; + if first_site { + // hit table 已发布后才允许暴露 0xcc;Armed 在 PTE 提交之前发布。 + site.state + .store(UprobeSiteState::Armed as u8, Ordering::Release); + } + let install_result = if first_site { + install_breakpoint_page(mm, &mut inner, &vma, page_base_addr, page_offset) + } else { + Ok(()) + }; + if let Err(e) = install_result { + // 回滚:移除表项 + 释放 slot + { + let mut list = mm.uprobe_list.lock_irqsave(); + if let Some(entries) = list.get_mut(&probe_vaddr) { + entries.retain(|x| !Arc::ptr_eq(x, &entry)); + } + } + site.state + .store(UprobeSiteState::Dead as u8, Ordering::Release); + return Err(e); + } + + drop(inner); + Ok(Some(UprobeHandle { + mm: Arc::downgrade(mm), + probe_vaddr, + entry: Some(entry), + })) +} + +// ──────────────────────── 内部实现 ──────────────────────── + +/// 从目标 mm 的页表读取 probe_vaddr 处的指令字节(最多 16 字节)。 +/// +/// `PageMapper::translate` 直接 walk 物理页表,不需要目标 mm 的 CR3 上下文, +/// 因此可跨进程读取。若页未 present 返回 `EFAULT`。 +fn read_user_opcode(mapper: &PageMapper, probe_vaddr: usize) -> Result { + let page_offset = probe_vaddr & (MMArch::PAGE_SIZE - 1); + let (paddr, _flags) = mapper + .translate(VirtAddr::new(probe_vaddr)) + .ok_or(SystemError::EFAULT)?; + let kva = unsafe { MMArch::phys_2_virt(paddr) }.ok_or(SystemError::EFAULT)?; + Ok(unsafe { *((kva.data() + page_offset) as *const u8) }) +} + +/// 确保 mm 有 XOL 区,并分配一个 slot,返回 slot 在页内偏移。 +fn ensure_xol_and_alloc_slot( + mm: &Arc, + inner: &mut RwSemWriteGuard<'_, InnerAddressSpace>, +) -> Result, SystemError> { + // 快速路径:XOL 已存在 → 直接分配 + { + let guard = mm.xol_area.lock_irqsave(); + if let Some(area) = guard.as_ref() { + return area.alloc_slot().map(Arc::new).ok_or(SystemError::ENOMEM); + } + } + + // 慢速路径:创建 XOL 页(匿名映射,R-X) + // map_anonymous 可能分配物理页(睡眠安全:此时未持有任何 SpinLock) + let prot = ProtFlags::PROT_READ | ProtFlags::PROT_EXEC; + let map_flags = MapFlags::MAP_PRIVATE | MapFlags::MAP_ANONYMOUS; + let page = inner.map_anonymous( + VirtAddr::new(0), // 让内核选择地址 + MMArch::PAGE_SIZE, + prot, + map_flags, + true, // round_to_min + true, // allocate_at_once(立即分配零页) + )?; + + // XOL is kernel-owned execution state, not an ordinary anonymous mapping. + // It must never be copied into a child mm or expanded/remapped as user data. + // VM_IO also makes MADV_DOFORK reject attempts to clear VM_DONTCOPY, matching + // the invariant that every mm owns exactly one independently managed XOL area. + let xol_vma = inner + .mappings + .contains(page.virt_address()) + .ok_or(SystemError::EFAULT)?; + { + let mut guard = xol_vma.lock(); + let special = + VmFlags::VM_DONTCOPY | VmFlags::VM_IO | VmFlags::VM_DONTEXPAND | VmFlags::VM_DONTDUMP; + let flags = *guard.vm_flags() | special; + guard.set_vm_flags(flags); + } + + // 获取 XOL 页物理地址(供 batch3 关中断路径写 slot 内容) + let page_paddr = inner + .user_mapper + .utable + .translate(page.virt_address()) + .map(|(pa, _)| pa) + .ok_or(SystemError::EFAULT)?; + + let owned_page = { + let mut pm = page_manager_lock(); + pm.get(&page_paddr).ok_or(SystemError::EFAULT)? + }; + let area = Arc::new(XolArea { + page_base: page.virt_address(), + page_paddr, + _page: owned_page, + generation: NEXT_XOL_GENERATION.fetch_add(1, Ordering::Relaxed), + slot_bitmap: SpinLock::new([0u64; XOL_BITMAP_WORDS]), + }); + let lease = area.alloc_slot().map(Arc::new).ok_or(SystemError::ENOMEM)?; + + let mut guard = mm.xol_area.lock_irqsave(); + if guard.is_none() { + *guard = Some(area); + } else { + // 因调用方持有 inner.write(),同 mm 的注册是串行的,此分支理论上不可达。 + // TODO: [stage2] unmap 冗余的 XOL 页避免泄漏 + return guard + .as_ref() + .unwrap() + .alloc_slot() + .map(Arc::new) + .ok_or(SystemError::ENOMEM); + } + Ok(lease) +} + +/// 安装 0xcc 断点页(复刻 do_wp_page 私有文件 COW)。 +/// +/// 若页已有断点(同一物理页上的另一个 uprobe),仅 patch 额外 0xcc 字节 + refcount++; +/// 否则 COW → patch → 单次 set_entry → rmap → flush_tlb_range。 +fn install_breakpoint_page( + mm: &Arc, + inner: &mut RwSemWriteGuard<'_, InnerAddressSpace>, + vma: &Arc, + page_base_addr: usize, + page_offset: usize, +) -> Result<(), SystemError> { + let address = VirtAddr::new(page_base_addr); + let end = VirtAddr::new(page_base_addr + MMArch::PAGE_SIZE); + + // ── 检查页是否已有断点(同页多 uprobe)── + let already_cowed = { + let pb = mm.uprobe_page_state.lock_irqsave(); + pb.contains_key(&page_base_addr) + }; + + if already_cowed { + // 页已私有化:在**当前映射页** patch 额外 0xcc 字节(translate 取实时 + // paddr——写缺页二次 COW 后仍是正确页),refcount++。 + let _pt_edit = mm.page_table_edit(); + let mapper = &mut inner.user_mapper.utable; + let (paddr, _) = mapper.translate(address).ok_or(SystemError::EFAULT)?; + let kva = unsafe { MMArch::phys_2_virt(paddr) }.ok_or(SystemError::EFAULT)?; + unsafe { + core::ptr::write_volatile((kva.data() + page_offset) as *mut u8, 0xcc); + } + let mut pb = mm.uprobe_page_state.lock_irqsave(); + if let Some(state) = pb.get_mut(&page_base_addr) { + state.refcount += 1; + } + return Ok(()); + } + + // ── 新 COW 断点页 ── + + // page_table_edit 锁(debug_assert IRQ 启用——注册在进程上下文) + let _pt_edit = mm.page_table_edit(); + let mapper = &mut inner.user_mapper.utable; + + // translate 取旧 paddr + flags + let (old_paddr, entry_flags) = mapper.translate(address).ok_or(SystemError::EFAULT)?; + + // 取旧 page(必须被 page_manager 追踪——File 页或 Normal 页) + let old_page = { + let mut pm = page_manager_lock(); + pm.get(&old_paddr).ok_or(SystemError::EFAULT)? + }; + + // COW:copy_page_as_normal → 私有 Normal 副本(type=Normal,不回写 page-cache — F7) + let new_page = { + let mut pm = page_manager_lock(); + pm.copy_page_as_normal(&old_paddr, mapper.allocator_mut()) + .map_err(|_| SystemError::ENOMEM)? + }; + + // patch 0xcc + patch_byte_in_phys(&new_page, page_offset, 0xcc)?; + + // 单次原子 set_entry(绝不制造瞬时空 PTE — F1/F2) + let table = mapper.get_table(address, 0).ok_or(SystemError::EFAULT)?; + let i = table.index_of(address).ok_or(SystemError::EFAULT)?; + unsafe { + table.set_entry(i, PageEntry::new(new_page.phys_address(), entry_flags)); + } + + // mm-aware TLB shootdown + mm.flush_tlb_range(address, end, MMArch::PAGE_SHIFT as u8, false); + + // rmap 账簿:attach 新副本,detach 旧页 + let vm_locked = vma.lock().vm_flags().contains(VmFlags::VM_LOCKED); + new_page.write().insert_vma(vma.clone(), vm_locked); + { + let mut old_guard = old_page.write(); + old_guard.remove_vma(vma.as_ref()); + } + InnerAddressSpace::remove_page_unevictable_if_unneeded(&old_page); + + // 记录页状态(页已私有化的标记) + let mut pb = mm.uprobe_page_state.lock_irqsave(); + pb.insert(page_base_addr, UprobePageState { refcount: 1 }); + + Ok(()) +} + +/// 在物理页的指定偏移写入一个字节(通过内核 direct-map)。 +fn patch_byte_in_phys(page: &Arc, offset: usize, byte: u8) -> Result<(), SystemError> { + let kva = unsafe { MMArch::phys_2_virt(page.phys_address()) }.ok_or(SystemError::EFAULT)?; + unsafe { + core::ptr::write_volatile((kva.data() + offset) as *mut u8, byte); + } + Ok(()) +} + +/// 注销内部实现(评审 R7/R8 重做)。 +/// +/// 顺序:移除表项 → **仅当该地址无剩余实例时**恢复断点字节 → 回收 slot → +/// 页级状态清理。 +fn uprobe_unregister_internal( + mm: &Arc, + probe_vaddr: usize, + entry: &Arc>, +) { + let (site, consumer_id) = { + let inst = entry.read(); + (inst.site.clone(), inst.consumer_id) + }; + uprobe_unregister_consumer_from_site(mm, probe_vaddr, &site, consumer_id); +} + +fn uprobe_unregister_consumer_from_site( + mm: &Arc, + probe_vaddr: usize, + site: &Arc, + consumer_id: u64, +) { + // Registration publishes memberships under mm.write -> uprobe_list. + // Use the same order so the atomic remove/last decision cannot be + // invalidated by a new same-address consumer before teardown commits. + let mut inner = mm.write(); + // Remove a non-last membership while holding the table lock. If this is + // the last membership, leave it published until the teardown owner has + // restored the opcode. Two concurrent removals can therefore never both + // decide they are non-last and strand an armed, ownerless breakpoint. + let (last_consumer, orig_first_byte) = { + let mut list = mm.uprobe_list.lock_irqsave(); + let Some(entries) = list.get_mut(&probe_vaddr) else { + return; + }; + let Some(remove_index) = entries.iter().position(|entry| { + let inst = entry.read(); + inst.consumer_id == consumer_id && Arc::ptr_eq(&inst.site, site) + }) else { + return; + }; + let last = entries + .iter() + .filter(|entry| Arc::ptr_eq(&entry.read().site, site)) + .count() + == 1; + if !last { + entries.remove(remove_index); + } + (last, site.definition.old_instruction[0]) + }; + + if !last_consumer { + rebuild_site_participants(mm, probe_vaddr, site); + return; + } + + let page_base_addr = probe_vaddr & !(MMArch::PAGE_SIZE - 1); + let page_offset = probe_vaddr & (MMArch::PAGE_SIZE - 1); + if last_consumer { + if site + .state + .compare_exchange( + UprobeSiteState::Armed as u8, + UprobeSiteState::Disarming as u8, + Ordering::AcqRel, + Ordering::Acquire, + ) + .is_err() + { + return; + } + *site.participants.write() = Arc::new(Vec::new()); + // Disarming 期间表项仍可命中;先恢复指令,再撤销 hit table。 + if site_mapping_still_matches(&inner, site) + && restore_breakpoint_byte(mm, &mut inner, page_base_addr, page_offset, orig_first_byte) + { + // The table remains visible while the original opcode is restored. + // A synchronous shootdown is also a CPU rendezvous: a CPU which + // already executed INT3 cannot acknowledge it until its irq-off + // #BP handler has acquired the table and its XOL execution guard. + mm.flush_tlb_range( + VirtAddr::new(page_base_addr), + VirtAddr::new(page_base_addr + MMArch::PAGE_SIZE), + MMArch::PAGE_SHIFT as u8, + false, + ); + } + } + { + let mut list = mm.uprobe_list.lock_irqsave(); + if let Some(entries) = list.get_mut(&probe_vaddr) { + entries.retain(|entry| { + let inst = entry.read(); + !(inst.consumer_id == consumer_id && Arc::ptr_eq(&inst.site, site)) + }); + if entries.is_empty() { + list.remove(&probe_vaddr); + } + } + } + if last_consumer { + site.state + .store(UprobeSiteState::Dead as u8, Ordering::Release); + let mut pb = mm.uprobe_page_state.lock_irqsave(); + if let Some(state) = pb.get_mut(&page_base_addr) { + state.refcount = state.refcount.saturating_sub(1); + if state.refcount == 0 { + pb.remove(&page_base_addr); + } + } + } +} + +fn rebuild_site_participants(mm: &AddressSpace, probe_vaddr: usize, site: &Arc) { + // Keep collection and publication in the same membership critical + // section. Otherwise an older refresh can publish after a newer + // SET_BPF/enable/disable/close refresh and resurrect stale callbacks. + let list = mm.uprobe_list.lock_irqsave(); + let participants = list + .get(&probe_vaddr) + .into_iter() + .flatten() + .filter_map(|entry| { + let instance = entry.read(); + Arc::ptr_eq(&instance.site, site) + .then(|| instance.consumer.runtime_snapshot()) + .flatten() + }) + .collect(); + // Publish while membership is still serialized, but defer destruction of + // the old callbacks until after the IRQ-off hit-table lock is released. + let old_participants = + core::mem::replace(&mut *site.participants.write(), Arc::new(participants)); + drop(list); + drop(old_participants); +} + +fn refresh_consumer_sites(consumer: &UprobeConsumer) { + let installed: Vec<_> = consumer + .sites + .lock_irqsave() + .iter() + .filter_map(|installed| { + Some(( + installed.mm.upgrade()?, + installed.vaddr, + installed.site.upgrade()?, + )) + }) + .collect(); + for (mm, vaddr, site) in installed { + if site.state() != UprobeSiteState::Dead { + rebuild_site_participants(&mm, vaddr, &site); + } + } +} + +/// 注销前按 Linux `register_for_each_vma()` 的方式在 mm 写锁下重验映射身份, +/// 防止 munmap 后同一虚址被无关文件复用时写坏新映射。 +fn site_mapping_still_matches(inner: &InnerAddressSpace, site: &UprobeSite) -> bool { + let Some(vma) = inner.mappings.contains(VirtAddr::new(site.probe_vaddr)) else { + return false; + }; + let Some(original_vma) = site.mapping.upgrade() else { + return false; + }; + if !Arc::ptr_eq(&vma, &original_vma) || vma.state_seq() != site.mapping_state_seq { + return false; + } + let guard = vma.lock(); + let Some(file) = guard.vm_file() else { + return false; + }; + let Some(pgoff) = guard.backing_page_offset() else { + return false; + }; + let Some(delta) = site.probe_vaddr.checked_sub(guard.region().start().data()) else { + return false; + }; + let Some(offset) = pgoff + .checked_mul(MMArch::PAGE_SIZE) + .and_then(|base| base.checked_add(delta)) + else { + return false; + }; + site.definition.matches_inode(&file.inode()) && offset == site.definition.offset() +} + +/// 在**当前映射页**上恢复断点原字节(评审 R8)。 +/// +/// 经 `translate` 取当前 paddr(可能是断点安装时的 COW 副本,也可能是程序 +/// 写缺页二次 COW 后的页),直接写回原首字节。不交换页映射——页上其他字节 +/// 的任何程序写入都保留。无 PTE 变更 → 无需 TLB flush(TLB 缓存翻译而非 +/// 内容;跨修改代码的串行化由 #BP 中断返回后的取指重取保证)。 +fn restore_breakpoint_byte( + mm: &Arc, + inner: &mut InnerAddressSpace, + page_base_addr: usize, + page_offset: usize, + orig_first_byte: u8, +) -> bool { + let _pt_edit = mm.page_table_edit(); + let mapper = &mut inner.user_mapper.utable; + if let Some((paddr, _)) = mapper.translate(VirtAddr::new(page_base_addr)) { + if let Some(kva) = unsafe { MMArch::phys_2_virt(paddr) } { + unsafe { + core::ptr::write_volatile((kva.data() + page_offset) as *mut u8, orig_first_byte); + } + return true; + } + } + // 页已被 munmap(translate 失败):无需恢复。 + false +} + +/// Remove every uprobe site whose instruction lies in `region` before a VMA/PTE +/// mutation commits. The caller owns `AddressSpace::write()`, so mapping +/// identity and the restored byte are checked against one stable VMA view. +/// +/// The XOL VMA is kernel-owned and immutable while the address space lives. +/// User VMA operations which overlap it are rejected before any probe byte or +/// mapping is changed. +pub(crate) fn uprobe_disarm_range_locked( + mm: &Arc, + inner: &mut InnerAddressSpace, + region: VirtRegion, +) -> Result<(), SystemError> { + let overlaps_xol = { + let area = mm.xol_area.lock_irqsave(); + area.as_ref().is_some_and(|area| { + let xol = VirtRegion::new(area.page_base(), MMArch::PAGE_SIZE); + xol.collide(®ion) + }) + }; + // This VM_IO|VM_DONTEXPAND mapping is an execution trampoline, not a + // user-remappable allocation. Rejecting overlap avoids both waiting under + // mm.write() and a rollback window in which all probes are withdrawn. + if overlaps_xol { + return Err(SystemError::EBUSY); + } + + let targets: Vec<(usize, Arc, u8)> = { + let list = mm.uprobe_list.lock_irqsave(); + list.range(region.start().data()..region.end().data()) + .filter_map(|(vaddr, entries)| { + let entry = entries.first()?.read(); + let old = entry.point.old_instruction[0]; + Some((*vaddr, entry.site.clone(), old)) + }) + .collect() + }; + + let mut disarmed = Vec::new(); + for (probe_vaddr, site, old_byte) in targets { + if site + .state + .compare_exchange( + UprobeSiteState::Armed as u8, + UprobeSiteState::Disarming as u8, + Ordering::AcqRel, + Ordering::Acquire, + ) + .is_err() + { + continue; + } + + // Keep the hit-table entry visible until the original byte is restored. + // A concurrent old #BP can therefore still find the site and execute its + // strongly held XOL lease. VMA withdrawal does not suppress callbacks + // for an instruction which already trapped before the mapping change. + if site_mapping_still_matches(inner, &site) { + let page_base = probe_vaddr & !(MMArch::PAGE_SIZE - 1); + let page_offset = probe_vaddr & (MMArch::PAGE_SIZE - 1); + restore_breakpoint_byte(mm, inner, page_base, page_offset, old_byte); + } + disarmed.push((probe_vaddr, site, old_byte)); + } + + if !disarmed.is_empty() { + // Besides invalidating translations, this synchronous shootdown is the + // grace point for CPUs which executed INT3 before the bytes above were + // restored. Such a CPU runs #BP with interrupts disabled and cannot + // acknowledge the IPI until it has acquired the hit-table entry and + // its strongly held XOL slot lease. + // The acknowledgement, not the flushed address coverage, provides + // the grace point. Flush one affected page here; the VMA operation + // performs its own range-wide TLB invalidation when it commits. + let rendezvous_page = disarmed[0].0 & !(MMArch::PAGE_SIZE - 1); + mm.flush_tlb_range( + VirtAddr::new(rendezvous_page), + VirtAddr::new(rendezvous_page + MMArch::PAGE_SIZE), + MMArch::PAGE_SHIFT as u8, + false, + ); + } + + for (probe_vaddr, site, _) in disarmed { + *site.participants.write() = Arc::new(Vec::new()); + { + let mut list = mm.uprobe_list.lock_irqsave(); + // Conditional removal prevents an old lifecycle delta from deleting + // a newer site installed at the same virtual address. + if list.get(&probe_vaddr).is_some_and(|entries| { + entries + .first() + .is_some_and(|entry| Arc::ptr_eq(&entry.read().site, &site)) + }) { + list.remove(&probe_vaddr); + } + } + site.state + .store(UprobeSiteState::Dead as u8, Ordering::Release); + let page_base = probe_vaddr & !(MMArch::PAGE_SIZE - 1); + let mut pages = mm.uprobe_page_state.lock_irqsave(); + if let Some(state) = pages.get_mut(&page_base) { + debug_assert!(state.refcount != 0, "uprobe page refcount underflow"); + if state.refcount <= 1 { + pages.remove(&page_base); + } else { + state.refcount -= 1; + } + } else { + debug_assert!(false, "armed uprobe site without page state"); + } + } + Ok(()) +} + +// ──────────────────────── 辅助:空操作 handler ──────────────────────── + +/// 空操作 handler(供 batch4 在仅需 event_callback 时占位)。 +pub fn noop_handler(_args: &dyn ProbeArgs) {} + +// ──────────────── 全局注册表与迟到应用(评审 R9) ──────────────── +// +// 注册的探针身份 = 文件 inode + 偏移(而非 open 时的映射快照)。新映射 +// (dlopen/mmap)、fork 产生新地址空间时,据此把已注册的探针**迟到安装** +// 到新的 mm;exec 换新 AddressSpace,实例表自然为空(探针不跨 exec)。 +// +// 消费者(perf event fd)close 时: +// 1. 从注册表移除该消费者(杜绝后续迟到安装); +// 2. drop 其「迟到句柄」(fork/mmap 路径安装的),复用 `UprobeHandle::Drop` +// 的逐 mm 注销(含评审 R7 的地址级字节恢复)。 +// 直接安装(open 时)的句柄仍由 `UprobePerfEvent::handles` 持有,drop 同理。 + +pub struct UprobeConsumerReg { + pub definition: Arc, + pub scope: UprobeConsumerScope, + pub pre_handler: fn(&dyn ProbeArgs), + pub post_handler: fn(&dyn ProbeArgs), + pub event_callback: Option>, + pub enabled: bool, +} + +/// 全局注册表:inode id → 文件偏移 → (消费者 id,回调)。 +/// 注册表值类型:某(inode, offset)上的消费者列表。 +type ConsumerList = Vec>; +/// 注册表类型:inode id → (文件偏移 → 消费者列表)。 +type RegistryMap = BTreeMap>; + +static UPROBE_REGISTRY: SpinLock = SpinLock::new(BTreeMap::new()); + +static NEXT_CONSUMER_ID: AtomicU64 = AtomicU64::new(1); +static ACTIVE_UPROBE_CONSUMERS: AtomicUsize = AtomicUsize::new(0); +static NEXT_TASK_SCOPE_ID: AtomicU64 = AtomicU64::new(1); +static UPROBE_TASK_SCOPES: SpinLock>> = + SpinLock::new(BTreeMap::new()); +static UPROBE_DEFINITIONS: SpinLock>> = + SpinLock::new(BTreeMap::new()); + +fn uprobe_registry_is_empty() -> bool { + ACTIVE_UPROBE_CONSUMERS.load(Ordering::Acquire) == 0 +} + +/// 分配新的消费者 id(每次 perf_event_open(uprobe) 一次)。 +pub fn uprobe_new_consumer_id() -> u64 { + NEXT_CONSUMER_ID.fetch_add(1, Ordering::Relaxed) +} + +/// 注册一个消费者探测点(inode + offset)。 +pub fn uprobe_registry_add( + inode_id: usize, + offset: usize, + consumer_id: u64, + reg: Arc, +) { + debug_assert_eq!(inode_id, reg.definition.inode_id()); + debug_assert_eq!(offset, reg.definition.offset()); + let consumer = UprobeConsumer::new( + consumer_id, + reg.definition.clone(), + match ®.scope { + UprobeConsumerScope::Task(task) => UprobeConsumerScope::Task(task.clone()), + UprobeConsumerScope::SystemWideAuthorized => UprobeConsumerScope::SystemWideAuthorized, + }, + UprobeConsumerRuntime { + pre_handler: reg.pre_handler, + post_handler: reg.post_handler, + event_callback: reg.event_callback.clone(), + enabled: reg.enabled, + }, + ); + uprobe_registry_add_consumer(consumer); +} + +pub fn uprobe_registry_add_consumer(consumer: Arc) { + if consumer.enabled.load(Ordering::Acquire) { + ACTIVE_UPROBE_CONSUMERS.fetch_add(1, Ordering::AcqRel); + } + let mut r = UPROBE_REGISTRY.lock_irqsave(); + r.entry(consumer.definition.inode_key) + .or_default() + .entry(consumer.definition.offset()) + .or_default() + .push(consumer); +} + +fn registry_consumer(consumer_id: u64) -> Option> { + let r = UPROBE_REGISTRY.lock_irqsave(); + r.values() + .flat_map(|offsets| offsets.values()) + .flatten() + .find(|consumer| consumer.id == consumer_id) + .cloned() +} + +/// 更新某消费者的 BPF 事件回调(PERF_EVENT_IOC_SET_BPF 时调用)。 +/// 迟到安装的实例据此取得与直接安装一致的回调。 +pub fn uprobe_registry_set_callback(consumer_id: u64, cb: Arc) { + // 不在 registry spinlock 内获取 consumer runtime 锁。 + if let Some(consumer) = registry_consumer(consumer_id) { + consumer.runtime.write().event_callback = Some(cb); + refresh_consumer_sites(&consumer); + } +} + +/// Update the single consumer-level enable state used by both already armed +/// and later-installed sites. +pub fn uprobe_registry_set_enabled(consumer_id: u64, enabled: bool) -> Result<(), SystemError> { + let consumer = registry_consumer(consumer_id).ok_or(SystemError::ENOENT)?; + let _lifecycle = consumer.lifecycle.lock(); + if consumer.closing.load(Ordering::Acquire) { + return Err(SystemError::ENOENT); + } + if consumer.enabled.load(Ordering::Acquire) == enabled { + return Ok(()); + } + if enabled { + consumer.runtime.write().enabled = true; + if !consumer.enabled.swap(true, Ordering::AcqRel) { + ACTIVE_UPROBE_CONSUMERS.fetch_add(1, Ordering::AcqRel); + } + if let Err(e) = apply_consumer_to_existing_mappings(&consumer) { + if consumer.enabled.swap(false, Ordering::AcqRel) { + ACTIVE_UPROBE_CONSUMERS.fetch_sub(1, Ordering::AcqRel); + } + consumer.runtime.write().enabled = false; + detach_consumer_sites(&consumer); + return Err(e); + } + } else { + if consumer.enabled.swap(false, Ordering::AcqRel) { + ACTIVE_UPROBE_CONSUMERS.fetch_sub(1, Ordering::AcqRel); + } + consumer.runtime.write().enabled = false; + consumer + .inflight_wait + .wait_until(|| (consumer.inflight.load(Ordering::Acquire) == 0).then_some(())); + detach_consumer_sites(&consumer); + } + Ok(()) +} + +fn detach_consumer_sites(consumer: &UprobeConsumer) { + let installed: Vec<_> = consumer + .sites + .lock_irqsave() + .iter() + .filter_map(|installed| { + Some(( + installed.mm.upgrade()?, + installed.vaddr, + installed.site.upgrade()?, + )) + }) + .collect(); + for (mm, vaddr, site) in installed { + uprobe_unregister_consumer_from_site(&mm, vaddr, &site, consumer.id); + } +} + +fn apply_consumer_to_existing_mappings(consumer: &Arc) -> Result<(), SystemError> { + let page_cache = consumer + .definition + .inode() + .page_cache() + .ok_or(SystemError::EINVAL)?; + for vma in page_cache.collect_file_vmas() { + let mapping = { + let guard = vma.lock(); + let Some(mm) = guard.address_space().and_then(|owner| owner.upgrade()) else { + continue; + }; + let Some(pgoff) = guard.backing_page_offset() else { + continue; + }; + let Some(file) = guard.vm_file() else { + continue; + }; + (mm, file, *guard.region(), pgoff) + }; + uprobe_apply_to_new_vma_inner( + &mapping.0, + &mapping.1, + mapping.2.start().data(), + mapping.2.size(), + mapping.3 << MMArch::PAGE_SHIFT, + true, + Some(consumer.id), + )?; + } + Ok(()) +} +/// 消费者关闭:移除注册表项 + drop 迟到句柄(逐 mm 注销)。 +pub fn uprobe_registry_remove_consumer(consumer_id: u64) { + let removed = { + let mut r = UPROBE_REGISTRY.lock_irqsave(); + let mut removed = None; + for (_, offsets) in r.iter_mut() { + for (_, consumers) in offsets.iter_mut() { + consumers.retain(|consumer| { + if consumer.id == consumer_id { + consumer.closing.store(true, Ordering::Release); + if consumer.enabled.swap(false, Ordering::AcqRel) { + ACTIVE_UPROBE_CONSUMERS.fetch_sub(1, Ordering::AcqRel); + } + removed = Some(consumer.clone()); + false + } else { + true + } + }); + } + offsets.retain(|_, consumers| !consumers.is_empty()); + } + r.retain(|_, offsets| !offsets.is_empty()); + removed + }; + let Some(consumer) = removed else { return }; + consumer + .inflight_wait + .wait_until(|| (consumer.inflight.load(Ordering::Acquire) == 0).then_some(())); + let sites = core::mem::take(&mut *consumer.sites.lock_irqsave()); + for installed in sites { + if let (Some(mm), Some(site)) = (installed.mm.upgrade(), installed.site.upgrade()) { + uprobe_unregister_consumer_from_site(&mm, installed.vaddr, &site, consumer.id); + } + } +} + +/// 对新映射的文件 VMA 迟到应用注册表中的探针(评审 R9:dlopen / 后续 mmap)。 +/// +/// 在 mmap 提交且地址空间写锁释放后调用(本函数内部自取 `mm.write()`)。 +/// `region_start/size` 为 VMA 的用户地址区间;`file_start_byte` 为 VMA 起始 +/// 地址对应的文件偏移(= `backing_pgoff << PAGE_SHIFT`)。 +pub fn uprobe_apply_to_new_vma( + mm: &Arc, + file: &Arc, + region_start: usize, + region_size: usize, + file_start_byte: usize, +) { + let _ = uprobe_apply_to_new_vma_inner( + mm, + file, + region_start, + region_size, + file_start_byte, + false, + None, + ); +} + +/// Initial perf registration must report a real installation failure while +/// still allowing registration when there is no matching mapping yet. +pub fn uprobe_apply_to_existing_vma( + mm: &Arc, + file: &Arc, + region_start: usize, + region_size: usize, + file_start_byte: usize, + consumer_id: u64, +) -> Result<(), SystemError> { + uprobe_apply_to_new_vma_inner( + mm, + file, + region_start, + region_size, + file_start_byte, + true, + Some(consumer_id), + ) +} + +fn uprobe_apply_to_new_vma_inner( + mm: &Arc, + file: &Arc, + region_start: usize, + region_size: usize, + file_start_byte: usize, + strict: bool, + only_consumer_id: Option, +) -> Result<(), SystemError> { + let inode = file.inode(); + let inode_id = inode.metadata().map(|md| md.inode_id.data()).unwrap_or(0); + let Some(page_cache) = inode.page_cache() else { + return Ok(()); + }; + let inode_key = Arc::as_ptr(&page_cache) as usize; + let region_file_end = file_start_byte + .checked_add(region_size) + .ok_or(SystemError::EINVAL)?; + // 锁内快照:落在新 VMA 文件区间内的消费者列表 + let matches: Vec<(usize, ConsumerList)> = { + let r = UPROBE_REGISTRY.lock_irqsave(); + let Some(offsets) = r.get(&inode_key) else { + return Ok(()); + }; + offsets + .iter() + .filter(|(off, _)| **off >= file_start_byte && **off < region_file_end) + .filter_map(|(off, consumers)| { + let consumers = if let Some(id) = only_consumer_id { + consumers + .iter() + .filter(|consumer| consumer.id == id) + .cloned() + .collect() + } else { + consumers.clone() + }; + (!consumers.is_empty()).then_some((*off, consumers)) + }) + .collect() + }; + if matches.is_empty() { + return Ok(()); + } + + for (offset, consumers) in matches { + let probe_vaddr = region_start + .checked_add(offset - file_start_byte) + .ok_or(SystemError::EINVAL)?; + for consumer in consumers { + if !consumer.scope.permits(mm) || consumer.closing.load(Ordering::Acquire) { + continue; + } + let consumer_id = consumer.id; + // 该消费者在此 mm 的该地址是否已有实例(fork 继承可能已装)? + let already = { + let list = mm.uprobe_list.lock_irqsave(); + list.get(&probe_vaddr) + .is_some_and(|es| es.iter().any(|e| e.read().consumer_id == consumer_id)) + }; + if already { + continue; + } + // Ordinary file mmap is lazy. Fault in only the page containing + // the breakpoint, and pin the retry to the VMA identity observed + // here so MAP_FIXED cannot redirect installation to a replacement. + let expected_vma = { + let inner = mm.read(); + inner + .mappings + .contains(VirtAddr::new(probe_vaddr)) + .and_then(|vma| { + let flags = *vma.lock().vm_flags(); + let valid_mask = VmFlags::VM_HUGETLB + | VmFlags::VM_MAYEXEC + | VmFlags::VM_MAYSHARE + | VmFlags::VM_WRITE; + ((flags & valid_mask) == VmFlags::VM_MAYEXEC).then(|| ExpectedProbeVma { + vma: Arc::downgrade(&vma), + state_seq: vma.state_seq(), + }) + }) + }; + let Some(expected_vma) = expected_vma else { + // A non-executable/shared/writable alias is not an install + // failure. Linux valid_vma() skips it and keeps the consumer + // registered for a later eligible mapping. + continue; + }; + let page_base = VirtAddr::new(probe_vaddr & !(MMArch::PAGE_SIZE - 1)); + if let Err(e) = mm.populate_range_post_commit( + page_base, + MMArch::PAGE_SIZE, + true, + false, + Some(expected_vma.vma.clone()), + ) { + log::debug!( + "uprobe fault-in {:x}+{:#x} in new vma failed: {:?}", + inode_id, + offset, + e + ); + if strict { + return Err(e); + } + continue; + } + match uprobe_register( + mm, + probe_vaddr, + noop_handler, + noop_handler, + consumer_id, + &expected_vma, + ) { + Ok(Some(handle)) => { + handle.persist(); + } + Ok(None) => continue, + Err(e) => { + log::debug!( + "uprobe late-apply {:x}+{:#x} in new vma failed: {:?}", + inode_id, + offset, + e + ); + // ENOENT is a consumer closing concurrently and therefore a + // successful absence, not a fork transaction failure. + if strict && e != SystemError::ENOENT { + return Err(e); + } + } + } + } + } + Ok(()) +} + +/// Re-evaluate all file VMAs intersecting `region` after a committed VMA +/// operation. The VMA snapshot is owned and the address-space lock is dropped +/// before fault-in/registration, preserving the registry -> mm lock boundary. +fn collect_file_vma_snapshot( + mm: &Arc, + region: VirtRegion, +) -> Vec<(Arc, usize, usize, usize)> { + { + let inner = mm.read(); + inner + .mappings + .conflicts(region) + .into_iter() + .filter_map(|vma| { + let guard = vma.lock(); + let file = guard.vm_file()?; + let pgoff = guard.backing_page_offset()?; + let vma_region = *guard.region(); + Some(( + file, + vma_region.start().data(), + vma_region.size(), + pgoff.checked_mul(MMArch::PAGE_SIZE)?, + )) + }) + .collect() + } +} + +pub(crate) fn uprobe_apply_to_range(mm: &Arc, region: VirtRegion) { + if uprobe_registry_is_empty() { + mm.uprobe_needs_full_reapply.store(false, Ordering::Release); + return; + } + let full_reapply = mm.uprobe_needs_full_reapply.swap(false, Ordering::AcqRel); + let region = if full_reapply { + VirtRegion::new(VirtAddr::new(0), MMArch::USER_END_VADDR.data()) + } else { + region + }; + let mut retry_full = false; + for (file, start, size, offset) in collect_file_vma_snapshot(mm, region) { + if full_reapply { + if uprobe_apply_to_new_vma_inner(mm, &file, start, size, offset, true, None).is_err() { + retry_full = true; + } + } else { + uprobe_apply_to_new_vma(mm, &file, start, size, offset); + } + } + if retry_full { + mm.uprobe_needs_full_reapply.store(true, Ordering::Release); + } +} + +pub(crate) fn uprobe_apply_to_all_vmas(mm: &Arc) { + uprobe_apply_to_range( + mm, + VirtRegion::new(VirtAddr::new(0), MMArch::USER_END_VADDR.data()), + ); +} + +/// Reconcile only the source and destination touched by mremap. The first +/// range automatically widens to the whole mm when an XOL VMA was invalidated. +pub(crate) fn uprobe_apply_to_mremap_ranges( + mm: &Arc, + old_vaddr: VirtAddr, + old_len: usize, + new_vaddr: VirtAddr, + new_len: usize, +) { + if old_len != 0 { + uprobe_apply_to_range(mm, VirtRegion::new(old_vaddr, old_len)); + } + if new_len != 0 && (new_vaddr != old_vaddr || new_len != old_len) { + uprobe_apply_to_range(mm, VirtRegion::new(new_vaddr, new_len)); + } +} + +fn uprobe_apply_to_all_vmas_strict(mm: &Arc) -> Result<(), SystemError> { + let all = VirtRegion::new(VirtAddr::new(0), MMArch::USER_END_VADDR.data()); + for (file, start, size, offset) in collect_file_vma_snapshot(mm, all) { + uprobe_apply_to_new_vma_inner(mm, &file, start, size, offset, true, None)?; + } + Ok(()) +} + +/// fork 时把父 mm 的探针继承到子 mm(评审 R9)。 +/// +/// 在 clone 完成、父 mm 全部锁释放后调用;子 mm 尚无运行线程。 +/// 子页经 fork 已含父页的 0xcc(共享只读映射),此处将其私有化并重建 +/// per-mm 实例(slot/表项),沿用父实例的 consumer_id(消费者 close 一并注销)。 +pub fn fork_inherit_uprobes( + parent_mm: &Arc, + child_mm: &Arc, +) -> Result<(), SystemError> { + // The child initially shares the parent's private breakpoint pages through + // normal fork COW. Restore every inherited 0xcc in one private copy per + // physical page before the child can run. Afterwards the registry applies + // only consumers whose scope actually permits the child mm (system-wide in + // phase 1; task events have inherit=0). + let snapshot: BTreeMap> = { + let list = parent_mm.uprobe_list.lock_irqsave(); + let mut pages = BTreeMap::>::new(); + for (vaddr, entries) in list.iter() { + let Some(entry) = entries.first() else { + continue; + }; + let old_byte = { + let entry = entry.read(); + entry.point.old_instruction[0] + }; + pages + .entry(*vaddr & !(MMArch::PAGE_SIZE - 1)) + .or_default() + .push((*vaddr & (MMArch::PAGE_SIZE - 1), old_byte)); + } + pages + }; + if snapshot.is_empty() { + return Ok(()); + } + + let mut inner = child_mm.write(); + for (page_base, patches) in snapshot { + let address = VirtAddr::new(page_base); + let Some(vma) = inner.mappings.contains(address) else { + // A VM_DONTCOPY source VMA intentionally has no child mapping. + continue; + }; + let (old_paddr, entry_flags) = inner + .user_mapper + .utable + .translate(address) + .ok_or(SystemError::EFAULT)?; + let old_page = { + let mut pages = page_manager_lock(); + pages.get(&old_paddr).ok_or(SystemError::EFAULT)? + }; + let new_page = { + let mapper = &mut inner.user_mapper.utable; + let mut pages = page_manager_lock(); + pages + .copy_page_as_normal(&old_paddr, mapper.allocator_mut()) + .map_err(|_| SystemError::ENOMEM)? + }; + for (offset, byte) in patches { + patch_byte_in_phys(&new_page, offset, byte)?; + } + { + let _pt_edit = child_mm.page_table_edit(); + let mapper = &mut inner.user_mapper.utable; + let table = mapper.get_table(address, 0).ok_or(SystemError::EFAULT)?; + let index = table.index_of(address).ok_or(SystemError::EFAULT)?; + unsafe { + table.set_entry(index, PageEntry::new(new_page.phys_address(), entry_flags)); + } + } + let vm_locked = vma.lock().vm_flags().contains(VmFlags::VM_LOCKED); + new_page.write().insert_vma(vma.clone(), vm_locked); + old_page.write().remove_vma(vma.as_ref()); + InnerAddressSpace::remove_page_unevictable_if_unneeded(&old_page); + child_mm.flush_tlb_range( + address, + VirtAddr::new(page_base + MMArch::PAGE_SIZE), + MMArch::PAGE_SHIFT as u8, + false, + ); + } + drop(inner); + + uprobe_apply_to_all_vmas_strict(child_mm)?; + Ok(()) +} diff --git a/kernel/src/mm/ucontext/vma_ops.rs b/kernel/src/mm/ucontext/vma_ops.rs index 3da9f86280..a928348953 100644 --- a/kernel/src/mm/ucontext/vma_ops.rs +++ b/kernel/src/mm/ucontext/vma_ops.rs @@ -125,6 +125,17 @@ impl InnerAddressSpace { }); } + // Uprobe bytes belong to the old VMA identity. Restore and detach + // their sites while that identity and its PTEs are still present; + // MAP_FIXED and mremap target replacement both converge here. + #[cfg(target_arch = "x86_64")] + if let Err(err) = super::uprobe::uprobe_disarm_range_locked(&mm, self, region_to_unmap) { + for plan in plans { + plan.split_lifecycle.rollback_into(&mut notifications); + } + return Err(VmaOpFailure { err, notifications }); + } + plans.reverse(); while let Some(plan) = plans.pop() { let cur_vma = match self.mappings.remove_vma(&plan.original_region) { @@ -374,7 +385,6 @@ impl InnerAddressSpace { let mm = self.outer_addr_space().ok_or(SystemError::EFAULT)?; let mut tlb = MmuGather::gather(&mm); - let mapper = &mut self.user_mapper.utable; let region = VirtRegion::new(start_page.virt_address(), page_count.bytes()); // debug!("mprotect: region: {:?}", region); @@ -444,6 +454,22 @@ impl InnerAddressSpace { }); } + if !plans.is_empty() { + #[cfg(target_arch = "x86_64")] + if let Err(err) = super::uprobe::uprobe_disarm_range_locked(&mm, self, region) { + for plan in plans { + plan.split_lifecycle + .rollback_into(&mut rollback_notifications); + } + return Err(VmaOpFailure { + err, + notifications: rollback_notifications, + }); + } + } + + let mapper = &mut self.user_mapper.utable; + for plan in plans { let r = match self.mappings.remove_vma(&plan.original_region) { Some(vma) => vma, @@ -881,8 +907,6 @@ impl InnerAddressSpace { let mm = self.outer_addr_space().ok_or(SystemError::EFAULT)?; let mut tlb = MmuGather::gather(&mm); - let mapper = &mut self.user_mapper.utable; - let region = VirtRegion::new(start_page.virt_address(), page_count.bytes()); let (regions, has_unmapped) = self.mappings.conflicts_with_unmapped(region); @@ -901,6 +925,9 @@ impl InnerAddressSpace { }; } + let drops_present_pages = + behavior == MadvFlags::MADV_DONTNEED || behavior == MadvFlags::MADV_DONTNEED_LOCKED; + if Self::madvise_uses_range_without_vma_split(behavior) { for r in regions { let (original_region, vm_flags) = { @@ -909,7 +936,6 @@ impl InnerAddressSpace { }; let intersection = original_region.intersect(®ion).unwrap(); - let _pt_edit = mm.page_table_edit(); match behavior { MadvFlags::MADV_DONTNEED | MadvFlags::MADV_DONTNEED_LOCKED => { if vm_flags.contains(VmFlags::VM_PFNMAP) @@ -919,9 +945,24 @@ impl InnerAddressSpace { tlb.finish(); return Err(SystemError::EINVAL.into()); } + #[cfg(target_arch = "x86_64")] + if drops_present_pages { + if let Err(err) = + super::uprobe::uprobe_disarm_range_locked(&mm, self, intersection) + { + tlb.finish(); + return Err(err.into()); + } + } + let _pt_edit = mm.page_table_edit(); + let mapper = &mut self.user_mapper.utable; r.unmap_range(intersection, mapper, &mut tlb, UnmapMappingMode::EvenCow); } - _ => r.do_madvise(behavior, mapper, &mut tlb), + _ => { + let _pt_edit = mm.page_table_edit(); + let mapper = &mut self.user_mapper.utable; + r.do_madvise(behavior, mapper, &mut tlb) + } } } tlb.finish(); @@ -932,6 +973,7 @@ impl InnerAddressSpace { }; } + let mapper = &mut self.user_mapper.utable; let mut plans: Vec = Vec::with_capacity(regions.len()); let mut rollback_notifications = VmaCloseNotifications::default(); for r in ®ions { diff --git a/kernel/src/perf/kprobe.rs b/kernel/src/perf/kprobe.rs index 6697a514e5..9971671a2d 100644 --- a/kernel/src/perf/kprobe.rs +++ b/kernel/src/perf/kprobe.rs @@ -154,7 +154,7 @@ impl PerfEventOps for KprobePerfEvent { } fn readable(&self) -> bool { - true + false } } diff --git a/kernel/src/perf/mod.rs b/kernel/src/perf/mod.rs index 24d41201b5..7a08fc5eae 100644 --- a/kernel/src/perf/mod.rs +++ b/kernel/src/perf/mod.rs @@ -2,8 +2,12 @@ mod bpf; mod kprobe; mod sys_perf_event_open; mod tracepoint; +#[cfg(target_arch = "x86_64")] +mod uprobe; mod util; +pub(crate) use util::{PERF_TYPE_KPROBE, PERF_TYPE_UPROBE}; + use crate::arch::MMArch; use crate::bpf::prog::BpfProg; use crate::filesystem::epoll::event_poll::EPollItemList; @@ -221,7 +225,7 @@ impl IndexNode for PerfEventInode { _buf: &mut [u8], _data: MutexGuard, ) -> Result { - panic!("read_at not implemented for PerfEvent"); + Err(SystemError::EOPNOTSUPP_OR_ENOTSUP) } fn write_at( @@ -231,7 +235,7 @@ impl IndexNode for PerfEventInode { _buf: &[u8], _data: MutexGuard, ) -> Result { - panic!("write_at not implemented for PerfEvent"); + Err(SystemError::EINVAL) } fn metadata(&self) -> Result { @@ -385,9 +389,17 @@ pub fn perf_event_open( pid: i32, cpu: i32, group_fd: i32, - flags: u32, + flags: usize, ) -> Result { let args = PerfProbeArgs::try_from(attr, pid, cpu, group_fd, flags)?; + if args.type_ == PERF_TYPE_KPROBE || args.type_ == PERF_TYPE_UPROBE { + let unsupported = PerfEventOpenFlags::PERF_FLAG_FD_NO_GROUP + | PerfEventOpenFlags::PERF_FLAG_FD_OUTPUT + | PerfEventOpenFlags::PERF_FLAG_PID_CGROUP; + if args.group_fd != -1 || args.flags.intersects(unsupported) { + return Err(SystemError::EOPNOTSUPP_OR_ENOTSUP); + } + } log::info!("perf_event_process: {:#?}", args); let file_mode = if args .flags @@ -400,13 +412,24 @@ pub fn perf_event_open( let cloexec = file_mode.contains(FileFlags::O_CLOEXEC); let event: Box = match args.type_ { - // Kprobe - // See /sys/bus/event_source/devices/kprobe/type - perf_type_id::PERF_TYPE_MAX => { + // Dynamic software PMUs are routed solely by their sysfs-advertised + // type. Probe names and paths are data, never dispatch metadata. + PERF_TYPE_KPROBE => { let kprobe_event = kprobe::perf_event_open_kprobe(args); Box::new(kprobe_event) } - perf_type_id::PERF_TYPE_SOFTWARE => { + PERF_TYPE_UPROBE => { + #[cfg(target_arch = "x86_64")] + { + let uprobe_event = uprobe::perf_event_open_uprobe(args)?; + Box::new(uprobe_event) + } + #[cfg(not(target_arch = "x86_64"))] + { + return Err(SystemError::ENOSYS); + } + } + ty if ty == perf_type_id::PERF_TYPE_SOFTWARE as u32 => { // For bpf prog output assert_eq!( args.config, @@ -419,13 +442,11 @@ pub fn perf_event_open( let bpf_event = bpf::perf_event_open_bpf(args); Box::new(bpf_event) } - perf_type_id::PERF_TYPE_TRACEPOINT => { + ty if ty == perf_type_id::PERF_TYPE_TRACEPOINT as u32 => { let tracepoint_event = tracepoint::perf_event_open_tracepoint(args)?; Box::new(tracepoint_event) } - _ => { - unimplemented!("perf_event_process: unknown type: {:?}", args); - } + _ => return Err(SystemError::ENOENT), }; let page_cache = event.page_cache(); diff --git a/kernel/src/perf/sys_perf_event_open.rs b/kernel/src/perf/sys_perf_event_open.rs index 027ad95c5b..1e67f1ae6b 100644 --- a/kernel/src/perf/sys_perf_event_open.rs +++ b/kernel/src/perf/sys_perf_event_open.rs @@ -1,12 +1,14 @@ -use crate::arch::interrupt::TrapFrame; use crate::arch::syscall::nr::SYS_PERF_EVENT_OPEN; +use crate::arch::{interrupt::TrapFrame, MMArch}; use crate::include::bindings::linux_bpf::perf_event_attr; +use crate::mm::MemoryManagementArch; use crate::perf::perf_event_open; use crate::syscall::table::FormattedSyscallParam; use crate::syscall::table::Syscall; -use crate::syscall::user_access::UserBufferReader; +use crate::syscall::user_access::{UserBufferReader, UserBufferWriter}; use alloc::string::ToString; use alloc::vec::Vec; +use core::cmp::min; use core::mem::size_of; use system_error::SystemError; @@ -16,6 +18,58 @@ use system_error::SystemError; /// performance event monitoring. pub struct SysPerfEventOpenHandle; +const PERF_ATTR_SIZE_VER0: usize = 64; + +fn report_supported_attr_size(attr: *const u8) -> Result<(), SystemError> { + let mut writer = UserBufferWriter::new(attr as *mut u8, 8, true)?; + writer.copy_to_user_protected(&(size_of::() as u32).to_ne_bytes(), 4)?; + Ok(()) +} + +/// Linux-compatible `perf_copy_attr`: short known versions are zero-extended, +/// while a longer userspace structure is accepted only when its unknown tail +/// is all zero. +fn copy_perf_event_attr(attr: *const u8) -> Result { + let header = UserBufferReader::new(attr, 8, true)?; + let mut size_bytes = [0u8; size_of::()]; + header.copy_from_user_protected(&mut size_bytes, 4)?; + let reported_size = u32::from_ne_bytes(size_bytes) as usize; + let user_size = if reported_size == 0 { + PERF_ATTR_SIZE_VER0 + } else { + reported_size + }; + + if !(PERF_ATTR_SIZE_VER0..=MMArch::PAGE_SIZE).contains(&user_size) { + report_supported_attr_size(attr)?; + return Err(SystemError::E2BIG); + } + + let reader = UserBufferReader::new(attr, user_size, true)?; + let mut kernel_attr: perf_event_attr = unsafe { core::mem::zeroed() }; + let local_size = size_of::(); + let copy_size = min(user_size, local_size); + let attr_bytes = unsafe { + core::slice::from_raw_parts_mut( + (&mut kernel_attr as *mut perf_event_attr).cast::(), + local_size, + ) + }; + reader.copy_from_user_protected(&mut attr_bytes[..copy_size], 0)?; + + if user_size > local_size { + let mut unknown_tail = alloc::vec![0u8; user_size - local_size]; + reader.copy_from_user_protected(&mut unknown_tail, local_size)?; + if unknown_tail.iter().any(|byte| *byte != 0) { + report_supported_attr_size(attr)?; + return Err(SystemError::E2BIG); + } + } + + kernel_attr.size = user_size as u32; + Ok(kernel_attr) +} + impl SysPerfEventOpenHandle { /// Extracts the attribute pointer from syscall arguments fn attr(args: &[usize]) -> *const u8 { @@ -38,8 +92,8 @@ impl SysPerfEventOpenHandle { } /// Extracts the flags from syscall arguments - fn flags(args: &[usize]) -> u32 { - args[4] as u32 + fn flags(args: &[usize]) -> usize { + args[4] } } @@ -72,12 +126,7 @@ impl Syscall for SysPerfEventOpenHandle { let group_fd = Self::group_fd(args); let flags = Self::flags(args); - let buf = UserBufferReader::new( - attr as *const perf_event_attr, - size_of::(), - true, - )?; - let attr = buf.buffer_protected(0)?.read_one::(0)?; + let attr = copy_perf_event_attr(attr)?; perf_event_open(&attr, pid, cpu, group_fd, flags) } diff --git a/kernel/src/perf/uprobe.rs b/kernel/src/perf/uprobe.rs new file mode 100644 index 0000000000..84a649f2a7 --- /dev/null +++ b/kernel/src/perf/uprobe.rs @@ -0,0 +1,367 @@ +//! perf_event_open 的 uprobe 分发实现(计划步骤 7 / batch4)。 +//! +//! 照 [`crate::perf::kprobe`] 的 `KprobePerfEvent` 结构,为用户态断点探针提供 +//! perf 接入:`perf_event_open` 按 event_source sysfs 公布的 uprobe PMU type +//! 分发到本模块;`config1`(name) 为路径,`config2`(offset) 为文件偏移。 +//! +//! - BPF 程序经 `PERF_EVENT_IOC_SET_BPF` → [`UprobePerfEvent::do_set_bpf_prog`] JIT 后 +//! 注入每个 per-mm 实例的 `event_callback`(评审 F10:复用 `BPF_PROG_TYPE_KPROBE`)。 +//! - 命中时由 batch3 的 `#BP` handler 调 `call_event_callback`,本模块的 +//! [`UprobePerfCallBack`] 保证 BPF 入口 `pt_regs.rip = break_address()`(原探针址, +//! 评审 F5,绝不暴露 XOL slot 地址)。 + +use super::Result; +use crate::arch::interrupt::TrapFrame; +use crate::arch::kprobe::KProbeContext; +use crate::bpf::helper::BPF_HELPER_FUN_SET; +use crate::bpf::prog::BpfProg; +use crate::filesystem::page_cache::PageCache; +use crate::filesystem::vfs::file::File; +use crate::filesystem::vfs::{ + fcntl::AtFlags, + utils::{user_resolved_path_at, ResolvedPath}, + FilePrivateData, FileSystem, IndexNode, VFS_MAX_FOLLOW_SYMLINK_TIMES, +}; +use crate::include::bindings::linux_bpf::bpf_prog_type; +use crate::libs::casting::DowncastArc; +use crate::libs::mutex::MutexGuard; +use crate::mm::ucontext::{ + noop_handler, uprobe_apply_to_existing_vma, uprobe_new_consumer_id, uprobe_registry_add, + uprobe_registry_remove_consumer, uprobe_registry_set_callback, uprobe_registry_set_enabled, + UprobeConsumerReg, UprobeConsumerScope, UprobeDefinition, UprobeTaskScope, +}; +use crate::mm::MemoryManagementArch; +use crate::perf::util::{PerfProbeArgs, PerfProbeConfig}; +use crate::perf::{BasicPerfEbpfCallBack, PerfEventOps}; +use crate::process::{ProcessManager, RawPid}; +use crate::smp::core::smp_get_processor_id; +use crate::smp::cpu::{smp_cpu_manager, ProcessorId}; +use alloc::boxed::Box; +use alloc::string::String; +use alloc::sync::Arc; +use alloc::vec::Vec; +use core::any::Any; +use core::mem::size_of; +use rbpf::EbpfVmRaw; +use system_error::SystemError; +use uprobe::{CallBackFunc, ProbeArgs}; + +/// 一次 `perf_event_open(uprobe)` 对应一个持久 consumer。per-mm site 由 +/// AddressSpace 命中表拥有,consumer 只保存弱索引用于 close 撤销。 +pub struct UprobePerfEvent { + _args: PerfProbeArgs, + // The mount owner must be released from perf-fd process context, never + // when an IRQ-side ActiveXol releases its last site reference. + _resolved_path: ResolvedPath, + /// 消费者 id(评审 R9):全局注册表与迟到句柄的归属键。 + consumer_id: u64, +} + +impl Drop for UprobePerfEvent { + /// 消费者关闭(fd 释放):从注册表移除(杜绝后续迟到安装)+ drop 迟到句柄 + /// (fork/mmap 路径安装的,逐 mm 注销)。直接安装的 `handles` 随本结构 + /// drop 自动注销(评审 R9)。 + fn drop(&mut self) { + uprobe_registry_remove_consumer(self.consumer_id); + } +} +impl core::fmt::Debug for UprobePerfEvent { + fn fmt(&self, f: &mut core::fmt::Formatter<'_>) -> core::fmt::Result { + f.debug_struct("UprobePerfEvent").finish_non_exhaustive() + } +} + +impl UprobePerfEvent { + /// JIT 编译 BPF 程序并注入到每个 per-mm 实例的 `event_callback`。 + /// + /// 同一个 `Arc` 共享给所有句柄(多 mm / 多映射共用一份 JIT + /// 产物),与 kprobe 的注入路径一致。 + pub fn do_set_bpf_prog(&self, prog_file: Arc) -> Result<()> { + let file = prog_file + .inode() + .downcast_arc::() + .ok_or(SystemError::EINVAL)?; + if file.prog_type() != bpf_prog_type::BPF_PROG_TYPE_KPROBE { + return Err(SystemError::EINVAL); + } + if file.is_sleepable() { + return Err(SystemError::EOPNOTSUPP_OR_ENOTSUP); + } + let prog_slice = file.insns(); + let prog_slice = + unsafe { core::slice::from_raw_parts(prog_slice.as_ptr(), prog_slice.len()) }; + let mut vm = EbpfVmRaw::new(Some(prog_slice)).map_err(|e| { + log::error!("create ebpf vm failed: {:?}", e); + SystemError::EINVAL + })?; + + for (id, f) in BPF_HELPER_FUN_SET.get() { + vm.register_helper(*id, *f) + .map_err(|_| SystemError::EINVAL)?; + } + + let callback: Arc; + + #[cfg(target_arch = "x86_64")] + { + use crate::perf::JITMem; + + log::info!("Using JIT compilation for BPF program on x86_64 architecture (uprobe)"); + let jit_mem = Box::new(JITMem::new()); + let jit_mem = Box::leak(jit_mem); + let jit_mem_addr = core::ptr::from_ref::(jit_mem) as usize; + vm.set_jit_exec_memory(jit_mem).unwrap(); + vm.jit_compile().unwrap(); + let basic_callback = BasicPerfEbpfCallBack::new(file, vm, jit_mem_addr); + callback = Arc::new(UprobePerfCallBack { + inner: basic_callback, + cpu: self._args.cpu, + }); + } + #[cfg(not(target_arch = "x86_64"))] + { + vm.register_allowed_memory(0..u64::MAX); + let basic_callback = BasicPerfEbpfCallBack::new(file, vm); + callback = Arc::new(UprobePerfCallBack { + inner: basic_callback, + cpu: self._args.cpu, + }); + } + + // callback 是 consumer 级单一事实源,现有与后续映射同时生效。 + uprobe_registry_set_callback(self.consumer_id, callback.clone()); + Ok(()) + } +} + +/// uprobe 的 eBPF 事件回调(镜像 kprobe 的 `KprobePerfCallBack`)。 +/// +/// **F5 不变量**:BPF 入口 `pt_regs.rip = break_address()`(原探针址)。即便 +/// batch3 传入的 TrapFrame.rip 仍是 int3 故障点(probe_vaddr+1),此处也强制把 +/// 暴露给 BPF 的 rip 改为原探针址,XOL slot 地址绝不外泄。 +pub struct UprobePerfCallBack { + inner: BasicPerfEbpfCallBack, + cpu: i32, +} + +impl CallBackFunc for UprobePerfCallBack { + fn call(&self, trap_frame: &dyn ProbeArgs) { + if self.cpu >= 0 && smp_get_processor_id().data() != self.cpu as u32 { + return; + } + // F5:BPF 看到的 rip 是原探针址(break_address),不是 XOL slot、也不是 rip+1。 + let probe_addr = trap_frame.break_address(); + let trap_frame = match trap_frame.as_any().downcast_ref::() { + Some(tf) => tf, + None => return, + }; + let mut pt_regs = KProbeContext::from(trap_frame); + pt_regs.rip = probe_addr as u64; + let probe_context = unsafe { + core::slice::from_raw_parts_mut( + &mut pt_regs as *mut KProbeContext as *mut u8, + size_of::(), + ) + }; + self.inner.call(probe_context); + } +} + +impl IndexNode for UprobePerfEvent { + fn read_at( + &self, + _offset: usize, + _len: usize, + _buf: &mut [u8], + _data: MutexGuard, + ) -> Result { + Err(SystemError::EOPNOTSUPP_OR_ENOTSUP) + } + + fn write_at( + &self, + _offset: usize, + _len: usize, + _buf: &[u8], + _data: MutexGuard, + ) -> Result { + Err(SystemError::EINVAL) + } + + fn fs(&self) -> Arc { + panic!("fs not implemented for PerfEvent"); + } + + fn as_any_ref(&self) -> &dyn Any { + self + } + + fn list(&self) -> Result> { + Err(SystemError::ENOSYS) + } + + fn page_cache(&self) -> Option> { + None + } + + fn absolute_path(&self) -> core::result::Result { + Ok(String::from("uprobe_perf_event")) + } +} + +impl PerfEventOps for UprobePerfEvent { + fn set_bpf_prog(&self, bpf_prog: Arc) -> Result<()> { + self.do_set_bpf_prog(bpf_prog) + } + fn enable(&self) -> Result<()> { + uprobe_registry_set_enabled(self.consumer_id, true) + } + fn disable(&self) -> Result<()> { + uprobe_registry_set_enabled(self.consumer_id, false) + } + + fn readable(&self) -> bool { + false + } +} + +/// 创建 uprobe perf event(照 `perf_event_open_kprobe`)。 +/// +/// - `config1`(name) = 二进制路径;`config2`(offset) = 文件偏移。 +/// - `pid >= 0`:仅目标进程的 mm(`pid == 0` = 当前进程);`pid == -1`:经 inode rmap +/// 遍历所有映射该文件的 mm(评审 B8)。 +pub fn perf_event_open_uprobe(args: PerfProbeArgs) -> Result { + // Linux perf accepts task events with cpu=-1 or a concrete CPU, and CPU + // events only with a concrete CPU. Values below -1 are never meaningful. + if args.pid < -1 + || args.cpu < -1 + || (args.pid == -1 && args.cpu == -1) + || (args.cpu >= 0 + && smp_cpu_manager() + .possible_cpus() + .get(ProcessorId::new(args.cpu as u32)) + != Some(true)) + { + return Err(SystemError::EINVAL); + } + // Linux 6.6 perf_uprobe_event_init() applies this PMU-wide gate before + // parsing or installing the probe. + if !crate::process::cred::capable(crate::process::cred::CAPFlags::CAP_SYS_ADMIN) { + return Err(SystemError::EACCES); + } + if args.inherit || args.enable_on_exec || args.remove_on_exec { + return Err(SystemError::EOPNOTSUPP_OR_ENOTSUP); + } + if args.config != PerfProbeConfig::Raw(0) { + return Err(SystemError::EOPNOTSUPP_OR_ENOTSUP); + } + + // Linux uprobe PMU ABI uses config1 exclusively for the pathname and + // config2 exclusively for the file offset. Do not reinterpret ':' in a + // valid filename as an out-of-band offset encoding. + let path = args.name.clone(); + let offset = usize::try_from(args.offset).map_err(|_| SystemError::EINVAL)?; + log::info!( + "create uprobe for path: {path}, offset: {:#x}, pid: {}", + offset, + args.pid + ); + + // path → inode → page_cache(inode rmap 入口) + let caller = ProcessManager::current_pcb(); + let (start, remaining) = user_resolved_path_at(&caller, AtFlags::AT_FDCWD.bits(), &path)?; + let resolved = start + .inode() + .lookup_follow_symlink_owned(&start, &remaining, VFS_MAX_FOLLOW_SYMLINK_TIMES, true) + .map_err(|e| { + log::warn!("uprobe: failed to look up path {path}: {:?}", e); + e + })?; + let inode = resolved.inode(); + let page_cache = inode.page_cache().ok_or_else(|| { + log::warn!("uprobe: target {path} has no page cache (not a regular mapped file)"); + SystemError::EINVAL + })?; + let definition = UprobeDefinition::new(inode.clone(), offset)?; + + // pid 语义(评审 R1):>=0 单 mm(需 ptrace 访问检查);==-1 全量(需特权); + // 其他负值非法(EINVAL)。 + let scope = if args.pid >= 0 { + let pcb = if args.pid == 0 { + // Do not round-trip through a raw pid: pid 0 denotes current even + // when the caller is nested in a PID namespace. + ProcessManager::current_pcb() + } else { + ProcessManager::find_task_by_vpid(RawPid::from(args.pid as usize)) + .ok_or(SystemError::ESRCH)? + }; + // The PMU-wide CAP_SYS_ADMIN gate mirrors Linux's privileged uprobe + // event_init path; do not layer the unrelated process_vm permission + // helper on top of it. + pcb.basic().user_vm().ok_or(SystemError::ESRCH)?; + UprobeConsumerScope::Task(UprobeTaskScope::new(&pcb)) + } else if args.pid == -1 { + // 系统级模式:向**所有**映射该文件的进程(含其他用户的)安装断点, + // PMU-wide CAP_SYS_ADMIN check above also covers system-wide events. + UprobeConsumerScope::SystemWideAuthorized + } else { + // pid < -1:Linux perf 语义不存在(-1 之外无系统级变体),EINVAL。 + return Err(SystemError::EINVAL); + }; + + // 消费者身份 + 注册表登记(评审 R9:fork/后续 mmap 迟到安装的依据)。 + let consumer_id = uprobe_new_consumer_id(); + let inode_id = definition.inode_id(); + uprobe_registry_add( + inode_id, + offset, + consumer_id, + Arc::new(UprobeConsumerReg { + definition, + scope, + pre_handler: noop_handler, + post_handler: noop_handler, + event_callback: None, + enabled: !args.disabled, + }), + ); + + // Existing mappings are a strict initial apply. Absence of a VMA is + // valid: the persistent inode+offset consumer will be installed by a later + // mmap/dlopen/exec hook. Installation failures cannot roll back an already + // valid perf event; they remain eligible for the next matching lifecycle. + if !args.disabled { + for vma in page_cache.collect_file_vmas() { + let mapping = { + let guard = vma.lock(); + let Some(mm) = guard.address_space().and_then(|owner| owner.upgrade()) else { + continue; + }; + let Some(pgoff) = guard.backing_page_offset() else { + continue; + }; + let Some(file) = guard.vm_file() else { + continue; + }; + (mm, file, *guard.region(), pgoff) + }; + if let Err(e) = uprobe_apply_to_existing_vma( + &mapping.0, + &mapping.1, + mapping.2.start().data(), + mapping.2.size(), + mapping.3 << crate::arch::MMArch::PAGE_SHIFT, + consumer_id, + ) { + uprobe_registry_remove_consumer(consumer_id); + return Err(e); + } + } + } + + Ok(UprobePerfEvent { + _args: args, + _resolved_path: resolved, + consumer_id, + }) +} diff --git a/kernel/src/perf/util.rs b/kernel/src/perf/util.rs index 7170fbd8d2..08d82a1aab 100644 --- a/kernel/src/perf/util.rs +++ b/kernel/src/perf/util.rs @@ -37,14 +37,27 @@ pub struct PerfProbeArgs { pub name: String, pub offset: u64, pub size: u32, - pub type_: perf_type_id, + /// Raw `perf_event_attr.type` value. Dynamic PMU types intentionally live + /// outside `perf_type_id`, whose `PERF_TYPE_MAX` member is not ABI. + pub type_: u32, pub pid: i32, pub cpu: i32, pub group_fd: i32, pub flags: PerfEventOpenFlags, pub sample_type: Option, + /// `perf_event_attr.disabled`:事件初始是否禁用(评审 R11a)。 + pub disabled: bool, + pub inherit: bool, + pub enable_on_exec: bool, + pub remove_on_exec: bool, } +/// DragonOS currently has no general PMU type allocator. Keep the two +/// software probe PMUs in the dynamic range and expose these exact values via +/// event_source sysfs. +pub const PERF_TYPE_KPROBE: u32 = perf_type_id::PERF_TYPE_MAX as u32; +pub const PERF_TYPE_UPROBE: u32 = PERF_TYPE_KPROBE + 1; + #[derive(Debug, Clone, Copy, PartialEq, Eq)] pub enum PerfProbeConfig { PerfSwIds(perf_sw_ids), @@ -57,17 +70,20 @@ impl PerfProbeArgs { pid: i32, cpu: i32, group_fd: i32, - flags: u32, + flags: usize, ) -> Result { - let ty = perf_type_id::from_u32(attr.type_).ok_or(SystemError::EINVAL)?; - let config = match ty { - perf_type_id::PERF_TYPE_TRACEPOINT => PerfProbeConfig::Raw(attr.config), - _ => { + if attr.__reserved_1() != 0 || attr.__reserved_2 != 0 || attr.__reserved_3 != 0 { + return Err(SystemError::EINVAL); + } + let ty = attr.type_; + let config = match perf_type_id::from_u32(ty) { + Some(perf_type_id::PERF_TYPE_SOFTWARE) => { let sw_id = perf_sw_ids::from_u32(attr.config as u32).ok_or(SystemError::EINVAL)?; PerfProbeConfig::PerfSwIds(sw_id) } + _ => PerfProbeConfig::Raw(attr.config), }; - let name = if ty == perf_type_id::PERF_TYPE_MAX { + let name = if ty == PERF_TYPE_KPROBE || ty == PERF_TYPE_UPROBE { let name_ptr = unsafe { attr.__bindgen_anon_3.config1 } as *const u8; let name = check_and_clone_cstr(name_ptr, None)?; name.into_string().map_err(|_| SystemError::EINVAL)? @@ -75,6 +91,7 @@ impl PerfProbeArgs { String::new() }; let sample_ty = perf_event_sample_format::from_u32(attr.sample_type as u32); + let raw_flags = u32::try_from(flags).map_err(|_| SystemError::EINVAL)?; let args = PerfProbeArgs { config, name, @@ -84,8 +101,12 @@ impl PerfProbeArgs { pid, cpu, group_fd, - flags: PerfEventOpenFlags::from_bits_truncate(flags), + flags: PerfEventOpenFlags::from_bits(raw_flags).ok_or(SystemError::EINVAL)?, sample_type: sample_ty, + disabled: attr.disabled() != 0, + inherit: attr.inherit() != 0, + enable_on_exec: attr.enable_on_exec() != 0, + remove_on_exec: attr.remove_on_exec() != 0, }; Ok(args) } diff --git a/kernel/src/process/exec.rs b/kernel/src/process/exec.rs index 8c24a94244..003409b69b 100644 --- a/kernel/src/process/exec.rs +++ b/kernel/src/process/exec.rs @@ -311,6 +311,12 @@ impl ExecParam { // TODO: Implement the remaining Linux logic. de_thread(&me).map_err(ExecError::SystemError)?; + // exec no longer returns to the old user context. Drop any in-flight + // XOL transaction before the old mm can be replaced so its site and + // slot lease cannot survive into the new image. + #[cfg(target_arch = "x86_64")] + crate::exception::uprobe::cleanup_task_active_xol(&me); + me.flags().remove(ProcessFlags::FORKNOEXEC); exec_task_namespaces().map_err(ExecError::SystemError)?; diff --git a/kernel/src/process/manager/exit.rs b/kernel/src/process/manager/exit.rs index 75b6096cc7..294b1207ff 100644 --- a/kernel/src/process/manager/exit.rs +++ b/kernel/src/process/manager/exit.rs @@ -378,6 +378,11 @@ impl ProcessManager { } } + // 退出不再返回旧用户态上下文;先释放 ActiveXol 对 site/slot/consumer + // 的强引用,再进入 mm teardown。该清理不需要也不应改 trapframe。 + #[cfg(target_arch = "x86_64")] + crate::exception::uprobe::cleanup_task_active_xol(¤t_pcb); + let pid: Arc; let raw_pid = current_pcb.raw_pid(); // log::debug!("[exit: {}]", raw_pid.data()); diff --git a/kernel/src/process/state.rs b/kernel/src/process/state.rs index 6aa4e36544..5ced4978ef 100644 --- a/kernel/src/process/state.rs +++ b/kernel/src/process/state.rs @@ -193,6 +193,12 @@ bitflags! { /// Process is waiting for an I/O operation to complete (used for iowait /// accounting). const IN_IOWAIT = 1 << 13; + /// uprobe XOL 单步窗口判别位(计划步骤 8,评审 F4)。 + /// + /// 由用户态 #BP handler 在重定向 rip 到 XOL slot 前置位;`do_debug` 检查并 + /// 清之以区分「XOL 单步完成的 #DB」与 ptrace/硬件断点 #DB。**仅作 #DB + /// 分发判别**,不参与 [`ProcessFlags::exit_to_user_mode_work`](非延迟工作)。 + const NEED_UPROBE = 1 << 14; /// PID links and visible-thread accounting have already been released. const PID_UNHASHED = 1 << 15; /// Task is currently traced by another task. diff --git a/kernel/src/process/task.rs b/kernel/src/process/task.rs index 6ad41a2094..aa7aa83ced 100644 --- a/kernel/src/process/task.rs +++ b/kernel/src/process/task.rs @@ -79,6 +79,13 @@ pub struct ProcessControlBlock { pub(super) rcu_read_depth: AtomicUsize, pub(super) flags: LockFreeFlags, + /// uprobe XOL 单步窗口的每线程活跃状态(仅 x86_64)。 + /// + /// #BP 重定向 rip 到 XOL slot 前保存(精确 slot_end、原始 TF、状态、site/slot + /// 强引用和命中 consumer 快照),#DB 完成或 abort 时取回。评审 R2/R3/R5: + /// 保存于执行线程而非 mm,使并发注销/信号投递不破坏恢复语义。 + #[cfg(target_arch = "x86_64")] + pub(crate) uprobe_ss: SpinLock>, /// Whether the current task has been counted in the global visible thread /// count. pub(super) visible_thread_accounted: AtomicBool, @@ -386,6 +393,8 @@ impl ProcessControlBlock { executable_path: RwLock::new(name), cmdline: RwLock::new(Vec::new()), rlimits: RwLock::new(Self::default_rlimits()), + #[cfg(target_arch = "x86_64")] + uprobe_ss: SpinLock::new(None), }; pcb.sig_info.write().set_tty(tty); diff --git a/user/apps/tests/dunitest/suites/normal/uprobe.cc b/user/apps/tests/dunitest/suites/normal/uprobe.cc new file mode 100644 index 0000000000..0d635e23aa --- /dev/null +++ b/user/apps/tests/dunitest/suites/normal/uprobe.cc @@ -0,0 +1,659 @@ +// uprobe 断点探针端到端测试(issue #2150 阶段一)。 +// +// 验证用户态经 perf_event_open 挂载 uprobe、触发被探测函数后进程存活 +// (#BP → XOL 单步 → #DB → 恢复 的命中路径不崩溃),并覆盖错误入参路径。 +// +// 内核侧接口(kernel/src/perf/uprobe.rs): +// - perf_event_attr.type 由 /sys/bus/event_source/devices/uprobe/type 提供 +// - perf_event_attr.config1 = 目标二进制路径 +// - perf_event_attr.config2 = 文件偏移 +// - syscall 参数 pid/cpu 决定 task 或 per-CPU scope + +#ifndef _GNU_SOURCE +#define _GNU_SOURCE +#endif + +#include + +#include +#include +#include +#include +#include +#include +#include +#include +#include + +#include +#include +#include +#include +#include + +#include + +#ifndef SYS_perf_event_open +#include +#define SYS_perf_event_open __NR_perf_event_open +#endif + +namespace { + +constexpr const char* UPROBE_TYPE_PATH = + "/sys/bus/event_source/devices/uprobe/type"; + +struct UprobePerfEventOptions { + pid_t pid = 0; + int cpu = -1; + bool disabled = false; + bool inherit = false; + bool enable_on_exec = false; + bool remove_on_exec = false; + __u64 config = 0; + int group_fd = -1; + unsigned long flags = 0; +}; + +class FdGuard { + public: + explicit FdGuard(int fd = -1) : fd_(fd) {} + ~FdGuard() { + if (fd_ >= 0) close(fd_); + } + + int get() const { return fd_; } + void close_now() { + if (fd_ >= 0) close(fd_); + fd_ = -1; + } + + private: + int fd_; +}; + +// 动态 PMU type 是用户态 ABI,测试不得依赖内核当前的分配顺序。 +bool read_uprobe_perf_type(__u32& type) { + std::ifstream type_file(UPROBE_TYPE_PATH); + unsigned long long parsed = 0; + if (!type_file.is_open()) { + errno = ENOENT; + return false; + } + if (!(type_file >> parsed) || + parsed > std::numeric_limits<__u32>::max()) { + errno = EINVAL; + return false; + } + type = static_cast<__u32>(parsed); + return true; +} + +// 一个简单的 noinline 目标函数,作为 uprobe 的挂载点。保持纯计算,避免 +// RIP-relative 操作数,降低 XOL 重定位的不确定性。 +__attribute__((noinline)) int uprobe_target(int x) { + asm volatile("" : "+r"(x) : : "memory"); // 防止内联/优化掉 + return x * 2 + 1; +} + +// 从 /proc/self/maps 解析 func 在所属可执行文件中的偏移。 +// 返回 false 表示解析失败(如 procfs 不支持该格式)。 +bool resolve_file_offset(const void* func, std::string& path, + unsigned long& offset) { + char exe_buf[4096]; + ssize_t n = readlink("/proc/self/exe", exe_buf, sizeof(exe_buf) - 1); + if (n <= 0) return false; + exe_buf[n] = '\0'; + path.assign(exe_buf); + + std::ifstream maps("/proc/self/maps"); + if (!maps.is_open()) return false; + + const auto func_addr = reinterpret_cast(func); + std::string line; + while (std::getline(maps, line)) { + unsigned long start = 0, end = 0, mapoff = 0; + char perms[8] = {}; + // 格式:start-end perms offset dev inode pathname + if (std::sscanf(line.c_str(), "%lx-%lx %7s %lx", &start, &end, perms, + &mapoff) != 4) + continue; + // 可执行段且 func 落在其中 + if (perms[0] == 'r' && perms[2] == 'x' && func_addr >= start && + func_addr < end) { + offset = mapoff + (func_addr - start); + return true; + } + } + return false; +} + +int open_uprobe_perf_event(const std::string& path, unsigned long offset, + const UprobePerfEventOptions& options = {}) { + __u32 type = 0; + if (!read_uprobe_perf_type(type)) return -1; + + struct perf_event_attr pe; + std::memset(&pe, 0, sizeof(pe)); + pe.size = sizeof(pe); + pe.type = type; + pe.config = options.config; + pe.config1 = reinterpret_cast<__u64>(path.c_str()); + pe.config2 = offset; + pe.disabled = options.disabled; + pe.inherit = options.inherit; + pe.enable_on_exec = options.enable_on_exec; + pe.remove_on_exec = options.remove_on_exec; + return static_cast(syscall(SYS_perf_event_open, &pe, options.pid, + options.cpu, options.group_fd, + options.flags)); +} + +constexpr unsigned char RAW_TARGET_CODE[] = { + 0x8d, 0x44, 0x3f, 0x01, // lea eax,[rdi+rdi+1] + 0xc3, // ret +}; + +int create_raw_code(char* path_template, const unsigned char* code, + size_t code_size) { + int fd = mkstemp(path_template); + if (fd < 0) return -1; + if (write(fd, code, code_size) != static_cast(code_size)) { + const int saved_errno = errno; + close(fd); + unlink(path_template); + errno = saved_errno; + return -1; + } + return fd; +} + +int create_raw_target(char* path_template) { + return create_raw_code(path_template, RAW_TARGET_CODE, + sizeof(RAW_TARGET_CODE)); +} + +} // namespace + +// 挂载 uprobe 到当前进程的目标函数,触发它,验证进程不崩溃且函数返回正确。 +// 这条用例是 uprobe 端到端的核心验证:#BP → XOL 单步 → #DB → 恢复。 +TEST(UprobeTest, RegisterAndTriggerSurvivesHit) { + std::string path; + unsigned long offset = 0; + ASSERT_TRUE(resolve_file_offset( + reinterpret_cast(&uprobe_target), path, offset)) + << "无法从 /proc/self/maps 解析目标函数偏移"; + + int fd = open_uprobe_perf_event(path, offset); + ASSERT_GE(fd, 0) << "perf_event_open(uprobe) 失败,errno=" << errno + << "(内核可能未启用 uprobe)"; + + ASSERT_GE(ioctl(fd, PERF_EVENT_IOC_ENABLE, 0), 0) + << "PERF_EVENT_IOC_ENABLE 失败,errno=" << errno; + + // 执行被探测函数:应命中 uprobe,经 XOL 单步原指令后正确返回。 + // 若 uprobe 命中路径有 bug,这里可能崩溃 / hang / SIGTRAP。 + volatile int result = uprobe_target(21); + EXPECT_EQ(result, 43); + + ioctl(fd, PERF_EVENT_IOC_DISABLE, 0); + close(fd); +} + +// 非法路径应被拒绝(返回负 errno)。 +TEST(UprobeTest, InvalidPathIsRejected) { + int fd = open_uprobe_perf_event("/nonexistent/path/to/binary", 0); + EXPECT_LT(fd, 0) << "非法路径不应成功挂载 uprobe"; + if (fd >= 0) close(fd); +} + +// 越界偏移应被拒绝。 +TEST(UprobeTest, InvalidOffsetIsRejected) { + std::string path; + unsigned long offset = 0; + if (!resolve_file_offset(reinterpret_cast(&uprobe_target), path, + offset)) { + GTEST_SKIP() << "无法解析目标函数偏移,跳过"; + } + int fd = open_uprobe_perf_event(path, 0xFFFFFFFFFFFFULL); + EXPECT_LT(fd, 0) << "越界偏移不应成功挂载"; + if (fd >= 0) close(fd); +} + +// 同一 uprobe 挂载后多次触发,验证 XOL 单步可重复且每次都正确返回。 +// 潜在 bug:XOL slot 内容被破坏、TF 未清导致单步循环、NEED_UPROBE 未清。 +TEST(UprobeTest, MultipleTriggersAllReturnCorrect) { + std::string path; + unsigned long offset = 0; + ASSERT_TRUE(resolve_file_offset( + reinterpret_cast(&uprobe_target), path, offset)) + << "无法从 /proc/self/maps 解析目标函数偏移"; + + int fd = open_uprobe_perf_event(path, offset); + ASSERT_GE(fd, 0) << "perf_event_open(uprobe) 失败,errno=" << errno; + ASSERT_GE(ioctl(fd, PERF_EVENT_IOC_ENABLE, 0), 0) << "ENABLE 失败"; + + // 连续触发 200 次:每次都必须经 #BP → XOL → #DB → 恢复并正确返回。 + for (int i = 0; i < 200; ++i) { + volatile int result = uprobe_target(i); + EXPECT_EQ(result, i * 2 + 1) << "第 " << i << " 次触发结果错误"; + } + + ioctl(fd, PERF_EVENT_IOC_DISABLE, 0); + close(fd); +} + +// close(fd) 触发注销(恢复原页)。注销后再调用函数应完全正常(无 0xcc 残留)。 +// 验证注销路径:移除表项 → 恢复断点页 → 回收 XOL slot。 +TEST(UprobeTest, UnregisterRestoresNormalExecution) { + std::string path; + unsigned long offset = 0; + ASSERT_TRUE(resolve_file_offset( + reinterpret_cast(&uprobe_target), path, offset)) + << "无法解析目标函数偏移"; + + { + int fd = open_uprobe_perf_event(path, offset); + ASSERT_GE(fd, 0) << "perf_event_open(uprobe) 失败,errno=" << errno; + ASSERT_GE(ioctl(fd, PERF_EVENT_IOC_ENABLE, 0), 0); + // 触发一次确认探针生效 + volatile int r = uprobe_target(5); + ASSERT_EQ(r, 11); + // close 触发 Drop → uprobe_unregister → 恢复原页 + close(fd); + } + + // 注销后:函数应直接执行,无断点介入,结果仍正确。 + for (int i = 0; i < 50; ++i) { + volatile int result = uprobe_target(i + 100); + EXPECT_EQ(result, (i + 100) * 2 + 1) << "注销后第 " << i << " 次结果错误"; + } +} + +// disabled 会撤销该 consumer;若它是最后一个,应恢复原指令。 +// 计数 ABI 尚未实现,这里只验证 disable/enable 生命周期不破坏控制流。 +TEST(UprobeTest, DisabledStillReturnsCorrectly) { + std::string path; + unsigned long offset = 0; + ASSERT_TRUE(resolve_file_offset( + reinterpret_cast(&uprobe_target), path, offset)) + << "无法解析目标函数偏移"; + + int fd = open_uprobe_perf_event(path, offset); + ASSERT_GE(fd, 0) << "perf_event_open(uprobe) 失败,errno=" << errno; + // 注册即 enable(perf 默认),先 disable 再测试 + ASSERT_GE(ioctl(fd, PERF_EVENT_IOC_DISABLE, 0), 0); + + // disabled 期间应直接执行原指令。 + for (int i = 0; i < 20; ++i) { + volatile int result = uprobe_target(i + 200); + EXPECT_EQ(result, (i + 200) * 2 + 1) << "disabled 第 " << i << " 次结果错误"; + } + + // 重新 enable:回调恢复(此处 noop_handler),函数结果仍须正确。 + ASSERT_GE(ioctl(fd, PERF_EVENT_IOC_ENABLE, 0), 0); + volatile int r = uprobe_target(7); + EXPECT_EQ(r, 15); + + close(fd); +} + +TEST(UprobeTest, EventSourceTypeIsPublished) { + __u32 type = 0; + ASSERT_TRUE(read_uprobe_perf_type(type)) + << "无法读取 " << UPROBE_TYPE_PATH << ",errno=" << errno; + EXPECT_GT(type, 0U); +} + +// perf fd 暂无可读计数,因此这里只验证 disabled 的可观察控制流语义: +// 事件以 disabled=1 创建后,被探测指令仍能正确执行,并可显式启用。 +TEST(UprobeTest, InitiallyDisabledCanBeEnabled) { + std::string path; + unsigned long offset = 0; + ASSERT_TRUE(resolve_file_offset( + reinterpret_cast(&uprobe_target), path, offset)) + << "无法解析目标函数偏移"; + + UprobePerfEventOptions options; + options.disabled = true; + FdGuard fd(open_uprobe_perf_event(path, offset, options)); + ASSERT_GE(fd.get(), 0) + << "disabled=1 的 perf_event_open 失败,errno=" << errno; + + for (int i = 0; i < 20; ++i) { + volatile int result = uprobe_target(i + 300); + EXPECT_EQ(result, (i + 300) * 2 + 1); + } + + ASSERT_GE(ioctl(fd.get(), PERF_EVENT_IOC_ENABLE, 0), 0) + << "初始 disabled 事件 ENABLE 失败,errno=" << errno; + volatile int result = uprobe_target(17); + EXPECT_EQ(result, 35); +} + +TEST(UprobeTest, InvalidPidCpuCombinationsAreRejected) { + std::string path; + unsigned long offset = 0; + ASSERT_TRUE(resolve_file_offset( + reinterpret_cast(&uprobe_target), path, offset)) + << "无法解析目标函数偏移"; + + auto expect_einval = [&](pid_t pid, int cpu) { + UprobePerfEventOptions options; + options.pid = pid; + options.cpu = cpu; + errno = 0; + int fd = open_uprobe_perf_event(path, offset, options); + int saved_errno = errno; + if (fd >= 0) { + close(fd); + ADD_FAILURE() << "非法 pid/cpu 组合意外成功:pid=" << pid + << ", cpu=" << cpu; + return; + } + EXPECT_EQ(saved_errno, EINVAL) + << "pid=" << pid << ", cpu=" << cpu; + }; + + expect_einval(-1, -1); + expect_einval(0, -2); + expect_einval(-2, 0); + expect_einval(0, std::numeric_limits::max()); +} + +TEST(UprobeTest, UnsupportedInheritanceAndExecFlagsAreRejected) { + std::string path; + unsigned long offset = 0; + ASSERT_TRUE(resolve_file_offset( + reinterpret_cast(&uprobe_target), path, offset)) + << "无法解析目标函数偏移"; + + auto expect_eopnotsupp = [&](const char* flag_name, + const UprobePerfEventOptions& options) { + errno = 0; + int fd = open_uprobe_perf_event(path, offset, options); + int saved_errno = errno; + if (fd >= 0) { + close(fd); + ADD_FAILURE() << flag_name << " 在 phase-1 不应被静默接受"; + return; + } + EXPECT_EQ(saved_errno, EOPNOTSUPP) << flag_name; + }; + + UprobePerfEventOptions options; + options.inherit = true; + expect_eopnotsupp("inherit", options); + + options = {}; + options.enable_on_exec = true; + expect_eopnotsupp("enable_on_exec", options); + + options = {}; + options.remove_on_exec = true; + expect_eopnotsupp("remove_on_exec", options); +} + +TEST(UprobeTest, UnsupportedConfigAndPerfCoreOptionsAreRejected) { + std::string path; + unsigned long offset = 0; + ASSERT_TRUE(resolve_file_offset( + reinterpret_cast(&uprobe_target), path, offset)); + + auto expect_eopnotsupp = [&](const char* name, + const UprobePerfEventOptions& options) { + errno = 0; + FdGuard fd(open_uprobe_perf_event(path, offset, options)); + EXPECT_LT(fd.get(), 0) << name << " 不应被静默接受"; + EXPECT_EQ(errno, EOPNOTSUPP) << name; + }; + + UprobePerfEventOptions options; + options.config = 1; // retprobe + expect_eopnotsupp("retprobe config", options); + + options = {}; + options.config = 1ULL << 32; // USDT ref_ctr_offset + expect_eopnotsupp("ref_ctr_offset config", options); + + options = {}; + options.group_fd = 0; + expect_eopnotsupp("group_fd", options); + + options = {}; + options.flags = PERF_FLAG_PID_CGROUP; + expect_eopnotsupp("PERF_FLAG_PID_CGROUP", options); +} + +TEST(UprobeTest, RelativePathUsesCurrentWorkingDirectory) { + std::string path; + unsigned long offset = 0; + ASSERT_TRUE(resolve_file_offset( + reinterpret_cast(&uprobe_target), path, offset)); + const auto slash = path.find_last_of('/'); + ASSERT_NE(slash, std::string::npos); + + FdGuard old_cwd(open(".", O_RDONLY | O_DIRECTORY)); + ASSERT_GE(old_cwd.get(), 0); + ASSERT_EQ(chdir(path.substr(0, slash).c_str()), 0); + FdGuard event(open_uprobe_perf_event(path.substr(slash + 1), offset)); + const int saved_errno = errno; + EXPECT_GE(event.get(), 0) << "相对路径应从 cwd 解析,errno=" << saved_errno; + ASSERT_EQ(fchdir(old_cwd.get()), 0); +} + +TEST(UprobeTest, UnsupportedReadReturnsErrorInsteadOfPanicking) { + std::string path; + unsigned long offset = 0; + ASSERT_TRUE(resolve_file_offset( + reinterpret_cast(&uprobe_target), path, offset)); + FdGuard fd(open_uprobe_perf_event(path, offset)); + ASSERT_GE(fd.get(), 0); + + __u64 count = 0; + errno = 0; + EXPECT_LT(read(fd.get(), &count, sizeof(count)), 0); + EXPECT_EQ(errno, EOPNOTSUPP); +} + +TEST(UprobeTest, ReadOnlyAliasIsSkippedAndLaterExecutableMapIsProbed) { + char path[] = "/tmp/uprobe_vma_XXXXXX"; + FdGuard file(create_raw_target(path)); + ASSERT_GE(file.get(), 0); + void* read_only = mmap(nullptr, 4096, PROT_READ, MAP_PRIVATE, file.get(), 0); + ASSERT_NE(read_only, MAP_FAILED); + + FdGuard event(open_uprobe_perf_event(path, 0)); + ASSERT_GE(event.get(), 0) + << "非可执行 alias 应被跳过而不是拒绝 consumer,errno=" << errno; + + void* executable = + mmap(nullptr, 4096, PROT_READ | PROT_EXEC, MAP_PRIVATE, file.get(), 0); + ASSERT_NE(executable, MAP_FAILED); + auto target = reinterpret_cast(executable); + EXPECT_EQ(target(21), 43); + + munmap(executable, 4096); + munmap(read_only, 4096); + unlink(path); +} + +TEST(UprobeTest, HardlinkAliasUsesCanonicalFileIdentity) { + char path[] = "/tmp/uprobe_alias_XXXXXX"; + FdGuard file(create_raw_target(path)); + ASSERT_GE(file.get(), 0); + const std::string alias = std::string(path) + ".link"; + ASSERT_EQ(link(path, alias.c_str()), 0); + + void* executable = + mmap(nullptr, 4096, PROT_READ | PROT_EXEC, MAP_PRIVATE, file.get(), 0); + ASSERT_NE(executable, MAP_FAILED); + FdGuard event(open_uprobe_perf_event(alias, 0)); + ASSERT_GE(event.get(), 0) << "hardlink alias 应命中同一 page cache,errno=" + << errno; + auto target = reinterpret_cast(executable); + EXPECT_EQ(target(9), 19); + + munmap(executable, 4096); + unlink(alias.c_str()); + unlink(path); +} + +TEST(UprobeTest, TmpfsExecutableUsesPageCacheInstructionBytes) { + char path[] = "/dev/shm/uprobe_tmpfs_XXXXXX"; + FdGuard file(create_raw_target(path)); + if (file.get() < 0 && (errno == ENOENT || errno == ENOSYS)) { + GTEST_SKIP() << "当前 rootfs 未提供 /dev/shm tmpfs"; + } + ASSERT_GE(file.get(), 0); + void* executable = + mmap(nullptr, 4096, PROT_READ | PROT_EXEC, MAP_PRIVATE, file.get(), 0); + if (executable == MAP_FAILED) { + // Some DragonOS tmpfs mounts are noexec. Registration must still be + // able to prepare the persistent definition from its page cache and + // wait for a future eligible mapping. + FdGuard event(open_uprobe_perf_event(path, 0)); + EXPECT_GE(event.get(), 0) + << "tmpfs definition 应从 page cache 读取,errno=" << errno; + unlink(path); + return; + } + FdGuard event(open_uprobe_perf_event(path, 0)); + ASSERT_GE(event.get(), 0) << "tmpfs uprobe 应从 page cache 读取,errno=" << errno; + auto target = reinterpret_cast(executable); + EXPECT_EQ(target(13), 27); + + munmap(executable, 4096); + unlink(path); +} + +// 同址 consumer 必须共享 site 生命周期:关闭第一个不能拆除第二个仍需要的断点; +// 关闭最后一个后继续执行不能触发无归属的 #BP。 +TEST(UprobeTest, SameAddressConsumersCloseIndependently) { + std::string path; + unsigned long offset = 0; + ASSERT_TRUE(resolve_file_offset( + reinterpret_cast(&uprobe_target), path, offset)) + << "无法解析目标函数偏移"; + + FdGuard first_fd(open_uprobe_perf_event(path, offset)); + ASSERT_GE(first_fd.get(), 0) + << "第一个 consumer 创建失败,errno=" << errno; + FdGuard second_fd(open_uprobe_perf_event(path, offset)); + ASSERT_GE(second_fd.get(), 0) + << "同址第二个 consumer 创建失败,errno=" << errno; + + ASSERT_GE(ioctl(first_fd.get(), PERF_EVENT_IOC_ENABLE, 0), 0); + ASSERT_GE(ioctl(second_fd.get(), PERF_EVENT_IOC_ENABLE, 0), 0); + EXPECT_EQ(uprobe_target(31), 63); + + first_fd.close_now(); + EXPECT_EQ(uprobe_target(32), 65) + << "关闭一个 consumer 不应破坏剩余 consumer 的 XOL 路径"; + + second_fd.close_now(); + for (int i = 0; i < 50; ++i) { + EXPECT_EQ(uprobe_target(i + 400), (i + 400) * 2 + 1) + << "最后一个 consumer 关闭后第 " << i << " 次执行错误"; + } +} + +TEST(UprobeTest, RepeatedStringInstructionIsRejected) { + constexpr unsigned char rep_movsb[] = { + 0xf3, 0xa4, // rep movsb + 0xc3, // ret + }; + char path[] = "/tmp/uprobe_rep_XXXXXX"; + FdGuard file(create_raw_code(path, rep_movsb, sizeof(rep_movsb))); + ASSERT_GE(file.get(), 0); + + errno = 0; + FdGuard event(open_uprobe_perf_event(path, 0)); + EXPECT_LT(event.get(), 0) + << "phase-1 XOL must reject repeated string instructions"; + EXPECT_EQ(errno, EINVAL); + unlink(path); +} + +TEST(UprobeTest, PushfInstructionIsRejected) { + constexpr unsigned char pushfq[] = { + 0x9c, // pushfq would expose the XOL single-step TF bit + 0x58, // pop rax + 0xc3, // ret + }; + char path[] = "/tmp/uprobe_pushf_XXXXXX"; + FdGuard file(create_raw_code(path, pushfq, sizeof(pushfq))); + ASSERT_GE(file.get(), 0); + + errno = 0; + FdGuard event(open_uprobe_perf_event(path, 0)); + EXPECT_LT(event.get(), 0) + << "phase-1 XOL must reject PUSHF because TF is instrumentation state"; + EXPECT_EQ(errno, EINVAL); + unlink(path); +} + +// Exercise the exact window where one CPU has executed INT3 while another +// CPU disables or closes the last consumer. A leaked ordinary SIGTRAP or a +// reused XOL slot terminates the test or produces a wrong result. +TEST(UprobeTest, ConcurrentTeardownDoesNotExposeRetiredBreakpoint) { + std::string path; + unsigned long offset = 0; + ASSERT_TRUE(resolve_file_offset( + reinterpret_cast(&uprobe_target), path, offset)) + << "无法解析目标函数偏移"; + + std::atomic stop{false}; + std::atomic bad_results{0}; + std::atomic completed_calls{0}; + std::thread runner([&]() { + int value = 1; + while (!stop.load(std::memory_order_acquire)) { + if (uprobe_target(value) != value * 2 + 1) { + bad_results.fetch_add(1, std::memory_order_relaxed); + } + completed_calls.fetch_add(1, std::memory_order_release); + value = value == 1000 ? 1 : value + 1; + } + }); + + int setup_failure_iteration = -1; + int setup_failure_errno = 0; + for (int i = 0; i < 200; ++i) { + FdGuard event(open_uprobe_perf_event(path, offset)); + if (event.get() < 0 || ioctl(event.get(), PERF_EVENT_IOC_ENABLE, 0) < 0) { + setup_failure_iteration = i; + setup_failure_errno = errno; + break; + } + const auto before = completed_calls.load(std::memory_order_acquire); + bool made_progress = false; + for (int spin = 0; spin < 10000; ++spin) { + if (completed_calls.load(std::memory_order_acquire) != before) { + made_progress = true; + break; + } + std::this_thread::yield(); + } + if (!made_progress || + ((i & 1) == 0 && ioctl(event.get(), PERF_EVENT_IOC_DISABLE, 0) < 0)) { + setup_failure_iteration = i; + setup_failure_errno = made_progress ? errno : ETIMEDOUT; + break; + } + // FdGuard closes the enabled or disabled final consumer here while + // the sibling continues to execute the target. + } + + stop.store(true, std::memory_order_release); + runner.join(); + EXPECT_EQ(setup_failure_iteration, -1) + << "iteration=" << setup_failure_iteration + << ", errno=" << setup_failure_errno; + EXPECT_EQ(bad_results.load(std::memory_order_relaxed), 0); +} + +int main(int argc, char** argv) { + ::testing::InitGoogleTest(&argc, argv); + return RUN_ALL_TESTS(); +}