kernel: add faulting state to process lifecycle by ppannuto · Pull Request #4948 · tock/tock

ppannuto · 2026-07-14T20:23:57Z

Pull Request Overview

The monolithic process object doesn't check process state every time it is accessed. Rather, we check process state at the beginning of the process event loop and operate assuming the process remains in that state until the next loop.

This means a process can't safely transition out of a runnable state in the middle of the loop (e.g., a command cannot directly restart or terminate a process, see #4872).

Instead, we mark a running process as Faulting while within an execution such that the next iteration through the loop the fault and its response can be safely handled.

This implements "path #2" from #4872 (comment)

Testing Strategy

This pull request was tested by compiling.

TODO or Help Wanted

Is this an agreeable solution to the problem?

Checklist

Ran make prepush.

PR Contents

Documentation

Updated the relevant files in /docs and the Book.
No documentation updates are required.

AI Use

No AI was used in this PR.
AI was used in this PR. I have read Tock's AI policy and have properly disclosed my AI use below.

The monolithic process object doesn't check process state every time it is accessed. Rather, we check process state at the beginning of the process event loop and operate assuming the process remains in that state until the next loop. This means a process can't safely transition out of a runnable state in the middle of the loop (e.g., a command cannot directly restart or terminate a process, see #4872). Instead, we mark a running process as `Faulting` while within an execution such that the next iteration through the loop the fault and its response can be safely handled.

bradjc

My sense is this would address the issue with a process faulting itself inside of a capsule but outside of the exit syscall.

However, I think I would prefer this to be integrated into the scheduler rather than the process state machine. That is, just like the scheduler can decide to run a process, it could decide to handle a faulted process, or handle a process that wants to restart. I would also like to see that scheduler include starting processes, verifying processes, or really anything that a process might want to do that would consume system-wide resources (e.g., time, energy). This would also be useful for processes that are stored in external nonvolatile storage and need to be loaded in to memory to run. Having this in the scheduler would allow us to write policies on when those operations should happen, and to be able to prioritize already running processes.

bradjc · 2026-07-14T20:48:00Z

+            // This is a bit brittle, but is a limitation of the way that process
+            // state is managed and the executor loop runs currently.
            match process.get_state() {
                process::State::Running => {


I'm not sure, we currently call process.set_fault_state(); within this match statement and it isn't a problem. Also in yield we call set_syscall_return_value() which changes the process's state. I think in this part of the state machine we can update a process's state.

Sometimes we safely can, yes. But within that same match is self.handle_syscall(resources, process, syscall);, which is the code path that eventually broke #4872.

bradjc · 2026-07-14T20:48:56Z

+    ForcedTerminate,
+
+    /// Catch-all for unknown faults.
+    Unknown,


What about ProcessFault? It is unknown with respect to what the process did, but not unknown that it wasn't initiated by a process manager.

If we liked this approach I was going to go through and eliminate Unknown if we can; there's a countable number of things (12 from a quick grep) that call set_faulting_state, but enough that I wasn't going to handle all the cases on first draft.

Updated to remove Unknown in f0f3108

ppannuto · 2026-07-14T21:11:47Z

I think I would prefer this to be integrated into the scheduler rather than the process state machine.

I thought about that, and actually started there, but our current scheduler interface is a bit awkward for this. In particular, it's probably the case that most schedulers would want to do the same thing with a "faulting" app, i.e., "handle it" the next time it would be scheduled to run. But currently that would require modifying every scheduler implementation. There's not a great place to put generic scheduling things except within the kernel's do_process loop.

brghena · 2026-07-15T15:41:13Z

That is, just like the scheduler can decide to run a process, it could decide to handle a faulted process, or handle a process that wants to restart.

I would also like to see that scheduler include starting processes, verifying processes, or really anything that a process might want to do that would consume system-wide resources (e.g., time, energy)

I do see the ideal of having the scheduler decide when is a good time to do system-wide work. But in practice, the scheduler doesn't know enough about the inner workings of applications and their priorities to determine the best possible time. Even if all the applications are currently yielded, the scheduler doesn't know if the next action a process is going to take is going to be time-sensitive (acknowledging a packet arrival for instance) or fudgeable (timer for printing some debug output).

So if the scheduler can't determine when is really the best time given the process workload, all it can use to determine is whether there is something to do or not. In which case it would presumably just do the work immediately once it's available. That would essentially be identical to putting it in the kernel loop.

Do you have an example where the scheduler would have insight into when to time system-wide work?

brghena · 2026-07-15T15:42:14Z

+                process::State::Faulting(reason) => {
+                    return_reason = process::StoppedExecutingReason::StoppedFaulted;
+                    process.resolve_faulting_state(reason);
+                    break;
+                }


There's not a great place to put generic scheduling things except within the kernel's do_process loop.

I was actually a little surprised to find this added to do_process and not to execute_kernel_work. I thought of process management as work the kernel needs to do, rather than something occurring the next time we go to schedule the process. I definitely think extensions like verifying process and starting new processes would be in the execute_kernel_work domain.

The problem is that execute_kernel_work is gated behind a scheduler decision of whether to do kernel work or process work: https://github.com/tock/tock/blob/master/kernel/src/kernel.rs#L362-L370

We can't put process lifecycle management in there, otherwise a scheduler could continually schedule process work for a faulting process, and the kernel would never actually get to handle the fault.

We can't put process lifecycle management in there, otherwise a scheduler could continually schedule process work for a faulting process, and the kernel would never actually get to handle the fault.

But isn't that already true today? A scheduler could only do process work or sleep, and never actually handle interrupts or run capsules?

I figured we'd just add "a faulting process exists" as part of the do_kernel_work_now check.

bradjc · 2026-07-15T17:06:46Z

Do you have an example where the scheduler would have insight into when to time system-wide work?

I think an easy answer could be "at night". Like "install this OS update at 2am tonight" kind of thing.

Our basic schedulers today aren't going to handle this in any interesting way. But opening the door by adding the API allows people to experiment, like Marshall's paper on batch scheduling for lower power Tock boards. If you had a board that required many frequent app loads (e.g., a platform without executable flash), I'm guessing you would be more interested in when you block for ~1000 ms to verify a new app.

ppannuto · 2026-07-17T21:42:56Z

Do you have an example where the scheduler would have insight into when to time system-wide work?

I think an easy answer could be "at night". Like "install this OS update at 2am tonight" kind of thing.

I think that's orthogonal from the process scheduler though. I would imagine something like "install this update" to be something that some system management app / capsule is responsible for, which already has mechanisms for setting alarms at specific times etc. I think of the Scheduler as much lower-level; I don't think the same code that's picking between round robin or managing MLFQ is doing high-level process lifetime work.

I do agree that it's a bit awkward that we don't have a canonical thing that manages the system. I believe the analog in Linux would be the init process (nowadays systemd), but it feels heavyweight for Tock to require a spawned process for this.

I would imagine once we have something like a system manager, stuff like the default fault policy goes away, and rather the manager is alerted that a process has faulted and then it deals with it from there. But that feels a much more substantial design/arch change, beyond addressing the immediate bug that's blocking the ENTS work

ppannuto · 2026-07-17T21:48:16Z

Hmm.. okay, I'm circling back on myself and wondering if something similar to DeferredCall tasks for "process lifecycle management" tasks could be added in execute_kernel_work and actually get to a lot of what Brad wants here ... I'll play with some sketches around that next week

bradjc · 2026-07-20T19:31:27Z

Do you have an example where the scheduler would have insight into when to time system-wide work?

I think an easy answer could be "at night". Like "install this OS update at 2am tonight" kind of thing.

I think that's orthogonal from the process scheduler though. I would imagine something like "install this update" to be something that some system management app / capsule is responsible for, which already has mechanisms for setting alarms at specific times etc. I think of the Scheduler as much lower-level; I don't think the same code that's picking between round robin or managing MLFQ is doing high-level process lifetime work.

But our scheduler isn't just a process scheduler. It chooses whether to run processes at all, and can choose to prioritize processes or the kernel. The proposed extensions from a while ago would have even allowed the scheduler to choose which interrupts to handle first.

bradjc · 2026-07-20T19:32:36Z

I would imagine once we have something like a system manager, stuff like the default fault policy goes away, and rather the manager is alerted that a process has faulted and then it deals with it from there. But that feels a much more substantial design/arch change, beyond addressing the immediate bug that's blocking the ENTS work

The very fast way to move beyond this is to just set a deferred call in the capsule so the fault downcall doesn't happen in response to an app's command.

github-actions Bot added the kernel label Jul 14, 2026

bradjc reviewed Jul 14, 2026

View reviewed changes

brghena reviewed Jul 15, 2026

View reviewed changes

kernel: replace FaultReason::Unknown with specific variants

f0f3108

ppannuto changed the title ~~[RFC] kernel: add faulting state to process lifecycle~~ kernel: add faulting state to process lifecycle Jul 17, 2026

Uh oh!

Uh oh!

Conversation

ppannuto commented Jul 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Overview

Testing Strategy

TODO or Help Wanted

Checklist

PR Contents

Documentation

AI Use

Uh oh!

bradjc left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ppannuto Jul 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ppannuto commented Jul 14, 2026

Uh oh!

brghena commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bradjc commented Jul 15, 2026

Uh oh!

ppannuto commented Jul 17, 2026

Uh oh!

ppannuto commented Jul 17, 2026

Uh oh!

bradjc commented Jul 20, 2026

Uh oh!

bradjc commented Jul 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ppannuto commented Jul 14, 2026 •

edited

Loading

ppannuto Jul 17, 2026 •

edited

Loading

brghena commented Jul 15, 2026 •

edited

Loading