fix(journaling): serialize recovery retries and preserve persistence failures

docs/site/src/content/docs/grains/journaling/runtime-behavior.md

@@ -62,13 +62,11 @@ Concurrent calls made while the same kind of write is queued can share that queu
 ## Safe-to-commit staging
 
 All interleaved callers share the manager's pending journal. Prepare fallible work, external acknowledgements,
-and proposed output in operation-local data. After establishing that an outcome is safe to commit, apply its
-mutations to durable state and initiate a write. Coordinate that transition with other interleaved operations
-which can affect the same decision. Any caller's write can include staged mutations from other calls.
-
-If an application error occurs after staging and makes those mutations unsafe to commit, end the activation's
-use of the manager and request deactivation. In-flight methods can retain local decisions and references
-across awaits; a fresh activation reconstructs both application and durable state together.
+and proposed output in operation-local data. After the final preparation await, check the relevant
+preconditions and apply the complete safe-to-commit update synchronously, then request an ordinary write.
+Orleans executes that synchronous block on a single activation thread. Another grain turn can run when
+the operation awaits, so keep shared state safe to commit at each await. Any caller's write can include
+staged mutations from other calls.
 
 ## Consistency and competing writers
 
@@ -80,7 +78,7 @@ Design commands to tolerate retries at the application boundary. Use operation i
 
 ## Storage failures
 
-A failed append, snapshot replacement, delete, or initialization permanently fences that manager instance.
+A failed append, snapshot replacement, or delete permanently fences that manager instance.
 Queued operations fault, and later write, delete, registration, and initialization requests fail explicitly.
 Existing in-memory state remains available to in-flight calls until deactivation completes. The grain runtime
 starts deactivation as part of handling the failure.
@@ -99,8 +97,42 @@ assigned lifetimes.
 Cancelling a write's cancellation token stops the caller's wait. An already queued write continues to its
 storage outcome, so the caller reconciles that outcome before retrying the command.
 
-An initialization failure preserves stored data for diagnosis. Restore the required format/codec registration
-or repair the backing data before creating a fresh manager or retrying activation.
+An initialization failure reports its error to that attempt's callers and leaves the manager uninitialized.
+The caller can retry <xref:Orleans.Journaling.IJournaledStateManager.InitializeAsync*> after a transient
+failure or after restoring the required format, codec, or backing data. Each attempt resets recovery
+bookkeeping and replays the journal from the beginning using the same registered state machines.
+Concurrent callers share the active attempt; writes and deletion become available after initialization
+succeeds. State registration stays closed once initialization has begun.
+
+Owner shutdown which cancels initial recovery cancels all initialization waiters and leaves the manager
+stopped. Disposal waits for the owned read to finish before releasing journal resources. Cancelling an
+individual initialization caller's token ends only its wait; owned recovery continues for other callers.
+
+## Custom state lifecycle
+
+Custom <xref:Orleans.Journaling.IStateMachine> implementations share the manager's single logical execution thread.
+Supply command codecs as constructor dependencies. The registration factory selects codecs keyed by the
+same write-format key used to configure the journal owner. Activation-owned state factories resolve those
+dependencies from the activation's services; standalone callers supply codecs with the appropriate lifetime.
+The codec is available when the state is constructed, including for an empty journal.
+During replay, <xref:Orleans.Journaling.JournalReplayContext.GetRequiredCommandCodec*> selects the codec
+for each entry's stored format.
+
+States synchronously encode their pending changes through <xref:Orleans.Journaling.IStateMachine.WritePendingEntries*>
+or their current contents through <xref:Orleans.Journaling.IStateMachine.WriteSnapshot*>.
+After storage acknowledges captured bytes, <xref:Orleans.Journaling.IStateMachine.OnWriteCompleted*>
+performs durable-completion bookkeeping. A zero-byte write completes without this callback.
+
+The journal owner keeps feature operations quiescent through deletion's storage and reset outcome,
+including when a caller cancels its wait. Successful deletion calls <xref:Orleans.Journaling.IStateMachine.Reset*>
+before completing deletion waiters.
+
+The manager records the first write or delete failure, fences further persistence, faults current
+and queued manager waiters, and requests grain deactivation. Features observe their write failures and
+complete their own operation waiters and resource cleanup through their operation and lifecycle ownership.
+Standalone callers own that cleanup explicitly. A previously captured write retains its actual storage
+outcome and acknowledgement bookkeeping. Owner-canceled initial recovery and idle shutdown complete
+through normal shutdown; admitted write/delete storage cancellation remains terminal.
 
 ## Compaction
 

src/Orleans.Journaling/IJournaledStateManager.cs

@@ -19,7 +19,11 @@ public interface IJournaledStateManager : IAsyncDisposable
     /// Initializes the state manager by replaying its journal.
     /// </summary>
     /// <remarks>
-    /// A failed initialization permanently fences this instance. Recover by creating a new manager and new state instances.
+    /// A recovery failure fails the current initialization attempt and leaves this instance uninitialized.
+    /// A subsequent call retries recovery from the beginning, resetting and replaying the registered state machines.
+    /// Writes become available after initialization succeeds. A manager fenced by a persistence failure requires a new instance.
+    /// Owner shutdown which cancels recovery cancels initialization and leaves this instance stopped.
+    /// Cancelling the caller's token ends only that caller's wait while owned recovery continues.
     /// </remarks>
     /// <param name="cancellationToken">The cancellation token.</param>
     /// <returns>A <see cref="ValueTask"/> which represents the operation.</returns>
@@ -58,7 +62,8 @@ public interface IJournaledStateManager : IAsyncDisposable
     /// Resets this instance, removing any persistent state.
     /// </summary>
     /// <remarks>
-    /// Quiesce other operations before deleting state: deletion resets every registered state machine.
+    /// The caller keeps other operations quiescent through completion: deletion resets every registered state machine.
+    /// Cancellation ends the caller's wait; an already queued deletion continues to its storage and reset outcome.
     /// A failed deletion permanently fences the manager and requests deactivation of its owning grain.
     /// </remarks>
     /// <param name="cancellationToken">The cancellation token.</param>

src/Orleans.Journaling/IStateMachine.cs

@@ -25,8 +25,8 @@ namespace Orleans.Journaling;
 /// rather than treating in-memory mutations as durable.
 /// </item>
 /// <item>
-/// A failed journal operation permanently fences the manager and requests grain deactivation.
-/// A new manager initializes new state instances by calling <see cref="Reset"/> and replaying durable entries.
+/// A failed write or delete permanently fences the manager and requests grain deactivation.
+/// Recovery calls <see cref="Reset"/> before replaying durable entries, including when initialization is retried.
 /// </item>
 /// </list>
 /// <para>

src/Orleans.Journaling/JournaledStateManager.cs

@@ -162,8 +162,9 @@ public async ValueTask InitializeAsync(CancellationToken cancellationToken = def
         lock (_lock)
         {
             ThrowIfFenced();
-            if (_workLoop is null)
+            if (_workLoop is null || _state is ManagerState.RecoveryFailed)
             {
+                _state = ManagerState.Unknown;
                 _workLoop = Start();
             }
 
@@ -191,9 +192,25 @@ private async Task WorkLoop()
         {
             await RecoverAsync(_shutdownCancellation.Token).ConfigureAwait(true);
         }
+        catch (OperationCanceledException) when (_shutdownCancellation.IsCancellationRequested)
+        {
+            return;
+        }
         catch (Exception exception)
         {
-            Fence(exception);
+            try
+            {
+                LogErrorProcessingWorkItems(_shared.Logger, exception);
+            }
+            finally
+            {
+                lock (_lock)
+                {
+                    _state = ManagerState.RecoveryFailed;
+                    FaultQueuedWorkItemsUnderLock(exception);
+                }
+            }
+
             return;
         }
 
@@ -548,6 +565,10 @@ private async Task WorkLoop()
                     }
                 }
             }
+            catch (OperationCanceledException) when (_shutdownCancellation.IsCancellationRequested)
+            {
+                return;
+            }
             catch (Exception exception)
             {
                 Fence(exception);
@@ -560,6 +581,11 @@ private void Fence(Exception exception)
     {
         lock (_lock)
         {
+            if (_state is ManagerState.Fenced)
+            {
+                return;
+            }
+
             _state = ManagerState.Fenced;
             _failure = exception;
         }
@@ -1173,6 +1199,7 @@ private sealed class RegisterStateWorkItem(string name) : WorkItem(name)
     private enum ManagerState : byte
     {
         Unknown,
+        RecoveryFailed,
         Ready,
         Fenced
     }

src/Orleans.Journaling/README.md

@@ -183,8 +183,9 @@ provider and requested state name.
 - `OnRecoveryCompleted`: finish reconstruction before application use.
 - `OnWriteCompleted`: publish effects that depend on storage acknowledgement.
 
-The recovery model uses fresh instances and replay. `JournalReplayContext.ResolveStateMachine`
-routes entries to the state machine for their stream.
+Recovery resets state machines and replays durable entries. A failed initialization can be retried on the
+same manager and registered states. `JournalReplayContext.ResolveStateMachine` routes entries to the
+state machine for their stream.
 
 `IJournaledStateManager` is independent of the grain-facing `IDurableStateManager` and extends
 `IAsyncDisposable`. Its owner API provides `RegisterStateMachine`, `TryGetStateMachine`,
@@ -255,13 +256,18 @@ durable state and await `WriteStateAsync`. One acknowledgement covers the manage
 batch, including changes staged by interleaved callers. Applications are responsible for sequencing that transition
 with other interleaved operations and for making uncertain-outcome retries idempotent.
 
-A failed journal operation permanently fences the manager, faults queued operations, and requests
+A failed write or delete permanently fences the manager, faults queued operations, and requests
 deactivation of the associated grain. In-flight calls retain their existing in-memory state while subsequent
 state-manager operations fail explicitly. A new activation recovers the actual durable outcome.
 For a manager created through `IJournaledStateManagerFactory`, dispose the failed instance and create
 another manager for the same `JournalId`, explicitly constructing and registering fresh state components
 before initialization and retiring the old components and dependencies according to their assigned lifetimes.
 
+An initialization failure reports its error to the attempt's callers and leaves the manager uninitialized.
+Call `InitializeAsync` again to retry from the beginning using the existing reset/replay contract.
+Concurrent callers share the active attempt, and writes become available after recovery succeeds.
+State registration stays closed after initialization first begins.
+
 Cancelling a caller's wait leaves an already queued write running. Observe durability through write
 acknowledgement or a fresh activation before deciding whether to retry an application command.