async fn maybe_synthesize_recovery_completion(
state: &Arc<AppState>,
clock: &ClockHandle,
event_log_tx: &EventLogTx,
rs: &mut ReducerState,
host: &str,
agent_current: &str,
at: DateTime<Utc>,
)Expand description
Scan active host_rollout_records for host; for each record whose
observed closure on the wire identifies a missed transition,
synthesize the event chain that advances the row to a state
consistent with the agent’s reality. Idempotent: if the state has
already advanced (e.g. concurrent agent emit), the record won’t
match and the synthesis is a no-op.
LOADBEARING: this is the CP-side half of architecture.md §305
acceptance gate 1 (“destroying the CP database and rebuilding
from empty state results in full fleet visibility within one
reconcile cycle, with zero operator intervention beyond restarting
the service”). The agent’s heartbeat carries current_closure on
every tick; CP rebuilds soft-state HRR rows from those inputs.
Four reachable recovery cases:
Activating+current == target: agent restarted mid-rollout, post-boot observed the activation took. SynthesiseRemoteActivationCompleted.Deferred+current == target: operator rebooted to finish a critical-component activation. Same synthesis.Pending+current == target: CP itself was wiped, the planner re-opened the rollout inPending, but the agent has been running the target closure all along. Synthesise the fullRemoteDispatchAck → RemoteActivationCompleted → RemoteConvergedchain.RemoteConverged’s soak-elapsed invariant is satisfied by stampingconverged_at = max(at, record.soak_due_at)— the soak window’s purpose (give probes time to fail) was exercised pre-wipe. Gated byevaluate_synth_gatesso a fleet bump that opens a Pending row for a host whose closure already matches doesn’t bypass channel-edges or wave-promotion ordering.Failed+current == current_closure_at_dispatch: the rollback’s switch-to-configuration restarted the agent mid-VerifyPoll, droppingLocalRollbackCompleted. SynthesiseRemoteRollbackCompleteon the heartbeat that observes the rolled-back closure; the canonicalFailedreducer arm (failed.rs::RemoteRollbackComplete) produces the quarantine- event_log + transition effects.