How it works

The reconcile loop

Every controller follows the same shape: watch a resource, compare desired to observed, take one safe step, record status, requeue. The operator decides; the agent performs host mutations; the agent reports; the operator rolls forward on the next pass.

sequenceDiagram
    autonumber
    participant U as nwctl / UI
    participant A as K8s API
    participant O as Operator (controller)
    participant N as Node agent
    participant H as Host / storage

    U->>A: apply CRD (intent)
    A-->>O: watch event
    O->>O: compare desired vs observed
    O->>A: patch .status (phase, next step)
    O->>N: delegate one host op
    N->>H: mutate (ip link / drbdadm / kairos)
    N-->>A: report observed state
    A-->>O: requeue
    O->>O: honor gates, then advance one step

These invariants are non-negotiable — each is a hard-won lesson from the DaemonSet's incident history:

• Idempotence. Every pass is safe to retry; never leave partial state.
• Honor explicit gates. Refuse to advance past a safety gate rather than racing it.
• Backpressure. If the storage layer is unhealthy, wait — don't queue more work.
• Fail loud, don't partially proceed. The motivating failure was one node flipping while the other didn't, and the next step firing anyway. Controllers stop instead.
• Status reflects observed, never desired.

The coordinated bond-flip machine

The most battle-hardened workflow is the single→multi-node storage upgrade: flipping the storage bond from active-backup to balance-rr so DRBD can replicate across nodes — without disrupting the VMs running on top. In the DaemonSet this is a 21-state machine persisted in a ConfigMap. Nodewright now implements it as a first-class BondModeFlipPlan controller — the 12-state forward path plus its automated recovery ladder (rollback, bounded retry, and escalate-when-unprovable), built and exercised end-to-end against a fake DRBD/KubeVirt harness. Each step delegates host work through the HostOperation protocol and re-checks the cluster lock before it mutates anything.

stateDiagram-v2
    direction TB
    [*] --> Idle
    Idle --> PreconditionsCheck: entry gates pass
    PreconditionsCheck --> AcquireLock
    AcquireLock --> SuspendMigration: lock held + leader snapshot
    SuspendMigration --> DisconnectDRBD: VMs pinned, VMIMs aborted
    DisconnectDRBD --> DrainSatellite
    DrainSatellite --> FlipNode1
    FlipNode1 --> FlipNode2: node1 verified balance-rr
    FlipNode2 --> ResumeDRBD: both on balance-rr
    ResumeDRBD --> RestoreMigration: gates pass + two replicas UpToDate
    RestoreMigration --> ReleaseLock: evictionStrategy restored
    ReleaseLock --> Complete: lock released
    Complete --> [*]

    Idle --> Halt: any gate failure
    ResumeDRBD --> Halt: link marginal / asymmetric
    Halt --> Recover: automated recovery
    Recover --> [*]
    note right of Halt
        Freeze in place: retain the lock,
        keep VMs pinned. Automated recovery
        (retry / rollback) takes over — and
        escalates to a human where it can't
        prove safety. See "Recovery" below.
    end note

What the diagram encodes — and why each guard exists:

Stage	The guard	Why
PreconditionsCheck	≥2 eligible nodes; every peer live	Never flip into a cluster that can't hold replicas.
SuspendMigration	abort in-flight VM migrations, pin evictionStrategy=None	If KubeVirt live-migrates a VM mid-flip, it can land on a node with asymmetric DRBD — data-availability risk.
DisconnectDRBD → DrainSatellite	disconnect replication, move storage pods off the node	Flip the bond on a quiesced storage path, not a live one.
FlipNode1 → FlipNode2	one node at a time; a failed flip halts	A half-flipped cluster is the original incident; the forward path never leaves one node flipped and silently proceeds.
ResumeDRBD	reconnect only after symmetry + a connectivity probe + the I6 consistency gate (see below) — and that gate runs before every reconnect, forward or recovery	Never resume replication over a marginal or asymmetric link, and never let a workload attach to a replica that isn't provably consistent.
Halt → recovery	any gate failure freezes in place (lock held, VMs pinned), then automated recovery retries or rolls back	The forward path stops safely; recovery then drives back toward a known-good state where it can prove safety — and escalates to a human wherever it can't (see Recovery).

The six safety invariants

Every guard above rolls up to six invariants the controller will not violate. Each is enforced in code and tested against the fake harness — they are the reason this workflow can be trusted with a live cluster's storage. The first invariant sits above the rest: preserving the last consistent copy of a PVC's data outranks availability, and availability outranks automation.

#	Invariant	How it's enforced
I1	Never flip both nodes at once	The two flip targets are snapshotted at AcquireLock — before drain empties eligibility — and the operator keeps at most one outstanding HostOperation per plan.
I2	Never resume DRBD over a marginal link	Two gates before resume — bond symmetry, then a single connectivity probe — must pass and stamp a resumeGatesPassedAt token; every later step refuses to act without it. No retry loop that could paper over a bad link.
I3	Never evict a VM mid-flip	Pin-first: every VM is set to evictionStrategy=None and in-flight migrations aborted before any host change; protection is re-asserted at the entry of every mutating step, and any breach halts.
I4	One writer only	An instance-fenced Lease, re-checked (renew-or-fail) immediately before every mutation. A lost lock halts — it never races another writer.
I5	Two replicas in sync before finishing	Both RestoreMigration and ReleaseLock independently re-run the two-replica sync gate; a negative result defers forever, never bypasses.
I6	Never trust "UpToDate" alone — prove consistency	The sync gate no longer believes DRBD's UpToDate string. It additionally requires zero out-of-sync sectors + matching generation-UUIDs + no split-brain, evaluated from both nodes and failing closed on anything unknown. This gate runs before every replication reconnect — forward and recovery. It closes the real incident where both replicas reported UpToDate while replication was only partial.

Data preservation is the number-one invariant

Everything else on this page serves one rule: never lose, and never corrupt, the last consistent copy of a PVC. If the controller cannot prove a replica is consistent, it treats it as unsafe — it will keep the workload pinned to the copy it can trust (even if that leaves only one copy) rather than let a VM attach to a possibly-divergent replica. Availability and automation both yield to this.

Recovery — automated, but bounded by proof

A halt is not the end of the story. When a gate fails, the plan freezes safely (lock held, VMs pinned) and an automated recovery ladder takes over. Its job is to get the cluster back to a known-good state without a human wherever it can — but only ever within what it can prove is safe.

stateDiagram-v2
    direction TB
    [*] --> Halted: gate failed, frozen safely
    Halted --> Dwell: let transients settle
    Dwell --> Retry: same rung, under the retry cap
    Retry --> Resumed: I6 gate proves consistency
    Dwell --> Rollback: forward not provable
    Rollback --> RevertedSafe: bond restored, I6-gated reconnect
    Retry --> Escalated: retry cap exhausted
    Rollback --> Escalated: rollback not provable
    Halted --> Escalated: split-brain / unknown state
    Resumed --> [*]
    RevertedSafe --> [*]
    Escalated --> [*]
    note right of Escalated
        Freeze and page a human.
        Lock retained, VMs stay pinned
        (eviction disabled, fail-over ignored),
        substrate untouched. No single-copy
        landing is forced here — that's B5/B6.
    end note

How the ladder behaves — and what it will not do:

Rung	What it does	The bound
Dwell	Waits out transient link/sync noise before acting	Cheap, reversible, no mutation.
Retry (same rung)	Re-attempts the failed step, up to a tunable cap	Every reconnect still passes the I6 gate; a retry can never paper over a bad link.
Rollback	Reverts the bond toward its pre-flip mode to restore the known-good topology	The reconnect on the way back is I6-gated too — rollback is not a trusted escape hatch.
Escalate	Freezes and pages a human	Reached whenever forward and backward are unprovable, the retry cap is exhausted, or a split-brain / unknown state appears.

Two design choices make this safe rather than merely convenient:

• Escalate-when-unprovable. Automation is bounded by proof, not by effort. The moment the controller can't prove a safe next step — either direction — it stops and escalates rather than guessing. Split-brain and unknown consistency escalate immediately, with no retry window.
• A frozen escalation is genuinely frozen. Escalation changes no phase, touches no lease, and mutates no host or storage. The cluster lock stays held and every VM stays pinned (eviction disabled, fail-over ignored) so nothing can migrate onto an unverified replica while a human investigates. Human recovery is an explicit, fingerprinted response — it is a reply to an escalation, not a trigger the controller can fire on its own.

What's deliberately deferred

Today, if a two-replica resume can't be proven, the plan halts and escalates — it does not yet automatically quarantine down to a single known-good copy and land there. That enforced single-copy landing is intentionally deferred to the hardware-lab tranches (B5/B6), where it can be exercised against real DRBD before it's trusted to run unattended.

This is why it's gated, not autonomous

Every step here mutates host networking or storage on a live cluster — high blast radius. The controller is over-specified with these guards as testable acceptance criteria; it ships default-off, installs no CRDs, and its dangerous paths are designed to run supervised with human sign-off, never as an unattended loop. The read-only observation bridge came first precisely because it carries none of that risk.