Components

Inside the operator

The operator is one binary (cmd/operator) hosting a controller-manager. Each CRD gets its own controller package; validation and defaulting live in an admission webhook; host work is delegated to the agent. Backends and operating systems sit behind capability interfaces so the same controllers run against different substrates.

Custom resources

Group nodewright.spectrocloud.com/v1alpha1. Workflow CRDs carry a .status.phase and a per-step progress list; status always reflects observed state, never desired.

CRD	Scope	Status	What it is
VMVolumeMigration	namespaced	built	Automates migrating a VM's volume to RWX (cold dd-copy / hot csi-clone) with snapshot bookends + checksum verification. The migration runbook, made declarative.
NodewrightCluster	cluster (singleton)	built	The cluster-level health rollup: Bootstrap · SingleNode · Upgrading · MultiNode · Degraded · Recovering.
NodewrightAgentStatus	cluster (per node)	built	Per-node observed state — role, heartbeat, storage/DRBD health.
BondModeFlipPlan	cluster	built ⭐	A coordinated bond-mode flip as a workflow — the 12-state forward path plus an automated recovery ladder (rollback / bounded retry / escalate-when-unprovable), with VM protection, DRBD coordination, and the I6 data-consistency gate that distrusts "UpToDate" before every reconnect. Built and fake-harness-tested.
HostOperation	cluster	built	One-shot, immutable operator→agent instruction — BondNetdevWrite · BootStageRun · NetworkReconfigure · ConnectivityProbe. Operator writes .spec, agent writes .status.
ReplicationUpgrade	cluster	planned	The single→multi-node storage upgrade, gated and staged.
EdgeProfile	cluster	planned	The substrate definition — topology, backend, OS, names, thresholds — as a signed, versioned artifact.

One dangerous workflow at a time

While any BondModeFlipPlan is active — including before it has even started — the operator holds off every VMVolumeMigration. The two mutate overlapping storage state, so they are mutually exclusive by construction, lowest-UID plan wins.

The observation bridge

The first increment shipped deliberately read-only. Rather than take over the DaemonSet, the operator observes it: it reads the DaemonSet's live state (a ConfigMap-backed 21-state machine plus per-node heartbeats) and its Kubernetes Events, and rolls that up into NodewrightCluster.status and NodewrightAgentStatus.

Why read-only first

It ships value in days, it cannot race or harm the DaemonSet, and it forces an accurate, typed model of the machine the operator will later take over. Low blast radius by construction.

The HostOperation protocol

When the operator needs a host mutated, it does not reach onto the node — it files a work order. HostOperation is a cluster-scoped, immutable, single-shot resource: the operator writes .spec (one primitive plus its typed parameters), the agent on the target node executes it and writes .status (phase, message, probe result). Retry is a new object, never an edit.

Four primitives, one discriminated union — BondNetdevWrite, BootStageRun, NetworkReconfigure, and a read-only ConnectivityProbe. Two rules make it safe:

• At most one non-terminal operation per plan, ever. The controller refuses to create a second while one is outstanding — this is the serialization point behind invariant I1 (never dual-flip).
• Deterministic, bounded names. Find-or-create is idempotent across restarts; a crashed controller never double-issues the same op.

Dormant in this phase

The protocol and the agent's four primitives are built and unit-tested, but the actuation seams have no production implementation — every primitive runs against an in-memory fake and fails closed (ErrNotWired) if asked to touch a real host. Real execution is gated to the hardware-lab phase.

Extensibility — three plug-in points, not a fork

A plug-in point is a clean boundary where you can swap the implementation without touching the code around it. The current customer's hardware is one option, never the hardcoded default. There are three:

• Storage provider (storage.Backend) — a pluggable module that teaches the operator how to work with one storage system (like a driver). Each declares static capabilities (SupportsClone(), SupportsRWX(), …) and its replication-coordination model (does it need host-level coordination like DRBD + bond flips, or does it self-replicate?). Controllers query these; the webhook rejects workflows a provider can't safely support.
• OS provider (os.OSProvider) — declares what host operations a given OS can perform (boot-recovery hooks, persistent network config, kernel tuning).
• EdgeProfile — bundles topology + storage + mode + OS choices, plus site-specific names and thresholds, into one signed profile. A new site or hardware shape is a new profile, not a code change.

Capabilities are compile-time static — declared by the Go type, never live-probed — so a provider outage can never silently change what the operator will admit.

Which backends are real vs. illustrative

Today there is one implemented storage provider: Piraeus/LINSTOR (open source), plus a degraded local-path for single-node. Portworx/Pure (paid) — and the ubuntu/rhel OS entries — appear in these diagrams as illustrative examples that prove the plug-in point is real, not shipped features or dated commitments. The paid Pure provider is a v1 direction behind the same interface; the interface is built now so LINSTOR-specific assumptions never leak into the core.