Architecture

This page describes the internal architecture of Trilio Site Recovery: how its Kubernetes operators, agents, webhooks, and custom resources are organized across three clusters, how reconciliation loops drive DR state forward, and how components interact during normal operation and failover. Understanding this architecture helps you reason about what runs where, why certain constraints exist (such as VM pinning and the three-cluster requirement), and how to interpret status conditions and logs when diagnosing problems.

ProtectionZone Controller (app.kubernetes.io/component: protectionzone-controller)

Runs on the TSR-Manager cluster in a dedicated Deployment. This is the bootstrap operator: it validates both workload cluster credentials, acquires an exclusivity lease on each workload cluster's trilio-site-recovery-system namespace, and then Helm-installs the TSR-Manager and workload control plane stacks from charts embedded directly in its binary. It also provisions the zone's default DRBDReplicationPolicy (<zoneName>-default with isDefault: true). The ProtectionZone controller runs as its own binary (cmd/protectionzone) because the per-zone control planes it deploys depend on it — it cannot itself be one of those per-zone deployments.

TSR-Manager Control Plane (app.kubernetes.io/component: manager-control-plane)

Runs on the TSR-Manager cluster in the zone namespace (dr-<name>). This is the central orchestrator: it hosts the reconcilers for ProtectionRequest, ProtectionGroup, FailoverRequest, TestFailover, ReplicationGroupStatus, RPOEvent, and DRBDReplicationPolicy. It reaches workload clusters exclusively through the kubeconfig Secrets stored in the zone namespace — it never receives inbound traffic from workload clusters. A single container named manager hosts all reconcilers; the pod shows 1/1. The manager binary is /tsr-manager.

Workload Control Plane (app.kubernetes.io/component: workload-control-plane)

Runs on each workload cluster in the trilio-site-recovery-system namespace. It hosts the DRBDResource placement reconciler (node binding, backing PVC provisioning, anchor pod management), the admission webhooks for DRBDResource and KubeVirt VirtualMachine objects, and exposes Prometheus metrics. It talks only to its own cluster's API server — it never contacts the TSR-Manager. The binary is /workload-manager.

drbd-node-agent DaemonSet (app.kubernetes.io/component: drbd-node-agent)

Runs one pod per schedulable worker node on each workload cluster. Named <release>-site-recovery-workload-control-plane-agent. This is the only component that touches the DRBD kernel module: it writes .res configuration files, calls drbdadm, provisions frontend PV-PVCs and DRBD metadata PVCs for the DRBDResources bound to its node, and manages the siterecovery.trilio.io/not-ready:NoSchedule startup taint. It runs with privileged: true and hostNetwork: true and is bound to the privileged SCC on OpenShift. The agent binary is part of the site-recovery-drbd-node-agent image; userspace DRBD tools are shipped inside that image.

TSR Web Console (site-recovery-frontend + site-recovery-backend)

Deployed via OLM on the TSR-Manager cluster. Provides dashboards for zone health, VM protection status, replication metrics, and one-click failover and test-failover operations. On OpenShift it is exposed through an OpenShift Route at https://trilio-site-recovery.<cluster apps domain>; browser access is authenticated via OpenShift OAuth. On upstream Kubernetes no ingress or authentication path is provided.

tsrctl

The only Site Recovery CLI tool. Supplied by Trilio with the release; contact Trilio support to obtain it. Manages zone kubeconfig credentials exclusively: zone-credential create, rotate, inspect, and delete. It does not drive DR operations — failover, protection, and test failover are all performed through CRDs.

tsr-gather (site-recovery-must-gather image)

Run via oc adm must-gather on the TSR-Manager cluster. Collects logs, CRs, and DRBD state from the TSR-Manager and all zone workload clusters into a single read-only archive. The entrypoint inside the image is /usr/bin/gather.

The following traces what happens when you protect a VM, illustrating how control flows through all three clusters.

Step 1 — Zone provisioning (done once per zone)

You create a ProtectionZone CR in a dr-<name> namespace on the TSR-Manager cluster. The ProtectionZone controller:

1. Reads the two kubeconfig Secrets named in spec.clusters[].kubeconfigSecretRef.
2. Probes both workload clusters (connectivity, authentication, RBAC via SelfSubjectAccessReview, KubeVirt presence, OpenShift SCC prerequisites).
3. Acquires a zone-exclusivity lease on each workload cluster's trilio-site-recovery-system namespace by writing siterecovery.trilio.io/owner-zone-namespace-uid and siterecovery.trilio.io/owner-zone annotations.
4. Helm-installs site-recovery-manager-control-plane into dr-<name> on the TSR-Manager cluster and site-recovery-workload-control-plane into trilio-site-recovery-system on both workload clusters (from charts embedded in the controller binary — no registry pull).
5. Creates the default DRBDReplicationPolicy (<zoneName>-default) in the zone namespace.
6. Sets ClustersValidated=True on the zone. Only then do webhooks on the TSR-Manager cluster accept new ProtectionGroup or ProtectionRequest creates.

Step 2 — VM protection

You create a ProtectionRequest CR in the zone namespace, specifying the VM name, namespace, and spec.sourceCluster. The TSR-Manager control plane's protectionrequest reconciler:

1. Looks up the VM on the source workload cluster through its cached client.
2. Selects the default DRBDReplicationPolicy for the zone and reads DRBD parameters.
3. Creates a DRBDResource CR in the VM's namespace on both workload clusters. The CR name follows the pattern drbd-<namespace>-<vm>-<hash8>.
4. The workload control plane on each cluster picks up its local DRBDResource, runs the placement reconciler to bind a node, and creates the anchor pod and DRBD metadata PVCs.
5. The drbd-node-agent on the bound node writes the DRBD .res file, calls drbdadm up, and provisions the frontend PVC. The agent removes the siterecovery.trilio.io/not-ready:NoSchedule taint from the node once DRBD converges (or the startup deadline passes).
6. The two DRBD node agents establish a TCP connection on a port in the range 7000–8000 (one port allocated per replicated VM volume). Replication uses Protocol C (synchronous, RPO = 0) exclusively.
7. As DRBD reaches Connected state the ProtectionRequest transitions through Pending → Validating → CreatingDRBD → Syncing → Protected.

Step 3 — Replication monitoring

The replicationmonitor reconciler in the TSR-Manager control plane polls each workload cluster at the ReplicationGroupStatus.spec.pollingIntervalSeconds interval (default 15 s). It aggregates per-VM DRBD metrics — out-of-sync bytes, connection state, current RPO — into the auto-created ReplicationGroupStatus CR (<protection-group-name>-replication in the zone namespace). On each status transition it emits an immutable RPOEvent record. The ReplicationGroupStatus.spec.rpoObjectiveSeconds value (derived from ProtectionGroup.spec.sla.rpoTargetSeconds) drives status.rpoStatus.

Step 4 — Planned failover

You create a FailoverRequest CR referencing the ProtectionGroup, specifying spec.targetCluster and spec.failoverType: planned. The failoverrequest reconciler:

1. Runs pre-failover hooks (Jobs on the target cluster).
2. Stops VMs on the source cluster.
3. Patches each member ProtectionRequest's spec.sourceCluster to the target cluster — this is the single field that constitutes the role flip.
4. The TSR-Manager control plane watches for DRBD Primary role to confirm on the target side via each cluster's DRBDResource status.
5. Starts VMs on the target cluster.
6. Runs post-failover hooks.
7. The FailoverRequest reaches state: Completed. Per-VM progress is visible in status.vmStatuses.

No step is automatic — every failover is initiated by creating a FailoverRequest. The TSR-Manager relays no storage traffic; DRBD replication is peer-to-peer between worker nodes on the two workload clusters.

Three-cluster topology is mandatory, not optional

The TSR-Manager cluster is a dedicated arbitrator: it holds all orchestration state, drives both workload clusters through kubeconfig Secrets, and runs no VM workloads. This separation means an outage on either workload cluster does not affect the TSR-Manager's ability to coordinate failover for the other. It also means the TSR-Manager's API server is never in the replication data path — DRBD traffic is peer-to-peer between worker nodes. The ProtectionZone CRD enforces exactly two workload clusters (minItems: 2, maxItems: 2) plus the TSR-Manager, with no override.

Helm charts are embedded in the operator binary

The three Helm charts (site-recovery-protectionzone-controller, site-recovery-manager-control-plane, site-recovery-workload-control-plane) are compiled into the ProtectionZone controller binary using go:embed. This means the controller is entirely self-contained: there is no chart repository, no registry pull at reconcile time, and no version skew between operator and chart. The ProtectionZone controller server-side-applies CRDs on every reconcile using field manager site-recovery-pz-controller with ForceOwnership, so out-of-band CRD edits are reverted within one reconcile cycle.

A separate binary for the ProtectionZone controller

The ProtectionZone controller runs as cmd/protectionzone, distinct from the TSR-Manager control plane (cmd/manager-control-plane). This is necessary because the TSR-Manager control plane is itself deployed per-zone by the ProtectionZone controller — a single binary could not both provision the deployment and be that deployment. The ProtectionZone controller's reconciler watches ProtectionZones cluster-wide (no namespace scope), while the TSR-Manager control plane is scoped to its zone namespace.

Workload control plane uses an unscoped cache

The workload control plane (cmd/workload-control-plane) deliberately does not scope its manager cache to a single namespace. Protected VMs and their associated DRBDResource CRs live in application namespaces across the cluster, not in a single operator namespace. Scoping to WATCH_NAMESPACE would leave the placement reconciler and the live-migration guard blind to VMs in other namespaces. WATCH_NAMESPACE is still validated for chart contract parity but is not applied to cache scoping.

VM pinning is a consequence of DRBD's block-device model

A protected VM is pinned to one node per cluster for the lifetime of the protection — nodeName is immutable once set. DRBD operates at the block-device layer on specific nodes; the metadata PVC, frontend PVC, and DRBD .res file are all node-local. Live migration and cold migration of a protected VM are denied by the workload control plane's admission webhooks. This also means oc adm drain of a node hosting a running protected VM will not complete.

Protocol C only; RPO is always 0

Synchronous replication (DRBD Protocol C) is the only supported mode. The admission webhook on ProtectionRequest rejects replicationProtocol: A with ReplicationProtocolA is not supported yet. The DRBDReplicationPolicy admission path accepts Protocol A at the schema level but the ProtectionRequest webhook is the enforcement point. A non-zero rpo with Protocol C is also rejected at admission. This means RPO is always 0 and is not tunable.

Zone-exclusivity lease prevents split-brain configuration

Each workload cluster can belong to only one zone at a time. When the ProtectionZone controller claims a workload cluster, it writes siterecovery.trilio.io/owner-zone-namespace-uid and siterecovery.trilio.io/owner-zone annotations on the trilio-site-recovery-system namespace using Kubernetes optimistic concurrency. A competing zone sees the annotation and fails with ClustersValidated=False, reason=ClusterAlreadyClaimed. The lease is never stolen — the controller reports the conflict and lets the operator resolve it.

Webhook certificates are managed by the operator, not cert-manager

The ProtectionZone controller provisions the webhook-server-cert Secret before deploying each stack. It re-issues the certificate whenever it is unparsable, missing a SAN, or within 30 days of expiry. There is no cert-manager integration and no customer action required for certificate lifecycle.

Scale ceiling of ~1000 protected VMs per deployment

The DRBD port allocator assigns one TCP port per replicated VM volume from the fixed range 7000–8000 inclusive (port 8000 is mandatory; a firewall rule that stops at 7999 silently prevents replication on one volume). This hard limits a single zone to approximately 1000 protected VMs. There is also a maximum of 15 replicated disks per VM. If your deployment requires more, you need multiple independent zones.

VM migration is not supported for protected VMs

Because DRBD operates at the node level, live migration and cold migration are denied at admission. If a node fails, the protected VM on that node cannot be migrated — the node-loss recovery path must be followed, which requires unprotecting and re-protecting the VM (with a full initial resync). This is a fundamental constraint of block-level replication and cannot be worked around within the current architecture.

Re-protection always requires a full resync

Unprotecting a VM tears down the DRBDResource pair. Re-protecting it allocates a new port, re-provisions DRBD metadata, and performs a full initial sync. There is no incremental re-attach path. Plan VM protection changes accordingly.

Upstream Kubernetes is not supported

The charts install on plain Kubernetes but the configuration is untested and unsupported. Three capabilities are OpenShift-specific and silently degrade on upstream Kubernetes: the DRBD node agent relies on the privileged SCC (on upstream Kubernetes you must supply your own Pod Security policy), the TSR Web Console requires an OpenShift Route and OAuth proxy (no ingress or authentication path is provided on upstream Kubernetes), and test-failover network isolation uses OVN-Kubernetes NetworkPolicy — on a non-OVN CNI, test VMs are not isolated and name resolution silently fails.

Single-Node OpenShift has no intra-cluster recovery

Single-Node OpenShift is supported for both workload and TSR-Manager clusters. However, if the single node of a workload cluster fails, the entire cluster side is lost. There is no alternative host to recover a copy onto within that cluster; node loss means following the node-loss recovery path.

DRBDReplicationPolicy is controller-managed

The default DRBDReplicationPolicy (<zoneName>-default, isDefault: true) is created and owned by the ProtectionZone controller. Only the isDefault policy in the ProtectionRequest's namespace is consulted. You cannot select a custom policy per ProtectionRequest in Tech Preview; custom policies are accepted by the API but the isDefault policy governs actual replication parameters.

FailoverRequest.spec.drainTimeoutSeconds has no effect

This field is accepted and range-checked by the admission webhook but is not consumed by the failover controller. It does not affect drain or failover behaviour and should not be relied upon.

Test failover is DRBD backend only

The TestFailover resource spins up isolated VM copies from DRBD snapshots in a controller-generated namespace (site-recovery-<protection-group>-<random> reported in status.testNamespace). This capability requires the DRBD storage backend. The test namespace is never user-provided and should not be constructed or patched manually.

The 5-minute failover wall-clock budget

The failover controller operates within a 5-minute wall-clock budget. Deployments with many VMs in a ProtectionGroup, slow storage, or slow hook Jobs may approach this limit. Design your ProtectionGroups and hooks to complete within this window.

RHEL CoreOS 10 nodes are not validated

The API accepts spec.clusters[].distributionName: RHEL-CoreOS-10 and an el10 module-loader image is built, but el10 nodes are not validated in Tech Preview. Do not use RHEL CoreOS 10 nodes on workload clusters. Mixed OS families within a single cluster are also not supported — a single module-loader image is deployed per cluster.