--- title: Architecture product: trilio-site-recovery-for-kubernetes doc_type: architecture version: 0.2.0 source: git2docs (code-derived, validation-filtered) canonical: https://docs-tsr.trilio.io/trilio-site-recovery-for-kubernetes/site-recovery/architecture --- # Architecture _Controller architecture, reconciliation loops, and component interaction for site recovery_ ## Overview This page describes the internal architecture of Trilio Site Recovery: how its Kubernetes operators, agents, webhooks, and custom resources are organized across three clusters, how reconciliation loops drive DR state forward, and how components interact during normal operation and failover. Understanding this architecture helps you reason about what runs where, why certain constraints exist (such as VM pinning and the three-cluster requirement), and how to interpret status conditions and logs when diagnosing problems. ## Architecture diagram ```mermaid flowchart TD subgraph TSR-Manager Cluster PZC["ProtectionZone Controller\n(site-recovery-protectionzone-controller)"] MCP["TSR-Manager Control Plane\n(site-recovery-manager-control-plane)"] UI["TSR Web Console\n(site-recovery-frontend + site-recovery-backend)"] PZ[(ProtectionZone CR)] PG[(ProtectionGroup CR)] PR[(ProtectionRequest CR)] FR[(FailoverRequest CR)] TFO[(TestFailover CR)] RGS[(ReplicationGroupStatus CR)] RPOE[(RPOEvent CR)] DRP[(DRBDReplicationPolicy CR)] Hook[(Hook CR)] KCS[(kubeconfig Secrets)] end subgraph "Workload Cluster 1 (Primary)" WCP1["Workload Control Plane\n(site-recovery-workload-control-plane)"] NA1["drbd-node-agent DaemonSet"] DRES1[(DRBDResource CRs)] VM1[KubeVirt VMs] PVC1[PVCs / PVs] end subgraph "Workload Cluster 2 (DR)" WCP2["Workload Control Plane\n(site-recovery-workload-control-plane)"] NA2["drbd-node-agent DaemonSet"] DRES2[(DRBDResource CRs)] VM2[KubeVirt VMs] PVC2[PVCs / PVs] end User([Platform Engineer / SRE]) -->|kubectl / oc / UI| TSR-Manager Cluster PZC -->|Helm deploy| WCP1 PZC -->|Helm deploy| WCP2 PZC -->|Helm deploy| MCP PZC -->|reads| KCS MCP -->|creates/updates via kubeconfig| DRES1 MCP -->|creates/updates via kubeconfig| DRES2 MCP -->|reconciles| PR MCP -->|reconciles| PG MCP -->|reconciles| FR MCP -->|reconciles| TFO MCP -->|auto-creates| RGS MCP -->|emits| RPOE WCP1 -->|admits/places| DRES1 WCP1 -->|guards| VM1 NA1 -->|owns DRBD kernel state| PVC1 WCP2 -->|admits/places| DRES2 WCP2 -->|guards| VM2 NA2 -->|owns DRBD kernel state| PVC2 NA1 <-->|DRBD Protocol C\nTCP 7000-8000| NA2 UI -->|REST API| MCP ``` ## Components **ProtectionZone Controller** (`app.kubernetes.io/component: protectionzone-controller`) Runs on the TSR-Manager cluster in a dedicated Deployment. This is the bootstrap operator: it validates both workload cluster credentials, acquires an exclusivity lease on each workload cluster's `trilio-site-recovery-system` namespace, and then Helm-installs the TSR-Manager and workload control plane stacks from charts embedded directly in its binary. It also provisions the zone's default `DRBDReplicationPolicy` (`-default` with `isDefault: true`). The ProtectionZone controller runs as its own binary (`cmd/protectionzone`) because the per-zone control planes it deploys depend on it — it cannot itself be one of those per-zone deployments. **TSR-Manager Control Plane** (`app.kubernetes.io/component: manager-control-plane`) Runs on the TSR-Manager cluster in the zone namespace (`dr-`). This is the central orchestrator: it hosts the reconcilers for `ProtectionRequest`, `ProtectionGroup`, `FailoverRequest`, `TestFailover`, `ReplicationGroupStatus`, `RPOEvent`, and `DRBDReplicationPolicy`. It reaches workload clusters exclusively through the kubeconfig Secrets stored in the zone namespace — it never receives inbound traffic from workload clusters. A single container named `manager` hosts all reconcilers; the pod shows `1/1`. The manager binary is `/tsr-manager`. **Workload Control Plane** (`app.kubernetes.io/component: workload-control-plane`) Runs on each workload cluster in the `trilio-site-recovery-system` namespace. It hosts the `DRBDResource` placement reconciler (node binding, backing PVC provisioning, anchor pod management), the admission webhooks for `DRBDResource` and KubeVirt `VirtualMachine` objects, and exposes Prometheus metrics. It talks only to its own cluster's API server — it never contacts the TSR-Manager. The binary is `/workload-manager`. **drbd-node-agent DaemonSet** (`app.kubernetes.io/component: drbd-node-agent`) Runs one pod per schedulable worker node on each workload cluster. Named `-site-recovery-workload-control-plane-agent`. This is the only component that touches the DRBD kernel module: it writes `.res` configuration files, calls `drbdadm`, provisions frontend PV-PVCs and DRBD metadata PVCs for the DRBDResources bound to its node, and manages the `siterecovery.trilio.io/not-ready:NoSchedule` startup taint. It runs with `privileged: true` and `hostNetwork: true` and is bound to the `privileged` SCC on OpenShift. The agent binary is part of the `site-recovery-drbd-node-agent` image; userspace DRBD tools are shipped inside that image. **TSR Web Console** (`site-recovery-frontend` + `site-recovery-backend`) Deployed via OLM on the TSR-Manager cluster. Provides dashboards for zone health, VM protection status, replication metrics, and one-click failover and test-failover operations. On OpenShift it is exposed through an OpenShift Route at `https://trilio-site-recovery.`; browser access is authenticated via OpenShift OAuth. On upstream Kubernetes no ingress or authentication path is provided. **tsrctl** The only Site Recovery CLI tool. Supplied by Trilio with the release; contact Trilio support to obtain it. Manages zone kubeconfig credentials exclusively: `zone-credential create`, `rotate`, `inspect`, and `delete`. It does not drive DR operations — failover, protection, and test failover are all performed through CRDs. **tsr-gather (site-recovery-must-gather image)** Run via `oc adm must-gather` on the TSR-Manager cluster. Collects logs, CRs, and DRBD state from the TSR-Manager and all zone workload clusters into a single read-only archive. The entrypoint inside the image is `/usr/bin/gather`. ## Data flow The following traces what happens when you protect a VM, illustrating how control flows through all three clusters. **Step 1 — Zone provisioning (done once per zone)** You create a `ProtectionZone` CR in a `dr-` namespace on the TSR-Manager cluster. The ProtectionZone controller: 1. Reads the two kubeconfig Secrets named in `spec.clusters[].kubeconfigSecretRef`. 2. Probes both workload clusters (connectivity, authentication, RBAC via `SelfSubjectAccessReview`, KubeVirt presence, OpenShift SCC prerequisites). 3. Acquires a zone-exclusivity lease on each workload cluster's `trilio-site-recovery-system` namespace by writing `siterecovery.trilio.io/owner-zone-namespace-uid` and `siterecovery.trilio.io/owner-zone` annotations. 4. Helm-installs `site-recovery-manager-control-plane` into `dr-` on the TSR-Manager cluster and `site-recovery-workload-control-plane` into `trilio-site-recovery-system` on both workload clusters (from charts embedded in the controller binary — no registry pull). 5. Creates the default `DRBDReplicationPolicy` (`-default`) in the zone namespace. 6. Sets `ClustersValidated=True` on the zone. Only then do webhooks on the TSR-Manager cluster accept new `ProtectionGroup` or `ProtectionRequest` creates. **Step 2 — VM protection** You create a `ProtectionRequest` CR in the zone namespace, specifying the VM name, namespace, and `spec.sourceCluster`. The TSR-Manager control plane's `protectionrequest` reconciler: 1. Looks up the VM on the source workload cluster through its cached client. 2. Selects the default `DRBDReplicationPolicy` for the zone and reads DRBD parameters. 3. Creates a `DRBDResource` CR in the VM's namespace on **both** workload clusters. The CR name follows the pattern `drbd---`. 4. The workload control plane on each cluster picks up its local `DRBDResource`, runs the placement reconciler to bind a node, and creates the anchor pod and DRBD metadata PVCs. 5. The drbd-node-agent on the bound node writes the DRBD `.res` file, calls `drbdadm up`, and provisions the frontend PVC. The agent removes the `siterecovery.trilio.io/not-ready:NoSchedule` taint from the node once DRBD converges (or the startup deadline passes). 6. The two DRBD node agents establish a TCP connection on a port in the range 7000–8000 (one port allocated per replicated VM volume). Replication uses Protocol C (synchronous, RPO = 0) exclusively. 7. As DRBD reaches `Connected` state the `ProtectionRequest` transitions through `Pending → Validating → CreatingDRBD → Syncing → Protected`. **Step 3 — Replication monitoring** The `replicationmonitor` reconciler in the TSR-Manager control plane polls each workload cluster at the `ReplicationGroupStatus.spec.pollingIntervalSeconds` interval (default 15 s). It aggregates per-VM DRBD metrics — out-of-sync bytes, connection state, current RPO — into the auto-created `ReplicationGroupStatus` CR (`-replication` in the zone namespace). On each status transition it emits an immutable `RPOEvent` record. The `ReplicationGroupStatus.spec.rpoObjectiveSeconds` value (derived from `ProtectionGroup.spec.sla.rpoTargetSeconds`) drives `status.rpoStatus`. **Step 4 — Planned failover** You create a `FailoverRequest` CR referencing the `ProtectionGroup`, specifying `spec.targetCluster` and `spec.failoverType: planned`. The `failoverrequest` reconciler: 1. Runs pre-failover hooks (Jobs on the target cluster). 2. Stops VMs on the source cluster. 3. Patches each member `ProtectionRequest`'s `spec.sourceCluster` to the target cluster — this is the single field that constitutes the role flip. 4. The TSR-Manager control plane watches for DRBD Primary role to confirm on the target side via each cluster's `DRBDResource` status. 5. Starts VMs on the target cluster. 6. Runs post-failover hooks. 7. The `FailoverRequest` reaches `state: Completed`. Per-VM progress is visible in `status.vmStatuses`. No step is automatic — every failover is initiated by creating a `FailoverRequest`. The TSR-Manager relays no storage traffic; DRBD replication is peer-to-peer between worker nodes on the two workload clusters. ## Design decisions **Three-cluster topology is mandatory, not optional** The TSR-Manager cluster is a dedicated arbitrator: it holds all orchestration state, drives both workload clusters through kubeconfig Secrets, and runs no VM workloads. This separation means an outage on either workload cluster does not affect the TSR-Manager's ability to coordinate failover for the other. It also means the TSR-Manager's API server is never in the replication data path — DRBD traffic is peer-to-peer between worker nodes. The `ProtectionZone` CRD enforces exactly two workload clusters (`minItems: 2`, `maxItems: 2`) plus the TSR-Manager, with no override. **Helm charts are embedded in the operator binary** The three Helm charts (`site-recovery-protectionzone-controller`, `site-recovery-manager-control-plane`, `site-recovery-workload-control-plane`) are compiled into the ProtectionZone controller binary using `go:embed`. This means the controller is entirely self-contained: there is no chart repository, no registry pull at reconcile time, and no version skew between operator and chart. The ProtectionZone controller server-side-applies CRDs on every reconcile using field manager `site-recovery-pz-controller` with `ForceOwnership`, so out-of-band CRD edits are reverted within one reconcile cycle. **A separate binary for the ProtectionZone controller** The ProtectionZone controller runs as `cmd/protectionzone`, distinct from the TSR-Manager control plane (`cmd/manager-control-plane`). This is necessary because the TSR-Manager control plane is itself deployed per-zone by the ProtectionZone controller — a single binary could not both provision the deployment and be that deployment. The ProtectionZone controller's reconciler watches ProtectionZones cluster-wide (no namespace scope), while the TSR-Manager control plane is scoped to its zone namespace. **Workload control plane uses an unscoped cache** The workload control plane (`cmd/workload-control-plane`) deliberately does not scope its manager cache to a single namespace. Protected VMs and their associated `DRBDResource` CRs live in application namespaces across the cluster, not in a single operator namespace. Scoping to `WATCH_NAMESPACE` would leave the placement reconciler and the live-migration guard blind to VMs in other namespaces. `WATCH_NAMESPACE` is still validated for chart contract parity but is not applied to cache scoping. **VM pinning is a consequence of DRBD's block-device model** A protected VM is pinned to one node per cluster for the lifetime of the protection — `nodeName` is immutable once set. DRBD operates at the block-device layer on specific nodes; the metadata PVC, frontend PVC, and DRBD `.res` file are all node-local. Live migration and cold migration of a protected VM are denied by the workload control plane's admission webhooks. This also means `oc adm drain` of a node hosting a running protected VM will not complete. **Protocol C only; RPO is always 0** Synchronous replication (DRBD Protocol C) is the only supported mode. The admission webhook on `ProtectionRequest` rejects `replicationProtocol: A` with `ReplicationProtocolA is not supported yet`. The `DRBDReplicationPolicy` admission path accepts Protocol A at the schema level but the `ProtectionRequest` webhook is the enforcement point. A non-zero `rpo` with Protocol C is also rejected at admission. This means RPO is always 0 and is not tunable. **Zone-exclusivity lease prevents split-brain configuration** Each workload cluster can belong to only one zone at a time. When the ProtectionZone controller claims a workload cluster, it writes `siterecovery.trilio.io/owner-zone-namespace-uid` and `siterecovery.trilio.io/owner-zone` annotations on the `trilio-site-recovery-system` namespace using Kubernetes optimistic concurrency. A competing zone sees the annotation and fails with `ClustersValidated=False, reason=ClusterAlreadyClaimed`. The lease is never stolen — the controller reports the conflict and lets the operator resolve it. **Webhook certificates are managed by the operator, not cert-manager** The ProtectionZone controller provisions the `webhook-server-cert` Secret before deploying each stack. It re-issues the certificate whenever it is unparsable, missing a SAN, or within 30 days of expiry. There is no cert-manager integration and no customer action required for certificate lifecycle. ## Trade-offs **Scale ceiling of ~1000 protected VMs per deployment** The DRBD port allocator assigns one TCP port per replicated VM volume from the fixed range 7000–8000 inclusive (port 8000 is mandatory; a firewall rule that stops at 7999 silently prevents replication on one volume). This hard limits a single zone to approximately 1000 protected VMs. There is also a maximum of 15 replicated disks per VM. If your deployment requires more, you need multiple independent zones. **VM migration is not supported for protected VMs** Because DRBD operates at the node level, live migration and cold migration are denied at admission. If a node fails, the protected VM on that node cannot be migrated — the node-loss recovery path must be followed, which requires unprotecting and re-protecting the VM (with a full initial resync). This is a fundamental constraint of block-level replication and cannot be worked around within the current architecture. **Re-protection always requires a full resync** Unprotecting a VM tears down the `DRBDResource` pair. Re-protecting it allocates a new port, re-provisions DRBD metadata, and performs a full initial sync. There is no incremental re-attach path. Plan VM protection changes accordingly. **Upstream Kubernetes is not supported** The charts install on plain Kubernetes but the configuration is untested and unsupported. Three capabilities are OpenShift-specific and silently degrade on upstream Kubernetes: the DRBD node agent relies on the `privileged` SCC (on upstream Kubernetes you must supply your own Pod Security policy), the TSR Web Console requires an OpenShift Route and OAuth proxy (no ingress or authentication path is provided on upstream Kubernetes), and test-failover network isolation uses OVN-Kubernetes NetworkPolicy — on a non-OVN CNI, test VMs are not isolated and name resolution silently fails. **Single-Node OpenShift has no intra-cluster recovery** Single-Node OpenShift is supported for both workload and TSR-Manager clusters. However, if the single node of a workload cluster fails, the entire cluster side is lost. There is no alternative host to recover a copy onto within that cluster; node loss means following the node-loss recovery path. **DRBDReplicationPolicy is controller-managed** The default `DRBDReplicationPolicy` (`-default`, `isDefault: true`) is created and owned by the ProtectionZone controller. Only the `isDefault` policy in the `ProtectionRequest`'s namespace is consulted. You cannot select a custom policy per `ProtectionRequest` in Tech Preview; custom policies are accepted by the API but the isDefault policy governs actual replication parameters. **`FailoverRequest.spec.drainTimeoutSeconds` has no effect** This field is accepted and range-checked by the admission webhook but is not consumed by the failover controller. It does not affect drain or failover behaviour and should not be relied upon. **Test failover is DRBD backend only** The `TestFailover` resource spins up isolated VM copies from DRBD snapshots in a controller-generated namespace (`site-recovery--` reported in `status.testNamespace`). This capability requires the DRBD storage backend. The test namespace is never user-provided and should not be constructed or patched manually. **The 5-minute failover wall-clock budget** The failover controller operates within a 5-minute wall-clock budget. Deployments with many VMs in a ProtectionGroup, slow storage, or slow hook Jobs may approach this limit. Design your ProtectionGroups and hooks to complete within this window. **RHEL CoreOS 10 nodes are not validated** The API accepts `spec.clusters[].distributionName: RHEL-CoreOS-10` and an el10 module-loader image is built, but el10 nodes are not validated in Tech Preview. Do not use RHEL CoreOS 10 nodes on workload clusters. Mixed OS families within a single cluster are also not supported — a single module-loader image is deployed per cluster.