--- title: Problem Being Solved product: trilio-site-recovery-for-kubernetes doc_type: concept version: 0.2.0 source: git2docs (code-derived, validation-filtered) canonical: https://docs-tsr.trilio.io/trilio-site-recovery-for-kubernetes/site-recovery/problem-statement --- # Problem Being Solved _What operational challenge this operator automates around site recovery and disaster recovery_ ## Overview This page explains the operational problem that Trilio Site Recovery solves: what happens to KubeVirt virtual machines when a cluster or site fails, why conventional Kubernetes mechanisms are insufficient for that scenario, and how Site Recovery automates the recovery workflow through a Kubernetes-native, CRD-driven operator. Understanding this context helps you make informed decisions about topology, replication configuration, and recovery procedures before you deploy. ## Content ## The gap between Kubernetes resilience and site-level recovery Kubernetes is designed to recover from node failures within a cluster. When a node becomes unavailable, the scheduler reschedules stateless pods onto healthy nodes within minutes. For KubeVirt virtual machines, however, this self-healing does not extend across cluster boundaries, and it does not protect the storage that backs a VM's disks. A KubeVirt VM's state lives in persistent volumes. If the cluster hosting those volumes becomes unavailable — because of a network partition, a datacenter outage, or a catastrophic hardware failure — Kubernetes has no built-in mechanism to: - Replicate the VM's block-level disk state to a second cluster in real time - Promote a standby copy of that disk to writeable at the secondary site - Start the VM at the secondary site with its full disk state intact - Do any of this within a recovery-time window that operational SLAs demand The result is that platform engineers managing KubeVirt workloads face a manual, error-prone recovery process under exactly the conditions when mistakes are most costly. ## What Site Recovery automates Site Recovery addresses this gap by continuously replicating the block devices backing each protected VM from a primary (active) workload cluster to a DR (standby) workload cluster using DRBD synchronous replication (Protocol C). Because Protocol C is synchronous, a write is not acknowledged to the guest until both sides confirm it — the DR copy is always current, and RPO is always zero. When a failover is needed, whether planned or unplanned, an operator creates a `FailoverRequest` custom resource. The TSR-Manager control plane then: 1. Stops the VM on the primary cluster (for a planned failover) or skips that step (for an unplanned one) 2. Promotes the DRBD device on the DR cluster from Secondary to Primary 3. Starts the VM on the DR cluster backed by its now-writeable disk copy 4. Runs any configured lifecycle hooks (pre-failover, post-failover) in the declared order Every step is declarative and observable through standard Kubernetes APIs. You can drive the same operations from `kubectl`, from a GitOps pipeline, or from the TSR web console on OpenShift. ## Why a dedicated management cluster is required Coordinating a failover requires a control plane that survives the failure of either workload cluster. A controller running on the primary cluster cannot reliably orchestrate its own failover, and a controller running on the DR cluster cannot reach the primary to verify its state when the primary is down. Site Recovery solves this with a dedicated TSR-Manager cluster — a third Kubernetes cluster that: - Hosts all orchestration controllers (protection, failover, test-failover, replication monitoring) - Stores kubeconfig credentials for reaching both workload clusters - Holds all TSR-Manager-side custom resources: `ProtectionZone`, `ProtectionGroup`, `ProtectionRequest`, `FailoverRequest`, `TestFailover`, `ReplicationGroupStatus`, `RPOEvent`, and `Hook` - Carries no VM workloads and relays no DRBD replication traffic The TSR-Manager cluster is not optional. It must be reachable from both workload clusters at all times, and its own availability directly determines whether orchestrated failover is possible. ## The operational challenges this addresses ### Continuous replication with zero manual intervention With unprotected VMs, achieving cross-cluster disk replication requires manual configuration of DRBD resources, connection endpoints, storage class mappings, and protocol parameters on both clusters — and keeping that configuration consistent as VMs are added, modified, or removed. Site Recovery automates this: creating a `ProtectionRequest` for a VM causes the controller to provision a `DRBDResource` on both workload clusters, wire the VM to DRBD-backed frontend PVCs, and drive DRBD to a fully synchronized `Protected` phase without any manual storage configuration. ### Coordinated multi-VM recovery Production workloads rarely consist of a single VM. A multi-tier application may have database VMs, application VMs, and load-balancer VMs that must fail over together and start in a specific order. A `ProtectionGroup` groups these VMs as an atomic unit, defines their boot ordering, and binds lifecycle hooks that run pre- and post-failover steps (such as notifying external systems or validating application health). A `FailoverRequest` against the group moves all members together. ### Validating DR readiness without disrupting production One of the hardest problems in disaster recovery is knowing whether your DR plan will work before you need it. Site Recovery provides a `TestFailover` mechanism that spins up isolated copies of protected VMs on the standby cluster using DRBD snapshots, inside a controller-generated namespace isolated by a default-deny `NetworkPolicy`. The test VMs run independently of production, can be validated through reachability probes and hook-driven verification jobs, and are cleaned up afterward. Production VMs are never touched. This makes regular DR drills operationally viable rather than disruptive. ### Consistent credential and access management Cross-cluster access requires kubeconfig credentials that must be provisioned, rotated, and revoked in a controlled way. The `tsrctl` CLI manages these zone credentials against per-zone `ServiceAccount`s in the `trilio-site-recovery-access` namespace on each workload cluster. Credentials are bound to a specific cluster by its `kube-system` Namespace UID, preventing accidental redirection to a different cluster. ### Observable replication health Before a failover, you need confidence that the DR copy is current and promotable. The replication monitor controller continuously polls DRBD state from the active workload cluster and writes aggregate metrics into a `ReplicationGroupStatus` CR per `ProtectionGroup`. When replication transitions occur — sync completed, connection lost, connection restored — the monitor emits immutable `RPOEvent` audit records. These feed the TSR web console dashboards on OpenShift and are queryable with `kubectl` at any time. ## Constraints that shape how you operate protected VMs Site Recovery's synchronous replication model imposes constraints you must plan for before protecting a VM: - **Node pinning:** A protected VM is pinned to one node per cluster for the lifetime of the protection. Live migration and cold migration of a protected VM are denied. Draining a node that hosts a running protected VM will not complete until the VM is unprotected first. - **Disk topology is frozen:** No disk additions, removals, resizing, or online volume expansion are permitted while a VM is protected. Re-protection requires a full unprotect followed by a complete initial resync. - **Round-trip latency:** Protocol C synchronous replication requires less than 10 ms round-trip latency between the primary and DR worker nodes. DRBD replication uses TCP ports 7000–8000 inclusive (one port per replicated VM volume); port 8000 is not optional. - **Scale ceiling:** A deployment supports approximately 1,000 protected VMs (one DRBD port per VM from the 7000–8000 range), up to 15 replicated disks per VM, and exactly two workload clusters per zone. - **Failover is always operator-initiated:** No failover happens automatically. Every failover — planned, unplanned, or test — is triggered by creating the appropriate custom resource. This is a deliberate design choice: automatic failover in a split-brain scenario can cause data loss or double-promotion, which is worse than the outage it tries to prevent. ## Tech Preview status Trilio Site Recovery is currently in Tech Preview. All capabilities described on this page are available and functional, but the product is tested on a specific set of configurations. DRBD with Protocol C synchronous replication is the only supported storage backend mode. OpenShift 4.17 and later is the validated platform. The TSR web console is available on OpenShift only, through an OAuth-proxy Route created automatically by the ProtectionZone controller. ## Examples The following examples illustrate the core custom resources that express the problem Site Recovery solves. They are not installation steps — see the installation workflow for the sequence in which to create these resources. **A ProtectionZone pairs two workload clusters with the TSR-Manager** ```yaml apiVersion: siterecovery.trilio.io/v1alpha1 kind: ProtectionZone metadata: name: prod-zone namespace: dr-prod-zone spec: displayName: "Production DR Zone" storageBackendMode: DRBD clusters: - name: cluster1 clusterID: cluster1 kubeconfigSecretRef: name: cluster1-kubeconfig key: kubeconfig - name: cluster2 clusterID: cluster2 kubeconfigSecretRef: name: cluster2-kubeconfig key: kubeconfig ``` **A ProtectionRequest declares that a single VM should be protected** Creating this resource causes the controller to provision DRBD replication for `my-database-vm` and drive it to `Protected` phase. The `sourceCluster` field identifies which cluster is currently active — patching it is how a per-VM role flip (failover) is expressed. ```yaml apiVersion: siterecovery.trilio.io/v1alpha1 kind: ProtectionRequest metadata: name: protect-my-database-vm namespace: dr-prod-zone spec: sourceCluster: cluster1 virtualMachine: name: my-database-vm namespace: production ``` **Watch the protection lifecycle progress** ```bash kubectl get pr protect-my-database-vm -n dr-prod-zone -w ``` The `phase` column progresses through: `Pending` → `Validating` → `CreatingDRBD` → `Syncing` → `Protected`. **A FailoverRequest moves an entire ProtectionGroup to the DR cluster** This is the single API call that triggers a planned failover of all VMs in the group. No manual steps are required before or after — the controller handles VM shutdown, DRBD promotion, and VM start in order. ```yaml apiVersion: siterecovery.trilio.io/v1alpha1 kind: FailoverRequest metadata: name: planned-failover-to-dr namespace: dr-prod-zone spec: protectionGroupRef: name: my-production-pg namespace: dr-prod-zone targetCluster: cluster2 failoverType: planned ``` **A TestFailover validates DR readiness without touching production VMs** ```yaml apiVersion: siterecovery.trilio.io/v1alpha1 kind: TestFailover metadata: name: quarterly-dr-drill namespace: dr-prod-zone spec: protectionGroupRef: name: my-production-pg namespace: dr-prod-zone cleanupPolicy: Manual retentionTime: 4h ``` After creation, the controller reports the isolated test namespace in `status.testNamespace`. The test progresses through phases — `Queued` → `CreatingSnapshots` → `CreatingVolumes` → `CreatingVMs` → `StartingVMs` → `Verifying` → `Succeeded` (or `Failed`) — without any production VM being paused or restarted. **Check replication health for a ProtectionGroup** The `ReplicationGroupStatus` CR is auto-created by the replication monitor. Its short name is `rgs`. ```bash kubectl get rgs my-production-pg-replication -n dr-prod-zone -o yaml ``` **List recent replication events for compliance review** ```bash kubectl get rpoe -n dr-prod-zone -l siterecovery.trilio.io/protection-group=my-production-pg ``` ## Related concepts - **Install and configure the control plane** — The sequence of steps to deploy the ProtectionZone controller via OLM, provision zone credentials, and create a ProtectionZone that registers your two workload clusters. - **ProtectionZone** — The top-level configuration object that pairs your primary and DR workload clusters and triggers automated deployment of the workload control plane stacks. - **ProtectionRequest** — The per-VM CR that declares protection and drives the DRBD replication lifecycle from initial provisioning through the `Protected` phase. - **ProtectionGroup** — The multi-VM grouping CR that enables coordinated failover with boot ordering and lifecycle hooks. - **Monitor replication health** — How to interpret `ReplicationGroupStatus` and `RPOEvent` CRs to assess whether your DR copy is current and your recovery point objective is being met. - **Run a test failover** — The step-by-step workflow for creating a `TestFailover`, monitoring its phases, interpreting verification results, and cleaning up test resources. - **Execute a planned failover** — The workflow for creating a `FailoverRequest` with `failoverType: planned`, monitoring `vmStatuses`, and confirming the target cluster is active. - **Execute an unplanned failover** — How the same `FailoverRequest` API expresses a force-promote failover when the primary site is unavailable.