--- title: Recovery Workflow product: trilio-site-recovery-for-kubernetes doc_type: guide version: 0.2.0 source: git2docs (code-derived, validation-filtered) canonical: https://docs-tsr.trilio.io/trilio-site-recovery-for-kubernetes/site-recovery/recovery-workflow --- # Recovery Workflow _End-to-end flow from triggering a recovery to completion, including state transitions_ ## Overview This page walks you through every step of the Site Recovery recovery workflow: from triggering a failover or test failover, through state transitions, to completion and optional failback. Understanding this workflow lets you execute planned failovers, respond to unplanned disasters, validate DR readiness without disrupting production, and return workloads to their original cluster after recovery. Every operation is declared through Kubernetes custom resources in the `siterecovery.trilio.io/v1alpha1` API group and can be driven by `kubectl` or the TSR web console. ## Prerequisites Before executing any recovery operation, confirm the following: - A TSR-Manager cluster running the Site Recovery control plane (required, not optional) - Two workload clusters registered in a `ProtectionZone` with `status.phase: Available` - The `ProtectionZone` condition `ClustersValidated=True` — webhooks block all ProtectionGroup and ProtectionRequest creates until this condition is met - At least one `ProtectionGroup` with member VMs in `status.state: Protected` - Each member VM has a corresponding `ProtectionRequest` in `status.phase: Protected` - `kubectl` configured to reach the TSR-Manager cluster - For test failovers: DRBD storage backend mode (TestFailover is supported in DRBD mode only) - For the TSR web console: OpenShift 4.17 or later on the TSR-Manager cluster (the console is OpenShift-only via an oauth-proxy Route) - `tsrctl` available — contact Trilio support to obtain it for your release - Firewall rules permitting TCP 7000–8000 inclusive between primary and DR worker nodes (one port per replicated VM; port 8000 is mandatory) ## Configuration Recovery operations are controlled by the fields on each custom resource. The key configuration points are: ### ProtectionGroup — grouping and boot order | Field | Default | Effect | |---|---|---| | `spec.virtualMachines[].bootOrder` | none | Assigns VMs to ordered boot batches during failover. VMs without a `bootOrder` form the final batch. | | `spec.virtualMachines[].name` / `.namespace` | required | Identifies each member VM. | | `spec.sla.rpoTargetSeconds` | 0 | RPO objective in seconds. Consumed by the replication monitor; Protocol C delivers RPO=0 and this field cannot tune that. | | `spec.sla.rtoTargetSeconds` | required | RTO objective in seconds. Recorded for compliance reporting. | | `spec.sourceCluster` | none | Protect-time home cluster. Used by the hook system to classify failover direction (failover vs failback). | | `spec.vmIdentity.macAddresses` | `Preserve` | Whether MAC addresses are preserved (`Preserve`) or cleared (`Clear`) on test-failover clones. | | `spec.hooks` | none | Lifecycle hook bindings. See the Hook CR for the job template. | ### FailoverRequest — planned vs unplanned | Field | Default | Effect | |---|---|---| | `spec.failoverType` | `planned` | `planned` requires the source cluster reachable and stops VMs gracefully. `unplanned` tolerates a dead source and treats its VMs as stopped. | | `spec.targetCluster` | required | `cluster1` or `cluster2` — the cluster to fail over to. | | `spec.batchBootTimeoutSeconds` | 300 | Per boot-order batch deadline. When a batch does not reach Running within this window, the controller proceeds to the next batch with a warning. | | `spec.skipHooks` | false | When true, all lifecycle hooks are skipped for this operation. | `spec.drainTimeoutSeconds` is accepted and range-checked by the admission webhook but is not consumed by the failover controller in Tech Preview; omit it from manifests. ### TestFailover — DR drill settings | Field | Default | Effect | |---|---|---| | `spec.cleanupPolicy` | `Automatic` | `Automatic` removes test resources after `spec.retentionTime` elapses. `Manual` requires an explicit cleanup request. | | `spec.retentionTime` | `2h` | How long test resources are kept before automatic cleanup. | | `spec.schedulingCheck` | `Fail` | `Fail` excludes VMs the DR scheduler cannot place. `Warn` records the failure but continues the drill. | | `spec.batchBootTimeoutSeconds` | 300 | Per boot-order batch deadline for test VM startup. | | `spec.verification.reachability` | none | Optional TCP or HTTP reachability probes run against test VMs during the Verifying phase. | ### ProtectionZone — node agent startup | Field | Default | Effect | |---|---|---| | `spec.nodeAgent.startupDeadlineSeconds` | 120 | How long the DRBD node agent taint (`siterecovery.trilio.io/not-ready:NoSchedule`) persists while waiting for DRBD resources to converge on a node. | | `spec.logging.level` | `info` | Log level for zone components. Accepted values: `error`, `info`, `debug`, `trace`. | ### Runtime log level (without restart) You can change log levels at runtime without restarting pods by patching the `site-recovery-logging` ConfigMap in the relevant namespace: ```bash # Increase TSR-Manager control plane verbosity kubectl -n dr-prod patch configmap site-recovery-logging --type=merge \ -p '{"data":{"level":"debug","level.control-plane":"trace"}}' # Increase node agent verbosity on a workload cluster kubectl -n trilio-site-recovery-system patch configmap site-recovery-logging \ --type=merge -p '{"data":{"level.drbd-node-agent":"debug"}}' ``` ## Usage ### Protect a virtual machine Create a `ProtectionRequest` on the TSR-Manager cluster in the zone namespace (`dr-`). The VM must be running on the source cluster when you protect it — the source node is captured at protect time and is immutable for the lifetime of the protection. ```yaml apiVersion: siterecovery.trilio.io/v1alpha1 kind: ProtectionRequest metadata: name: web-vm-pr namespace: dr-prod spec: sourceCluster: cluster1 virtualMachine: name: web-vm namespace: app-ns ``` ```bash kubectl apply -f protection-request.yaml ``` Watch the protection lifecycle: ```bash kubectl -n dr-prod get pr web-vm-pr -w ``` Phases: `Pending` → `Validating` → `CreatingDRBD` → `Syncing` → `Protected` ### Monitor replication health The replication monitor auto-creates a `ReplicationGroupStatus` named `-replication` in the zone namespace: ```bash kubectl -n dr-prod get rgs kubectl -n dr-prod describe rgs my-pg-replication ``` Check RPO events (immutable audit records): ```bash kubectl -n dr-prod get rpoe ``` Inspect DRBD state on a workload cluster (DRBDResource lives on the workload cluster, not TSR-Manager): ```bash # Run this against the workload cluster, not TSR-Manager kubectl --kubeconfig ~/.kube/workload-a get dres -n app-ns ``` ### Run a test failover (DR drill) A TestFailover spins up isolated VM copies from DRBD snapshots in a network-fenced namespace on the standby cluster. It does not touch production VMs. ```yaml apiVersion: siterecovery.trilio.io/v1alpha1 kind: TestFailover metadata: name: drill-2024-q4 namespace: dr-prod spec: protectionGroupRef: name: my-pg namespace: dr-prod cleanupPolicy: Manual retentionTime: 4h schedulingCheck: Fail ``` ```bash kubectl apply -f test-failover.yaml kubectl -n dr-prod get tfo drill-2024-q4 -w ``` Phases: `Queued` → `PreparingEnvironment` → `CreatingSnapshots` → `CreatingVolumes` → `CreatingVMs` → `StartingVMs` → `Verifying` → `Succeeded` (or `Failed`) The generated test namespace is reported in `status.testNamespace`. To trigger cleanup after a Manual policy test: ```bash kubectl -n dr-prod patch tfo drill-2024-q4 --type=merge \ -p '{"spec":{"cleanupRequested":true}}' ``` ### Execute a planned failover A planned failover gracefully stops VMs on the source cluster, waits for DRBD locks to settle, then starts VMs on the target. Every failover requires a `FailoverRequest`; failover is never automatic. ```yaml apiVersion: siterecovery.trilio.io/v1alpha1 kind: FailoverRequest metadata: name: failover-to-dr namespace: dr-prod spec: protectionGroupRef: name: my-pg namespace: dr-prod targetCluster: cluster2 failoverType: planned ``` ```bash kubectl apply -f failover-request.yaml kubectl -n dr-prod get fr failover-to-dr -w ``` Phases: `Validating` → `RunningPreHooks` → `StoppingSource` → `WaitingForLocks` → `RemovingTaints` → `StartingTarget` → `Monitoring` → `RunningPostHooks` → `Completed` Monitor per-VM status: ```bash kubectl -n dr-prod get fr failover-to-dr -o jsonpath='{.status.vmStatuses}' ``` ### Execute an unplanned failover When the source cluster is unreachable, set `spec.failoverType: unplanned`. The controller tolerates a dead source and treats its VMs as crash-consistent stopped: ```yaml apiVersion: siterecovery.trilio.io/v1alpha1 kind: FailoverRequest metadata: name: emergency-failover namespace: dr-prod spec: protectionGroupRef: name: my-pg namespace: dr-prod targetCluster: cluster2 failoverType: unplanned ``` ### Fail back to the original cluster Failback is a failover in the reverse direction. Once VMs are running on `cluster2` and replication has resynchronized, create a new `FailoverRequest` targeting `cluster1`: ```yaml apiVersion: siterecovery.trilio.io/v1alpha1 kind: FailoverRequest metadata: name: failback-to-primary namespace: dr-prod spec: protectionGroupRef: name: my-pg namespace: dr-prod targetCluster: cluster1 failoverType: planned ``` ### Rotate zone credentials Rotate before the 30-day expiry warning appears on the zone. Obtain `tsrctl` from Trilio support, then: ```bash tsrctl zone-credential rotate --refresh-rbac ``` After rotation, upload the new kubeconfig files as Secrets: ```bash kubectl -n dr-prod create secret generic cluster1-kubeconfig \ --from-file=kubeconfig=./tsr-zone-prod-workload-a.kubeconfig \ --dry-run=client -o yaml | kubectl apply -f - ``` ### Collect a support bundle Run on the TSR-Manager cluster. The bundle collects logs, CRs, and DRBD state from the TSR-Manager and all zone workload clusters: ```bash oc adm must-gather --image=quay.io/triliovault/site-recovery-must-gather ``` No `-- ` override is needed; the image entrypoint (`/usr/bin/gather`) runs automatically. For a tighter log window: ```bash oc adm must-gather --image=quay.io/triliovault/site-recovery-must-gather \ -- /usr/bin/gather --since 2h ``` ## Examples ### Example 1: Create a ProtectionGroup with boot ordering VMs are started in ascending `bootOrder` batches after failover. VMs without a `bootOrder` start last. ```yaml apiVersion: siterecovery.trilio.io/v1alpha1 kind: ProtectionGroup metadata: name: three-tier-app namespace: dr-prod spec: sourceCluster: cluster1 virtualMachines: - name: db-vm namespace: app-ns bootOrder: 1 - name: app-vm namespace: app-ns bootOrder: 2 - name: web-vm namespace: app-ns ``` During failover, `db-vm` starts first. The controller waits for its VMI to reach Running before starting `app-vm`. `web-vm` starts in the final unordered batch. ### Example 2: TestFailover with TCP reachability probes After test VMs are Running, the controller runs the specified probes from inside the fenced test namespace. ```yaml apiVersion: siterecovery.trilio.io/v1alpha1 kind: TestFailover metadata: name: drill-with-probes namespace: dr-prod spec: protectionGroupRef: name: three-tier-app namespace: dr-prod cleanupPolicy: Automatic retentionTime: 2h schedulingCheck: Fail verification: reachability: - type: tcp port: 5432 vmName: db-vm - type: http port: 8080 endpoint: /health expectedStatus: 200 vmName: app-vm ``` The `vmName` field scopes each probe to a specific VM. A probe with no `vmName` runs against all test VMs. A TCP probe must not include `endpoint` or `expectedStatus`. ### Example 3: Inspect per-VM failover status ```bash kubectl -n dr-prod get fr failover-to-dr \ -o jsonpath='{range .status.vmStatuses[*]}{.name}{"\t"}{.phase}{"\t"}{.error}{"\n"}{end}' ``` ### Example 4: Check zone credential expiry ```bash tsrctl zone-credential inspect ./tsr-zone-prod-workload-a.kubeconfig ``` ### Example 5: View ReplicationGroupStatus for a ProtectionGroup The `ReplicationGroupStatus` is auto-created as `-replication`: ```bash kubectl -n dr-prod get rgs three-tier-app-replication -o yaml ``` Key status fields: `status.rpoStatus`, `status.aggregated.outOfSyncBytes`, `status.aggregated.connectionState`, `status.currentRPOSeconds`. ### Example 6: List protected VMs on a workload cluster DRBDResource CRs live on the workload cluster (not TSR-Manager). Use the label stamped on each protected VM: ```bash # On the workload cluster — list protected VMs kubectl --kubeconfig ~/.kube/workload-a get vm -A \ -l siterecovery.trilio.io/protected-vm=true ``` ### Example 7: Revoke a leaked zone credential If a kubeconfig is compromised, rotate and revoke all previous tokens: ```bash tsrctl zone-credential rotate --revoke-previous ``` After uploading the new kubeconfig Secret, the zone re-validates automatically. ### Example 8: Adjust log verbosity at runtime ```bash # Enable trace logging for the TSR-Manager control plane without a pod restart kubectl -n dr-prod patch configmap site-recovery-logging --type=merge \ -p '{"data":{"level.control-plane":"trace"}}' ``` ## Troubleshooting ### ProtectionZone stuck in Degraded — ClustersValidated=False **Symptom:** `kubectl -n dr-prod get pz prod` shows `status.phase: Degraded`; the `ClustersValidated` condition is `False`. **Likely causes and fixes:** | Reason | Fix | |---|---| | `ClusterAlreadyClaimed` | Another zone already holds a lease on one of the workload clusters. Check `siterecovery.trilio.io/owner-zone` annotation on the workload cluster's `trilio-site-recovery-system` namespace. Release the stale lease only if the owning zone no longer exists. | | `InsufficientClusterPermissions` | The zone credential lacks required RBAC. Run `tsrctl zone-credential rotate --refresh-rbac` and re-upload the new Secret. | | `ClusterIdentityChanged` | The zone credential now reaches a cluster with a different `kube-system` UID than when the zone was created. There is no override; restore the original cluster or recreate the zone. | | `KubeVirtNotInstalled` | The workload cluster does not serve `kubevirt.io/v1`. Install OpenShift Virtualization (KubeVirt) on that cluster. | | `SCCPrerequisiteMissing` | The stock `privileged` SCC or its ClusterRole is absent on an OpenShift workload cluster. | ### ProtectionRequest stuck in Syncing or CreatingDRBD **Symptom:** `kubectl -n dr-prod get pr ` shows phase `Syncing` or `CreatingDRBD` for longer than expected. **Likely causes:** - DRBD replication ports 7000–8000 (inclusive) not open between primary and DR worker nodes. Port 8000 is mandatory; a rule stopping at 7999 silently prevents one volume from replicating. - DRBD kernel module not loaded on a worker node. Check the node agent pod logs: ```bash # Find the node agent pod on the workload cluster kubectl -n trilio-site-recovery-system get pods \ -l app.kubernetes.io/component=drbd-node-agent kubectl -n trilio-site-recovery-system logs -c drbd-module-loader ``` - The DRBDResource on the workload cluster is in a degraded state: ```bash kubectl --kubeconfig ~/.kube/workload-a -n app-ns get dres -o wide ``` ### FailoverRequest stuck in WaitingForLocks **Symptom:** `kubectl -n dr-prod get fr ` shows `status.phase: WaitingForLocks` with reason `DRBDPromotionPending`. **Cause:** The failover controller patches `spec.sourceCluster` on each member `ProtectionRequest` and waits for the ProtectionRequest controller to confirm the promoted copy is DRBD Primary by updating `status.activeCluster`. If one or more ProtectionRequests never confirm, the failover holds until the 5-minute wall-clock deadline fails it. **Fix:** Check each member ProtectionRequest: ```bash kubectl -n dr-prod get pr -o wide ``` Look for `status.phase` not in `Protected` or a `Ready=False` condition with reason `RoleConflict`. If the ProtectionRequest itself is stuck, check the TSR-Manager control plane logs: ```bash kubectl -n dr-prod logs deploy/-site-recovery-manager-control-plane \ -l app.kubernetes.io/component=manager-control-plane ``` ### TestFailover fails at CreatingSnapshots — SnapshotClassNotFound **Symptom:** `status.phase: Failed`, condition `SnapshotsReady=False` with reason `SnapshotClassNotFound`. **Cause:** No `VolumeSnapshotClass` on the target (standby) workload cluster has a `driver` field matching the provisioner of the VM's `StorageClass`. **Fix:** Install a `VolumeSnapshotClass` on the standby cluster whose `driver` matches the provisioner of the StorageClass used by the VM's PVCs. Annotate it as the default if there are multiple: ```yaml annotations: snapshot.storage.kubernetes.io/is-default-class: "true" ``` ### TestFailover stuck — SchedulingProbePending **Symptom:** `status.conditions` shows `ConditionValidated=Unknown` with reason `SchedulingProbePending`. **Cause:** The scheduling probe pods submitted to the target (standby) cluster have not received a scheduler verdict. This is normal for up to the probe deadline; if it persists, the DR scheduler may be unavailable or the target node affinity cannot be satisfied. **Fix:** Check probe pods on the target workload cluster: ```bash kubectl --kubeconfig ~/.kube/workload-b -n app-ns get pods \ -l siterecovery.trilio.io/scheduling-probe-owner-uid= ``` Inspect the pod's `status.conditions` for the `PodScheduled` condition message. ### Node agent stuck in Init — Init:CrashLoopBackOff **Symptom:** DRBD node agent pods show `Init:CrashLoopBackOff` or `Init:Error`. **Cause:** The `drbd-module-loader` init container could not find a compatible kmod for this node's kernel. **Fix:** 1. Check the node's kernel version: `kubectl get node -o jsonpath='{.status.nodeInfo.kernelVersion}'` 2. Inspect init container logs: `kubectl -n trilio-site-recovery-system logs -c drbd-module-loader` 3. If the kernel is newer than the mirrored image, contact Trilio support to trigger a mirror refresh. Once the image is updated, delete the stuck pod to force a fresh pull. 4. A foreign DRBD module loaded with `usermode_helper` enabled requires a node reboot after removing whatever loaded it. ### Zone namespace stuck Terminating **Symptom:** `kubectl get namespace dr-prod` shows `Terminating` indefinitely. **Cause:** Deleting the zone namespace before deleting the ProtectionZone CR removes the zone's webhook Service, leaving the cluster-scoped webhook configurations pointing at a deleted Service. This blocks teardown. **Fix:** Delete the stale webhook configurations: ```bash kubectl delete validatingwebhookconfiguration \ site-recovery-manager-control-plane-dr-prod-validating-webhook-configuration kubectl delete mutatingwebhookconfiguration \ site-recovery-manager-control-plane-dr-prod-mutating-webhook-configuration ``` Then clear finalizers on remaining CRs in the namespace: ```bash for kind in protectiongroups protectionrequests testfailovers; do for obj in $(kubectl -n dr-prod get "$kind" -o name); do kubectl -n dr-prod patch "$obj" --type=merge \ -p '{"metadata":{"finalizers":null}}' done done ```