Recovery Workflow

This page walks you through every step of the Site Recovery recovery workflow: from triggering a failover or test failover, through state transitions, to completion and optional failback. Understanding this workflow lets you execute planned failovers, respond to unplanned disasters, validate DR readiness without disrupting production, and return workloads to their original cluster after recovery. Every operation is declared through Kubernetes custom resources in the siterecovery.trilio.io/v1alpha1 API group and can be driven by kubectl or the TSR web console.

Before executing any recovery operation, confirm the following:

• A TSR-Manager cluster running the Site Recovery control plane (required, not optional)
• Two workload clusters registered in a ProtectionZone with status.phase: Available
• The ProtectionZone condition ClustersValidated=True — webhooks block all ProtectionGroup and ProtectionRequest creates until this condition is met
• At least one ProtectionGroup with member VMs in status.state: Protected
• Each member VM has a corresponding ProtectionRequest in status.phase: Protected
• kubectl configured to reach the TSR-Manager cluster
• For test failovers: DRBD storage backend mode (TestFailover is supported in DRBD mode only)
• For the TSR web console: OpenShift 4.17 or later on the TSR-Manager cluster (the console is OpenShift-only via an oauth-proxy Route)
• tsrctl available — contact Trilio support to obtain it for your release
• Firewall rules permitting TCP 7000–8000 inclusive between primary and DR worker nodes (one port per replicated VM; port 8000 is mandatory)

Site Recovery is installed as a Kubernetes operator via OLM on OpenShift. The control plane Helm charts are embedded in the operator binary and deployed automatically by the ProtectionZone controller — you do not run helm install directly.

1. Install the operator via OLM

Create a CatalogSource, OperatorGroup, and Subscription on the TSR-Manager cluster in the trilio-site-recovery-system namespace:

apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: site-recovery-catalog
  namespace: trilio-site-recovery-system
spec:
  sourceType: grpc
  image: quay.io/triliodata/site-recovery-catalog
  displayName: Trilio Site Recovery
  publisher: Trilio
---
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: site-recovery-og
  namespace: trilio-site-recovery-system
spec: {}
---
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: site-recovery
  namespace: trilio-site-recovery-system
spec:
  channel: stable
  name: site-recovery
  source: site-recovery-catalog
  sourceNamespace: trilio-site-recovery-system

kubectl apply -f catalogsource.yaml

Wait for the CSV to reach Succeeded:

kubectl -n trilio-site-recovery-system get csv

2. Provision zone credentials with tsrctl

Obtain tsrctl from Trilio support. Then create credentials for each workload cluster:

tsrctl zone-credential create \
  --zone prod \
  --kubeconfig ~/.kube/workload-a,~/.kube/workload-b

This writes tsr-zone-prod-<cluster-name>.kubeconfig for each cluster.

3. Create the zone namespace and upload credentials

kubectl create namespace dr-prod

kubectl -n dr-prod create secret generic cluster1-kubeconfig \
  --from-file=kubeconfig=./tsr-zone-prod-workload-a.kubeconfig

kubectl -n dr-prod create secret generic cluster2-kubeconfig \
  --from-file=kubeconfig=./tsr-zone-prod-workload-b.kubeconfig

4. Create the ProtectionZone

apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionZone
metadata:
  name: prod
  namespace: dr-prod
spec:
  displayName: "Production DR Zone"
  storageBackendMode: DRBD
  clusters:
    - name: cluster1
      clusterID: cluster1
      distributionName: RHEL-CoreOS-9
      kubeconfigSecretRef:
        name: cluster1-kubeconfig
        key: kubeconfig
    - name: cluster2
      clusterID: cluster2
      distributionName: RHEL-CoreOS-9
      kubeconfigSecretRef:
        name: cluster2-kubeconfig
        key: kubeconfig

kubectl apply -f protectionzone.yaml

The ProtectionZone controller validates both clusters, installs the TSR-Manager and workload control plane stacks, and deploys the DRBD node agent DaemonSet on every worker node of both workload clusters. Monitor progress:

kubectl -n dr-prod get pz prod -w

Wait for status.phase: Available and ClustersValidated=True before proceeding.

Recovery operations are controlled by the fields on each custom resource. The key configuration points are:

ProtectionGroup — grouping and boot order

FailoverRequest — planned vs unplanned

spec.drainTimeoutSeconds is accepted and range-checked by the admission webhook but is not consumed by the failover controller in Tech Preview; omit it from manifests.

TestFailover — DR drill settings

ProtectionZone — node agent startup

Runtime log level (without restart)

You can change log levels at runtime without restarting pods by patching the site-recovery-logging ConfigMap in the relevant namespace:

# Increase TSR-Manager control plane verbosity
kubectl -n dr-prod patch configmap site-recovery-logging --type=merge \
  -p '{"data":{"level":"debug","level.control-plane":"trace"}}'

# Increase node agent verbosity on a workload cluster
kubectl -n trilio-site-recovery-system patch configmap site-recovery-logging \
  --type=merge -p '{"data":{"level.drbd-node-agent":"debug"}}'

Protect a virtual machine

Create a ProtectionRequest on the TSR-Manager cluster in the zone namespace (dr-<zone-name>). The VM must be running on the source cluster when you protect it — the source node is captured at protect time and is immutable for the lifetime of the protection.

apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionRequest
metadata:
  name: web-vm-pr
  namespace: dr-prod
spec:
  sourceCluster: cluster1
  virtualMachine:
    name: web-vm
    namespace: app-ns

kubectl apply -f protection-request.yaml

Watch the protection lifecycle:

kubectl -n dr-prod get pr web-vm-pr -w

Phases: Pending → Validating → CreatingDRBD → Syncing → Protected

Monitor replication health

The replication monitor auto-creates a ReplicationGroupStatus named <protection-group-name>-replication in the zone namespace:

kubectl -n dr-prod get rgs
kubectl -n dr-prod describe rgs my-pg-replication

Check RPO events (immutable audit records):

kubectl -n dr-prod get rpoe

Inspect DRBD state on a workload cluster (DRBDResource lives on the workload cluster, not TSR-Manager):

# Run this against the workload cluster, not TSR-Manager
kubectl --kubeconfig ~/.kube/workload-a get dres -n app-ns

Run a test failover (DR drill)

A TestFailover spins up isolated VM copies from DRBD snapshots in a network-fenced namespace on the standby cluster. It does not touch production VMs.

apiVersion: siterecovery.trilio.io/v1alpha1
kind: TestFailover
metadata:
  name: drill-2024-q4
  namespace: dr-prod
spec:
  protectionGroupRef:
    name: my-pg
    namespace: dr-prod
  cleanupPolicy: Manual
  retentionTime: 4h
  schedulingCheck: Fail

kubectl apply -f test-failover.yaml
kubectl -n dr-prod get tfo drill-2024-q4 -w

Phases: Queued → PreparingEnvironment → CreatingSnapshots → CreatingVolumes → CreatingVMs → StartingVMs → Verifying → Succeeded (or Failed)

The generated test namespace is reported in status.testNamespace. To trigger cleanup after a Manual policy test:

kubectl -n dr-prod patch tfo drill-2024-q4 --type=merge \
  -p '{"spec":{"cleanupRequested":true}}'

Execute a planned failover

A planned failover gracefully stops VMs on the source cluster, waits for DRBD locks to settle, then starts VMs on the target. Every failover requires a FailoverRequest; failover is never automatic.

apiVersion: siterecovery.trilio.io/v1alpha1
kind: FailoverRequest
metadata:
  name: failover-to-dr
  namespace: dr-prod
spec:
  protectionGroupRef:
    name: my-pg
    namespace: dr-prod
  targetCluster: cluster2
  failoverType: planned

kubectl apply -f failover-request.yaml
kubectl -n dr-prod get fr failover-to-dr -w

Phases: Validating → RunningPreHooks → StoppingSource → WaitingForLocks → RemovingTaints → StartingTarget → Monitoring → RunningPostHooks → Completed

Monitor per-VM status:

kubectl -n dr-prod get fr failover-to-dr -o jsonpath='{.status.vmStatuses}'

Execute an unplanned failover

When the source cluster is unreachable, set spec.failoverType: unplanned. The controller tolerates a dead source and treats its VMs as crash-consistent stopped:

apiVersion: siterecovery.trilio.io/v1alpha1
kind: FailoverRequest
metadata:
  name: emergency-failover
  namespace: dr-prod
spec:
  protectionGroupRef:
    name: my-pg
    namespace: dr-prod
  targetCluster: cluster2
  failoverType: unplanned

Fail back to the original cluster

Failback is a failover in the reverse direction. Once VMs are running on cluster2 and replication has resynchronized, create a new FailoverRequest targeting cluster1:

apiVersion: siterecovery.trilio.io/v1alpha1
kind: FailoverRequest
metadata:
  name: failback-to-primary
  namespace: dr-prod
spec:
  protectionGroupRef:
    name: my-pg
    namespace: dr-prod
  targetCluster: cluster1
  failoverType: planned

Rotate zone credentials

Rotate before the 30-day expiry warning appears on the zone. Obtain tsrctl from Trilio support, then:

tsrctl zone-credential rotate --refresh-rbac

After rotation, upload the new kubeconfig files as Secrets:

kubectl -n dr-prod create secret generic cluster1-kubeconfig \
  --from-file=kubeconfig=./tsr-zone-prod-workload-a.kubeconfig \
  --dry-run=client -o yaml | kubectl apply -f -

Collect a support bundle

Run on the TSR-Manager cluster. The bundle collects logs, CRs, and DRBD state from the TSR-Manager and all zone workload clusters:

oc adm must-gather --image=quay.io/triliovault/site-recovery-must-gather

No -- <command> override is needed; the image entrypoint (/usr/bin/gather) runs automatically. For a tighter log window:

oc adm must-gather --image=quay.io/triliovault/site-recovery-must-gather \
  -- /usr/bin/gather --since 2h

Example 1: Create a ProtectionGroup with boot ordering

VMs are started in ascending bootOrder batches after failover. VMs without a bootOrder start last.

apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionGroup
metadata:
  name: three-tier-app
  namespace: dr-prod
spec:
  sourceCluster: cluster1
  virtualMachines:
    - name: db-vm
      namespace: app-ns
      bootOrder: 1
    - name: app-vm
      namespace: app-ns
      bootOrder: 2
    - name: web-vm
      namespace: app-ns

During failover, db-vm starts first. The controller waits for its VMI to reach Running before starting app-vm. web-vm starts in the final unordered batch.

Example 2: TestFailover with TCP reachability probes

After test VMs are Running, the controller runs the specified probes from inside the fenced test namespace.

apiVersion: siterecovery.trilio.io/v1alpha1
kind: TestFailover
metadata:
  name: drill-with-probes
  namespace: dr-prod
spec:
  protectionGroupRef:
    name: three-tier-app
    namespace: dr-prod
  cleanupPolicy: Automatic
  retentionTime: 2h
  schedulingCheck: Fail
  verification:
    reachability:
      - type: tcp
        port: 5432
        vmName: db-vm
      - type: http
        port: 8080
        endpoint: /health
        expectedStatus: 200
        vmName: app-vm

The vmName field scopes each probe to a specific VM. A probe with no vmName runs against all test VMs. A TCP probe must not include endpoint or expectedStatus.

Example 3: Inspect per-VM failover status

kubectl -n dr-prod get fr failover-to-dr \
  -o jsonpath='{range .status.vmStatuses[*]}{.name}{"\t"}{.phase}{"\t"}{.error}{"\n"}{end}'

Example 4: Check zone credential expiry

tsrctl zone-credential inspect ./tsr-zone-prod-workload-a.kubeconfig

Example 5: View ReplicationGroupStatus for a ProtectionGroup

The ReplicationGroupStatus is auto-created as <pg-name>-replication:

kubectl -n dr-prod get rgs three-tier-app-replication -o yaml

Key status fields: status.rpoStatus, status.aggregated.outOfSyncBytes, status.aggregated.connectionState, status.currentRPOSeconds.

Example 6: List protected VMs on a workload cluster

DRBDResource CRs live on the workload cluster (not TSR-Manager). Use the label stamped on each protected VM:

# On the workload cluster — list protected VMs
kubectl --kubeconfig ~/.kube/workload-a get vm -A \
  -l siterecovery.trilio.io/protected-vm=true

Example 7: Revoke a leaked zone credential

If a kubeconfig is compromised, rotate and revoke all previous tokens:

tsrctl zone-credential rotate --revoke-previous

After uploading the new kubeconfig Secret, the zone re-validates automatically.

Example 8: Adjust log verbosity at runtime

# Enable trace logging for the TSR-Manager control plane without a pod restart
kubectl -n dr-prod patch configmap site-recovery-logging --type=merge \
  -p '{"data":{"level.control-plane":"trace"}}'

ProtectionZone stuck in Degraded — ClustersValidated=False

Symptom: kubectl -n dr-prod get pz prod shows status.phase: Degraded; the ClustersValidated condition is False.

Likely causes and fixes:

ProtectionRequest stuck in Syncing or CreatingDRBD

Symptom: kubectl -n dr-prod get pr <name> shows phase Syncing or CreatingDRBD for longer than expected.

Likely causes:

• DRBD replication ports 7000–8000 (inclusive) not open between primary and DR worker nodes. Port 8000 is mandatory; a rule stopping at 7999 silently prevents one volume from replicating.
• DRBD kernel module not loaded on a worker node. Check the node agent pod logs:
  Copy

  # Find the node agent pod on the workload cluster
  kubectl -n trilio-site-recovery-system get pods \
    -l app.kubernetes.io/component=drbd-node-agent
  kubectl -n trilio-site-recovery-system logs <agent-pod> -c drbd-module-loader
  

• The DRBDResource on the workload cluster is in a degraded state:
  Copy

  kubectl --kubeconfig ~/.kube/workload-a -n app-ns get dres -o wide
  

FailoverRequest stuck in WaitingForLocks

Symptom: kubectl -n dr-prod get fr <name> shows status.phase: WaitingForLocks with reason DRBDPromotionPending.

Cause: The failover controller patches spec.sourceCluster on each member ProtectionRequest and waits for the ProtectionRequest controller to confirm the promoted copy is DRBD Primary by updating status.activeCluster. If one or more ProtectionRequests never confirm, the failover holds until the 5-minute wall-clock deadline fails it.

Fix: Check each member ProtectionRequest:

kubectl -n dr-prod get pr -o wide

Look for status.phase not in Protected or a Ready=False condition with reason RoleConflict. If the ProtectionRequest itself is stuck, check the TSR-Manager control plane logs:

kubectl -n dr-prod logs deploy/<release>-site-recovery-manager-control-plane \
  -l app.kubernetes.io/component=manager-control-plane

TestFailover fails at CreatingSnapshots — SnapshotClassNotFound

Symptom: status.phase: Failed, condition SnapshotsReady=False with reason SnapshotClassNotFound.

Cause: No VolumeSnapshotClass on the target (standby) workload cluster has a driver field matching the provisioner of the VM's StorageClass.

Fix: Install a VolumeSnapshotClass on the standby cluster whose driver matches the provisioner of the StorageClass used by the VM's PVCs. Annotate it as the default if there are multiple:

annotations:
  snapshot.storage.kubernetes.io/is-default-class: "true"

TestFailover stuck — SchedulingProbePending

Symptom: status.conditions shows ConditionValidated=Unknown with reason SchedulingProbePending.

Cause: The scheduling probe pods submitted to the target (standby) cluster have not received a scheduler verdict. This is normal for up to the probe deadline; if it persists, the DR scheduler may be unavailable or the target node affinity cannot be satisfied.

Fix: Check probe pods on the target workload cluster:

kubectl --kubeconfig ~/.kube/workload-b -n app-ns get pods \
  -l siterecovery.trilio.io/scheduling-probe-owner-uid=<tfo-uid>

Inspect the pod's status.conditions for the PodScheduled condition message.

Node agent stuck in Init — Init:CrashLoopBackOff

Symptom: DRBD node agent pods show Init:CrashLoopBackOff or Init:Error.

Cause: The drbd-module-loader init container could not find a compatible kmod for this node's kernel.

Fix:

1. Check the node's kernel version: kubectl get node <node> -o jsonpath='{.status.nodeInfo.kernelVersion}'
2. Inspect init container logs: kubectl -n trilio-site-recovery-system logs <agent-pod> -c drbd-module-loader
3. If the kernel is newer than the mirrored image, contact Trilio support to trigger a mirror refresh. Once the image is updated, delete the stuck pod to force a fresh pull.
4. A foreign DRBD module loaded with usermode_helper enabled requires a node reboot after removing whatever loaded it.

Zone namespace stuck Terminating

Symptom: kubectl get namespace dr-prod shows Terminating indefinitely.

Cause: Deleting the zone namespace before deleting the ProtectionZone CR removes the zone's webhook Service, leaving the cluster-scoped webhook configurations pointing at a deleted Service. This blocks teardown.

Fix: Delete the stale webhook configurations:

kubectl delete validatingwebhookconfiguration \
  site-recovery-manager-control-plane-dr-prod-validating-webhook-configuration
kubectl delete mutatingwebhookconfiguration \
  site-recovery-manager-control-plane-dr-prod-mutating-webhook-configuration

Then clear finalizers on remaining CRs in the namespace:

for kind in protectiongroups protectionrequests testfailovers; do
  for obj in $(kubectl -n dr-prod get "$kind" -o name); do
    kubectl -n dr-prod patch "$obj" --type=merge \
      -p '{"metadata":{"finalizers":null}}'
  done
done