Recovery Operation Failures

This page helps you diagnose and recover from failed or partially completed Site Recovery operations — specifically FailoverRequest, TestFailover, and ProtectionRequest lifecycles. Understanding what the controller does when an operation fails, what state is left behind, and which manual steps are safe is critical for restoring protection without data loss. Because every DR operation is declared through Kubernetes custom resources and the controller is re-entrant rather than rollback-oriented, recovery usually means understanding current CR phase, inspecting per-VM status, and deciding whether to resume, re-create, or manually remediate.

Before using this guide you need:

• kubectl access to the TSR-Manager cluster with read/write permissions on the siterecovery.trilio.io API group
• oc CLI if collecting a support bundle via oc adm must-gather
• The zone's protection namespace (dr-<zone-name>) and the workload clusters' trilio-site-recovery-system namespace
• Familiarity with ProtectionZone, ProtectionGroup, ProtectionRequest, FailoverRequest, and TestFailover CRs
• OpenShift 4.17 or later on the TSR-Manager cluster and both workload clusters (Tech Preview supported floor)

No additional installation is required for diagnostics. All commands below run against the TSR-Manager cluster using kubectl (and oc for support bundle collection), which are the only CLIs you need for recovery operations. Ensure your kubeconfig is pointed at the TSR-Manager cluster before running any command in this guide.

# Verify you are targeting the TSR-Manager cluster
kubectl get pz -A

If you need a support bundle to share with Trilio support:

oc adm must-gather --image=quay.io/triliovault/site-recovery-must-gather:<release-tag>

Substitute <release-tag> with the pinned release tag supplied by Trilio with your release.

The following CR fields are most relevant when diagnosing and recovering from failed operations.

FailoverRequest

TestFailover

ProtectionRequest

site-recovery-logging ConfigMap (runtime log level, no pod restart)

Create or patch this ConfigMap in the relevant namespace (TSR-Manager zone namespace dr-<name> or workload trilio-site-recovery-system) to increase verbosity during diagnosis:

apiVersion: v1
kind: ConfigMap
metadata:
  name: site-recovery-logging
  namespace: dr-<zone-name>   # or trilio-site-recovery-system on a workload cluster
data:
  level: debug                       # global: error | info | debug | trace
  level.control-plane: debug
  level.drbd-node-agent: info
  level.protectionzone: debug

Identifying a failed operation

All DR operations expose their state through status.state and status.phase. Check these first.

FailoverRequest

# List all FailoverRequests in the zone namespace
kubectl get fr -n dr-<zone-name>

The printed columns are: Protection Group, Target, State, Phase, Age.

Key state values:

• status.state: Pending | InProgress | Completed | Failed | Cancelled
• status.phase: Initializing | Validating | RunningPreHooks | StoppingSource | WaitingForLocks | RemovingTaints | StartingTarget | Monitoring | RunningPostHooks | Completed | Failed

For per-VM detail:

kubectl get fr <fr-name> -n dr-<zone-name> -o jsonpath='{.status.vmStatuses}' | jq .

Note: status.vmStatuses is the only per-VM detail field; there is no status.failedVMs field and no status.message field.

TestFailover

kubectl get tfo -n dr-<zone-name>

The printed columns are: Protection Group, Phase, Result, Passed, Scheduling, Age.

Key phase values (exact): Queued | PreparingEnvironment | CreatingSnapshots | CreatingVolumes | CreatingVMs | StartingVMs | Verifying | Succeeded | Failed | CleaningUp | Cleaned

To find the isolated test namespace (never guess or construct this yourself — always read it from status):

kubectl get tfo <tfo-name> -n dr-<zone-name> \
  -o jsonpath='{.status.testNamespace}'

ProtectionRequest

kubectl get pr -n dr-<zone-name>

The printed columns are: VM, Namespace, Phase, Sync, Age.

Exact phase values: Pending | Validating | CreatingDRBD | Syncing | ReadyToActivate | Activating | Protected | Unprotecting | Failed

There is no message field on ProtectionRequest status; use conditions and events:

kubectl describe pr <pr-name> -n dr-<zone-name>

Inspecting hook failures

Hook statuses are embedded in the FailoverRequest or TestFailover status:

kubectl get fr <fr-name> -n dr-<zone-name> \
  -o jsonpath='{.status.hookStatuses}' | jq .

Each hook status carries: phase (Pending | Running | Succeeded | Failed | TimedOut | Skipped), message, attempts, exitCode, and jobName. If a hook Job is still present, inspect it:

kubectl logs job/<job-name> -n <hook-job-namespace>

Checking DRBD replication state during a stalled failover

DRBDResource CRs live on each workload cluster, not on the TSR-Manager cluster.

# Run against the relevant workload cluster
kubectl get dres -n <vm-namespace> --context <workload-cluster-context>

The printed columns are: Side, Role, Node, State, Phase, Age.

A stalled failover commonly shows status.role still as Secondary on the target side and connectionState as Disconnected or Connecting. Check sync progress:

kubectl get dres <resource-name> -n <vm-namespace> \
  --context <workload-cluster-context> \
  -o jsonpath='{.status.syncProgress}'

Checking replication health via ReplicationGroupStatus

The ReplicationGroupStatus is auto-created per ProtectionGroup on the TSR-Manager cluster, named <protection-group-name>-replication:

kubectl get rgs -n dr-<zone-name>

The printed columns are: Protection-Group, RPO-Status, Out-Of-Sync, Synced, Connection, Last-Sync, Age.

# Inspect aggregated metrics
kubectl get rgs <pg-name>-replication -n dr-<zone-name> -o jsonpath='{.status}' | jq .

Filter ReplicationGroupStatus by ProtectionGroup using the label (field selectors on spec/status do not work on custom resources):

kubectl get rgs -n dr-<zone-name> \
  -l siterecovery.trilio.io/protection-group=<pg-name>

Finding RPO violations

kubectl get rpoe -n dr-<zone-name> \
  -l siterecovery.trilio.io/protection-group=<pg-name>

RPOEvent CRs are immutable audit records. The printed columns are: Protection-Group, Event-Type, RPO-Seconds, Out-Of-Sync, Reason, Timestamp, Age.

Example 1 — Inspect a failed FailoverRequest and its per-VM statuses

# Get the overall state
kubectl get fr my-failover -n dr-zone1

# Get per-VM detail (status.vmStatuses is the authoritative per-VM field)
kubectl get fr my-failover -n dr-zone1 \
  -o jsonpath='{range .status.vmStatuses[*]}{.name}{"\t"}{.phase}{"\t"}{.error}{"\n"}{end}'

# Describe for events and conditions
kubectl describe fr my-failover -n dr-zone1

Example 2 — Find the test namespace for a stuck TestFailover

# Read the controller-generated test namespace from status (never construct it manually)
TEST_NS=$(kubectl get tfo my-drill -n dr-zone1 \
  -o jsonpath='{.status.testNamespace}')
echo "Test namespace: $TEST_NS"

# List test VMs in that namespace (on the standby workload cluster)
kubectl get vm -n "$TEST_NS" --context <standby-cluster-context>

Example 3 — Retry a TestFailover with Manual cleanup after failure

When a TestFailover fails at Verifying and you want to inspect test VMs before cleanup:

cat <<EOF | kubectl apply -f -
apiVersion: siterecovery.trilio.io/v1alpha1
kind: TestFailover
metadata:
  name: drill-inspect
  namespace: dr-zone1
spec:
  protectionGroupRef:
    name: pg-webapp
    namespace: dr-zone1
  cleanupPolicy: Manual
  schedulingCheck: Warn
  skipHooks: false
EOF

With cleanupPolicy: Manual, test resources remain in status.testNamespace after failure so you can SSH into VMs or inspect logs. When done:

# Signal cleanup by patching cleanupRequested
kubectl patch tfo drill-inspect -n dr-zone1 \
  --type merge -p '{"spec":{"cleanupRequested":true}}'

Example 4 — Skip hooks on a blocked FailoverRequest

If a pre-failover hook is blocking the operation and you need to proceed:

cat <<EOF | kubectl apply -f -
apiVersion: siterecovery.trilio.io/v1alpha1
kind: FailoverRequest
metadata:
  name: failover-nohooks
  namespace: dr-zone1
spec:
  protectionGroupRef:
    name: pg-webapp
    namespace: dr-zone1
  targetCluster: cluster2
  failoverType: planned
  skipHooks: true
EOF

Example 5 — Check DRBD resource state on a workload cluster

# List all DRBDResources for a VM namespace on the primary workload cluster
kubectl get dres -n my-vm-namespace --context primary-cluster

# Inspect connection and sync state of a specific resource
kubectl get dres drbd-my-vm-namespace-my-vm-a1b2c3d4 \
  -n my-vm-namespace \
  --context primary-cluster \
  -o jsonpath='{"Role: "}{.status.role}{"\nState: "}{.status.connectionState}{"\nPhase: "}{.status.phase}{"\nSync: "}{.status.syncProgress.percentage}{"% \nBytes remaining: "}{.status.syncProgress.bytesRemaining}{"\n"}'

Example 6 — Increase log verbosity on the TSR-Manager control plane

kubectl patch configmap site-recovery-logging \
  -n dr-zone1 \
  --type merge \
  -p '{"data":{"level":"debug","level.control-plane":"debug"}}'

Changes take effect at the next log-level watch interval without pod restarts.

Example 7 — Filter protected VMs by label on a workload cluster

# The siterecovery.trilio.io/protected-vm=true label is stamped on the
# KubeVirt VirtualMachine object on the workload cluster only
kubectl get vm -A \
  -l siterecovery.trilio.io/protected-vm=true \
  --context primary-cluster

Example 8 — Collect a support bundle

# Run on the TSR-Manager cluster; the must-gather image reaches through to
# all zone workload clusters automatically
oc adm must-gather \
  --image=quay.io/triliovault/site-recovery-must-gather:<release-tag>

The archive is written to the current directory. Contact Trilio support and attach the archive.

Issue: FailoverRequest stuck in WaitingForLocks or StoppingSource

Symptom: kubectl get fr shows Phase: StoppingSource or Phase: WaitingForLocks for an extended period. Per-VM statuses show Stopping but do not progress.

Likely cause: A running protected VM on the source cluster cannot be stopped because the node hosting it cannot be drained. A protected VM is pinned to one node for the lifetime of its protection; oc adm drain of that node will not complete while the VM is running and protected.

Fix:

1. Identify which VMs are stuck using status.vmStatuses:
   Copy

   kubectl get fr <fr-name> -n dr-<zone-name> \
     -o jsonpath='{range .status.vmStatuses[*]}{.name}{"\t"}{.phase}{"\n"}{end}'
   

2. For a planned failover, verify the source-side VM is reachable and not held by an external process.
3. If the source cluster is unavailable and you need to proceed, delete the stuck FailoverRequest only after confirming no VM is in an ambiguous state where it may be running on both clusters, then create a new FailoverRequest with failoverType: unplanned. An unplanned failover force-promotes the DR side without waiting for the source to stop.

Caution: Do not start VMs on the source cluster after the DR side has been promoted. Do not delete a stuck FailoverRequest while VMs are in an unknown state — the controller is re-entrant and will resume where it left off if the CR is preserved.

Issue: FailoverRequest in Failed state, hook in Failed or TimedOut

Symptom: status.state: Failed, status.phase: RunningPreHooks or RunningPostHooks. A hook status shows phase: Failed or phase: TimedOut.

Likely cause: A Hook Job exceeded its timeout, returned a non-zero exit code, or the hook Job pod could not be scheduled.

Fix:

1. Identify the failing hook and its Job:
   Copy

   kubectl get fr <fr-name> -n dr-<zone-name> \
     -o jsonpath='{range .status.hookStatuses[?(@.phase=="Failed")]}{.hookName}{"\t"}{.jobName}{"\t"}{.message}{"\n"}{end}'
   

2. Inspect the Job logs:
   Copy

   kubectl logs job/<job-name> -n <hook-job-namespace>
   

3. Fix the underlying issue in the Hook Job image or configuration.
4. If the hook is non-critical and you need to proceed now, create a new FailoverRequest with spec.skipHooks: true. Be aware this bypasses all hooks including verification steps.
5. A Hook CR is frozen against edits while an operation is running it. Wait for the operation to reach a terminal state before editing the Hook spec.

Issue: ProtectionRequest stuck in Syncing for an extended period

Symptom: kubectl get pr shows Phase: Syncing with a non-advancing Sync percentage.

Likely cause: DRBD replication is stalled. Common causes: network interruption between primary and DR worker nodes on TCP ports 7000–8000, the DRBD connection is Disconnected or Connecting, or a node agent pod is not running.

Fix:

1. Check the DRBDResource on both workload clusters:
   Copy

   kubectl get dres -n <vm-namespace> --context <primary-cluster>
   kubectl get dres -n <vm-namespace> --context <dr-cluster>
   

2. Look for connectionState: Disconnected or phase: Degraded.
3. Verify network connectivity between worker nodes on TCP 7000–8000. Port 8000 is mandatory — a firewall stopping at 7999 will silently prevent replication on one volume.
4. Check that the drbd-node-agent DaemonSet pods are running on both clusters:
   Copy

   kubectl get pods -n trilio-site-recovery-system \
     -l app.kubernetes.io/component=drbd-node-agent \
     --context <workload-cluster>
   

5. Check node-agent logs for the specific node hosting the VM. Because kubectl logs does not accept --field-selector, use two steps:
   Copy

   # Step 1: get the agent pod on the node
   AGENT_POD=$(kubectl get pods -n trilio-site-recovery-system \
     --field-selector spec.nodeName=<node-name> \
     -l app.kubernetes.io/component=drbd-node-agent \
     -o jsonpath='{.items[0].metadata.name}' \
     --context <workload-cluster>)
   # Step 2: stream logs
   kubectl logs "$AGENT_POD" -n trilio-site-recovery-system \
     --context <workload-cluster>
   

Issue: ProtectionRequest in Failed phase

Symptom: kubectl get pr shows Phase: Failed. kubectl describe pr shows a condition with reason indicating the failure.

Likely cause: DRBD provisioning failed, the VM reference is invalid, or a DRBD role conflict was detected.

Common conditions and fixes:

• Reason RoleConflict on the ProtectionRequest Ready condition: Both sides of the DRBD pair are claiming the Primary role simultaneously. This can occur after an unplanned failover if the original source cluster comes back online. Do not start VMs on the original source cluster. Investigate DRBD state on both workload clusters and ensure only the intended active cluster holds the Primary role.

• Phase stays Failed after fixing the underlying issue: Re-protect always requires a full unprotect followed by a full initial resync. Delete the ProtectionRequest and wait for Unprotecting to complete before creating a new one. The admission webhook will reject a new ProtectionRequest for the same VM while the previous one (including one in Terminating) still exists.

  Copy

  kubectl delete pr <pr-name> -n dr-<zone-name>
  # Wait until the object is fully gone (finalizers removed)
  kubectl get pr <pr-name> -n dr-<zone-name>
  # Then create a new ProtectionRequest
  

Issue: TestFailover stuck in CreatingVolumes or CreatingVMs

Symptom: kubectl get tfo shows Phase: CreatingVolumes or Phase: CreatingVMs with no progress.

Likely cause: DRBD snapshots could not be taken (the replication may be in a degraded state), or the DR cluster cannot schedule the test VMs due to resource constraints. spec.schedulingCheck: Fail will abort when a VM cannot be placed.

Fix:

1. Check the TestFailover conditions:
   Copy

   kubectl describe tfo <tfo-name> -n dr-<zone-name>
   

2. Check the scheduling verdict:
   Copy

   kubectl get tfo <tfo-name> -n dr-<zone-name> \
     -o jsonpath='{.status.schedulingCheck}' | jq .
   

3. Check replication health before running a drill — a degraded replication state means snapshots may be inconsistent:
   Copy

   kubectl get rgs <pg-name>-replication -n dr-<zone-name> \
     -o jsonpath='{.status.aggregated}' | jq .
   

4. If the DR cluster lacks capacity, either free resources or set spec.schedulingCheck: Warn to get a result with a scheduling warning rather than an abort.
5. After investigating, clean up the failed TestFailover. If you used cleanupPolicy: Manual, patch spec.cleanupRequested: true to trigger cleanup. Do not manually patch finalizers to force deletion of test resources.

Issue: ProtectionZone ClustersValidated condition is False, blocking new ProtectionGroup or ProtectionRequest creates

Symptom: Creating a ProtectionGroup or ProtectionRequest is rejected by the webhook with a message referencing zone validation.

Likely cause: The ProtectionZone ClustersValidated condition is not True. This is the gate for all PG and PR admission. Common reasons:

# Check the zone conditions
kubectl get pz <zone-name> -n dr-<zone-name> \
  -o jsonpath='{.status.conditions}' | jq .

Issue: drbd-node-agent pods not starting; node has siterecovery.trilio.io/not-ready:NoSchedule taint

Symptom: Worker nodes remain tainted and VMs cannot be scheduled by the DR system. DaemonSet pods show Pending or CrashLoopBackOff.

Likely cause: The node agent applies the siterecovery.trilio.io/not-ready:NoSchedule taint to its own node during startup until DRBD converges. If the agent cannot start (missing DRBD kernel module, SCC admission denied, image pull failure), the taint is never removed.

Fix:

1. Check DaemonSet pod status:
   Copy

   kubectl get pods -n trilio-site-recovery-system \
     -l app.kubernetes.io/component=drbd-node-agent \
     --context <workload-cluster>
   

2. Describe a failing pod for events:
   Copy

   kubectl describe pod <agent-pod-name> -n trilio-site-recovery-system \
     --context <workload-cluster>
   

3. Common causes:
   • DRBD kernel module (drbd9) not loaded — verify the module loader init container succeeded.
   • SCC admission denied — check for SCCAdmissionDenied reason on the ProtectionZone WorkloadStacksReady condition.
   • Image pull error — verify the workload cluster can reach quay.io/triliovault.
4. Once the agent starts successfully, it removes the taint automatically when DRBD converges. Do not remove the taint manually — the agent re-applies it on restart if DRBD has not converged.