---
title: Recovery Operation Failures
product: trilio-site-recovery-for-kubernetes
doc_type: guide
version: 0.2.0
source: git2docs (code-derived, validation-filtered)
canonical: https://docs-tsr.trilio.io/trilio-site-recovery-for-kubernetes/site-recovery/troubleshooting-recovery-failures
---

# Recovery Operation Failures

_Diagnosing failed or partially completed recovery operations, rollback behavior_

## Overview

This page helps you diagnose and recover from failed or partially completed Site Recovery operations — specifically FailoverRequest, TestFailover, and ProtectionRequest lifecycles. Understanding what the controller does when an operation fails, what state is left behind, and which manual steps are safe is critical for restoring protection without data loss. Because every DR operation is declared through Kubernetes custom resources and the controller is re-entrant rather than rollback-oriented, recovery usually means understanding current CR phase, inspecting per-VM status, and deciding whether to resume, re-create, or manually remediate.

## Prerequisites

Before using this guide you need:

- `kubectl` access to the TSR-Manager cluster with read/write permissions on the `siterecovery.trilio.io` API group
- `oc` CLI if collecting a support bundle via `oc adm must-gather`
- The zone's protection namespace (`dr-<zone-name>`) and the workload clusters' `trilio-site-recovery-system` namespace
- Familiarity with ProtectionZone, ProtectionGroup, ProtectionRequest, FailoverRequest, and TestFailover CRs
- OpenShift 4.17 or later on the TSR-Manager cluster and both workload clusters (Tech Preview supported floor)

## Installation

No additional installation is required for diagnostics. All commands below run against the TSR-Manager cluster using `kubectl` (and `oc` for support bundle collection), which are the only CLIs you need for recovery operations. Ensure your kubeconfig is pointed at the TSR-Manager cluster before running any command in this guide.

```bash
# Verify you are targeting the TSR-Manager cluster
kubectl get pz -A
```

If you need a support bundle to share with Trilio support:

```bash
oc adm must-gather --image=quay.io/triliovault/site-recovery-must-gather:<release-tag>
```

Substitute `<release-tag>` with the pinned release tag supplied by Trilio with your release.

## Configuration

The following CR fields are most relevant when diagnosing and recovering from failed operations.

**FailoverRequest**

| Field | Default | Effect |
|---|---|---|
| `spec.failoverType` | `planned` | `planned` gracefully stops VMs on the source before promoting the DR side; `unplanned` force-promotes without stopping the source first |
| `spec.targetCluster` | *(required)* | `cluster1` or `cluster2`; identifies which workload cluster becomes active |
| `spec.skipHooks` | `false` | When `true`, all pre- and post-failover hooks are skipped — useful when a hook failure is blocking the operation |
| `spec.batchBootTimeoutSeconds` | `300` | Timeout per boot-order batch before the controller marks the batch failed |
| `spec.drainTimeoutSeconds` | `300` | Accepted and range-checked at admission but not consumed by the failover controller in Tech Preview; has no effect on drain or failover behaviour |

**TestFailover**

| Field | Default | Effect |
|---|---|---|
| `spec.cleanupPolicy` | `Automatic` | `Automatic` destroys test resources after the drill; `Manual` parks them for inspection after failure |
| `spec.schedulingCheck` | `Fail` | `Fail` aborts the test if DR-side scheduling cannot place a VM; `Warn` records the failure but continues |
| `spec.skipHooks` | `false` | Skips all test hooks — useful to bypass a failing verification hook |
| `spec.retentionTime` | `2h` | How long `Automatic`-policy test resources are retained before cleanup begins |
| `spec.verification.reachability[].timeout` | `30s` | Per-probe timeout; capped at 10 minutes by the webhook |

**ProtectionRequest**

| Field | Effect |
|---|---|
| `spec.sourceCluster` | Patching this to the DR cluster triggers the role flip that constitutes a failover for that VM; the FailoverRequest controller patches it for you during a group failover |

**site-recovery-logging ConfigMap** (runtime log level, no pod restart)

Create or patch this ConfigMap in the relevant namespace (TSR-Manager zone namespace `dr-<name>` or workload `trilio-site-recovery-system`) to increase verbosity during diagnosis:

```yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: site-recovery-logging
  namespace: dr-<zone-name>   # or trilio-site-recovery-system on a workload cluster
data:
  level: debug                       # global: error | info | debug | trace
  level.control-plane: debug
  level.drbd-node-agent: info
  level.protectionzone: debug
```

## Usage

### Identifying a failed operation

All DR operations expose their state through `status.state` and `status.phase`. Check these first.

**FailoverRequest**

```bash
# List all FailoverRequests in the zone namespace
kubectl get fr -n dr-<zone-name>
```

The printed columns are: `Protection Group`, `Target`, `State`, `Phase`, `Age`.

Key state values:
- `status.state`: `Pending | InProgress | Completed | Failed | Cancelled`
- `status.phase`: `Initializing | Validating | RunningPreHooks | StoppingSource | WaitingForLocks | RemovingTaints | StartingTarget | Monitoring | RunningPostHooks | Completed | Failed`

For per-VM detail:

```bash
kubectl get fr <fr-name> -n dr-<zone-name> -o jsonpath='{.status.vmStatuses}' | jq .
```

Note: `status.vmStatuses` is the only per-VM detail field; there is no `status.failedVMs` field and no `status.message` field.

**TestFailover**

```bash
kubectl get tfo -n dr-<zone-name>
```

The printed columns are: `Protection Group`, `Phase`, `Result`, `Passed`, `Scheduling`, `Age`.

Key phase values (exact): `Queued | PreparingEnvironment | CreatingSnapshots | CreatingVolumes | CreatingVMs | StartingVMs | Verifying | Succeeded | Failed | CleaningUp | Cleaned`

To find the isolated test namespace (never guess or construct this yourself — always read it from status):

```bash
kubectl get tfo <tfo-name> -n dr-<zone-name> \
  -o jsonpath='{.status.testNamespace}'
```

**ProtectionRequest**

```bash
kubectl get pr -n dr-<zone-name>
```

The printed columns are: `VM`, `Namespace`, `Phase`, `Sync`, `Age`.

Exact phase values: `Pending | Validating | CreatingDRBD | Syncing | ReadyToActivate | Activating | Protected | Unprotecting | Failed`

There is no `message` field on ProtectionRequest status; use conditions and events:

```bash
kubectl describe pr <pr-name> -n dr-<zone-name>
```

### Inspecting hook failures

Hook statuses are embedded in the FailoverRequest or TestFailover status:

```bash
kubectl get fr <fr-name> -n dr-<zone-name> \
  -o jsonpath='{.status.hookStatuses}' | jq .
```

Each hook status carries: `phase` (`Pending | Running | Succeeded | Failed | TimedOut | Skipped`), `message`, `attempts`, `exitCode`, and `jobName`. If a hook Job is still present, inspect it:

```bash
kubectl logs job/<job-name> -n <hook-job-namespace>
```

### Checking DRBD replication state during a stalled failover

DRBDResource CRs live on each workload cluster, not on the TSR-Manager cluster.

```bash
# Run against the relevant workload cluster
kubectl get dres -n <vm-namespace> --context <workload-cluster-context>
```

The printed columns are: `Side`, `Role`, `Node`, `State`, `Phase`, `Age`.

A stalled failover commonly shows `status.role` still as `Secondary` on the target side and `connectionState` as `Disconnected` or `Connecting`. Check sync progress:

```bash
kubectl get dres <resource-name> -n <vm-namespace> \
  --context <workload-cluster-context> \
  -o jsonpath='{.status.syncProgress}'
```

### Checking replication health via ReplicationGroupStatus

The ReplicationGroupStatus is auto-created per ProtectionGroup on the TSR-Manager cluster, named `<protection-group-name>-replication`:

```bash
kubectl get rgs -n dr-<zone-name>
```

The printed columns are: `Protection-Group`, `RPO-Status`, `Out-Of-Sync`, `Synced`, `Connection`, `Last-Sync`, `Age`.

```bash
# Inspect aggregated metrics
kubectl get rgs <pg-name>-replication -n dr-<zone-name> -o jsonpath='{.status}' | jq .
```

Filter ReplicationGroupStatus by ProtectionGroup using the label (field selectors on spec/status do not work on custom resources):

```bash
kubectl get rgs -n dr-<zone-name> \
  -l siterecovery.trilio.io/protection-group=<pg-name>
```

### Finding RPO violations

```bash
kubectl get rpoe -n dr-<zone-name> \
  -l siterecovery.trilio.io/protection-group=<pg-name>
```

RPOEvent CRs are immutable audit records. The printed columns are: `Protection-Group`, `Event-Type`, `RPO-Seconds`, `Out-Of-Sync`, `Reason`, `Timestamp`, `Age`.

## Examples

### Example 1 — Inspect a failed FailoverRequest and its per-VM statuses

```bash
# Get the overall state
kubectl get fr my-failover -n dr-zone1

# Get per-VM detail (status.vmStatuses is the authoritative per-VM field)
kubectl get fr my-failover -n dr-zone1 \
  -o jsonpath='{range .status.vmStatuses[*]}{.name}{"\t"}{.phase}{"\t"}{.error}{"\n"}{end}'

# Describe for events and conditions
kubectl describe fr my-failover -n dr-zone1
```

### Example 2 — Find the test namespace for a stuck TestFailover

```bash
# Read the controller-generated test namespace from status (never construct it manually)
TEST_NS=$(kubectl get tfo my-drill -n dr-zone1 \
  -o jsonpath='{.status.testNamespace}')
echo "Test namespace: $TEST_NS"

# List test VMs in that namespace (on the standby workload cluster)
kubectl get vm -n "$TEST_NS" --context <standby-cluster-context>
```

### Example 3 — Retry a TestFailover with Manual cleanup after failure

When a TestFailover fails at `Verifying` and you want to inspect test VMs before cleanup:

```bash
cat <<EOF | kubectl apply -f -
apiVersion: siterecovery.trilio.io/v1alpha1
kind: TestFailover
metadata:
  name: drill-inspect
  namespace: dr-zone1
spec:
  protectionGroupRef:
    name: pg-webapp
    namespace: dr-zone1
  cleanupPolicy: Manual
  schedulingCheck: Warn
  skipHooks: false
EOF
```

With `cleanupPolicy: Manual`, test resources remain in `status.testNamespace` after failure so you can SSH into VMs or inspect logs. When done:

```bash
# Signal cleanup by patching cleanupRequested
kubectl patch tfo drill-inspect -n dr-zone1 \
  --type merge -p '{"spec":{"cleanupRequested":true}}'
```

### Example 4 — Skip hooks on a blocked FailoverRequest

If a pre-failover hook is blocking the operation and you need to proceed:

```bash
cat <<EOF | kubectl apply -f -
apiVersion: siterecovery.trilio.io/v1alpha1
kind: FailoverRequest
metadata:
  name: failover-nohooks
  namespace: dr-zone1
spec:
  protectionGroupRef:
    name: pg-webapp
    namespace: dr-zone1
  targetCluster: cluster2
  failoverType: planned
  skipHooks: true
EOF
```

### Example 5 — Check DRBD resource state on a workload cluster

```bash
# List all DRBDResources for a VM namespace on the primary workload cluster
kubectl get dres -n my-vm-namespace --context primary-cluster

# Inspect connection and sync state of a specific resource
kubectl get dres drbd-my-vm-namespace-my-vm-a1b2c3d4 \
  -n my-vm-namespace \
  --context primary-cluster \
  -o jsonpath='{"Role: "}{.status.role}{"\nState: "}{.status.connectionState}{"\nPhase: "}{.status.phase}{"\nSync: "}{.status.syncProgress.percentage}{"% \nBytes remaining: "}{.status.syncProgress.bytesRemaining}{"\n"}'
```

### Example 6 — Increase log verbosity on the TSR-Manager control plane

```bash
kubectl patch configmap site-recovery-logging \
  -n dr-zone1 \
  --type merge \
  -p '{"data":{"level":"debug","level.control-plane":"debug"}}'
```

Changes take effect at the next log-level watch interval without pod restarts.

### Example 7 — Filter protected VMs by label on a workload cluster

```bash
# The siterecovery.trilio.io/protected-vm=true label is stamped on the
# KubeVirt VirtualMachine object on the workload cluster only
kubectl get vm -A \
  -l siterecovery.trilio.io/protected-vm=true \
  --context primary-cluster
```

### Example 8 — Collect a support bundle

```bash
# Run on the TSR-Manager cluster; the must-gather image reaches through to
# all zone workload clusters automatically
oc adm must-gather \
  --image=quay.io/triliovault/site-recovery-must-gather:<release-tag>
```

The archive is written to the current directory. Contact Trilio support and attach the archive.

## Troubleshooting

### Issue: FailoverRequest stuck in `WaitingForLocks` or `StoppingSource`

**Symptom:** `kubectl get fr` shows `Phase: StoppingSource` or `Phase: WaitingForLocks` for an extended period. Per-VM statuses show `Stopping` but do not progress.

**Likely cause:** A running protected VM on the source cluster cannot be stopped because the node hosting it cannot be drained. A protected VM is pinned to one node for the lifetime of its protection; `oc adm drain` of that node will not complete while the VM is running and protected.

**Fix:**
1. Identify which VMs are stuck using `status.vmStatuses`:
   ```bash
   kubectl get fr <fr-name> -n dr-<zone-name> \
     -o jsonpath='{range .status.vmStatuses[*]}{.name}{"\t"}{.phase}{"\n"}{end}'
   ```
2. For a planned failover, verify the source-side VM is reachable and not held by an external process.
3. If the source cluster is unavailable and you need to proceed, delete the stuck FailoverRequest **only after confirming no VM is in an ambiguous state where it may be running on both clusters**, then create a new FailoverRequest with `failoverType: unplanned`. An unplanned failover force-promotes the DR side without waiting for the source to stop.

> **Caution:** Do not start VMs on the source cluster after the DR side has been promoted. Do not delete a stuck FailoverRequest while VMs are in an unknown state — the controller is re-entrant and will resume where it left off if the CR is preserved.

---

### Issue: FailoverRequest in `Failed` state, hook in `Failed` or `TimedOut`

**Symptom:** `status.state: Failed`, `status.phase: RunningPreHooks` or `RunningPostHooks`. A hook status shows `phase: Failed` or `phase: TimedOut`.

**Likely cause:** A Hook Job exceeded its timeout, returned a non-zero exit code, or the hook Job pod could not be scheduled.

**Fix:**
1. Identify the failing hook and its Job:
   ```bash
   kubectl get fr <fr-name> -n dr-<zone-name> \
     -o jsonpath='{range .status.hookStatuses[?(@.phase=="Failed")]}{.hookName}{"\t"}{.jobName}{"\t"}{.message}{"\n"}{end}'
   ```
2. Inspect the Job logs:
   ```bash
   kubectl logs job/<job-name> -n <hook-job-namespace>
   ```
3. Fix the underlying issue in the Hook Job image or configuration.
4. If the hook is non-critical and you need to proceed now, create a new FailoverRequest with `spec.skipHooks: true`. Be aware this bypasses all hooks including verification steps.
5. A Hook CR is frozen against edits while an operation is running it. Wait for the operation to reach a terminal state before editing the Hook spec.

---

### Issue: ProtectionRequest stuck in `Syncing` for an extended period

**Symptom:** `kubectl get pr` shows `Phase: Syncing` with a non-advancing `Sync` percentage.

**Likely cause:** DRBD replication is stalled. Common causes: network interruption between primary and DR worker nodes on TCP ports 7000–8000, the DRBD connection is `Disconnected` or `Connecting`, or a node agent pod is not running.

**Fix:**
1. Check the DRBDResource on both workload clusters:
   ```bash
   kubectl get dres -n <vm-namespace> --context <primary-cluster>
   kubectl get dres -n <vm-namespace> --context <dr-cluster>
   ```
2. Look for `connectionState: Disconnected` or `phase: Degraded`.
3. Verify network connectivity between worker nodes on TCP 7000–8000. Port 8000 is mandatory — a firewall stopping at 7999 will silently prevent replication on one volume.
4. Check that the drbd-node-agent DaemonSet pods are running on both clusters:
   ```bash
   kubectl get pods -n trilio-site-recovery-system \
     -l app.kubernetes.io/component=drbd-node-agent \
     --context <workload-cluster>
   ```
5. Check node-agent logs for the specific node hosting the VM. Because `kubectl logs` does not accept `--field-selector`, use two steps:
   ```bash
   # Step 1: get the agent pod on the node
   AGENT_POD=$(kubectl get pods -n trilio-site-recovery-system \
     --field-selector spec.nodeName=<node-name> \
     -l app.kubernetes.io/component=drbd-node-agent \
     -o jsonpath='{.items[0].metadata.name}' \
     --context <workload-cluster>)
   # Step 2: stream logs
   kubectl logs "$AGENT_POD" -n trilio-site-recovery-system \
     --context <workload-cluster>
   ```

---

### Issue: ProtectionRequest in `Failed` phase

**Symptom:** `kubectl get pr` shows `Phase: Failed`. `kubectl describe pr` shows a condition with reason indicating the failure.

**Likely cause:** DRBD provisioning failed, the VM reference is invalid, or a DRBD role conflict was detected.

**Common conditions and fixes:**

- **Reason `RoleConflict`** on the ProtectionRequest `Ready` condition: Both sides of the DRBD pair are claiming the Primary role simultaneously. This can occur after an unplanned failover if the original source cluster comes back online. Do not start VMs on the original source cluster. Investigate DRBD state on both workload clusters and ensure only the intended active cluster holds the Primary role.

- **Phase stays `Failed` after fixing the underlying issue:** Re-protect always requires a full unprotect followed by a full initial resync. Delete the ProtectionRequest and wait for `Unprotecting` to complete before creating a new one. The admission webhook will reject a new ProtectionRequest for the same VM while the previous one (including one in `Terminating`) still exists.

  ```bash
  kubectl delete pr <pr-name> -n dr-<zone-name>
  # Wait until the object is fully gone (finalizers removed)
  kubectl get pr <pr-name> -n dr-<zone-name>
  # Then create a new ProtectionRequest
  ```

---

### Issue: TestFailover stuck in `CreatingVolumes` or `CreatingVMs`

**Symptom:** `kubectl get tfo` shows `Phase: CreatingVolumes` or `Phase: CreatingVMs` with no progress.

**Likely cause:** DRBD snapshots could not be taken (the replication may be in a degraded state), or the DR cluster cannot schedule the test VMs due to resource constraints. `spec.schedulingCheck: Fail` will abort when a VM cannot be placed.

**Fix:**
1. Check the TestFailover conditions:
   ```bash
   kubectl describe tfo <tfo-name> -n dr-<zone-name>
   ```
2. Check the scheduling verdict:
   ```bash
   kubectl get tfo <tfo-name> -n dr-<zone-name> \
     -o jsonpath='{.status.schedulingCheck}' | jq .
   ```
3. Check replication health before running a drill — a degraded replication state means snapshots may be inconsistent:
   ```bash
   kubectl get rgs <pg-name>-replication -n dr-<zone-name> \
     -o jsonpath='{.status.aggregated}' | jq .
   ```
4. If the DR cluster lacks capacity, either free resources or set `spec.schedulingCheck: Warn` to get a result with a scheduling warning rather than an abort.
5. After investigating, clean up the failed TestFailover. If you used `cleanupPolicy: Manual`, patch `spec.cleanupRequested: true` to trigger cleanup. Do not manually patch finalizers to force deletion of test resources.

---

### Issue: ProtectionZone `ClustersValidated` condition is False, blocking new ProtectionGroup or ProtectionRequest creates

**Symptom:** Creating a ProtectionGroup or ProtectionRequest is rejected by the webhook with a message referencing zone validation.

**Likely cause:** The ProtectionZone `ClustersValidated` condition is not `True`. This is the gate for all PG and PR admission. Common reasons:

| Reason | Meaning | Fix |
|---|---|---|
| `ClusterAlreadyClaimed` | A workload cluster is already registered to another zone | Remove the zone-exclusivity annotation or use a different cluster |
| `InsufficientClusterPermissions` | Zone credential authenticates but RBAC check failed | Rotate credentials: `tsrctl zone-credential rotate --refresh-rbac` (contact Trilio support to obtain `tsrctl`) |
| `ClusterIdentityChanged` | Credential now reaches a cluster with a different `kube-system` UID | No override exists; restore the original cluster or re-create the zone |
| `SCCPrerequisiteMissing` | Required SCC is missing on the workload cluster | Apply the required SCC before creating the zone |
| `KubeVirtNotInstalled` | KubeVirt is not installed on the workload cluster | Install KubeVirt |

```bash
# Check the zone conditions
kubectl get pz <zone-name> -n dr-<zone-name> \
  -o jsonpath='{.status.conditions}' | jq .
```

---

### Issue: drbd-node-agent pods not starting; node has `siterecovery.trilio.io/not-ready:NoSchedule` taint

**Symptom:** Worker nodes remain tainted and VMs cannot be scheduled by the DR system. DaemonSet pods show `Pending` or `CrashLoopBackOff`.

**Likely cause:** The node agent applies the `siterecovery.trilio.io/not-ready:NoSchedule` taint to its own node during startup until DRBD converges. If the agent cannot start (missing DRBD kernel module, SCC admission denied, image pull failure), the taint is never removed.

**Fix:**
1. Check DaemonSet pod status:
   ```bash
   kubectl get pods -n trilio-site-recovery-system \
     -l app.kubernetes.io/component=drbd-node-agent \
     --context <workload-cluster>
   ```
2. Describe a failing pod for events:
   ```bash
   kubectl describe pod <agent-pod-name> -n trilio-site-recovery-system \
     --context <workload-cluster>
   ```
3. Common causes:
   - DRBD kernel module (`drbd9`) not loaded — verify the module loader init container succeeded.
   - SCC admission denied — check for `SCCAdmissionDenied` reason on the ProtectionZone `WorkloadStacksReady` condition.
   - Image pull error — verify the workload cluster can reach `quay.io/triliovault`.
4. Once the agent starts successfully, it removes the taint automatically when DRBD converges. Do not remove the taint manually — the agent re-applies it on restart if DRBD has not converged.