--- title: Troubleshooting product: trilio-site-recovery-for-kubernetes doc_type: runbook version: 0.2.0 source: git2docs (code-derived, validation-filtered) canonical: https://docs-tsr.trilio.io/trilio-site-recovery-for-kubernetes/site-recovery/troubleshooting --- # Troubleshooting _Common controller errors, debugging steps, and log analysis for site recovery operations_ ## Objective Diagnose and resolve common controller errors, interpret log output, and unblock stalled Site Recovery operations across the TSR-Manager and workload clusters. ## Scope This runbook covers: - Identifying and interpreting error conditions on ProtectionZone, ProtectionRequest, FailoverRequest, and TestFailover CRs - Reading controller logs on the TSR-Manager cluster and workload clusters - Diagnosing degraded DRBD replication and node agent problems - Resolving stalled or failed protection, failover, and test-failover operations - Adjusting runtime log levels without pod restarts This runbook does **not** cover: - Initial installation or zone provisioning (see the install runbook) - Zone credential rotation (see the zone-credential runbook) - Node loss and DRBDResource rebind (see the node-loss runbook) - Support bundle collection (see the tsr-gather runbook) - DRBD kernel module loading failures (see the drbd-module-loading runbook) ## Prerequisites Before working through this runbook, ensure you have: - `kubectl` and `oc` (OpenShift CLI) installed and configured with access to the TSR-Manager cluster and both workload clusters - Sufficient RBAC on the TSR-Manager cluster to read and describe all Site Recovery CRs (`protectionzones`, `protectiongroups`, `protectionrequests`, `failoverrequests`, `testfailovers`, `drbdreplicationpolicies`, `replicationgroupstatuses`, `rpoevents`, `hooks`) and to read pod logs in the zone namespace (`dr-`) - Sufficient RBAC on each workload cluster to read pods, logs, DRBDResource CRs, and node objects in `trilio-site-recovery-system` and VM namespaces - The zone namespace on TSR-Manager (`dr-`) and the workload namespace (`trilio-site-recovery-system`) identified for the zone under investigation - OpenShift 4.17 or later on the TSR-Manager cluster and both workload clusters ## Steps ### 1. Identify which component is reporting the problem Site Recovery state is expressed through Kubernetes conditions and phases on its CRs. Start by reading the phase and conditions of the top-level object that prompted the investigation. **ProtectionZone** (lives in `dr-` on TSR-Manager): ```bash kubectl -n dr- get pz -o yaml ``` Key fields: - `status.phase`: one of `Provisioning`, `Available`, `Unavailable`, `Degraded`, `Deleting` - `status.conditions[].type` and `.reason`: the conditions below name the specific gate that is unmet - `status.clusters[].status` and `.reason`: per-cluster reachability and why it failed Common `ClustersValidated=False` reasons and what they mean: | Reason | Meaning | First action | |---|---|---| | `ClusterAlreadyClaimed` | A workload cluster is leased to a different zone | See the zone-cluster-lease runbook | | `InsufficientClusterPermissions` | The zone credential authenticates but fails RBAC checks | Run `tsrctl zone-credential rotate --zone --kubeconfig , --refresh-rbac` | | `ClusterIdentityChanged` | The credential now reaches a cluster with a different `kube-system` UID | Restore the original cluster; there is no override | | `SCCPrerequisiteMissing` | The `privileged` SCC or its ClusterRole is absent on a workload cluster | Restore the SCC; this is a platform prerequisite | | `KubeVirtNotInstalled` | KubeVirt is not installed on a workload cluster | Install OpenShift Virtualization on that cluster | While `ClustersValidated` is not `True`, the TSR-Manager webhooks reject all ProtectionGroup and ProtectionRequest creates or spec changes for this zone. --- ### 2. Read controller logs All TSR-Manager-side controllers (protection, failover, test-failover, replication monitor, pg-sync) run in the **manager-control-plane** pod in the zone namespace. There is one container named `manager`, so the pod shows `1/1`. **Find the manager pod:** ```bash kubectl -n dr- get pods -l app.kubernetes.io/component=manager-control-plane ``` **Stream logs:** ```bash kubectl -n dr- logs -l app.kubernetes.io/component=manager-control-plane -c manager --follow ``` **Tail recent logs with a time window:** ```bash kubectl -n dr- logs -l app.kubernetes.io/component=manager-control-plane -c manager --since=1h ``` **Find the workload control-plane pod on a workload cluster** (the placement reconciler, admission webhooks, and DRBD resource controller run here): ```bash kubectl --context -n trilio-site-recovery-system get pods \ -l app.kubernetes.io/component=workload-control-plane ``` **Find node agent pods** (one per worker node; named `-site-recovery-workload-control-plane-agent-`): ```bash kubectl --context -n trilio-site-recovery-system get pods \ -l app.kubernetes.io/component=drbd-node-agent ``` To read logs from the agent on a specific node, first resolve the pod name: ```bash POD=$(kubectl --context -n trilio-site-recovery-system get pods \ --field-selector spec.nodeName= \ -l app.kubernetes.io/component=drbd-node-agent \ -o jsonpath='{.items[0].metadata.name}') kubectl --context -n trilio-site-recovery-system logs "$POD" ``` --- ### 3. Increase log verbosity at runtime You can change the log level of any Site Recovery component without restarting pods. The `site-recovery-logging` ConfigMap in the relevant namespace controls the level. Accepted values: `error`, `info`, `debug`, `trace` **Increase TSR-Manager control plane verbosity** (ConfigMap lives in the zone namespace): ```bash kubectl -n dr- create configmap site-recovery-logging \ --from-literal=level=debug \ --from-literal=level.control-plane=debug \ --dry-run=client -o yaml | kubectl apply -f - ``` **Increase workload control plane and node agent verbosity** (ConfigMap lives in `trilio-site-recovery-system` on the workload cluster): ```bash kubectl --context -n trilio-site-recovery-system \ create configmap site-recovery-logging \ --from-literal=level=debug \ --from-literal=level.drbd-node-agent=debug \ --dry-run=client -o yaml | kubectl apply -f - ``` Changes take effect within approximately 15 seconds without pod restarts. Revert by deleting the key or setting it back to `info`. --- ### 4. Diagnose a stalled ProtectionRequest ProtectionRequest phases in order: `Pending` → `Validating` → `CreatingDRBD` → `Syncing` → `ReadyToActivate` → `Activating` → `Protected`. Teardown runs through `Unprotecting`. `Failed` is terminal. **Check the current phase and conditions:** ```bash kubectl -n dr- get protect kubectl -n dr- describe protectionrequest ``` The `kubectl get` output columns are: VM, Namespace, Phase, Sync, Age. **Common stall causes by phase:** | Phase | What to look for | |---|---| | `Validating` | The VM must be running when you protect it; a stopped VM causes validation failure. The DRBD metadata StorageClass must not be `WaitForFirstConsumer`. Check conditions for the specific reason. | | `CreatingDRBD` | The DRBDResource pair is being created on both workload clusters. Check node agent logs and DRBDResource status on both workload clusters (see step 6). | | `Syncing` | Initial sync is in progress. Check `status.syncProgress` and `status.slowestVolumeSyncProgress`. A `Syncing` phase that shows 0% progress for more than a few minutes suggests a DRBD connectivity problem — check ports 7000–8000 TCP between primary and DR worker nodes. | | `ReadyToActivate` | Initial sync is complete; the controller is switching the VM to frontend PVCs. If stuck here, check for admission webhook errors or node agent logs on both sides. | | `Failed` | Read `status.conditions` for the specific reason. Check controller logs for the reconcile error that set this phase. | Note: `ReadyToActivate` means the initial sync finished and the VM is being switched — this is a normal operator-action state, not a stall by itself. A VM stays in `ReadyToActivate` only if the activation step is blocked. **Check conditions on the ProtectionRequest:** ```bash kubectl -n dr- get protectionrequest \ -o jsonpath='{range .status.conditions[*]}{.type}: {.status} ({.reason}) — {.message}{"\n"}{end}' ``` If the `Ready` condition shows `reason: RoleConflict`, a DRBD Primary/Secondary role disagreement has been detected between the two DRBDResource copies. This requires investigation of DRBD state on both nodes — see step 6. **Check DRBDResource status on both workload clusters** (DRBDResource CRs live on the workload clusters, not TSR-Manager): ```bash # On the primary workload cluster, in the VM's namespace: kubectl --context -n get dres kubectl --context -n describe drbdresource # On the DR workload cluster: kubectl --context -n get dres kubectl --context -n describe drbdresource ``` The DRBDResource name follows the pattern `drbd---`. The `kubectl get dres` output columns are: Side, Role, Node, State, Phase, Age. A healthy protected VM shows the source-side copy with `Role: Primary` and `State: Connected`, and the target-side copy with `Role: Secondary` and `State: Connected`. **Important:** Do not run `kubectl get drbdresource` against the TSR-Manager cluster — the `drbdresources` CRD is installed only on workload clusters. Running it against TSR-Manager returns `server doesn't have a resource type`. --- ### 5. Diagnose a stalled or failed FailoverRequest FailoverRequest has two status fields: `status.state` (overall lifecycle: `Pending`, `InProgress`, `Completed`, `Failed`, `Cancelled`) and `status.phase` (granular step: `Initializing`, `Validating`, `RunningPreHooks`, `StoppingSource`, `WaitingForLocks`, `RemovingTaints`, `StartingTarget`, `Monitoring`, `RunningPostHooks`, `Completed`, `Failed`). **Check state and phase:** ```bash kubectl -n dr- get fr kubectl -n dr- describe failoverrequest ``` The `kubectl get` output columns are: Protection Group, Target, State, Phase, Age. There is no TYPE or STATUS column. **Per-VM detail** is in `status.vmStatuses`, not a top-level `failedVMs` field (which does not exist): ```bash kubectl -n dr- get failoverrequest \ -o jsonpath='{range .status.vmStatuses[*]}{.name}: {.phase} — source={.sourceStatus} target={.targetStatus}{"\n"}{.error}{"\n"}{end}' ``` **Hook status** (if hooks are configured): ```bash kubectl -n dr- get failoverrequest \ -o jsonpath='{range .status.hookStatuses[*]}{.name} ({.type}): {.phase} — {.message}{"\n"}{end}' ``` **Common stall causes by phase:** | Phase | What to look for | |---|---| | `RunningPreHooks` | A hook Job has not completed. Read its logs on the workload cluster in the namespace shown in `status.hookStatuses[].namespace`. If the hook is broken, recreate the FailoverRequest with `spec.skipHooks: true`. | | `StoppingSource` | The source-side VMs are not stopping. Check VMI status on the source workload cluster. For an `unplanned` failover, source VMs may be unreachable — this is expected. | | `WaitingForLocks` | The DRBD Primary role has not been confirmed on the target side. Check DRBDResource status on both clusters (step 6). A planned failover waits for sync to complete before promoting; if the connection is lost, resolve DRBD connectivity first. | | `StartingTarget` | Target-side VMs are not reaching Running. Check VMI and pod status on the target workload cluster in the VM namespaces. | | `RunningPostHooks` | A post-hook Job has not completed. The infrastructure failover has already happened at this point — the FailoverRequest may be marked Failed, but the VMs are running on the target. Do not attempt to roll back manually. | **Critical caution:** Do not delete a stuck FailoverRequest while VMs are in an unknown state. The failover controller is re-entrant — it resumes from where it left off on the next reconcile. Deleting the CR while a role flip is mid-flight can leave DRBD in an inconsistent state. If a FailoverRequest is stuck, wait for the controller to time out or resolve the underlying cause first. **After a DR-side promotion, do not start VMs on the primary cluster.** Only one side should have VMs running at a time. Starting VMs on the demoted primary while the DR side is also running can corrupt data. --- ### 6. Diagnose degraded DRBD replication **Check the ReplicationGroupStatus** (auto-created by the replication monitor, one per ProtectionGroup, named `-replication`, on TSR-Manager): ```bash kubectl -n dr- get rgs kubectl -n dr- describe replicationgroupstatus -replication ``` The `kubectl get` output columns are: Protection-Group, RPO-Status, Out-Of-Sync, Synced, Connection, Last-Sync, Age. Key status fields: - `status.aggregated.connectionState`: `Connected`, `Degraded`, or `Disconnected` — worst-case across all VMs in the group - `status.aggregated.outOfSyncBytes`: total bytes not yet replicated - `status.rpoStatus`: `Met`, `Degraded`, `Violated`, or `Unknown` **Check RPOEvent records** for a history of replication state transitions: ```bash kubectl -n dr- get rpoe -l siterecovery.trilio.io/protection-group= ``` **Check DRBDResource connection state** on both workload clusters. DRBD replication requires TCP ports 7000–8000 inclusive (one port per protected VM) to be open between primary and DR worker nodes. Port 8000 is not optional — a firewall rule that stops at 7999 will silently prevent replication for one VM volume. To verify the allocated port for a VM: ```bash kubectl -n dr- get protectionrequest \ -o jsonpath='{.status.drbdResourceRef.port}{"\n"}' ``` **Check node agent status** — the node agent applies a `siterecovery.trilio.io/not-ready:NoSchedule` taint to its node during startup. If this taint persists beyond the `startupDeadlineSeconds` (default 120s), the agent emits a `DRBDStartupDeadlineExceeded` Warning event on the Node and the DRBDResource carries `NodeStartupGate=False`. This clears automatically when DRBD converges. ```bash # Check for the startup taint on workload nodes: kubectl --context get nodes \ -o custom-columns='NAME:.metadata.name,TAINTS:.spec.taints[*].key' # Check node events for startup deadline warnings: kubectl --context get events --field-selector reason=DRBDStartupDeadlineExceeded ``` **Check for a DRBD node agent pod stuck in Init** (the `drbd-module-loader` init container failed to load the kernel module): ```bash kubectl --context -n trilio-site-recovery-system get pods \ -l app.kubernetes.io/component=drbd-node-agent ``` If a pod shows `Init:Error` or `Init:CrashLoopBackOff`: ```bash kubectl --context -n trilio-site-recovery-system logs -c drbd-module-loader ``` Common causes: no compatible kmod RPM for this kernel version (mirror lag), or a foreign DRBD module already loaded with `usermode_helper` enabled (requires node reboot). See the drbd-module-loading runbook for detailed resolution steps. --- ### 7. Diagnose a stalled or failed TestFailover TestFailover phases in order: `Queued` → `PreparingEnvironment` → `CreatingSnapshots` → `CreatingVolumes` → `CreatingVMs` → `StartingVMs` → `Verifying` → `Succeeded`. Cleanup runs through `CleaningUp` → `Cleaned`. `Failed` is terminal. **Check the phase, result, and scheduling verdict:** ```bash kubectl -n dr- get tfo kubectl -n dr- describe testfailover ``` The `kubectl get` output columns are: Protection Group, Phase, Result, Passed, Scheduling, Age. **Read the test namespace** — this is a controller-generated value, never user-provided: ```bash kubectl -n dr- get testfailover \ -o jsonpath='{.status.testNamespace}{"\n"}' ``` The test namespace is named `site-recovery--` and is created on the target workload cluster. **Common stall causes by phase:** | Phase | What to look for | |---|---| | `Queued` | Validation is pending. Check `status.conditions[type=Validated]` — a `FailoverRequest` in progress for the same ProtectionGroup blocks a TestFailover. | | `PreparingEnvironment` | The test namespace or network isolation policies could not be created. `EnvironmentReady=False` with reason `NamespaceCreateFailed` or `NetworkIsolationFailed` indicates an RBAC or admission problem on the target workload cluster. This phase also runs pre-test hooks; `reason: PreTestHookFailed` means a hook aborted before any snapshot. | | `CreatingSnapshots` | VolumeSnapshot creation failed or is not becoming ready. Check `status.conditions[type=SnapshotsReady]`. Reason `SnapshotClassNotFound` means no VolumeSnapshotClass on the standby cluster matches the volume's StorageClass provisioner — this requires a VolumeSnapshotClass to be installed on that cluster. Reason `SnapshotError` means the storage layer is reporting an error; the phase waits up to 5 minutes before failing. | | `CreatingVolumes` | Test PVCs could not be created from the snapshots. Check conditions and events on the target workload cluster in the test namespace. | | `StartingVMs` | Test VMs are not reaching Running. The test namespace is network-fenced (default-deny NetworkPolicy); test VMs can only communicate with each other and with cluster DNS. Check VMI status in the test namespace on the target cluster. | | `Verifying` | Automatic boot/agent/filesystem checks or optional reachability probes are failing. Check `status.verificationResults.checks` for which check failed and why. | | `Failed` | Check `status.conditions` and `status.verificationResults`. The `status.result` and `status.resultMessage` survive cleanup and describe the outcome even after the test resources are removed. | **Check scheduling probe results** (for `schedulingCheck` failures): ```bash kubectl -n dr- get testfailover \ -o jsonpath='{.status.schedulingCheck}{"\n"}' ``` If `verdict` is `Failed` or `Warned`, `status.schedulingCheck.excludedVMs` names the VMs the DR scheduler could not place, with the scheduler's own message. **A TestFailover that is concurrent with a FailoverRequest for the same ProtectionGroup** is refused at validation (`reason: FailoverInProgress`). Wait for the FailoverRequest to complete before starting a test. **To trigger early cleanup** of a running or completed test while keeping the record: ```bash kubectl -n dr- patch testfailover --type=merge \ -p '{"spec":{"cleanupRequested":true}}' ``` This moves the phase to `CleaningUp` then `Cleaned` and preserves `status.result`. Do **not** patch `metadata.finalizers` to force deletion of a test namespace — test namespace cleanup is driven by the controller's cleanup pipeline, not by namespace finalizers, and patching finalizers directly will leave snapshots, PersistentVolumes, and VolumeSnapshotContents behind. --- ### 8. Diagnose a degraded node (node agent not converging) A protected VM is pinned to one node per cluster for the lifetime of the protection. Live migration and cold migration of a protected VM are denied by the admission webhook. Running `oc adm drain` on a node hosting a running protected VM will not complete. If a node is showing `NodeStartupGate=False` or a DRBDResource is in `Degraded` or `Failed` phase on a workload cluster: 1. Read the DRBDResource conditions on that workload cluster: ```bash kubectl --context -n get dres \ -o jsonpath='{range .status.conditions[*]}{.type}: {.status} ({.reason}) — {.message}{"\n"}{end}' ``` 2. Check the condition `PlacementReady` — this gates metadata-PVC provisioning. If `PlacementReady=False`, the anchor pod for this resource may not have landed on the expected node yet. 3. Read the node agent log for that specific node (see step 2 for how to locate the pod by node name). 4. Check whether the metadata PVCs exist. They are named `drbd-md--v` on the workload cluster in the VM's namespace: ```bash kubectl --context -n get pvc | grep drbd-md- ``` 5. If a DRBDResource is stuck in `Deleting` because its bound node is gone (the node agent finalizer `siterecovery.trilio.io/drbdresource-agent` cannot be cleared by a non-existent node), follow the node-loss runbook. Do **not** patch the placement finalizer (`siterecovery.trilio.io/drbdresource-placement`) directly — doing so leaks frontend PVCs and PersistentVolumes permanently. --- ### 9. Check the ProtectionZone controller The ProtectionZone controller (the operator that validates zones and installs workload stacks) runs as its own deployment on TSR-Manager: ```bash kubectl -n get pods \ -l app.kubernetes.io/component=protectionzone-controller kubectl -n logs \ -l app.kubernetes.io/component=protectionzone-controller --follow ``` The operator namespace is where the OLM Subscription was created, typically the same namespace as the CSV. If the ProtectionZone controller has restarted, it re-applies all CRDs with server-side-apply and `ForceOwnership` on each reconcile. Any out-of-band CRD edits are reverted within one reconcile cycle. ## Verification After resolving the root cause, verify the following: **ProtectionZone is Available:** ```bash kubectl -n dr- get pz ``` `status.phase` should be `Available` and `ClustersValidated=True`. **ProtectionRequest is Protected:** ```bash kubectl -n dr- get protect ``` All entries should show `Phase: Protected` and `Sync: 100` (or the sync column showing the current replication percentage). **DRBDResources are Connected and Consistent on both workload clusters:** ```bash kubectl --context -n get dres kubectl --context -n get dres ``` Expect `State: Connected` and `Phase: Consistent` on both copies. **ReplicationGroupStatus shows Connected and RPO Met:** ```bash kubectl -n dr- get rgs ``` Expect `Connection: Connected` and `RPO-Status: Met`. **FailoverRequest Completed:** ```bash kubectl -n dr- get fr ``` Expect `State: Completed` and `Phase: Completed`. **TestFailover Succeeded (or Cleaned after success):** ```bash kubectl -n dr- get tfo ``` Expect `Phase: Succeeded` (or `Cleaned` if cleanup ran) and `Result: Succeeded`. **No protected VMs carry the startup taint on either workload cluster:** ```bash kubectl --context get nodes \ -o custom-columns='NAME:.metadata.name,TAINTS:.spec.taints[*].key' | grep not-ready ``` If no output is returned, no nodes are tainted by the node agent. ## Rollback The actions in this runbook are primarily diagnostic (reading logs and status). The only mutating actions are: **Reverting log level changes:** Delete the `site-recovery-logging` ConfigMap in the affected namespace to return all components to their startup log level: ```bash # On TSR-Manager, in the zone namespace: kubectl -n dr- delete configmap site-recovery-logging # On workload clusters: kubectl --context -n trilio-site-recovery-system delete configmap site-recovery-logging ``` Or set specific keys back to `info`: ```bash kubectl -n dr- create configmap site-recovery-logging \ --from-literal=level=info \ --dry-run=client -o yaml | kubectl apply -f - ``` **Reverting a `cleanupRequested: true` patch on a TestFailover:** The `cleanupRequested` field can be set but never unset. If you triggered cleanup prematurely, the test resources (VMs, PVCs, snapshots, test namespace) will be deleted. The TestFailover CR itself and its `status.result` and `status.resultMessage` are preserved. **If you patched DRBDResource finalizers manually:** If you incorrectly removed the agent finalizer (`siterecovery.trilio.io/drbdresource-agent`) on a DRBDResource whose node is still alive, the placement reconciler will proceed with storage teardown without the DRBD device having been cleanly torn down on the kernel side. Contact Trilio support immediately with the DRBDResource name, namespace, and the node it was bound to. **Re-protecting a VM after unprotect:** Re-protect always requires a full initial sync. There is no resume-from-checkpoint path. Create a new ProtectionRequest for the VM after the previous one has fully deleted (the `Unprotecting` phase has completed and the CR is gone). ## Escalation If this runbook does not resolve the issue, contact Trilio support with the following information: 1. **A support bundle collected with `oc adm must-gather`** from the TSR-Manager cluster, covering the affected zones. Collect this as close to the incident time as possible: ```bash oc adm must-gather --image=quay.io/triliovault/site-recovery-must-gather: ``` Include `--since 2h` (or a longer window if the incident is older) to capture relevant log history. Contact Trilio support to obtain the correct pinned image tag for your release. 2. **The output of the following commands**, run against the TSR-Manager cluster, identifying the affected zone and operation: ```bash kubectl -n dr- get pz,pg,pr,fr,tfo,rgs,rpoe -o wide kubectl -n dr- describe protectionzone kubectl -n dr- describe ``` 3. **The output of `kubectl get dres`** from both workload clusters in the affected VM namespaces: ```bash kubectl --context -n get dres -o yaml kubectl --context -n get dres -o yaml ``` 4. **Node kernel versions** for affected workload nodes: ```bash oc --context get nodes \ -o custom-columns='NAME:.metadata.name,KERNEL:.status.nodeInfo.kernelVersion,OS:.status.nodeInfo.osImage' ``` 5. **A description of the sequence of events** that led to the current state, including any manual interventions (CR patches, pod restarts, namespace deletions) performed before opening the ticket. Do not share kubeconfig files or the contents of zone credential Secrets. The support bundle redacts all Secret data, kubeconfigs, and cloud-init payloads.