Troubleshooting

Diagnose and resolve common controller errors, interpret log output, and unblock stalled Site Recovery operations across the TSR-Manager and workload clusters.

This runbook covers:

• Identifying and interpreting error conditions on ProtectionZone, ProtectionRequest, FailoverRequest, and TestFailover CRs
• Reading controller logs on the TSR-Manager cluster and workload clusters
• Diagnosing degraded DRBD replication and node agent problems
• Resolving stalled or failed protection, failover, and test-failover operations
• Adjusting runtime log levels without pod restarts

This runbook does not cover:

• Initial installation or zone provisioning (see the install runbook)
• Zone credential rotation (see the zone-credential runbook)
• Node loss and DRBDResource rebind (see the node-loss runbook)
• Support bundle collection (see the tsr-gather runbook)
• DRBD kernel module loading failures (see the drbd-module-loading runbook)

Before working through this runbook, ensure you have:

• kubectl and oc (OpenShift CLI) installed and configured with access to the TSR-Manager cluster and both workload clusters
• Sufficient RBAC on the TSR-Manager cluster to read and describe all Site Recovery CRs (protectionzones, protectiongroups, protectionrequests, failoverrequests, testfailovers, drbdreplicationpolicies, replicationgroupstatuses, rpoevents, hooks) and to read pod logs in the zone namespace (dr-<zone-name>)
• Sufficient RBAC on each workload cluster to read pods, logs, DRBDResource CRs, and node objects in trilio-site-recovery-system and VM namespaces
• The zone namespace on TSR-Manager (dr-<zone-name>) and the workload namespace (trilio-site-recovery-system) identified for the zone under investigation
• OpenShift 4.17 or later on the TSR-Manager cluster and both workload clusters

1. Identify which component is reporting the problem

Site Recovery state is expressed through Kubernetes conditions and phases on its CRs. Start by reading the phase and conditions of the top-level object that prompted the investigation.

ProtectionZone (lives in dr-<zone-name> on TSR-Manager):

kubectl -n dr-<zone-name> get pz <zone-name> -o yaml

Key fields:

• status.phase: one of Provisioning, Available, Unavailable, Degraded, Deleting
• status.conditions[].type and .reason: the conditions below name the specific gate that is unmet
• status.clusters[].status and .reason: per-cluster reachability and why it failed

Common ClustersValidated=False reasons and what they mean:

While ClustersValidated is not True, the TSR-Manager webhooks reject all ProtectionGroup and ProtectionRequest creates or spec changes for this zone.

2. Read controller logs

All TSR-Manager-side controllers (protection, failover, test-failover, replication monitor, pg-sync) run in the manager-control-plane pod in the zone namespace. There is one container named manager, so the pod shows 1/1.

Find the manager pod:

kubectl -n dr-<zone-name> get pods -l app.kubernetes.io/component=manager-control-plane

Stream logs:

kubectl -n dr-<zone-name> logs -l app.kubernetes.io/component=manager-control-plane -c manager --follow

Tail recent logs with a time window:

kubectl -n dr-<zone-name> logs -l app.kubernetes.io/component=manager-control-plane -c manager --since=1h

Find the workload control-plane pod on a workload cluster (the placement reconciler, admission webhooks, and DRBD resource controller run here):

kubectl --context <workload-context> -n trilio-site-recovery-system get pods \
  -l app.kubernetes.io/component=workload-control-plane

Find node agent pods (one per worker node; named <release>-site-recovery-workload-control-plane-agent-<suffix>):

kubectl --context <workload-context> -n trilio-site-recovery-system get pods \
  -l app.kubernetes.io/component=drbd-node-agent

To read logs from the agent on a specific node, first resolve the pod name:

POD=$(kubectl --context <workload-context> -n trilio-site-recovery-system get pods \
  --field-selector spec.nodeName=<node-name> \
  -l app.kubernetes.io/component=drbd-node-agent \
  -o jsonpath='{.items[0].metadata.name}')
kubectl --context <workload-context> -n trilio-site-recovery-system logs "$POD"

3. Increase log verbosity at runtime

You can change the log level of any Site Recovery component without restarting pods. The site-recovery-logging ConfigMap in the relevant namespace controls the level.

Accepted values: error, info, debug, trace

Increase TSR-Manager control plane verbosity (ConfigMap lives in the zone namespace):

kubectl -n dr-<zone-name> create configmap site-recovery-logging \
  --from-literal=level=debug \
  --from-literal=level.control-plane=debug \
  --dry-run=client -o yaml | kubectl apply -f -

Increase workload control plane and node agent verbosity (ConfigMap lives in trilio-site-recovery-system on the workload cluster):

kubectl --context <workload-context> -n trilio-site-recovery-system \
  create configmap site-recovery-logging \
  --from-literal=level=debug \
  --from-literal=level.drbd-node-agent=debug \
  --dry-run=client -o yaml | kubectl apply -f -

Changes take effect within approximately 15 seconds without pod restarts. Revert by deleting the key or setting it back to info.

4. Diagnose a stalled ProtectionRequest

ProtectionRequest phases in order: Pending → Validating → CreatingDRBD → Syncing → ReadyToActivate → Activating → Protected. Teardown runs through Unprotecting. Failed is terminal.

Check the current phase and conditions:

kubectl -n dr-<zone-name> get protect <pr-name>
kubectl -n dr-<zone-name> describe protectionrequest <pr-name>

The kubectl get output columns are: VM, Namespace, Phase, Sync, Age.

Common stall causes by phase:

Note: ReadyToActivate means the initial sync finished and the VM is being switched — this is a normal operator-action state, not a stall by itself. A VM stays in ReadyToActivate only if the activation step is blocked.

Check conditions on the ProtectionRequest:

kubectl -n dr-<zone-name> get protectionrequest <pr-name> \
  -o jsonpath='{range .status.conditions[*]}{.type}: {.status} ({.reason}) — {.message}{"\n"}{end}'

If the Ready condition shows reason: RoleConflict, a DRBD Primary/Secondary role disagreement has been detected between the two DRBDResource copies. This requires investigation of DRBD state on both nodes — see step 6.

Check DRBDResource status on both workload clusters (DRBDResource CRs live on the workload clusters, not TSR-Manager):

# On the primary workload cluster, in the VM's namespace:
kubectl --context <cluster1-context> -n <vm-namespace> get dres
kubectl --context <cluster1-context> -n <vm-namespace> describe drbdresource <drbd-resource-name>

# On the DR workload cluster:
kubectl --context <cluster2-context> -n <vm-namespace> get dres
kubectl --context <cluster2-context> -n <vm-namespace> describe drbdresource <drbd-resource-name>

The DRBDResource name follows the pattern drbd-<namespace>-<vm>-<hash8>. The kubectl get dres output columns are: Side, Role, Node, State, Phase, Age.

A healthy protected VM shows the source-side copy with Role: Primary and State: Connected, and the target-side copy with Role: Secondary and State: Connected.

Important: Do not run kubectl get drbdresource against the TSR-Manager cluster — the drbdresources CRD is installed only on workload clusters. Running it against TSR-Manager returns server doesn't have a resource type.

5. Diagnose a stalled or failed FailoverRequest

FailoverRequest has two status fields: status.state (overall lifecycle: Pending, InProgress, Completed, Failed, Cancelled) and status.phase (granular step: Initializing, Validating, RunningPreHooks, StoppingSource, WaitingForLocks, RemovingTaints, StartingTarget, Monitoring, RunningPostHooks, Completed, Failed).

Check state and phase:

kubectl -n dr-<zone-name> get fr <fr-name>
kubectl -n dr-<zone-name> describe failoverrequest <fr-name>

The kubectl get output columns are: Protection Group, Target, State, Phase, Age. There is no TYPE or STATUS column.

Per-VM detail is in status.vmStatuses, not a top-level failedVMs field (which does not exist):

kubectl -n dr-<zone-name> get failoverrequest <fr-name> \
  -o jsonpath='{range .status.vmStatuses[*]}{.name}: {.phase} — source={.sourceStatus} target={.targetStatus}{"\n"}{.error}{"\n"}{end}'

Hook status (if hooks are configured):

kubectl -n dr-<zone-name> get failoverrequest <fr-name> \
  -o jsonpath='{range .status.hookStatuses[*]}{.name} ({.type}): {.phase} — {.message}{"\n"}{end}'

Common stall causes by phase:

Critical caution: Do not delete a stuck FailoverRequest while VMs are in an unknown state. The failover controller is re-entrant — it resumes from where it left off on the next reconcile. Deleting the CR while a role flip is mid-flight can leave DRBD in an inconsistent state. If a FailoverRequest is stuck, wait for the controller to time out or resolve the underlying cause first.

After a DR-side promotion, do not start VMs on the primary cluster. Only one side should have VMs running at a time. Starting VMs on the demoted primary while the DR side is also running can corrupt data.

6. Diagnose degraded DRBD replication

Check the ReplicationGroupStatus (auto-created by the replication monitor, one per ProtectionGroup, named <pg-name>-replication, on TSR-Manager):

kubectl -n dr-<zone-name> get rgs
kubectl -n dr-<zone-name> describe replicationgroupstatus <pg-name>-replication

The kubectl get output columns are: Protection-Group, RPO-Status, Out-Of-Sync, Synced, Connection, Last-Sync, Age.

Key status fields:

• status.aggregated.connectionState: Connected, Degraded, or Disconnected — worst-case across all VMs in the group
• status.aggregated.outOfSyncBytes: total bytes not yet replicated
• status.rpoStatus: Met, Degraded, Violated, or Unknown

Check RPOEvent records for a history of replication state transitions:

kubectl -n dr-<zone-name> get rpoe -l siterecovery.trilio.io/protection-group=<pg-name>

Check DRBDResource connection state on both workload clusters. DRBD replication requires TCP ports 7000–8000 inclusive (one port per protected VM) to be open between primary and DR worker nodes. Port 8000 is not optional — a firewall rule that stops at 7999 will silently prevent replication for one VM volume.

To verify the allocated port for a VM:

kubectl -n dr-<zone-name> get protectionrequest <pr-name> \
  -o jsonpath='{.status.drbdResourceRef.port}{"\n"}'

Check node agent status — the node agent applies a siterecovery.trilio.io/not-ready:NoSchedule taint to its node during startup. If this taint persists beyond the startupDeadlineSeconds (default 120s), the agent emits a DRBDStartupDeadlineExceeded Warning event on the Node and the DRBDResource carries NodeStartupGate=False. This clears automatically when DRBD converges.

# Check for the startup taint on workload nodes:
kubectl --context <workload-context> get nodes \
  -o custom-columns='NAME:.metadata.name,TAINTS:.spec.taints[*].key'

# Check node events for startup deadline warnings:
kubectl --context <workload-context> get events --field-selector reason=DRBDStartupDeadlineExceeded

Check for a DRBD node agent pod stuck in Init (the drbd-module-loader init container failed to load the kernel module):

kubectl --context <workload-context> -n trilio-site-recovery-system get pods \
  -l app.kubernetes.io/component=drbd-node-agent

If a pod shows Init:Error or Init:CrashLoopBackOff:

kubectl --context <workload-context> -n trilio-site-recovery-system logs <agent-pod> -c drbd-module-loader

Common causes: no compatible kmod RPM for this kernel version (mirror lag), or a foreign DRBD module already loaded with usermode_helper enabled (requires node reboot). See the drbd-module-loading runbook for detailed resolution steps.

7. Diagnose a stalled or failed TestFailover

TestFailover phases in order: Queued → PreparingEnvironment → CreatingSnapshots → CreatingVolumes → CreatingVMs → StartingVMs → Verifying → Succeeded. Cleanup runs through CleaningUp → Cleaned. Failed is terminal.

Check the phase, result, and scheduling verdict:

kubectl -n dr-<zone-name> get tfo <tfo-name>
kubectl -n dr-<zone-name> describe testfailover <tfo-name>

The kubectl get output columns are: Protection Group, Phase, Result, Passed, Scheduling, Age.

Read the test namespace — this is a controller-generated value, never user-provided:

kubectl -n dr-<zone-name> get testfailover <tfo-name> \
  -o jsonpath='{.status.testNamespace}{"\n"}'

The test namespace is named site-recovery-<protection-group>-<random> and is created on the target workload cluster.

Common stall causes by phase:

Check scheduling probe results (for schedulingCheck failures):

kubectl -n dr-<zone-name> get testfailover <tfo-name> \
  -o jsonpath='{.status.schedulingCheck}{"\n"}'

If verdict is Failed or Warned, status.schedulingCheck.excludedVMs names the VMs the DR scheduler could not place, with the scheduler's own message.

A TestFailover that is concurrent with a FailoverRequest for the same ProtectionGroup is refused at validation (reason: FailoverInProgress). Wait for the FailoverRequest to complete before starting a test.

To trigger early cleanup of a running or completed test while keeping the record:

kubectl -n dr-<zone-name> patch testfailover <tfo-name> --type=merge \
  -p '{"spec":{"cleanupRequested":true}}'

This moves the phase to CleaningUp then Cleaned and preserves status.result.

Do not patch metadata.finalizers to force deletion of a test namespace — test namespace cleanup is driven by the controller's cleanup pipeline, not by namespace finalizers, and patching finalizers directly will leave snapshots, PersistentVolumes, and VolumeSnapshotContents behind.

8. Diagnose a degraded node (node agent not converging)

A protected VM is pinned to one node per cluster for the lifetime of the protection. Live migration and cold migration of a protected VM are denied by the admission webhook. Running oc adm drain on a node hosting a running protected VM will not complete.

If a node is showing NodeStartupGate=False or a DRBDResource is in Degraded or Failed phase on a workload cluster:

1. Read the DRBDResource conditions on that workload cluster:

kubectl --context <workload-context> -n <vm-namespace> get dres <drbd-resource-name> \
  -o jsonpath='{range .status.conditions[*]}{.type}: {.status} ({.reason}) — {.message}{"\n"}{end}'

2. Check the condition PlacementReady — this gates metadata-PVC provisioning. If PlacementReady=False, the anchor pod for this resource may not have landed on the expected node yet.

3. Read the node agent log for that specific node (see step 2 for how to locate the pod by node name).

4. Check whether the metadata PVCs exist. They are named drbd-md-<resourceName>-v<volumeNumber> on the workload cluster in the VM's namespace:

kubectl --context <workload-context> -n <vm-namespace> get pvc | grep drbd-md-

5. If a DRBDResource is stuck in Deleting because its bound node is gone (the node agent finalizer siterecovery.trilio.io/drbdresource-agent cannot be cleared by a non-existent node), follow the node-loss runbook. Do not patch the placement finalizer (siterecovery.trilio.io/drbdresource-placement) directly — doing so leaks frontend PVCs and PersistentVolumes permanently.

9. Check the ProtectionZone controller

The ProtectionZone controller (the operator that validates zones and installs workload stacks) runs as its own deployment on TSR-Manager:

kubectl -n <operator-namespace> get pods \
  -l app.kubernetes.io/component=protectionzone-controller
kubectl -n <operator-namespace> logs \
  -l app.kubernetes.io/component=protectionzone-controller --follow

The operator namespace is where the OLM Subscription was created, typically the same namespace as the CSV.

If the ProtectionZone controller has restarted, it re-applies all CRDs with server-side-apply and ForceOwnership on each reconcile. Any out-of-band CRD edits are reverted within one reconcile cycle.

After resolving the root cause, verify the following:

ProtectionZone is Available:

kubectl -n dr-<zone-name> get pz <zone-name>

status.phase should be Available and ClustersValidated=True.

ProtectionRequest is Protected:

kubectl -n dr-<zone-name> get protect

All entries should show Phase: Protected and Sync: 100 (or the sync column showing the current replication percentage).

DRBDResources are Connected and Consistent on both workload clusters:

kubectl --context <cluster1-context> -n <vm-namespace> get dres
kubectl --context <cluster2-context> -n <vm-namespace> get dres

Expect State: Connected and Phase: Consistent on both copies.

ReplicationGroupStatus shows Connected and RPO Met:

kubectl -n dr-<zone-name> get rgs

Expect Connection: Connected and RPO-Status: Met.

FailoverRequest Completed:

kubectl -n dr-<zone-name> get fr <fr-name>

Expect State: Completed and Phase: Completed.

TestFailover Succeeded (or Cleaned after success):

kubectl -n dr-<zone-name> get tfo <tfo-name>

Expect Phase: Succeeded (or Cleaned if cleanup ran) and Result: Succeeded.

No protected VMs carry the startup taint on either workload cluster:

kubectl --context <workload-context> get nodes \
  -o custom-columns='NAME:.metadata.name,TAINTS:.spec.taints[*].key' | grep not-ready

If no output is returned, no nodes are tainted by the node agent.

The actions in this runbook are primarily diagnostic (reading logs and status). The only mutating actions are:

Reverting log level changes:

Delete the site-recovery-logging ConfigMap in the affected namespace to return all components to their startup log level:

# On TSR-Manager, in the zone namespace:
kubectl -n dr-<zone-name> delete configmap site-recovery-logging

# On workload clusters:
kubectl --context <workload-context> -n trilio-site-recovery-system delete configmap site-recovery-logging

Or set specific keys back to info:

kubectl -n dr-<zone-name> create configmap site-recovery-logging \
  --from-literal=level=info \
  --dry-run=client -o yaml | kubectl apply -f -

Reverting a cleanupRequested: true patch on a TestFailover:

The cleanupRequested field can be set but never unset. If you triggered cleanup prematurely, the test resources (VMs, PVCs, snapshots, test namespace) will be deleted. The TestFailover CR itself and its status.result and status.resultMessage are preserved.

If you patched DRBDResource finalizers manually:

If you incorrectly removed the agent finalizer (siterecovery.trilio.io/drbdresource-agent) on a DRBDResource whose node is still alive, the placement reconciler will proceed with storage teardown without the DRBD device having been cleanly torn down on the kernel side. Contact Trilio support immediately with the DRBDResource name, namespace, and the node it was bound to.

Re-protecting a VM after unprotect:

Re-protect always requires a full initial sync. There is no resume-from-checkpoint path. Create a new ProtectionRequest for the VM after the previous one has fully deleted (the Unprotecting phase has completed and the CR is gone).

If this runbook does not resolve the issue, contact Trilio support with the following information:

1. A support bundle collected with oc adm must-gather from the TSR-Manager cluster, covering the affected zones. Collect this as close to the incident time as possible:

oc adm must-gather --image=quay.io/triliovault/site-recovery-must-gather:<release-tag>

Include --since 2h (or a longer window if the incident is older) to capture relevant log history. Contact Trilio support to obtain the correct pinned image tag for your release.

2. The output of the following commands, run against the TSR-Manager cluster, identifying the affected zone and operation:

kubectl -n dr-<zone-name> get pz,pg,pr,fr,tfo,rgs,rpoe -o wide
kubectl -n dr-<zone-name> describe protectionzone <zone-name>
kubectl -n dr-<zone-name> describe <affected-cr-kind> <affected-cr-name>

3. The output of kubectl get dres from both workload clusters in the affected VM namespaces:

kubectl --context <cluster1-context> -n <vm-namespace> get dres -o yaml
kubectl --context <cluster2-context> -n <vm-namespace> get dres -o yaml

4. Node kernel versions for affected workload nodes:

oc --context <workload-context> get nodes \
  -o custom-columns='NAME:.metadata.name,KERNEL:.status.nodeInfo.kernelVersion,OS:.status.nodeInfo.osImage'

5. A description of the sequence of events that led to the current state, including any manual interventions (CR patches, pod restarts, namespace deletions) performed before opening the ticket.

Do not share kubeconfig files or the contents of zone credential Secrets. The support bundle redacts all Secret data, kubeconfigs, and cloud-init payloads.