Trilio Site Recovery for Kubernetes
Docs/site-recovery/Troubleshooting/Reconciliation Issues
Guide

Reconciliation Issues

Resources stuck in pending or in-progress states, reconciliation loop errors

Overview

This page covers resources that become stuck in pending or in-progress states, reconciliation loop errors, and the diagnostic steps to identify and resolve them. Site Recovery controllers run continuously on the TSR-Manager cluster and on each workload cluster; when a reconciler cannot make progress it requeues and logs the reason, leaving the resource's status.phase or status.conditions as the primary signal. Understanding which controller owns which phase, where its logs live, and what conditions gate forward progress lets you resolve most stalls without opening a support case.

Prerequisites

Before using this guide:

  • Access to the TSR-Manager cluster and both workload clusters with kubectl (or oc) configured for each
  • Permissions to read pods, logs, events, and all siterecovery.trilio.io/v1alpha1 custom resources on the TSR-Manager cluster, and DRBDResource CRs on workload clusters
  • The oc CLI for OpenShift-specific commands (log retrieval via label selectors, must-gather)
  • Familiarity with the Site Recovery namespace layout: TSR-Manager components and all TSR-Manager CRs live in dr-<zone-name>; workload components live in trilio-site-recovery-system on each workload cluster
  • tsrctl — supplied by Trilio with the release; contact Trilio support to obtain it
  • OpenShift 4.17 or later on all three clusters
Installation

No installation is required to use this guide. The diagnostic steps below use kubectl, oc, and tsrctl against clusters that are already running Site Recovery. If a controller pod is missing or in CrashLoopBackOff, follow the ProtectionZone installation guide to re-provision the zone before proceeding.

Configuration

Several configuration knobs affect reconciliation behaviour and are relevant when diagnosing stalls.

Runtime log level

Log verbosity is controlled per namespace by the site-recovery-logging ConfigMap. Raising the level to debug or trace produces per-reconcile detail without restarting any pod.

apiVersion: v1
kind: ConfigMap
metadata:
  name: site-recovery-logging
  namespace: dr-<zone-name>         # TSR-Manager components
data:
  level: debug                      # global: error | info | debug | trace
  level.control-plane: trace        # override for the manager-control-plane reconcilers
  level.protectionzone: debug       # override for the ProtectionZone controller

Apply the same ConfigMap in trilio-site-recovery-system on a workload cluster to raise the node-agent or workload-control-plane log level:

apiVersion: v1
kind: ConfigMap
metadata:
  name: site-recovery-logging
  namespace: trilio-site-recovery-system
data:
  level: debug
  level.drbd-node-agent: trace
  level.control-plane: debug

Accepted values: error, info, debug, trace. There is no warn level.

Node agent startup deadline

The siterecovery.trilio.io/not-ready:NoSchedule taint is held on each worker node until DRBD converges or the deadline passes. The default is 120 seconds. If your DRBD initial sync is slow and you see NodeStartupGate=False / StartupDeadlineExceeded shortly after reboots, raise the deadline:

spec:
  nodeAgent:
    startupDeadlineSeconds: 300   # 10–3600 seconds; changing this rolls the DaemonSet

Leader election

Leader election is enabled by default (leaderElect: true). Only one replica of the TSR-Manager control plane holds the leader lease at any time. If you see log messages from multiple pods disagreeing about resource state, confirm that only one pod holds the lease.

kubectl get lease -n dr-<zone-name>
Usage

The general diagnostic workflow is:

  1. Identify the stuck resource and its current status.phase and status.conditions.
  2. Determine which controller is responsible and retrieve its logs.
  3. Inspect related resources (DRBDResource on workload clusters, zone conditions, node agent pods).
  4. Apply the fix and watch the resource reconcile forward.

Locating controller pods

TSR-Manager control plane (all TSR-Manager-side reconcilers — ProtectionRequest, FailoverRequest, TestFailover, ProtectionGroup sync, replication monitor):

kubectl get pods -n dr-<zone-name> \
  -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=manager-control-plane

The pod runs a single container named manager and reports 1/1.

ProtectionZone controller (zone validation, workload stack installation):

kubectl get pods -n dr-<zone-name> \
  -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=protectionzone-controller

Workload control plane (DRBDResource placement, admission webhooks — run this on each workload cluster):

kubectl get pods -n trilio-site-recovery-system \
  -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=workload-control-plane

DRBD node agent (DRBD kernel state, PVC provisioning — run this on each workload cluster):

kubectl get pods -n trilio-site-recovery-system \
  -l app.kubernetes.io/component=drbd-node-agent

To get the agent pod on a specific node:

NODE=<node-name>
POD=$(kubectl get pods -n trilio-site-recovery-system \
  --field-selector spec.nodeName=${NODE} \
  -l app.kubernetes.io/component=drbd-node-agent \
  -o jsonpath='{.items[0].metadata.name}')
kubectl logs -n trilio-site-recovery-system ${POD}

Reading resource status

Always check status.conditions alongside status.phase — the phase is a summary; the conditions carry the reason and message:

# ProtectionRequest (on TSR-Manager cluster, in dr-<zone-name>)
kubectl get pr <name> -n dr-<zone-name> -o yaml

# DRBDResource (on the relevant workload cluster, in the VM's namespace)
kubectl get dres -n <vm-namespace> -o wide

# ProtectionZone
kubectl get pz -n dr-<zone-name> -o yaml

# ReplicationGroupStatus
kubectl get rgs <pg-name>-replication -n dr-<zone-name> -o yaml

Filtering events

# Events on the TSR-Manager cluster scoped to a zone namespace
kubectl get events -n dr-<zone-name> --sort-by=.lastTimestamp

# Events on a workload cluster for a specific node
kubectl get events -n trilio-site-recovery-system \
  --field-selector involvedObject.kind=Node,involvedObject.name=<node-name>
Examples

Example 1 — ProtectionRequest stuck in CreatingDRBD

Check the ProtectionRequest conditions:

kubectl get pr <pr-name> -n dr-<zone-name> -o jsonpath='{.status.conditions}' | python3 -m json.tool

Check the corresponding DRBDResource on each workload cluster (the resource is named drbd-<vm-namespace>-<vm-name>-<hash8>):

# On the primary workload cluster
kubectl get dres -n <vm-namespace> -o wide
kubectl describe dres <drbd-resource-name> -n <vm-namespace>

Check node agent logs on the node where the DRBDResource was placed:

NODE=$(kubectl get dres <drbd-resource-name> -n <vm-namespace> \
  -o jsonpath='{.spec.nodeName}')
POD=$(kubectl get pods -n trilio-site-recovery-system \
  --field-selector spec.nodeName=${NODE} \
  -l app.kubernetes.io/component=drbd-node-agent \
  -o jsonpath='{.items[0].metadata.name}')
kubectl logs -n trilio-site-recovery-system ${POD} --tail=100

Example 2 — ProtectionZone stuck in Provisioning or showing ClustersValidated=False

kubectl get pz <zone-name> -n dr-<zone-name> -o yaml

Look for the ClustersValidated condition and its reason field. Cross-reference the reason with the troubleshooting table below.

Raise log verbosity on the ProtectionZone controller to trace validation:

kubectl patch configmap site-recovery-logging -n dr-<zone-name> \
  --type merge -p '{"data":{"level.protectionzone":"trace"}}'

kubectl logs -n dr-<zone-name> \
  -l app.kubernetes.io/component=protectionzone-controller --follow

Example 3 — TestFailover stuck in CreatingVMs

Find the test namespace from the TestFailover status (never guess or construct it manually):

TEST_NS=$(kubectl get tfo <tfo-name> -n dr-<zone-name> \
  -o jsonpath='{.status.testNamespace}')
echo "Test namespace: ${TEST_NS}"

Inspect pods and events in the test namespace on the standby workload cluster:

kubectl get pods -n ${TEST_NS}
kubectl get events -n ${TEST_NS} --sort-by=.lastTimestamp

Example 4 — FailoverRequest stuck in WaitingForLocks or StoppingSource

Inspect the FailoverRequest status — per-VM detail is in status.vmStatuses:

kubectl get fr <fr-name> -n dr-<zone-name> \
  -o jsonpath='{.status.vmStatuses}' | python3 -m json.tool

kubectl get fr <fr-name> -n dr-<zone-name> \
  -o jsonpath='{.status.phase}'

Check TSR-Manager control plane logs for the failover reconciler:

kubectl logs -n dr-<zone-name> \
  -l app.kubernetes.io/component=manager-control-plane \
  --tail=200 | grep -i failover

Example 5 — Inspect zone credential status

tsrctl zone-credential inspect --zone <zone-name>

If the credential is expiring or the zone reports InsufficientClusterPermissions, rotate it with RBAC refresh:

tsrctl zone-credential rotate --refresh-rbac

If the previous credential should be immediately revoked:

tsrctl zone-credential rotate --refresh-rbac --revoke-previous
Troubleshooting

Use status.conditions[].reason as the primary lookup key. The phase tells you where the resource is stuck; the condition reason tells you why.

ProtectionZone issues

Symptom: ClustersValidated=False, reason KubeVirtNotInstalled

Cause: The workload cluster does not serve the kubevirt.io/v1 API group.

Fix: Install OpenShift Virtualization (or KubeVirt) on the affected workload cluster. The zone controller re-checks automatically. Verify:

kubectl api-resources --api-group=kubevirt.io

Symptom: ClustersValidated=False, reason InsufficientClusterPermissions

Cause: The zone credential authenticates but fails the RBAC sweep. The condition message names the missing verbs.

Fix: Rotate the credential with --refresh-rbac to re-apply the expected ClusterRole:

tsrctl zone-credential rotate --refresh-rbac

Symptom: ClustersValidated=False, reason ClusterIdentityChanged

Cause: The zone credential now reaches a cluster whose kube-system Namespace UID differs from the UID recorded when the zone first claimed that cluster. This cannot be overridden.

Fix: Restore the original cluster, or delete and re-create the ProtectionZone (which triggers a full re-protection of all VMs). Contact Trilio support before proceeding.

Symptom: ClustersValidated=False, reason ClusterAlreadyClaimed

Cause: The workload cluster's trilio-site-recovery-system Namespace carries a siterecovery.trilio.io/owner-zone-namespace-uid annotation pointing to a different zone.

Fix: Each workload cluster can belong to exactly one zone at a time. Identify the owning zone:

kubectl get namespace trilio-site-recovery-system \
  -o jsonpath='{.metadata.annotations}'

Delete the owning zone (and unprotect its VMs first) before re-claiming the cluster.

Symptom: ClustersValidated=False, reason SCCPrerequisiteMissing

Cause: The privileged SCC or its associated ClusterRole is missing on the workload cluster.

Fix: Confirm the SCC exists on the OpenShift workload cluster:

oc get scc privileged

If it is missing, restore it via the OpenShift documentation for your version. The zone controller re-validates automatically.

Symptom: WorkloadStacksReady=False, reason SCCAdmissionDenied

Cause: The node-agent DaemonSet was rejected by OpenShift SCC admission after the workload stack was installed.

Fix: Check the DaemonSet events on the workload cluster:

RELEASE=<release-name>
kubectl describe daemonset ${RELEASE}-site-recovery-workload-control-plane-agent \
  -n trilio-site-recovery-system

Ensure the agent ServiceAccount is bound to the privileged SCC. The ProtectionZone controller provisions this binding; if it was removed out of band, delete and re-create the ProtectionZone to re-provision.

Symptom: CredentialsValid condition with reason CredentialExpiring

Cause: Advisory only — the zone credential's expiry is within the warning window. The zone phase is not changed.

Fix: Rotate the credential before it expires:

tsrctl zone-credential rotate --revoke-previous

ProtectionRequest issues

Symptom: Phase Pending — does not advance

Cause: The ClustersValidated condition on the zone is not True. The TSR-Manager webhook refuses new ProtectionGroup and ProtectionRequest creates while this gate is open.

Fix: Resolve the zone condition first (see ProtectionZone issues above), then re-apply the ProtectionRequest.

Symptom: Phase Validating — stuck for more than a few minutes

Cause: The scheduling probe pod on the DR workload cluster is not completing. The VM may have resource requests or node affinities that no DR node can satisfy.

Fix: Check status.schedulingProbe on the ProtectionRequest:

kubectl get pr <pr-name> -n dr-<zone-name> \
  -o jsonpath='{.status.schedulingProbe}' | python3 -m json.tool

If outcome is Unschedulable, the refusedNodes array shows per-node rejection reasons. Adjust the DR cluster's node capacity, node labels, or the VM's scheduling constraints.

Symptom: Phase CreatingDRBD — stuck for more than a few minutes

Cause: The node agent on the source or DR workload cluster has not yet provisioned the DRBDResource. Common sub-causes:

  • The DRBDResource's ConditionPlacementReady is False — the node agent has not yet written the .res file.
  • The metadata PVC (drbd-md-<resourceName>-v<volumeNumber>) has not bound.
  • The DRBD kernel module failed to load (init container exit code non-zero).

Fix:

# Check DRBDResource phase on the primary workload cluster
kubectl get dres -n <vm-namespace> -o wide

# Describe the DRBDResource for condition detail
kubectl describe dres <drbd-resource-name> -n <vm-namespace>

# Check metadata PVCs
kubectl get pvc -n <vm-namespace> | grep drbd-md-

# Check node agent init container logs for the relevant node
NODE=$(kubectl get dres <drbd-resource-name> -n <vm-namespace> \
  -o jsonpath='{.spec.nodeName}')
POD=$(kubectl get pods -n trilio-site-recovery-system \
  --field-selector spec.nodeName=${NODE} \
  -l app.kubernetes.io/component=drbd-node-agent \
  -o jsonpath='{.items[0].metadata.name}')
kubectl logs -n trilio-site-recovery-system ${POD} -c module-loader
kubectl logs -n trilio-site-recovery-system ${POD}

If the module loader failed, confirm the node's kernel version is el9:

oc get node <node-name> -o jsonpath='{.status.nodeInfo.kernelVersion}'

RHEL 10-based nodes (el10) are not validated in Tech Preview.

Symptom: Phase Syncing — sync never completes, DRBDResource connectionState is StandAlone or Disconnected

Cause: DRBD cannot establish a TCP connection between primary and DR worker nodes. The most common cause is a firewall rule that does not cover the full TCP port range 7000–8000 inclusive.

Fix: Verify that TCP ports 7000–8000 (inclusive) are open between primary and DR worker node IPs. Port 8000 is mandatory — a rule that stops at 7999 silently prevents replication for one volume.

# Check the port assigned to the DRBDResource
kubectl get dres <drbd-resource-name> -n <vm-namespace> \
  -o jsonpath='{.spec.port}'

# Check connection state
kubectl get dres <drbd-resource-name> -n <vm-namespace> \
  -o jsonpath='{.status.connectionState}'

Also check node agent logs on both sides for connection timeout messages.

Symptom: Phase Syncing — Ready=False, reason RoleConflict

Cause: Both sides of a DRBDResource are reporting Primary role simultaneously. This can happen after an unplanned failover if the source cluster recovers while the DR cluster is already promoted.

Fix: This requires manual intervention. Check status.conditions on the ProtectionRequest and status.role on both DRBDResources (one on each workload cluster). Do not attempt to patch the DRBDResource role directly. Contact Trilio support for guided recovery.

Symptom: Phase Activating — does not reach Protected

Cause: The DRBD role flip (Secondary → Primary) on the target cluster has not completed, or the VM failed to start after the PVC handoff.

Fix:

# Check DRBDResource role on both workload clusters
kubectl get dres -n <vm-namespace> -o wide

# Check VM status on the target cluster
kubectl get vm <vm-name> -n <vm-namespace>
kubectl get events -n <vm-namespace> --sort-by=.lastTimestamp

# Check TSR-Manager control plane logs
kubectl logs -n dr-<zone-name> \
  -l app.kubernetes.io/component=manager-control-plane \
  --tail=200 | grep -i activat

FailoverRequest issues

Symptom: state=InProgress, phase does not advance past StoppingSource

Cause (planned failover): The source VM did not stop cleanly within the failover window. Protected VMs are pinned to a node; if the node is under pressure or the VM's shutdown hook is slow, the stop may not complete.

Fix: Check status.vmStatuses for the stuck VM:

kubectl get fr <fr-name> -n dr-<zone-name> \
  -o jsonpath='{.status.vmStatuses}' | python3 -m json.tool

Check the VM's events on the source workload cluster:

kubectl get events -n <vm-namespace> \
  --field-selector involvedObject.name=<vm-name> --sort-by=.lastTimestamp

For an unplanned failover where the source cluster is unreachable, the controller force-promotes the DR side without stopping the source. If you issued a planned failover but the source is down, delete the FailoverRequest and re-create it with failoverType: unplanned.

Symptom: state=InProgress, phase=RunningPreHooks — hook never completes

Cause: A Hook Job is hanging, has exhausted retries, or its onFailure: Abort setting caused the failover to halt.

Fix: Check status.hookStatuses on the FailoverRequest:

kubectl get fr <fr-name> -n dr-<zone-name> \
  -o jsonpath='{.status.hookStatuses}' | python3 -m json.tool

Find the hook's Job in its configured namespace and inspect its pod logs:

kubectl get jobs -n <hook-namespace>
kubectl logs -n <hook-namespace> job/<job-name>

If the hook is permanently failing and you need to proceed, re-create the FailoverRequest with skipHooks: true. This bypasses all hooks.

Symptom: state=Failed

Cause: A fatal error during failover — possible causes include hook abort, DRBD promotion failure, or the 5-minute failover wall-clock budget expiring.

Fix: Check conditions and status.vmStatuses:

kubectl get fr <fr-name> -n dr-<zone-name> -o yaml

Check TSR-Manager control plane logs:

kubectl logs -n dr-<zone-name> \
  -l app.kubernetes.io/component=manager-control-plane \
  --tail=500 | grep -i 'failover\|error\|failed'

A failed FailoverRequest is terminal — delete it and create a new one after resolving the underlying issue.

TestFailover issues

Symptom: Phase CreatingSnapshots — stuck

Cause: DRBD snapshot creation on the standby workload cluster failed. The DRBDResource on the standby side may be Disconnected or not yet Consistent.

Fix: Check the standby-side DRBDResource phase and connection state:

# On the standby workload cluster
kubectl get dres -n <vm-namespace> -o wide

If connectionState is not Connected, resolve the DRBD connectivity issue (firewall, node agent) before retrying the TestFailover.

Symptom: Phase CreatingVMs or StartingVMs — stuck

Cause: Test VMs in the controller-generated test namespace failed to schedule or start.

Fix: Retrieve the test namespace from the TestFailover status:

TEST_NS=$(kubectl get tfo <tfo-name> -n dr-<zone-name> \
  -o jsonpath='{.status.testNamespace}')
kubectl get pods -n ${TEST_NS}
kubectl get events -n ${TEST_NS} --sort-by=.lastTimestamp

If VMs are unschedulable, check status.schedulingCheck on the TestFailover for per-VM probe outcomes:

kubectl get tfo <tfo-name> -n dr-<zone-name> \
  -o jsonpath='{.status.schedulingCheck}' | python3 -m json.tool

Symptom: Phase Verifying — stuck or result=Failed

Cause: A test-verification hook or a reachability check in spec.verification.reachability did not pass.

Fix: Check status.hookStatuses and status.verificationResults:

kubectl get tfo <tfo-name> -n dr-<zone-name> \
  -o jsonpath='{.status.verificationResults}' | python3 -m json.tool

kubectl get tfo <tfo-name> -n dr-<zone-name> \
  -o jsonpath='{.status.hookStatuses}' | python3 -m json.tool

Symptom: TestFailover is Succeeded or Failed but test resources remain

Cause: cleanupPolicy: Manual was set, or cleanupPolicy: Automatic with a retentionTime that has not yet elapsed.

Fix: Trigger cleanup by setting spec.cleanupRequested: true:

kubectl patch tfo <tfo-name> -n dr-<zone-name> \
  --type merge -p '{"spec":{"cleanupRequested":true}}'

Do not patch or remove finalizers to force deletion — this leaves DRBD snapshots and test namespace resources orphaned on the workload cluster.

Node agent issues

Symptom: Node has siterecovery.trilio.io/not-ready:NoSchedule taint — not removed

Cause: The node agent did not converge all DRBDResources on this node within startupDeadlineSeconds. The DRBDResource on this node will show NodeStartupGate=False / StartupDeadlineExceeded.

Fix: This is a time-bounded condition — the taint has already been removed if the deadline has passed (the agent does not re-taint after the deadline unless all resources go StandAlone). Check whether the condition has self-cleared:

kubectl get dres -n <vm-namespace> -o yaml | grep -A5 NodeStartupGate

If the condition is persistent, check DRBD connectivity and metadata PVC binding on the node (see CreatingDRBD troubleshooting above). Increasing spec.nodeAgent.startupDeadlineSeconds on the ProtectionZone gives the agent more time before the gate opens.

Symptom: Node agent pod in Init:Error or Init:CrashLoopBackOff

Cause: The DRBD module loader init container failed to load the kernel module. The node's kernel ABI may not match any precompiled module in the image, or the node is running an unsupported kernel family.

Fix:

NODE=<node-name>
POD=$(kubectl get pods -n trilio-site-recovery-system \
  --field-selector spec.nodeName=${NODE} \
  -l app.kubernetes.io/component=drbd-node-agent \
  -o jsonpath='{.items[0].metadata.name}')
kubectl logs -n trilio-site-recovery-system ${POD} -c module-loader

Confirm the node's kernel version ends in .el9_*:

oc get node ${NODE} -o jsonpath='{.status.nodeInfo.kernelVersion}'

RHEL 10-based (el10) nodes are not validated in Tech Preview. Mixed OS families within one workload cluster are not supported.

Symptom: A foreign DRBD module is already loaded with usermode_helper enabled

Cause: Another tool (for example, LINBIT DRBD Operator) has loaded the DRBD module with usermode_helper enabled. Site Recovery requires usermode_helper=disabled and treats the presence of another DRBD usermode helper as a hard failure.

Fix: Remove the conflicting DRBD module and reboot the node. Do not attempt to unload the module without a reboot — the kernel module state cannot be safely reset in place.

Collecting a support bundle

If you cannot resolve an issue with the steps above, collect a diagnostic bundle from the TSR-Manager cluster and send it to Trilio support:

oc adm must-gather \
  --image=quay.io/triliovault/site-recovery-must-gather:<release-tag> \
  --dest-dir=./site-recovery-bundle

The gather entrypoint inside the image collects logs, CRs, and DRBD state from the TSR-Manager cluster and all zone workload clusters into a single read-only archive in ./site-recovery-bundle. Replace <release-tag> with the pinned tag supplied with your release.