--- title: Reconciliation Issues product: trilio-site-recovery-for-kubernetes doc_type: guide version: 0.2.0 source: git2docs (code-derived, validation-filtered) canonical: https://docs-tsr.trilio.io/trilio-site-recovery-for-kubernetes/site-recovery/troubleshooting-reconciliation --- # Reconciliation Issues _Resources stuck in pending or in-progress states, reconciliation loop errors_ ## Overview This page covers resources that become stuck in pending or in-progress states, reconciliation loop errors, and the diagnostic steps to identify and resolve them. Site Recovery controllers run continuously on the TSR-Manager cluster and on each workload cluster; when a reconciler cannot make progress it requeues and logs the reason, leaving the resource's `status.phase` or `status.conditions` as the primary signal. Understanding which controller owns which phase, where its logs live, and what conditions gate forward progress lets you resolve most stalls without opening a support case. ## Prerequisites Before using this guide: - Access to the TSR-Manager cluster and both workload clusters with `kubectl` (or `oc`) configured for each - Permissions to read pods, logs, events, and all `siterecovery.trilio.io/v1alpha1` custom resources on the TSR-Manager cluster, and DRBDResource CRs on workload clusters - The `oc` CLI for OpenShift-specific commands (log retrieval via label selectors, must-gather) - Familiarity with the Site Recovery namespace layout: TSR-Manager components and all TSR-Manager CRs live in `dr-`; workload components live in `trilio-site-recovery-system` on each workload cluster - `tsrctl` — supplied by Trilio with the release; contact Trilio support to obtain it - OpenShift 4.17 or later on all three clusters ## Installation No installation is required to use this guide. The diagnostic steps below use `kubectl`, `oc`, and `tsrctl` against clusters that are already running Site Recovery. If a controller pod is missing or in `CrashLoopBackOff`, follow the ProtectionZone installation guide to re-provision the zone before proceeding. ## Configuration Several configuration knobs affect reconciliation behaviour and are relevant when diagnosing stalls. ### Runtime log level Log verbosity is controlled per namespace by the `site-recovery-logging` ConfigMap. Raising the level to `debug` or `trace` produces per-reconcile detail without restarting any pod. ```yaml apiVersion: v1 kind: ConfigMap metadata: name: site-recovery-logging namespace: dr- # TSR-Manager components data: level: debug # global: error | info | debug | trace level.control-plane: trace # override for the manager-control-plane reconcilers level.protectionzone: debug # override for the ProtectionZone controller ``` Apply the same ConfigMap in `trilio-site-recovery-system` on a workload cluster to raise the node-agent or workload-control-plane log level: ```yaml apiVersion: v1 kind: ConfigMap metadata: name: site-recovery-logging namespace: trilio-site-recovery-system data: level: debug level.drbd-node-agent: trace level.control-plane: debug ``` Accepted values: `error`, `info`, `debug`, `trace`. There is no `warn` level. ### Node agent startup deadline The `siterecovery.trilio.io/not-ready:NoSchedule` taint is held on each worker node until DRBD converges or the deadline passes. The default is 120 seconds. If your DRBD initial sync is slow and you see `NodeStartupGate=False / StartupDeadlineExceeded` shortly after reboots, raise the deadline: ```yaml spec: nodeAgent: startupDeadlineSeconds: 300 # 10–3600 seconds; changing this rolls the DaemonSet ``` ### Leader election Leader election is enabled by default (`leaderElect: true`). Only one replica of the TSR-Manager control plane holds the leader lease at any time. If you see log messages from multiple pods disagreeing about resource state, confirm that only one pod holds the lease. ```bash kubectl get lease -n dr- ``` ## Usage The general diagnostic workflow is: 1. Identify the stuck resource and its current `status.phase` and `status.conditions`. 2. Determine which controller is responsible and retrieve its logs. 3. Inspect related resources (DRBDResource on workload clusters, zone conditions, node agent pods). 4. Apply the fix and watch the resource reconcile forward. ### Locating controller pods **TSR-Manager control plane** (all TSR-Manager-side reconcilers — ProtectionRequest, FailoverRequest, TestFailover, ProtectionGroup sync, replication monitor): ```bash kubectl get pods -n dr- \ -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=manager-control-plane ``` The pod runs a single container named `manager` and reports `1/1`. **ProtectionZone controller** (zone validation, workload stack installation): ```bash kubectl get pods -n dr- \ -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=protectionzone-controller ``` **Workload control plane** (DRBDResource placement, admission webhooks — run this on each workload cluster): ```bash kubectl get pods -n trilio-site-recovery-system \ -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=workload-control-plane ``` **DRBD node agent** (DRBD kernel state, PVC provisioning — run this on each workload cluster): ```bash kubectl get pods -n trilio-site-recovery-system \ -l app.kubernetes.io/component=drbd-node-agent ``` To get the agent pod on a specific node: ```bash NODE= POD=$(kubectl get pods -n trilio-site-recovery-system \ --field-selector spec.nodeName=${NODE} \ -l app.kubernetes.io/component=drbd-node-agent \ -o jsonpath='{.items[0].metadata.name}') kubectl logs -n trilio-site-recovery-system ${POD} ``` ### Reading resource status Always check `status.conditions` alongside `status.phase` — the phase is a summary; the conditions carry the reason and message: ```bash # ProtectionRequest (on TSR-Manager cluster, in dr-) kubectl get pr -n dr- -o yaml # DRBDResource (on the relevant workload cluster, in the VM's namespace) kubectl get dres -n -o wide # ProtectionZone kubectl get pz -n dr- -o yaml # ReplicationGroupStatus kubectl get rgs -replication -n dr- -o yaml ``` ### Filtering events ```bash # Events on the TSR-Manager cluster scoped to a zone namespace kubectl get events -n dr- --sort-by=.lastTimestamp # Events on a workload cluster for a specific node kubectl get events -n trilio-site-recovery-system \ --field-selector involvedObject.kind=Node,involvedObject.name= ``` ## Troubleshooting Use `status.conditions[].reason` as the primary lookup key. The phase tells you where the resource is stuck; the condition reason tells you why. --- ### ProtectionZone issues #### Symptom: `ClustersValidated=False`, reason `KubeVirtNotInstalled` **Cause:** The workload cluster does not serve the `kubevirt.io/v1` API group. **Fix:** Install OpenShift Virtualization (or KubeVirt) on the affected workload cluster. The zone controller re-checks automatically. Verify: ```bash kubectl api-resources --api-group=kubevirt.io ``` --- #### Symptom: `ClustersValidated=False`, reason `InsufficientClusterPermissions` **Cause:** The zone credential authenticates but fails the RBAC sweep. The condition message names the missing verbs. **Fix:** Rotate the credential with `--refresh-rbac` to re-apply the expected ClusterRole: ```bash tsrctl zone-credential rotate --refresh-rbac ``` --- #### Symptom: `ClustersValidated=False`, reason `ClusterIdentityChanged` **Cause:** The zone credential now reaches a cluster whose `kube-system` Namespace UID differs from the UID recorded when the zone first claimed that cluster. This cannot be overridden. **Fix:** Restore the original cluster, or delete and re-create the ProtectionZone (which triggers a full re-protection of all VMs). Contact Trilio support before proceeding. --- #### Symptom: `ClustersValidated=False`, reason `ClusterAlreadyClaimed` **Cause:** The workload cluster's `trilio-site-recovery-system` Namespace carries a `siterecovery.trilio.io/owner-zone-namespace-uid` annotation pointing to a different zone. **Fix:** Each workload cluster can belong to exactly one zone at a time. Identify the owning zone: ```bash kubectl get namespace trilio-site-recovery-system \ -o jsonpath='{.metadata.annotations}' ``` Delete the owning zone (and unprotect its VMs first) before re-claiming the cluster. --- #### Symptom: `ClustersValidated=False`, reason `SCCPrerequisiteMissing` **Cause:** The `privileged` SCC or its associated ClusterRole is missing on the workload cluster. **Fix:** Confirm the SCC exists on the OpenShift workload cluster: ```bash oc get scc privileged ``` If it is missing, restore it via the OpenShift documentation for your version. The zone controller re-validates automatically. --- #### Symptom: `WorkloadStacksReady=False`, reason `SCCAdmissionDenied` **Cause:** The node-agent DaemonSet was rejected by OpenShift SCC admission after the workload stack was installed. **Fix:** Check the DaemonSet events on the workload cluster: ```bash RELEASE= kubectl describe daemonset ${RELEASE}-site-recovery-workload-control-plane-agent \ -n trilio-site-recovery-system ``` Ensure the agent ServiceAccount is bound to the `privileged` SCC. The ProtectionZone controller provisions this binding; if it was removed out of band, delete and re-create the ProtectionZone to re-provision. --- #### Symptom: `CredentialsValid` condition with reason `CredentialExpiring` **Cause:** Advisory only — the zone credential's expiry is within the warning window. The zone phase is not changed. **Fix:** Rotate the credential before it expires: ```bash tsrctl zone-credential rotate --revoke-previous ``` --- ### ProtectionRequest issues #### Symptom: Phase `Pending` — does not advance **Cause:** The `ClustersValidated` condition on the zone is not `True`. The TSR-Manager webhook refuses new ProtectionGroup and ProtectionRequest creates while this gate is open. **Fix:** Resolve the zone condition first (see ProtectionZone issues above), then re-apply the ProtectionRequest. --- #### Symptom: Phase `Validating` — stuck for more than a few minutes **Cause:** The scheduling probe pod on the DR workload cluster is not completing. The VM may have resource requests or node affinities that no DR node can satisfy. **Fix:** Check `status.schedulingProbe` on the ProtectionRequest: ```bash kubectl get pr -n dr- \ -o jsonpath='{.status.schedulingProbe}' | python3 -m json.tool ``` If `outcome` is `Unschedulable`, the `refusedNodes` array shows per-node rejection reasons. Adjust the DR cluster's node capacity, node labels, or the VM's scheduling constraints. --- #### Symptom: Phase `CreatingDRBD` — stuck for more than a few minutes **Cause:** The node agent on the source or DR workload cluster has not yet provisioned the DRBDResource. Common sub-causes: - The DRBDResource's `ConditionPlacementReady` is `False` — the node agent has not yet written the `.res` file. - The metadata PVC (`drbd-md--v`) has not bound. - The DRBD kernel module failed to load (init container exit code non-zero). **Fix:** ```bash # Check DRBDResource phase on the primary workload cluster kubectl get dres -n -o wide # Describe the DRBDResource for condition detail kubectl describe dres -n # Check metadata PVCs kubectl get pvc -n | grep drbd-md- # Check node agent init container logs for the relevant node NODE=$(kubectl get dres -n \ -o jsonpath='{.spec.nodeName}') POD=$(kubectl get pods -n trilio-site-recovery-system \ --field-selector spec.nodeName=${NODE} \ -l app.kubernetes.io/component=drbd-node-agent \ -o jsonpath='{.items[0].metadata.name}') kubectl logs -n trilio-site-recovery-system ${POD} -c module-loader kubectl logs -n trilio-site-recovery-system ${POD} ``` If the module loader failed, confirm the node's kernel version is `el9`: ```bash oc get node -o jsonpath='{.status.nodeInfo.kernelVersion}' ``` RHEL 10-based nodes (`el10`) are not validated in Tech Preview. --- #### Symptom: Phase `Syncing` — sync never completes, DRBDResource `connectionState` is `StandAlone` or `Disconnected` **Cause:** DRBD cannot establish a TCP connection between primary and DR worker nodes. The most common cause is a firewall rule that does not cover the full TCP port range 7000–8000 inclusive. **Fix:** Verify that TCP ports 7000–8000 (inclusive) are open between primary and DR worker node IPs. Port 8000 is mandatory — a rule that stops at 7999 silently prevents replication for one volume. ```bash # Check the port assigned to the DRBDResource kubectl get dres -n \ -o jsonpath='{.spec.port}' # Check connection state kubectl get dres -n \ -o jsonpath='{.status.connectionState}' ``` Also check node agent logs on both sides for connection timeout messages. --- #### Symptom: Phase `Syncing` — `Ready=False`, reason `RoleConflict` **Cause:** Both sides of a DRBDResource are reporting Primary role simultaneously. This can happen after an unplanned failover if the source cluster recovers while the DR cluster is already promoted. **Fix:** This requires manual intervention. Check `status.conditions` on the ProtectionRequest and `status.role` on both DRBDResources (one on each workload cluster). Do not attempt to patch the DRBDResource role directly. Contact Trilio support for guided recovery. --- #### Symptom: Phase `Activating` — does not reach `Protected` **Cause:** The DRBD role flip (Secondary → Primary) on the target cluster has not completed, or the VM failed to start after the PVC handoff. **Fix:** ```bash # Check DRBDResource role on both workload clusters kubectl get dres -n -o wide # Check VM status on the target cluster kubectl get vm -n kubectl get events -n --sort-by=.lastTimestamp # Check TSR-Manager control plane logs kubectl logs -n dr- \ -l app.kubernetes.io/component=manager-control-plane \ --tail=200 | grep -i activat ``` --- ### FailoverRequest issues #### Symptom: `state=InProgress`, `phase` does not advance past `StoppingSource` **Cause (planned failover):** The source VM did not stop cleanly within the failover window. Protected VMs are pinned to a node; if the node is under pressure or the VM's shutdown hook is slow, the stop may not complete. **Fix:** Check `status.vmStatuses` for the stuck VM: ```bash kubectl get fr -n dr- \ -o jsonpath='{.status.vmStatuses}' | python3 -m json.tool ``` Check the VM's events on the source workload cluster: ```bash kubectl get events -n \ --field-selector involvedObject.name= --sort-by=.lastTimestamp ``` For an unplanned failover where the source cluster is unreachable, the controller force-promotes the DR side without stopping the source. If you issued a `planned` failover but the source is down, delete the FailoverRequest and re-create it with `failoverType: unplanned`. --- #### Symptom: `state=InProgress`, `phase=RunningPreHooks` — hook never completes **Cause:** A Hook Job is hanging, has exhausted retries, or its `onFailure: Abort` setting caused the failover to halt. **Fix:** Check `status.hookStatuses` on the FailoverRequest: ```bash kubectl get fr -n dr- \ -o jsonpath='{.status.hookStatuses}' | python3 -m json.tool ``` Find the hook's Job in its configured namespace and inspect its pod logs: ```bash kubectl get jobs -n kubectl logs -n job/ ``` If the hook is permanently failing and you need to proceed, re-create the FailoverRequest with `skipHooks: true`. This bypasses all hooks. --- #### Symptom: `state=Failed` **Cause:** A fatal error during failover — possible causes include hook abort, DRBD promotion failure, or the 5-minute failover wall-clock budget expiring. **Fix:** Check conditions and `status.vmStatuses`: ```bash kubectl get fr -n dr- -o yaml ``` Check TSR-Manager control plane logs: ```bash kubectl logs -n dr- \ -l app.kubernetes.io/component=manager-control-plane \ --tail=500 | grep -i 'failover\|error\|failed' ``` A failed FailoverRequest is terminal — delete it and create a new one after resolving the underlying issue. --- ### TestFailover issues #### Symptom: Phase `CreatingSnapshots` — stuck **Cause:** DRBD snapshot creation on the standby workload cluster failed. The DRBDResource on the standby side may be `Disconnected` or not yet `Consistent`. **Fix:** Check the standby-side DRBDResource phase and connection state: ```bash # On the standby workload cluster kubectl get dres -n -o wide ``` If `connectionState` is not `Connected`, resolve the DRBD connectivity issue (firewall, node agent) before retrying the TestFailover. --- #### Symptom: Phase `CreatingVMs` or `StartingVMs` — stuck **Cause:** Test VMs in the controller-generated test namespace failed to schedule or start. **Fix:** Retrieve the test namespace from the TestFailover status: ```bash TEST_NS=$(kubectl get tfo -n dr- \ -o jsonpath='{.status.testNamespace}') kubectl get pods -n ${TEST_NS} kubectl get events -n ${TEST_NS} --sort-by=.lastTimestamp ``` If VMs are unschedulable, check `status.schedulingCheck` on the TestFailover for per-VM probe outcomes: ```bash kubectl get tfo -n dr- \ -o jsonpath='{.status.schedulingCheck}' | python3 -m json.tool ``` --- #### Symptom: Phase `Verifying` — stuck or `result=Failed` **Cause:** A `test-verification` hook or a reachability check in `spec.verification.reachability` did not pass. **Fix:** Check `status.hookStatuses` and `status.verificationResults`: ```bash kubectl get tfo -n dr- \ -o jsonpath='{.status.verificationResults}' | python3 -m json.tool kubectl get tfo -n dr- \ -o jsonpath='{.status.hookStatuses}' | python3 -m json.tool ``` --- #### Symptom: TestFailover is `Succeeded` or `Failed` but test resources remain **Cause:** `cleanupPolicy: Manual` was set, or `cleanupPolicy: Automatic` with a `retentionTime` that has not yet elapsed. **Fix:** Trigger cleanup by setting `spec.cleanupRequested: true`: ```bash kubectl patch tfo -n dr- \ --type merge -p '{"spec":{"cleanupRequested":true}}' ``` Do not patch or remove finalizers to force deletion — this leaves DRBD snapshots and test namespace resources orphaned on the workload cluster. --- ### Node agent issues #### Symptom: Node has `siterecovery.trilio.io/not-ready:NoSchedule` taint — not removed **Cause:** The node agent did not converge all DRBDResources on this node within `startupDeadlineSeconds`. The DRBDResource on this node will show `NodeStartupGate=False / StartupDeadlineExceeded`. **Fix:** This is a time-bounded condition — the taint has already been removed if the deadline has passed (the agent does not re-taint after the deadline unless all resources go StandAlone). Check whether the condition has self-cleared: ```bash kubectl get dres -n -o yaml | grep -A5 NodeStartupGate ``` If the condition is persistent, check DRBD connectivity and metadata PVC binding on the node (see `CreatingDRBD` troubleshooting above). Increasing `spec.nodeAgent.startupDeadlineSeconds` on the ProtectionZone gives the agent more time before the gate opens. --- #### Symptom: Node agent pod in `Init:Error` or `Init:CrashLoopBackOff` **Cause:** The DRBD module loader init container failed to load the kernel module. The node's kernel ABI may not match any precompiled module in the image, or the node is running an unsupported kernel family. **Fix:** ```bash NODE= POD=$(kubectl get pods -n trilio-site-recovery-system \ --field-selector spec.nodeName=${NODE} \ -l app.kubernetes.io/component=drbd-node-agent \ -o jsonpath='{.items[0].metadata.name}') kubectl logs -n trilio-site-recovery-system ${POD} -c module-loader ``` Confirm the node's kernel version ends in `.el9_*`: ```bash oc get node ${NODE} -o jsonpath='{.status.nodeInfo.kernelVersion}' ``` RHEL 10-based (`el10`) nodes are not validated in Tech Preview. Mixed OS families within one workload cluster are not supported. --- #### Symptom: A foreign DRBD module is already loaded with `usermode_helper` enabled **Cause:** Another tool (for example, LINBIT DRBD Operator) has loaded the DRBD module with `usermode_helper` enabled. Site Recovery requires `usermode_helper=disabled` and treats the presence of another DRBD usermode helper as a hard failure. **Fix:** Remove the conflicting DRBD module and reboot the node. Do not attempt to unload the module without a reboot — the kernel module state cannot be safely reset in place. --- ### Collecting a support bundle If you cannot resolve an issue with the steps above, collect a diagnostic bundle from the TSR-Manager cluster and send it to Trilio support: ```bash oc adm must-gather \ --image=quay.io/triliovault/site-recovery-must-gather: \ --dest-dir=./site-recovery-bundle ``` The `gather` entrypoint inside the image collects logs, CRs, and DRBD state from the TSR-Manager cluster and all zone workload clusters into a single read-only archive in `./site-recovery-bundle`. Replace `` with the pinned tag supplied with your release.