--- title: Observability product: trilio-site-recovery-for-kubernetes doc_type: guide version: 0.2.0 source: git2docs (code-derived, validation-filtered) canonical: https://docs-tsr.trilio.io/trilio-site-recovery-for-kubernetes/site-recovery/observability --- # Observability _Metrics, logging, and monitoring for site recovery operations_ ## Overview This page covers the observability surface of Trilio Site Recovery: how to read replication health from custom resources, how to tune runtime log levels without restarting pods, how to interpret RPO events and replication group status, and how to collect a diagnostic support bundle when something goes wrong. Observability in Site Recovery is CR-driven — the primary signals are the `ReplicationGroupStatus` and `RPOEvent` objects on the TSR-Manager cluster, supplemented by `DRBDResource` status on each workload cluster and the `site-recovery-logging` ConfigMap for runtime log control. ## Prerequisites Before using the observability features described on this page: - A working Site Recovery deployment with at least one `ProtectionZone` in `Available` phase and at least one `ProtectionGroup` with protected VMs. - `kubectl` configured with a context pointing at the **TSR-Manager cluster** for querying TSR-Manager-side CRs (`ReplicationGroupStatus`, `RPOEvent`, `ProtectionGroup`, `ProtectionRequest`). - A separate `kubectl` context pointing at each **workload cluster** for querying `DRBDResource` objects — these CRDs are installed only on workload clusters, not on the TSR-Manager cluster. - OpenShift CLI (`oc`) if you intend to collect a support bundle via `oc adm must-gather`. - Cluster-read access on the TSR-Manager cluster (the `trilio-site-recovery-viewer` ClusterRole is sufficient for read-only observability). ## Installation No additional installation is required for observability. All monitoring objects are created automatically by the Site Recovery controllers: 1. **ReplicationGroupStatus** — the replication monitor controller auto-creates one `ReplicationGroupStatus` CR per `ProtectionGroup`, named `-replication`, in the zone namespace (`dr-`) on the TSR-Manager cluster. It appears as soon as the first VM in the group reaches the `Protected` phase. 2. **RPOEvent** — the replication monitor emits an `RPOEvent` CR on the TSR-Manager cluster on every replication status transition (sync completed, connection lost, connection restored). These are immutable audit records and accumulate over time. 3. **DRBDResource** — the workload control plane creates one `DRBDResource` per protected VM per workload cluster, in the VM's own namespace on that cluster. Named `drbd---`. 4. **Logging ConfigMap** — the `site-recovery-logging` ConfigMap is rendered per namespace by the ProtectionZone controller. You do not create it; you edit it to change log levels at runtime. To verify that the replication monitor objects exist after protecting a VM: ```bash # On the TSR-Manager cluster, in the zone namespace kubectl -n dr- get replicationgroupstatuses kubectl -n dr- get rpoevents ``` To verify DRBDResource objects on a workload cluster: ```bash # On a workload cluster, in the VM's namespace kubectl -n get drbdresources ``` ## Configuration ### Runtime log levels Site Recovery components read the `site-recovery-logging` ConfigMap in their own namespace approximately every 15 seconds and apply log level changes without a pod restart. **ConfigMap name:** `site-recovery-logging` **Accepted level values:** `error`, `info`, `debug`, `trace` > **Note:** There is no `warn` level. The logging framework uses a two-class model (info and error). Use `debug` or `trace` for verbose output. **Keys:** | Key | Scope | Example | |---|---|---| | `level` | Global default for all components in this namespace | `level: debug` | | `level.control-plane` | TSR-Manager control plane only | `level.control-plane: trace` | | `level.drbd-node-agent` | DRBD node agent on workload clusters | `level.drbd-node-agent: debug` | | `level.protectionzone` | ProtectionZone controller | `level.protectionzone: info` | To set the global level for the TSR-Manager control plane namespace: ```bash kubectl -n dr- patch configmap site-recovery-logging \ --type=merge -p '{"data":{"level":"debug"}}' ``` To set a per-component override while keeping global level at `info`: ```bash kubectl -n dr- patch configmap site-recovery-logging \ --type=merge -p '{"data":{"level":"info","level.control-plane":"debug"}}' ``` The ConfigMap also exists on each workload cluster in the `trilio-site-recovery-system` namespace and controls the node agent log level there: ```bash kubectl --context \ -n trilio-site-recovery-system patch configmap site-recovery-logging \ --type=merge -p '{"data":{"level.drbd-node-agent":"debug"}}' ``` ### ReplicationGroupStatus polling interval The `ReplicationGroupStatus` spec field `pollingIntervalSeconds` controls how often the replication monitor scrapes DRBD metrics from the active workload cluster. The default is 15 seconds. Valid range: 5–300 seconds. This value is set by the controller and is not intended to be edited by operators — the controller manages the object. ### RPO objective `ProtectionGroup.spec.sla.rpoTargetSeconds` sets the RPO objective in seconds. The replication monitor reads this value and uses it to compute `ReplicationGroupStatus.status.rpoStatus` (`Met`, `Degraded`, `Violated`, or `Unknown`) and to drive `RPOEvent` emission on threshold crossings. Because Site Recovery uses Protocol C synchronous replication exclusively, RPO is always 0 at the DRBD level — this field governs the alerting threshold, not the replication mode. ## Usage ### Checking overall replication health The fastest single-command health check for a protection group is: ```bash kubectl -n dr- get rgs ``` The `rgs` short name returns all `ReplicationGroupStatus` objects. Key columns: - **RPO-Status** — `Met`, `Degraded`, `Violated`, or `Unknown` - **Out-Of-Sync** — bytes currently out of sync across all volumes - **Synced** — number of volumes in the `Established` replication state - **Connection** — `Connected`, `Degraded`, or `Disconnected` - **Last-Sync** — timestamp of the last full sync To inspect a specific protection group's replication status in detail: ```bash kubectl -n dr- get rgs -replication -o yaml ``` The `status.volumeReplicationStatuses` array lists per-volume metrics including `outOfSyncBytes`, `replicationState`, `estimatedRPOSeconds`, and `lastSyncedAt`. ### Reading RPO events RPO events are immutable audit records of every replication status transition. Use them for SLA compliance review and post-incident analysis: ```bash # List all RPO events for a zone, newest first kubectl -n dr- get rpoe --sort-by=.spec.timestamp # Filter by protection group using the label kubectl -n dr- get rpoe \ -l siterecovery.trilio.io/protection-group= ``` The `spec.eventType` field records what happened. Supported values: `SyncCompleted`, `RPODegraded`, `RPOViolated`, `ConnectionLost`, `ConnectionRestored`, `ResyncStarted`, `ResyncCompleted`, `VolumeAdded`, `VolumeRemoved`, `MetadataUndersized`, `ResizePending`, `ResizeComplete`. The `spec.violationReason` field (when `eventType` is `RPOViolated`) names the cause: `ConnectionLost`, `NetworkTimeout`, `SyncLagExceeded`, `ProtocolCOutOfSync`, `ResyncInProgress`, or `Unknown`. > **Important:** Field selectors on `spec.*` fields of custom resources are not supported. Always filter RPO events using the `siterecovery.trilio.io/protection-group` label, not `--field-selector spec.protectionGroupRef.name=…`. ### Checking ProtectionRequest phase and sync progress ```bash # List all protection requests in a zone namespace with sync progress kubectl -n dr- get pr ``` The `Sync` printer column shows `status.syncProgress` as a percentage. Valid `Phase` values are: `Pending`, `Validating`, `CreatingDRBD`, `Syncing`, `ReadyToActivate`, `Activating`, `Protected`, `Unprotecting`, `Failed`. For detailed sync status on a specific VM: ```bash kubectl -n dr- get pr -o yaml ``` Check `status.slowestVolumeSyncProgress` for the bottleneck volume and `status.schedulingProbe` for DR placement outcome. ### Checking DRBDResource state on workload clusters DRBDResource CRs live on workload clusters, not on the TSR-Manager cluster. Run these commands with a workload cluster context: ```bash # List all DRBDResource objects in the VM's namespace kubectl --context -n get dres ``` Key columns: **Side** (`Source` or `Target`), **Role** (`Primary`, `Secondary`, or `Unknown`), **State** (connection state), **Phase**. To view per-volume replication details: ```bash kubectl --context -n \ get dres -o jsonpath='{.status.volumes}' | jq . ``` Each volume entry in `status.volumes` reports `replicationState` (one of `Established`, `SyncSource`, `SyncTarget`, `PausedSyncS`, `PausedSyncT`, `Off`, `Unknown`), `diskState`, and `syncProgress`. ### Checking zone and cluster health The ProtectionZone CR's status reports per-cluster reachability: ```bash kubectl -n dr- get pz kubectl -n dr- get pz -o yaml ``` In the output, `status.clusters[].status` is `Reachable`, `Unreachable`, or `Degraded`. The `status.clusters[].reason` and `status.clusters[].message` fields name the cause when status is not `Reachable`. The `ClustersValidated` condition on the ProtectionZone must be `True` for protection operations to proceed. While it is `False`, the TSR-Manager webhooks block new ProtectionGroup and ProtectionRequest creates. ### Monitoring node agent health on workload clusters The DRBD node agent DaemonSet is identified by the label `app.kubernetes.io/component=drbd-node-agent`: ```bash # List node agent pods kubectl --context \ -n trilio-site-recovery-system get pods \ -l app.kubernetes.io/component=drbd-node-agent # Get logs from the agent on a specific node (two-step) AGENT_POD=$(kubectl --context \ -n trilio-site-recovery-system get pods \ --field-selector spec.nodeName= \ -l app.kubernetes.io/component=drbd-node-agent \ -o jsonpath='{.items[0].metadata.name}') kubectl --context \ -n trilio-site-recovery-system logs $AGENT_POD ``` The node agent applies a `siterecovery.trilio.io/not-ready:NoSchedule` taint to its node during startup. This taint is removed once all DRBD resources on the node converge, or when the startup deadline (default 120 seconds) expires. If the deadline expires before convergence, the node's DRBDResource carries the condition `NodeStartupGate=False` with reason `StartupDeadlineExceeded`. ### Collecting a support bundle Collect a full diagnostic bundle from the TSR-Manager cluster using `oc adm must-gather`. This collects logs, CRs, and DRBD state from the TSR-Manager and reaches through to all zone workload clusters: ```bash oc adm must-gather --image=quay.io/triliovault/site-recovery-must-gather: ``` Contact Trilio support to obtain the correct pinned release tag for your deployment. The entrypoint inside the image is `/usr/bin/gather`. To pass flags: ```bash # Limit the log window to the last 2 hours oc adm must-gather --image=quay.io/triliovault/site-recovery-must-gather: \ -- /usr/bin/gather --since 2h # Collect from one zone only oc adm must-gather --image=quay.io/triliovault/site-recovery-must-gather: \ -- /usr/bin/gather --zone # TSR-Manager only, no reach-through to workload clusters oc adm must-gather --image=quay.io/triliovault/site-recovery-must-gather: \ -- /usr/bin/gather --local-only # Increase the log size cap (default 100Mi per container) oc adm must-gather --image=quay.io/triliovault/site-recovery-must-gather: \ -- /usr/bin/gather --max-log-size 1Gi ``` Output is written to `./must-gather.local./`. Review `SUMMARY.md` for a zone-level overview and `manifest.yaml` for per-cluster inventory and any permission gaps. > **Collect on incident detection, not after.** Pod logs are bounded and a restarted container's previous log may be lost. A bundle taken days after an incident may be missing the evidence needed to diagnose it. ### Zone credential permissions for support bundles The zone credential must include the support-bundle read permissions for the bundle to be complete. If the credential was minted before these reads were added, the bundle will be marked `partial` with a `permissionGaps` entry in `manifest.yaml`. To add the missing permissions to the existing credential without uploading a new kubeconfig: ```bash tsrctl zone-credential rotate --refresh-rbac ``` This re-applies the ClusterRole behind the existing ServiceAccount so the current token gains the support-bundle reads. Then collect the bundle again. ## Examples ### Example 1: Check replication health for all protection groups in a zone ```bash kubectl -n dr-prod get rgs ``` ### Example 2: Inspect per-volume replication status for a specific protection group ```bash kubectl -n dr-prod get rgs pg-webservers-replication -o yaml ``` Key fields to review in the output: - `status.rpoStatus` — overall RPO status (`Met`, `Degraded`, `Violated`, `Unknown`) - `status.aggregated.connectionState` — `Connected`, `Degraded`, or `Disconnected` - `status.aggregated.outOfSyncBytes` — bytes not yet replicated - `status.aggregated.syncedVolumeCount` — volumes in `Established` state - `status.lastFullySyncedAt` — timestamp of last complete sync - `status.volumeReplicationStatuses[]` — per-volume breakdown ### Example 3: List RPO events for a protection group using the label selector ```bash kubectl -n dr-prod get rpoe \ -l siterecovery.trilio.io/protection-group=pg-webservers \ --sort-by=.spec.timestamp ``` ### Example 4: Check sync progress for all protected VMs in a zone ```bash kubectl -n dr-prod get pr ``` The `Sync` column shows the percentage complete. A value less than 100 in the `Syncing` phase is normal during initial replication. `Protected` phase at 100% indicates healthy replication. ### Example 5: View DRBD connection state for a specific VM on the primary workload cluster ```bash # Find the DRBDResource name (format: drbd---) kubectl --context primary-cluster -n vm-prod get dres # Inspect the specific resource kubectl --context primary-cluster -n vm-prod \ get dres drbd-vm-prod-web-frontend-a1b2c3d4 -o yaml ``` ### Example 6: Watch ProtectionRequest phase transitions for a VM ```bash kubectl -n dr-prod get pr -w ``` ### Example 7: Enable debug logging for the TSR-Manager control plane without restarting ```bash kubectl -n dr-prod patch configmap site-recovery-logging \ --type=merge \ -p '{"data":{"level":"info","level.control-plane":"debug"}}' ``` The change takes effect within approximately 15 seconds. To revert: ```bash kubectl -n dr-prod patch configmap site-recovery-logging \ --type=merge \ -p '{"data":{"level":"info","level.control-plane":"info"}}' ``` ### Example 8: Enable debug logging on the DRBD node agent on a workload cluster ```bash kubectl --context workload-primary \ -n trilio-site-recovery-system patch configmap site-recovery-logging \ --type=merge \ -p '{"data":{"level.drbd-node-agent":"debug"}}' ``` ### Example 9: Collect a support bundle scoped to the last 2 hours ```bash oc adm must-gather \ --image=quay.io/triliovault/site-recovery-must-gather: \ -- /usr/bin/gather --since 2h ``` ### Example 10: Check the ProtectionZone condition gates ```bash kubectl -n dr-prod get pz prod -o jsonpath='{.status.conditions}' | jq . ``` Look for `ClustersValidated` — if its `status` is not `True`, the `reason` field will name one of: `ClusterAlreadyClaimed`, `InsufficientClusterPermissions`, `ClusterIdentityChanged`, `SCCPrerequisiteMissing`, `KubeVirtNotInstalled`. ### Example 11: Find node agent pods on a specific node and tail their logs ```bash AGENT_POD=$(kubectl --context workload-primary \ -n trilio-site-recovery-system get pods \ --field-selector spec.nodeName=worker-node-1 \ -l app.kubernetes.io/component=drbd-node-agent \ -o jsonpath='{.items[0].metadata.name}') kubectl --context workload-primary \ -n trilio-site-recovery-system logs $AGENT_POD --tail=100 ``` ### Example 12: Rotate zone credentials to add support-bundle read permissions ```bash tsrctl zone-credential rotate --refresh-rbac ``` ## Troubleshooting ### ReplicationGroupStatus shows `rpoStatus: Violated` **Symptom:** `kubectl -n dr- get rgs` shows `RPO-Status: Violated` and `Out-Of-Sync` is non-zero. **Likely causes:** - Network connectivity between primary and DR worker nodes has degraded or been lost. - A DRBD node agent is not running on the node hosting the protected VM. - The DRBD kernel module was unloaded (e.g. after a node reboot without the agent restarting cleanly). **Fix:** 1. Check `status.aggregated.connectionState` on the `ReplicationGroupStatus` — `Disconnected` points to a network or agent issue. 2. Check `DRBDResource` connection state on the workload cluster: `kubectl --context -n get dres`. 3. Verify the node agent is running: `kubectl --context -n trilio-site-recovery-system get pods -l app.kubernetes.io/component=drbd-node-agent`. 4. If a node agent pod is in `Init:Error` or `Init:CrashLoopBackOff`, the DRBD kernel module failed to load. Check the `drbd-module-loader` init container logs: `kubectl --context -n trilio-site-recovery-system logs -c drbd-module-loader`. Contact Trilio support if the module image has no compatible kmod for the current kernel. 5. Verify TCP ports 7000–8000 inclusive are open between primary and DR worker nodes. A firewall that stops at 7999 will silently prevent replication on one volume. --- ### `ReplicationGroupStatus` is missing (no RGS object exists) **Symptom:** `kubectl -n dr- get rgs` returns no objects, but a ProtectionGroup exists. **Likely cause:** No VM in the ProtectionGroup has reached the `Protected` phase yet. The replication monitor creates the `ReplicationGroupStatus` object only after the first VM is fully protected. **Fix:** Check the ProtectionRequest phases: `kubectl -n dr- get pr`. If any PR is stuck in `Syncing` or `CreatingDRBD`, investigate on the workload cluster by checking the corresponding `DRBDResource`. --- ### `kubectl get dres` returns `server doesn't have a resource type` **Symptom:** Running `kubectl get dres` against the TSR-Manager cluster fails with a resource-type error. **Cause:** The `DRBDResource` CRD is installed only on workload clusters. It is not registered on the TSR-Manager cluster. **Fix:** Run `kubectl get dres` with a workload cluster context: `kubectl --context -n get dres`. --- ### `ProtectionZone` shows `ClustersValidated=False` with reason `InsufficientClusterPermissions` **Symptom:** `kubectl -n dr- get pz` shows `Degraded` or the `ClustersValidated` condition is `False`. **Likely cause:** The zone credential lacks required RBAC permissions, often because the credential was minted before a new permission was required. **Fix:** Use the newest `tsrctl` (supplied by Trilio with the release — contact Trilio support to obtain it) and rotate the credential with `--refresh-rbac`: ```bash tsrctl zone-credential rotate --refresh-rbac ``` This re-applies the ClusterRole without issuing a new token. The zone re-validates within its next reconcile cycle. --- ### `ProtectionZone` shows `ClustersValidated=False` with reason `ClusterIdentityChanged` **Symptom:** The zone condition shows `ClusterIdentityChanged`. **Cause:** The zone credential now reaches a cluster whose `kube-system` Namespace UID differs from the UID recorded when the zone first claimed that cluster. This means the credential was rotated to point at a different physical cluster. **Fix:** There is no override. Restore the original cluster's credentials or delete and re-create the zone. Do not attempt to clear this condition by editing the ProtectionZone CR. --- ### Node agent pod stuck with `siterecovery.trilio.io/not-ready:NoSchedule` taint not clearing **Symptom:** A node remains tainted with `siterecovery.trilio.io/not-ready:NoSchedule` longer than expected, and the DRBDResource shows `NodeStartupGate=False` with reason `StartupDeadlineExceeded`. **Likely cause:** DRBD could not bring up one or more resources on this node within the startup deadline (default 120 seconds). The taint was removed at the deadline but the condition persists until the resources converge. **Fix:** This condition clears itself when replication converges — no action is required unless it persists indefinitely. To investigate: 1. Check the node agent logs for `drbdadm` errors. 2. Verify the peer workload cluster is reachable from this node on ports 7000–8000. 3. Check the `DRBDResource` conditions on the workload cluster for `ConnectionHealthy=False` or `DrbdadmFailed`. If the startup deadline is consistently too short for your environment, increase it on the ProtectionZone: `ProtectionZone.spec.nodeAgent.startupDeadlineSeconds` (valid range: 10–3600). --- ### Support bundle shows `status: partial` with `permissionGaps` **Symptom:** After running `oc adm must-gather`, `manifest.yaml` shows `status: partial` and lists permission gaps for one or more workload clusters. **Cause:** The zone credential for that workload cluster was minted before the support-bundle read permissions were added. The bundle collector collected what it could but could not read certain objects. **Fix:** ```bash tsrctl zone-credential rotate --refresh-rbac ``` Then collect the bundle again. The `--refresh-rbac` flag re-applies the ClusterRole to the existing ServiceAccount without requiring a new kubeconfig upload. --- ### Log level change in `site-recovery-logging` ConfigMap has no effect **Symptom:** After patching the ConfigMap, component log output does not change. **Likely causes:** - You edited the ConfigMap in the wrong namespace. TSR-Manager control plane components read the ConfigMap from the zone namespace (`dr-`). Workload components read it from `trilio-site-recovery-system` on the workload cluster. - You used the level value `warn`, which is not accepted. Valid values are `error`, `info`, `debug`, `trace`. - The ConfigMap key name is wrong. The global key is `level`; per-component keys are `level.control-plane`, `level.drbd-node-agent`, and `level.protectionzone`. **Fix:** Verify the namespace and key spelling, then allow up to 15 seconds for the change to take effect. --- ### `kubectl get rpoe` returns no results when filtering by field selector **Symptom:** `kubectl -n dr- get rpoe --field-selector spec.protectionGroupRef.name=` returns nothing. **Cause:** Field selectors on `spec.*` fields of custom resources are not supported. Only `metadata.name` and `metadata.namespace` work as field selectors on CRDs. **Fix:** Use the label selector instead: ```bash kubectl -n dr- get rpoe \ -l siterecovery.trilio.io/protection-group= ```