Guide

Observability

Metrics, logging, and monitoring for site recovery operations

Overview

This page covers the observability surface of Trilio Site Recovery: how to read replication health from custom resources, how to tune runtime log levels without restarting pods, how to interpret RPO events and replication group status, and how to collect a diagnostic support bundle when something goes wrong. Observability in Site Recovery is CR-driven — the primary signals are the ReplicationGroupStatus and RPOEvent objects on the TSR-Manager cluster, supplemented by DRBDResource status on each workload cluster and the site-recovery-logging ConfigMap for runtime log control.

Prerequisites

Before using the observability features described on this page:

A working Site Recovery deployment with at least one ProtectionZone in Available phase and at least one ProtectionGroup with protected VMs.
kubectl configured with a context pointing at the TSR-Manager cluster for querying TSR-Manager-side CRs (ReplicationGroupStatus, RPOEvent, ProtectionGroup, ProtectionRequest).
A separate kubectl context pointing at each workload cluster for querying DRBDResource objects — these CRDs are installed only on workload clusters, not on the TSR-Manager cluster.
OpenShift CLI (oc) if you intend to collect a support bundle via oc adm must-gather.
Cluster-read access on the TSR-Manager cluster (the trilio-site-recovery-viewer ClusterRole is sufficient for read-only observability).

Installation

No additional installation is required for observability. All monitoring objects are created automatically by the Site Recovery controllers:

ReplicationGroupStatus — the replication monitor controller auto-creates one ReplicationGroupStatus CR per ProtectionGroup, named <protection-group-name>-replication, in the zone namespace (dr-<zone-name>) on the TSR-Manager cluster. It appears as soon as the first VM in the group reaches the Protected phase.
RPOEvent — the replication monitor emits an RPOEvent CR on the TSR-Manager cluster on every replication status transition (sync completed, connection lost, connection restored). These are immutable audit records and accumulate over time.
DRBDResource — the workload control plane creates one DRBDResource per protected VM per workload cluster, in the VM's own namespace on that cluster. Named drbd-<namespace>-<vm>-<hash8>.
Logging ConfigMap — the site-recovery-logging ConfigMap is rendered per namespace by the ProtectionZone controller. You do not create it; you edit it to change log levels at runtime.

To verify that the replication monitor objects exist after protecting a VM:

# On the TSR-Manager cluster, in the zone namespace
kubectl -n dr-<zone-name> get replicationgroupstatuses
kubectl -n dr-<zone-name> get rpoevents

To verify DRBDResource objects on a workload cluster:

# On a workload cluster, in the VM's namespace
kubectl -n <vm-namespace> get drbdresources

Configuration

Runtime log levels

Site Recovery components read the site-recovery-logging ConfigMap in their own namespace approximately every 15 seconds and apply log level changes without a pod restart.

ConfigMap name: site-recovery-logging

Accepted level values: error, info, debug, trace

Note: There is no warn level. The logging framework uses a two-class model (info and error). Use debug or trace for verbose output.

Keys:

Key	Scope	Example
`level`	Global default for all components in this namespace	`level: debug`
`level.control-plane`	TSR-Manager control plane only	`level.control-plane: trace`
`level.drbd-node-agent`	DRBD node agent on workload clusters	`level.drbd-node-agent: debug`
`level.protectionzone`	ProtectionZone controller	`level.protectionzone: info`

To set the global level for the TSR-Manager control plane namespace:

kubectl -n dr-<zone-name> patch configmap site-recovery-logging \
  --type=merge -p '{"data":{"level":"debug"}}'

To set a per-component override while keeping global level at info:

kubectl -n dr-<zone-name> patch configmap site-recovery-logging \
  --type=merge -p '{"data":{"level":"info","level.control-plane":"debug"}}'

The ConfigMap also exists on each workload cluster in the trilio-site-recovery-system namespace and controls the node agent log level there:

kubectl --context <workload-cluster> \
  -n trilio-site-recovery-system patch configmap site-recovery-logging \
  --type=merge -p '{"data":{"level.drbd-node-agent":"debug"}}'

ReplicationGroupStatus polling interval

The ReplicationGroupStatus spec field pollingIntervalSeconds controls how often the replication monitor scrapes DRBD metrics from the active workload cluster. The default is 15 seconds. Valid range: 5–300 seconds. This value is set by the controller and is not intended to be edited by operators — the controller manages the object.

RPO objective

ProtectionGroup.spec.sla.rpoTargetSeconds sets the RPO objective in seconds. The replication monitor reads this value and uses it to compute ReplicationGroupStatus.status.rpoStatus (Met, Degraded, Violated, or Unknown) and to drive RPOEvent emission on threshold crossings. Because Site Recovery uses Protocol C synchronous replication exclusively, RPO is always 0 at the DRBD level — this field governs the alerting threshold, not the replication mode.

Usage

Checking overall replication health

The fastest single-command health check for a protection group is:

kubectl -n dr-<zone-name> get rgs

The rgs short name returns all ReplicationGroupStatus objects. Key columns:

RPO-Status — Met, Degraded, Violated, or Unknown
Out-Of-Sync — bytes currently out of sync across all volumes
Synced — number of volumes in the Established replication state
Connection — Connected, Degraded, or Disconnected
Last-Sync — timestamp of the last full sync

To inspect a specific protection group's replication status in detail:

kubectl -n dr-<zone-name> get rgs <protection-group-name>-replication -o yaml

The status.volumeReplicationStatuses array lists per-volume metrics including outOfSyncBytes, replicationState, estimatedRPOSeconds, and lastSyncedAt.

Reading RPO events

RPO events are immutable audit records of every replication status transition. Use them for SLA compliance review and post-incident analysis:

# List all RPO events for a zone, newest first
kubectl -n dr-<zone-name> get rpoe --sort-by=.spec.timestamp

# Filter by protection group using the label
kubectl -n dr-<zone-name> get rpoe \
  -l siterecovery.trilio.io/protection-group=<pg-name>

The spec.eventType field records what happened. Supported values: SyncCompleted, RPODegraded, RPOViolated, ConnectionLost, ConnectionRestored, ResyncStarted, ResyncCompleted, VolumeAdded, VolumeRemoved, MetadataUndersized, ResizePending, ResizeComplete.

The spec.violationReason field (when eventType is RPOViolated) names the cause: ConnectionLost, NetworkTimeout, SyncLagExceeded, ProtocolCOutOfSync, ResyncInProgress, or Unknown.

Important: Field selectors on spec.* fields of custom resources are not supported. Always filter RPO events using the siterecovery.trilio.io/protection-group label, not --field-selector spec.protectionGroupRef.name=….

Checking ProtectionRequest phase and sync progress

# List all protection requests in a zone namespace with sync progress
kubectl -n dr-<zone-name> get pr

The Sync printer column shows status.syncProgress as a percentage. Valid Phase values are: Pending, Validating, CreatingDRBD, Syncing, ReadyToActivate, Activating, Protected, Unprotecting, Failed.

For detailed sync status on a specific VM:

kubectl -n dr-<zone-name> get pr <pr-name> -o yaml

Check status.slowestVolumeSyncProgress for the bottleneck volume and status.schedulingProbe for DR placement outcome.

Checking DRBDResource state on workload clusters

DRBDResource CRs live on workload clusters, not on the TSR-Manager cluster. Run these commands with a workload cluster context:

# List all DRBDResource objects in the VM's namespace
kubectl --context <workload-cluster> -n <vm-namespace> get dres

Key columns: Side (Source or Target), Role (Primary, Secondary, or Unknown), State (connection state), Phase.

To view per-volume replication details:

kubectl --context <workload-cluster> -n <vm-namespace> \
  get dres <drbd-resource-name> -o jsonpath='{.status.volumes}' | jq .

Each volume entry in status.volumes reports replicationState (one of Established, SyncSource, SyncTarget, PausedSyncS, PausedSyncT, Off, Unknown), diskState, and syncProgress.

Checking zone and cluster health

The ProtectionZone CR's status reports per-cluster reachability:

kubectl -n dr-<zone-name> get pz
kubectl -n dr-<zone-name> get pz <zone-name> -o yaml

In the output, status.clusters[].status is Reachable, Unreachable, or Degraded. The status.clusters[].reason and status.clusters[].message fields name the cause when status is not Reachable.

The ClustersValidated condition on the ProtectionZone must be True for protection operations to proceed. While it is False, the TSR-Manager webhooks block new ProtectionGroup and ProtectionRequest creates.

Monitoring node agent health on workload clusters

The DRBD node agent DaemonSet is identified by the label app.kubernetes.io/component=drbd-node-agent:

# List node agent pods
kubectl --context <workload-cluster> \
  -n trilio-site-recovery-system get pods \
  -l app.kubernetes.io/component=drbd-node-agent

# Get logs from the agent on a specific node (two-step)
AGENT_POD=$(kubectl --context <workload-cluster> \
  -n trilio-site-recovery-system get pods \
  --field-selector spec.nodeName=<node-name> \
  -l app.kubernetes.io/component=drbd-node-agent \
  -o jsonpath='{.items[0].metadata.name}')

kubectl --context <workload-cluster> \
  -n trilio-site-recovery-system logs $AGENT_POD

The node agent applies a siterecovery.trilio.io/not-ready:NoSchedule taint to its node during startup. This taint is removed once all DRBD resources on the node converge, or when the startup deadline (default 120 seconds) expires. If the deadline expires before convergence, the node's DRBDResource carries the condition NodeStartupGate=False with reason StartupDeadlineExceeded.

Collecting a support bundle

Collect a full diagnostic bundle from the TSR-Manager cluster using oc adm must-gather. This collects logs, CRs, and DRBD state from the TSR-Manager and reaches through to all zone workload clusters:

oc adm must-gather --image=quay.io/triliovault/site-recovery-must-gather:<release-tag>

Contact Trilio support to obtain the correct pinned release tag for your deployment.

The entrypoint inside the image is /usr/bin/gather. To pass flags:

# Limit the log window to the last 2 hours
oc adm must-gather --image=quay.io/triliovault/site-recovery-must-gather:<release-tag> \
  -- /usr/bin/gather --since 2h

# Collect from one zone only
oc adm must-gather --image=quay.io/triliovault/site-recovery-must-gather:<release-tag> \
  -- /usr/bin/gather --zone <zone-name>

# TSR-Manager only, no reach-through to workload clusters
oc adm must-gather --image=quay.io/triliovault/site-recovery-must-gather:<release-tag> \
  -- /usr/bin/gather --local-only

# Increase the log size cap (default 100Mi per container)
oc adm must-gather --image=quay.io/triliovault/site-recovery-must-gather:<release-tag> \
  -- /usr/bin/gather --max-log-size 1Gi

Output is written to ./must-gather.local.<hash>/. Review SUMMARY.md for a zone-level overview and manifest.yaml for per-cluster inventory and any permission gaps.

Collect on incident detection, not after. Pod logs are bounded and a restarted container's previous log may be lost. A bundle taken days after an incident may be missing the evidence needed to diagnose it.

Zone credential permissions for support bundles

The zone credential must include the support-bundle read permissions for the bundle to be complete. If the credential was minted before these reads were added, the bundle will be marked partial with a permissionGaps entry in manifest.yaml.

To add the missing permissions to the existing credential without uploading a new kubeconfig:

tsrctl zone-credential rotate --refresh-rbac

This re-applies the ClusterRole behind the existing ServiceAccount so the current token gains the support-bundle reads. Then collect the bundle again.

Examples

Example 1: Check replication health for all protection groups in a zone

kubectl -n dr-prod get rgs

Example 2: Inspect per-volume replication status for a specific protection group

kubectl -n dr-prod get rgs pg-webservers-replication -o yaml

Key fields to review in the output:

status.rpoStatus — overall RPO status (Met, Degraded, Violated, Unknown)
status.aggregated.connectionState — Connected, Degraded, or Disconnected
status.aggregated.outOfSyncBytes — bytes not yet replicated
status.aggregated.syncedVolumeCount — volumes in Established state
status.lastFullySyncedAt — timestamp of last complete sync
status.volumeReplicationStatuses[] — per-volume breakdown

Example 3: List RPO events for a protection group using the label selector

kubectl -n dr-prod get rpoe \
  -l siterecovery.trilio.io/protection-group=pg-webservers \
  --sort-by=.spec.timestamp

Example 4: Check sync progress for all protected VMs in a zone

kubectl -n dr-prod get pr

The Sync column shows the percentage complete. A value less than 100 in the Syncing phase is normal during initial replication. Protected phase at 100% indicates healthy replication.

Example 5: View DRBD connection state for a specific VM on the primary workload cluster

# Find the DRBDResource name (format: drbd-<namespace>-<vm>-<hash8>)
kubectl --context primary-cluster -n vm-prod get dres

# Inspect the specific resource
kubectl --context primary-cluster -n vm-prod \
  get dres drbd-vm-prod-web-frontend-a1b2c3d4 -o yaml

Example 6: Watch ProtectionRequest phase transitions for a VM

kubectl -n dr-prod get pr -w

Example 7: Enable debug logging for the TSR-Manager control plane without restarting

kubectl -n dr-prod patch configmap site-recovery-logging \
  --type=merge \
  -p '{"data":{"level":"info","level.control-plane":"debug"}}'

The change takes effect within approximately 15 seconds. To revert:

kubectl -n dr-prod patch configmap site-recovery-logging \
  --type=merge \
  -p '{"data":{"level":"info","level.control-plane":"info"}}'

Example 8: Enable debug logging on the DRBD node agent on a workload cluster

kubectl --context workload-primary \
  -n trilio-site-recovery-system patch configmap site-recovery-logging \
  --type=merge \
  -p '{"data":{"level.drbd-node-agent":"debug"}}'

Example 9: Collect a support bundle scoped to the last 2 hours

oc adm must-gather \
  --image=quay.io/triliovault/site-recovery-must-gather:<release-tag> \
  -- /usr/bin/gather --since 2h

Example 10: Check the ProtectionZone condition gates

kubectl -n dr-prod get pz prod -o jsonpath='{.status.conditions}' | jq .

Look for ClustersValidated — if its status is not True, the reason field will name one of: ClusterAlreadyClaimed, InsufficientClusterPermissions, ClusterIdentityChanged, SCCPrerequisiteMissing, KubeVirtNotInstalled.

Example 11: Find node agent pods on a specific node and tail their logs

AGENT_POD=$(kubectl --context workload-primary \
  -n trilio-site-recovery-system get pods \
  --field-selector spec.nodeName=worker-node-1 \
  -l app.kubernetes.io/component=drbd-node-agent \
  -o jsonpath='{.items[0].metadata.name}')

kubectl --context workload-primary \
  -n trilio-site-recovery-system logs $AGENT_POD --tail=100

Example 12: Rotate zone credentials to add support-bundle read permissions

tsrctl zone-credential rotate --refresh-rbac

Troubleshooting

ReplicationGroupStatus shows `rpoStatus: Violated`

Symptom: kubectl -n dr-<zone> get rgs shows RPO-Status: Violated and Out-Of-Sync is non-zero.

Likely causes:

Network connectivity between primary and DR worker nodes has degraded or been lost.
A DRBD node agent is not running on the node hosting the protected VM.
The DRBD kernel module was unloaded (e.g. after a node reboot without the agent restarting cleanly).

Fix:

Check status.aggregated.connectionState on the ReplicationGroupStatus — Disconnected points to a network or agent issue.
Check DRBDResource connection state on the workload cluster: kubectl --context <workload> -n <vm-ns> get dres.
Verify the node agent is running: kubectl --context <workload> -n trilio-site-recovery-system get pods -l app.kubernetes.io/component=drbd-node-agent.
If a node agent pod is in Init:Error or Init:CrashLoopBackOff, the DRBD kernel module failed to load. Check the drbd-module-loader init container logs: kubectl --context <workload> -n trilio-site-recovery-system logs <agent-pod> -c drbd-module-loader. Contact Trilio support if the module image has no compatible kmod for the current kernel.
Verify TCP ports 7000–8000 inclusive are open between primary and DR worker nodes. A firewall that stops at 7999 will silently prevent replication on one volume.

`ReplicationGroupStatus` is missing (no RGS object exists)

Symptom: kubectl -n dr-<zone> get rgs returns no objects, but a ProtectionGroup exists.

Likely cause: No VM in the ProtectionGroup has reached the Protected phase yet. The replication monitor creates the ReplicationGroupStatus object only after the first VM is fully protected.

Fix: Check the ProtectionRequest phases: kubectl -n dr-<zone> get pr. If any PR is stuck in Syncing or CreatingDRBD, investigate on the workload cluster by checking the corresponding DRBDResource.

`kubectl get dres` returns `server doesn't have a resource type`

Symptom: Running kubectl get dres against the TSR-Manager cluster fails with a resource-type error.

Cause: The DRBDResource CRD is installed only on workload clusters. It is not registered on the TSR-Manager cluster.

Fix: Run kubectl get dres with a workload cluster context: kubectl --context <workload-cluster> -n <vm-namespace> get dres.

`ProtectionZone` shows `ClustersValidated=False` with reason `InsufficientClusterPermissions`

Symptom: kubectl -n dr-<zone> get pz shows Degraded or the ClustersValidated condition is False.

Likely cause: The zone credential lacks required RBAC permissions, often because the credential was minted before a new permission was required.

Fix: Use the newest tsrctl (supplied by Trilio with the release — contact Trilio support to obtain it) and rotate the credential with --refresh-rbac:

tsrctl zone-credential rotate --refresh-rbac

This re-applies the ClusterRole without issuing a new token. The zone re-validates within its next reconcile cycle.

`ProtectionZone` shows `ClustersValidated=False` with reason `ClusterIdentityChanged`

Symptom: The zone condition shows ClusterIdentityChanged.

Cause: The zone credential now reaches a cluster whose kube-system Namespace UID differs from the UID recorded when the zone first claimed that cluster. This means the credential was rotated to point at a different physical cluster.

Fix: There is no override. Restore the original cluster's credentials or delete and re-create the zone. Do not attempt to clear this condition by editing the ProtectionZone CR.

Node agent pod stuck with `siterecovery.trilio.io/not-ready:NoSchedule` taint not clearing

Symptom: A node remains tainted with siterecovery.trilio.io/not-ready:NoSchedule longer than expected, and the DRBDResource shows NodeStartupGate=False with reason StartupDeadlineExceeded.

Likely cause: DRBD could not bring up one or more resources on this node within the startup deadline (default 120 seconds). The taint was removed at the deadline but the condition persists until the resources converge.

Fix: This condition clears itself when replication converges — no action is required unless it persists indefinitely. To investigate:

Check the node agent logs for drbdadm errors.
Verify the peer workload cluster is reachable from this node on ports 7000–8000.
Check the DRBDResource conditions on the workload cluster for ConnectionHealthy=False or DrbdadmFailed.

If the startup deadline is consistently too short for your environment, increase it on the ProtectionZone: ProtectionZone.spec.nodeAgent.startupDeadlineSeconds (valid range: 10–3600).

Support bundle shows `status: partial` with `permissionGaps`

Symptom: After running oc adm must-gather, manifest.yaml shows status: partial and lists permission gaps for one or more workload clusters.

Cause: The zone credential for that workload cluster was minted before the support-bundle read permissions were added. The bundle collector collected what it could but could not read certain objects.

Fix:

tsrctl zone-credential rotate --refresh-rbac

Then collect the bundle again. The --refresh-rbac flag re-applies the ClusterRole to the existing ServiceAccount without requiring a new kubeconfig upload.

Log level change in `site-recovery-logging` ConfigMap has no effect

Symptom: After patching the ConfigMap, component log output does not change.