Metrics

Trilio Site Recovery exposes Prometheus-compatible metrics from both the TSR-Manager control plane and the workload control plane. These metrics let you track replication health, monitor failover and test-failover outcomes, and alert on RPO violations — without polling Kubernetes custom resources directly. This page describes where metrics are served, how to configure the metrics endpoint, and how to use the ReplicationGroupStatus and RPOEvent custom resources alongside metrics to build a complete observability picture for your protected workloads.

Before configuring metrics collection, ensure you have:

• Trilio Site Recovery installed on your TSR-Manager cluster and both workload clusters (the ProtectionZone must be in Available phase with ClustersValidated=True)
• A Prometheus or Prometheus-compatible scraping stack reachable from the TSR-Manager cluster and each workload cluster
• kubectl access to the TSR-Manager cluster (dr-<zone-name> namespace) and each workload cluster (trilio-site-recovery-system namespace)
• OpenShift 4.17 or later on all three clusters (TSR-Manager, primary workload, DR workload)
• Familiarity with the ReplicationGroupStatus (short name: rgs) and RPOEvent (short name: rpoe) custom resources, which are the primary observability objects on the TSR-Manager cluster

The metrics endpoint is enabled by default on all three Site Recovery controller binaries: the TSR-Manager control plane, the ProtectionZone controller, and each workload control plane. No additional installation step is required beyond installing the operator.

Verify the metrics endpoint is active

On the TSR-Manager cluster (replace dr-<zone-name> with your zone namespace):

# List TSR-Manager control-plane pods
kubectl get pods -n dr-<zone-name> \
  -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=manager-control-plane

# List ProtectionZone controller pods
kubectl get pods -n dr-<zone-name> \
  -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=protectionzone-controller

On each workload cluster:

# List workload control-plane pods
kubectl get pods -n trilio-site-recovery-system \
  -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=workload-control-plane

The health probe for the TSR-Manager control plane and the ProtectionZone controller is bound to :8081. The workload control plane uses the same default. The metrics server binds to the address set by the --metrics-bind-address flag (default 0, meaning disabled unless the chart enables it).

Check whether the metrics server is enabled in your deployment

The chart-level switch is metrics.enabled (default true) with the port controlled by metrics.port (default 8443) and TLS controlled by metrics.secure (default true). When metrics.secure=true, the endpoint requires authentication and authorization through OpenShift's built-in bearer-token flow:

# Inspect the running deployment to confirm the metrics address flag
kubectl get deployment -n dr-<zone-name> \
  -l app.kubernetes.io/component=manager-control-plane \
  -o jsonpath='{.items[0].spec.template.spec.containers[0].args}'

If the flag shows --metrics-bind-address=0, the metrics server is disabled for that component and you must update the deployment configuration (done through the ProtectionZone CR — see Configuration below).

All metrics configuration is chart-managed and applied automatically by the ProtectionZone controller. You do not run helm install or helm upgrade directly. The following options govern metrics behavior and are set in the embedded charts.

Metrics server options

When metrics.secure=true, the endpoint uses the same WithAuthenticationAndAuthorization filter as the control-plane binary, so your Prometheus scrape job must present a valid bearer token with sufficient RBAC permissions.

Logging configuration (affects log-derived observability)

Runtime log verbosity is controlled through the site-recovery-logging ConfigMap in each component's namespace. Changes take effect without a pod restart.

# Increase TSR-Manager control-plane log level to debug
kubectl patch configmap site-recovery-logging -n dr-<zone-name> \
  --type merge \
  -p '{"data":{"level":"debug","level.control-plane":"debug"}}'

Accepted values for level, level.control-plane, level.drbd-node-agent, and level.protectionzone: error, info, debug, trace. There is no warn level.

Health probe address

The health probe for all three control-plane components binds to :8081. This is the liveness and readiness probe endpoint — it is separate from the Prometheus metrics endpoint and does not require authentication:

# Manually probe the health endpoint of a TSR-Manager control-plane pod
kubectl exec -n dr-<zone-name> <pod-name> -- \
  wget -qO- http://localhost:8081/healthz

DRBD node agent metrics

The drbd-node-agent DaemonSet exposes its metrics on a separate port configured by agent.metricsPort (default 18091) and its health probe on agent.probePort (default 18090). The agent runs on the host network, so these ports must not conflict with other node-local listeners. The agent metrics server does not use the authentication/authorization filter and defaults to disabled (--metrics-bind-address=0). Enable it by setting agent.metricsPort to a non-zero port in the zone configuration.

Polling interval for ReplicationGroupStatus

The replication monitor polls DRBD state on the active workload cluster at the interval set in the auto-created ReplicationGroupStatus CR (spec.pollingIntervalSeconds, default 15, minimum 5, maximum 300). Lowering this value increases the freshness of status.aggregated.outOfSyncBytes, status.currentRPOSeconds, and status.rpoStatus, at the cost of more API requests to the workload cluster:

Site Recovery's primary observability surface is two custom resources on the TSR-Manager cluster: ReplicationGroupStatus and RPOEvent. Use kubectl against the TSR-Manager cluster to query them.

Check replication health for all protection groups

# List all ReplicationGroupStatus objects (short name: rgs)
kubectl get rgs -n dr-<zone-name>

The printed columns are: Protection-Group, RPO-Status, Out-Of-Sync, Synced, Connection, Last-Sync, Age.

Inspect per-volume replication state

# Describe a specific ReplicationGroupStatus
kubectl describe rgs <protection-group-name>-replication -n dr-<zone-name>

The status.volumeReplicationStatuses array lists each replicated volume's outOfSyncBytes, connectionState, replicationState, lastSyncedAt, and estimatedRPOSeconds.

Monitor RPO events (audit trail)

# List all RPO events for a zone (short name: rpoe)
kubectl get rpoe -n dr-<zone-name>

The printed columns are: Protection-Group, Event-Type, RPO-Seconds, Out-Of-Sync, Reason, Timestamp, Age.

Filtering by protection group using the siterecovery.trilio.io/protection-group label:

kubectl get rpoe -n dr-<zone-name> \
  -l siterecovery.trilio.io/protection-group=<protection-group-name>

Important: Field selectors on spec.* or status.* fields are not supported for custom resources. Always filter RPOEvent and ReplicationGroupStatus objects using the siterecovery.trilio.io/protection-group label, not --field-selector.

Check DRBDResource state on a workload cluster

DRBDResource CRs live on the workload clusters, not on the TSR-Manager cluster. Use the context for the relevant workload cluster:

# List DRBDResources on the primary workload cluster
kubectl get dres -n <vm-namespace> --context <cluster1-context>

The printed columns are: Side, Role, Node, State, Phase, Age.

# Describe a specific DRBDResource
kubectl describe dres drbd-<namespace>-<vm>-<hash8> -n <vm-namespace> --context <cluster1-context>

Key fields to inspect:

• status.connectionState: Connected | Connecting | Disconnected | StandAlone | Unknown
• status.role: Primary | Secondary | Unknown
• status.phase: Pending | Provisioning | ReadyToActivate | Syncing | Consistent | Degraded | Failed | Deleting
• status.syncProgress.bytesRemaining and status.syncProgress.percentage

Inspect zone and protection-group health via the TSR web console API

The TSR web console backend exposes REST endpoints you can query programmatically. These are OpenShift-only (the Route is created by the ProtectionZone controller on OpenShift; upstream Kubernetes has no supported browser or API access path):

# Get overall zone health
curl -sk -H "Authorization: Bearer $(oc whoami -t)" \
  https://trilio-site-recovery.<cluster-ingress-domain>/api/protection-zones/<zone_id>/health

# Get per-protection-group health
curl -sk -H "Authorization: Bearer $(oc whoami -t)" \
  https://trilio-site-recovery.<cluster-ingress-domain>/api/protection-zones/<zone_id>/protection-groups/<pg_name>/health

# Get zone-wide replication status
curl -sk -H "Authorization: Bearer $(oc whoami -t)" \
  https://trilio-site-recovery.<cluster-ingress-domain>/api/protection-zones/<zone_id>/replication-status

# Get per-protection-group replication status
curl -sk -H "Authorization: Bearer $(oc whoami -t)" \
  https://trilio-site-recovery.<cluster-ingress-domain>/api/protection-zones/<zone_id>/protection-groups/<pg_name>/replication-status

# Get RPO events for the zone
curl -sk -H "Authorization: Bearer $(oc whoami -t)" \
  https://trilio-site-recovery.<cluster-ingress-domain>/api/protection-zones/<zone_id>/rpo-events

# Get RPO compliance summary
curl -sk -H "Authorization: Bearer $(oc whoami -t)" \
  https://trilio-site-recovery.<cluster-ingress-domain>/api/protection-zones/<zone_id>/rpo-compliance

# Get dashboard statistics
curl -sk -H "Authorization: Bearer $(oc whoami -t)" \
  https://trilio-site-recovery.<cluster-ingress-domain>/api/protection-zones/<zone_id>/dashboard/stats

# Get operational history
curl -sk -H "Authorization: Bearer $(oc whoami -t)" \
  https://trilio-site-recovery.<cluster-ingress-domain>/api/protection-zones/<zone_id>/operations/stats

Check zone credentials expiry

The ProtectionZone controller sets a CredentialExpiring reason on the CredentialsValid condition when a zone credential's token is approaching expiry. This is advisory and does not change the zone phase:

kubectl describe pz <zone-name> -n dr-<zone-name> | grep -A5 CredentialsValid

To rotate a credential before it expires, contact Trilio support to obtain the tsrctl CLI, then run:

tsrctl zone-credential rotate --refresh-rbac

Example 1: View aggregated replication health for a protection group

kubectl get rgs my-app-group-replication -n dr-prod -o yaml

This returns the full ReplicationGroupStatus object including status.aggregated.outOfSyncBytes, status.aggregated.connectionState, status.currentRPOSeconds, status.rpoStatus, and per-volume breakdown in status.volumeReplicationStatuses.

Example 2: List all RPO violation events for a specific protection group

kubectl get rpoe -n dr-prod \
  -l siterecovery.trilio.io/protection-group=my-app-group \
  --sort-by=.spec.timestamp

Each RPOEvent is immutable once created and serves as the audit record. The spec.eventType field indicates what triggered the event (for example, RPOViolated, ConnectionLost, SyncCompleted).

Example 3: Watch replication sync progress during initial protection

# On the primary workload cluster
kubectl get pr -n dr-prod --watch

The printed columns for ProtectionRequest are: VM, Namespace, Phase, Sync, Age. The Sync column reflects status.syncProgress as a percentage. The phase transitions are: Pending → Validating → CreatingDRBD → Syncing → ReadyToActivate → Activating → Protected.

Example 4: Inspect DRBD connection state on a workload cluster

# List all DRBDResources in the VM's namespace on the primary workload cluster
kubectl get dres -n my-vms --context primary-cluster-context

The State column reflects status.connectionState. A value of Connected with Role of Primary and Phase of Consistent indicates healthy replication.

Example 5: Check RPO compliance via the TSR web console API

curl -sk -H "Authorization: Bearer $(oc whoami -t)" \
  https://trilio-site-recovery.<cluster-ingress-domain>/api/protection-zones/prod/rpo-compliance

Example 6: Retrieve RPO event history for a specific protection group

curl -sk -H "Authorization: Bearer $(oc whoami -t)" \
  "https://trilio-site-recovery.<cluster-ingress-domain>/api/protection-zones/prod/protection-groups/my-app-group/rpo-history"

Example 7: Run a health check and view results

# Trigger a health check run
curl -sk -X POST -H "Authorization: Bearer $(oc whoami -t)" \
  https://trilio-site-recovery.<cluster-ingress-domain>/api/protection-zones/prod/health/run

# Retrieve health check results
curl -sk -H "Authorization: Bearer $(oc whoami -t)" \
  https://trilio-site-recovery.<cluster-ingress-domain>/api/protection-zones/prod/health/checks

Example 8: Increase log verbosity on the drbd-node-agent for a specific node

The site-recovery-logging ConfigMap in trilio-site-recovery-system on the workload cluster controls agent log level at runtime:

kubectl patch configmap site-recovery-logging \
  -n trilio-site-recovery-system \
  --context workload-cluster-context \
  --type merge \
  -p '{"data":{"level.drbd-node-agent":"debug"}}'

To find the agent pod on a specific node:

# Step 1: get the pod name for a specific node
AGENT_POD=$(kubectl get pods -n trilio-site-recovery-system \
  --context workload-cluster-context \
  --field-selector spec.nodeName=<node-name> \
  -l app.kubernetes.io/component=drbd-node-agent \
  -o jsonpath='{.items[0].metadata.name}')

# Step 2: tail its logs
kubectl logs -n trilio-site-recovery-system \
  --context workload-cluster-context \
  $AGENT_POD --follow

Symptom: kubectl get rgs returns no objects

Likely cause: You are querying the wrong cluster or namespace. ReplicationGroupStatus objects are TSR-Manager-side CRs and live only on the TSR-Manager cluster in the dr-<zone-name> namespace. Running kubectl get rgs against a workload cluster returns server doesn't have a resource type.

Fix: Confirm you are targeting the TSR-Manager cluster and the correct zone namespace:

kubectl get rgs -n dr-<zone-name> --context tsr-manager-context

Symptom: ReplicationGroupStatus shows rpoStatus: Violated or connectionState: Disconnected

Likely cause: DRBD replication between the primary and DR worker nodes has been interrupted. Common causes include network connectivity loss on TCP ports 7000–8000, a node reboot, or a DRBD kernel module issue.

Fix:

1. Check the DRBDResource status on both workload clusters:
   Copy

   kubectl get dres -n <vm-namespace> --context primary-cluster-context
   kubectl get dres -n <vm-namespace> --context dr-cluster-context
   

2. Verify TCP ports 7000–8000 (inclusive) are open between the primary and DR worker nodes. Port 8000 is required; a firewall rule that stops at 7999 will silently prevent replication on the last allocatable volume.
3. Check drbd-node-agent logs on the affected node:
   Copy

   AGENT_POD=$(kubectl get pods -n trilio-site-recovery-system \
     --context primary-cluster-context \
     --field-selector spec.nodeName=<affected-node> \
     -l app.kubernetes.io/component=drbd-node-agent \
     -o jsonpath='{.items[0].metadata.name}')
   kubectl logs -n trilio-site-recovery-system \
     --context primary-cluster-context $AGENT_POD
   

Symptom: Metrics endpoint returns 401 Unauthorized

Likely cause: The metrics server is running with metrics.secure=true (the default), which requires a valid bearer token with RBAC access.

Fix: Include a valid bearer token in your scrape configuration. On OpenShift:

curl -sk -H "Authorization: Bearer $(oc whoami -t)" \
  https://<pod-ip>:8443/metrics

Alternatively, set metrics.secure=false by updating your zone configuration — but this is not recommended for production environments.

Symptom: RPOEvent objects are accumulating and cannot be deleted

Likely cause: RPOEvent objects are fully immutable and can only be deleted by the replication-monitor service account. Attempts to delete them with a regular user account or kubectl delete will be rejected by the admission webhook.

Fix: RPOEvent retention is managed automatically by the replication monitor. Do not attempt to delete these objects manually. If the namespace is stuck in Terminating state, the Kubernetes garbage collector and namespace controller are exempt from this restriction and will clean them up during namespace deletion.

Symptom: ProtectionZone shows CredentialExpiring condition

Likely cause: One or more zone credentials are approaching their expiry time. This is an advisory condition that does not affect zone availability.

Fix: Contact Trilio support to obtain the tsrctl CLI, then rotate the expiring credential:

tsrctl zone-credential rotate --refresh-rbac

The --refresh-rbac flag re-applies the RBAC rules in addition to rotating the token, which is useful if cluster permissions have changed since the credential was first created.

Symptom: kubectl get dres against the TSR-Manager cluster returns server doesn't have a resource type

Likely cause: DRBDResource CRDs are only registered on workload clusters, not on the TSR-Manager cluster. The TSR-Manager cluster registers 9 CRDs; workload clusters each register 1 (drbdresources).

Fix: Run kubectl get dres against the appropriate workload cluster context:

kubectl get dres -n <vm-namespace> --context <workload-cluster-context>

Symptom: drbd-node-agent pod stays in Pending and the node has a siterecovery.trilio.io/not-ready:NoSchedule taint

Likely cause: The drbd-node-agent applies this taint to its own node during startup while DRBD converges. This is expected behavior. The taint is removed once DRBD reaches a stable state within the startup deadline (default 120 seconds, configurable via agent.startupDeadlineSeconds).

Fix: Wait for the agent to converge. If the taint persists beyond the startup deadline, check agent logs for DRBD initialization errors:

kubectl logs -n trilio-site-recovery-system \
  --context workload-cluster-context \
  -l app.kubernetes.io/component=drbd-node-agent \
  --field-selector spec.nodeName=<node-name>

Symptom: ProtectionRequest is stuck in Syncing phase and sync percentage is not advancing

Likely cause: DRBD resync may be stalled due to a network issue, insufficient bandwidth, or a disk I/O error on one side.

Fix:

1. Check status.slowestVolumeSyncProgress on the ProtectionRequest:
   Copy

   kubectl get pr <name> -n dr-<zone-name> -o jsonpath='{.status.slowestVolumeSyncProgress}'
   

2. Inspect the corresponding DRBDResource on the workload cluster for status.volumes[].diskState and status.volumes[].replicationState.
3. Review drbd-node-agent logs on both the primary and DR side for the affected node.