--- title: Metrics product: trilio-site-recovery-for-kubernetes doc_type: guide version: 0.2.0 source: git2docs (code-derived, validation-filtered) canonical: https://docs-tsr.trilio.io/trilio-site-recovery-for-kubernetes/site-recovery/metrics --- # Metrics _Prometheus metrics exposed by the controller, including recovery success rates and duration_ ## Overview Trilio Site Recovery exposes Prometheus-compatible metrics from both the TSR-Manager control plane and the workload control plane. These metrics let you track replication health, monitor failover and test-failover outcomes, and alert on RPO violations — without polling Kubernetes custom resources directly. This page describes where metrics are served, how to configure the metrics endpoint, and how to use the `ReplicationGroupStatus` and `RPOEvent` custom resources alongside metrics to build a complete observability picture for your protected workloads. ## Prerequisites Before configuring metrics collection, ensure you have: - Trilio Site Recovery installed on your TSR-Manager cluster and both workload clusters (the ProtectionZone must be in `Available` phase with `ClustersValidated=True`) - A Prometheus or Prometheus-compatible scraping stack reachable from the TSR-Manager cluster and each workload cluster - `kubectl` access to the TSR-Manager cluster (`dr-` namespace) and each workload cluster (`trilio-site-recovery-system` namespace) - OpenShift 4.17 or later on all three clusters (TSR-Manager, primary workload, DR workload) - Familiarity with the `ReplicationGroupStatus` (short name: `rgs`) and `RPOEvent` (short name: `rpoe`) custom resources, which are the primary observability objects on the TSR-Manager cluster ## Installation The metrics endpoint is enabled by default on all three Site Recovery controller binaries: the TSR-Manager control plane, the ProtectionZone controller, and each workload control plane. No additional installation step is required beyond installing the operator. ### Verify the metrics endpoint is active **On the TSR-Manager cluster** (replace `dr-` with your zone namespace): ```bash # List TSR-Manager control-plane pods kubectl get pods -n dr- \ -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=manager-control-plane ``` ```bash # List ProtectionZone controller pods kubectl get pods -n dr- \ -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=protectionzone-controller ``` **On each workload cluster**: ```bash # List workload control-plane pods kubectl get pods -n trilio-site-recovery-system \ -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=workload-control-plane ``` The health probe for the TSR-Manager control plane and the ProtectionZone controller is bound to `:8081`. The workload control plane uses the same default. The metrics server binds to the address set by the `--metrics-bind-address` flag (default `0`, meaning disabled unless the chart enables it). ### Check whether the metrics server is enabled in your deployment The chart-level switch is `metrics.enabled` (default `true`) with the port controlled by `metrics.port` (default `8443`) and TLS controlled by `metrics.secure` (default `true`). When `metrics.secure=true`, the endpoint requires authentication and authorization through OpenShift's built-in bearer-token flow: ```bash # Inspect the running deployment to confirm the metrics address flag kubectl get deployment -n dr- \ -l app.kubernetes.io/component=manager-control-plane \ -o jsonpath='{.items[0].spec.template.spec.containers[0].args}' ``` If the flag shows `--metrics-bind-address=0`, the metrics server is disabled for that component and you must update the deployment configuration (done through the ProtectionZone CR — see Configuration below). ## Configuration All metrics configuration is chart-managed and applied automatically by the ProtectionZone controller. You do not run `helm install` or `helm upgrade` directly. The following options govern metrics behavior and are set in the embedded charts. ### Metrics server options | Config key | Default | Effect | |---|---|---| | `metrics.enabled` | `true` | Enables or disables the Prometheus metrics endpoint | | `metrics.port` | `8443` | Port the metrics HTTPS server listens on | | `metrics.secure` | `true` | When `true`, serves metrics over HTTPS with authentication and authorization. Set to `false` for plain HTTP (not recommended in production) | When `metrics.secure=true`, the endpoint uses the same `WithAuthenticationAndAuthorization` filter as the control-plane binary, so your Prometheus scrape job must present a valid bearer token with sufficient RBAC permissions. ### Logging configuration (affects log-derived observability) Runtime log verbosity is controlled through the `site-recovery-logging` ConfigMap in each component's namespace. Changes take effect without a pod restart. ```bash # Increase TSR-Manager control-plane log level to debug kubectl patch configmap site-recovery-logging -n dr- \ --type merge \ -p '{"data":{"level":"debug","level.control-plane":"debug"}}' ``` Accepted values for `level`, `level.control-plane`, `level.drbd-node-agent`, and `level.protectionzone`: `error`, `info`, `debug`, `trace`. There is no `warn` level. ### Health probe address The health probe for all three control-plane components binds to `:8081`. This is the liveness and readiness probe endpoint — it is separate from the Prometheus metrics endpoint and does not require authentication: ```bash # Manually probe the health endpoint of a TSR-Manager control-plane pod kubectl exec -n dr- -- \ wget -qO- http://localhost:8081/healthz ``` ### DRBD node agent metrics The drbd-node-agent DaemonSet exposes its metrics on a separate port configured by `agent.metricsPort` (default `18091`) and its health probe on `agent.probePort` (default `18090`). The agent runs on the host network, so these ports must not conflict with other node-local listeners. The agent metrics server does not use the authentication/authorization filter and defaults to disabled (`--metrics-bind-address=0`). Enable it by setting `agent.metricsPort` to a non-zero port in the zone configuration. ### Polling interval for ReplicationGroupStatus The replication monitor polls DRBD state on the active workload cluster at the interval set in the auto-created `ReplicationGroupStatus` CR (`spec.pollingIntervalSeconds`, default `15`, minimum `5`, maximum `300`). Lowering this value increases the freshness of `status.aggregated.outOfSyncBytes`, `status.currentRPOSeconds`, and `status.rpoStatus`, at the cost of more API requests to the workload cluster: ## Usage Site Recovery's primary observability surface is two custom resources on the TSR-Manager cluster: `ReplicationGroupStatus` and `RPOEvent`. Use `kubectl` against the TSR-Manager cluster to query them. ### Check replication health for all protection groups ```bash # List all ReplicationGroupStatus objects (short name: rgs) kubectl get rgs -n dr- ``` The printed columns are: `Protection-Group`, `RPO-Status`, `Out-Of-Sync`, `Synced`, `Connection`, `Last-Sync`, `Age`. ### Inspect per-volume replication state ```bash # Describe a specific ReplicationGroupStatus kubectl describe rgs -replication -n dr- ``` The `status.volumeReplicationStatuses` array lists each replicated volume's `outOfSyncBytes`, `connectionState`, `replicationState`, `lastSyncedAt`, and `estimatedRPOSeconds`. ### Monitor RPO events (audit trail) ```bash # List all RPO events for a zone (short name: rpoe) kubectl get rpoe -n dr- ``` The printed columns are: `Protection-Group`, `Event-Type`, `RPO-Seconds`, `Out-Of-Sync`, `Reason`, `Timestamp`, `Age`. Filtering by protection group using the `siterecovery.trilio.io/protection-group` label: ```bash kubectl get rpoe -n dr- \ -l siterecovery.trilio.io/protection-group= ``` > **Important:** Field selectors on `spec.*` or `status.*` fields are not supported for custom resources. Always filter `RPOEvent` and `ReplicationGroupStatus` objects using the `siterecovery.trilio.io/protection-group` label, not `--field-selector`. ### Check DRBDResource state on a workload cluster `DRBDResource` CRs live on the workload clusters, not on the TSR-Manager cluster. Use the context for the relevant workload cluster: ```bash # List DRBDResources on the primary workload cluster kubectl get dres -n --context ``` The printed columns are: `Side`, `Role`, `Node`, `State`, `Phase`, `Age`. ```bash # Describe a specific DRBDResource kubectl describe dres drbd--- -n --context ``` Key fields to inspect: - `status.connectionState`: `Connected` | `Connecting` | `Disconnected` | `StandAlone` | `Unknown` - `status.role`: `Primary` | `Secondary` | `Unknown` - `status.phase`: `Pending` | `Provisioning` | `ReadyToActivate` | `Syncing` | `Consistent` | `Degraded` | `Failed` | `Deleting` - `status.syncProgress.bytesRemaining` and `status.syncProgress.percentage` ### Inspect zone and protection-group health via the TSR web console API The TSR web console backend exposes REST endpoints you can query programmatically. These are OpenShift-only (the Route is created by the ProtectionZone controller on OpenShift; upstream Kubernetes has no supported browser or API access path): ```bash # Get overall zone health curl -sk -H "Authorization: Bearer $(oc whoami -t)" \ https://trilio-site-recovery./api/protection-zones//health ``` ```bash # Get per-protection-group health curl -sk -H "Authorization: Bearer $(oc whoami -t)" \ https://trilio-site-recovery./api/protection-zones//protection-groups//health ``` ```bash # Get zone-wide replication status curl -sk -H "Authorization: Bearer $(oc whoami -t)" \ https://trilio-site-recovery./api/protection-zones//replication-status ``` ```bash # Get per-protection-group replication status curl -sk -H "Authorization: Bearer $(oc whoami -t)" \ https://trilio-site-recovery./api/protection-zones//protection-groups//replication-status ``` ```bash # Get RPO events for the zone curl -sk -H "Authorization: Bearer $(oc whoami -t)" \ https://trilio-site-recovery./api/protection-zones//rpo-events ``` ```bash # Get RPO compliance summary curl -sk -H "Authorization: Bearer $(oc whoami -t)" \ https://trilio-site-recovery./api/protection-zones//rpo-compliance ``` ```bash # Get dashboard statistics curl -sk -H "Authorization: Bearer $(oc whoami -t)" \ https://trilio-site-recovery./api/protection-zones//dashboard/stats ``` ```bash # Get operational history curl -sk -H "Authorization: Bearer $(oc whoami -t)" \ https://trilio-site-recovery./api/protection-zones//operations/stats ``` ### Check zone credentials expiry The ProtectionZone controller sets a `CredentialExpiring` reason on the `CredentialsValid` condition when a zone credential's token is approaching expiry. This is advisory and does not change the zone phase: ```bash kubectl describe pz -n dr- | grep -A5 CredentialsValid ``` To rotate a credential before it expires, contact Trilio support to obtain the `tsrctl` CLI, then run: ```bash tsrctl zone-credential rotate --refresh-rbac ``` ## Examples ### Example 1: View aggregated replication health for a protection group ```bash kubectl get rgs my-app-group-replication -n dr-prod -o yaml ``` This returns the full `ReplicationGroupStatus` object including `status.aggregated.outOfSyncBytes`, `status.aggregated.connectionState`, `status.currentRPOSeconds`, `status.rpoStatus`, and per-volume breakdown in `status.volumeReplicationStatuses`. ### Example 2: List all RPO violation events for a specific protection group ```bash kubectl get rpoe -n dr-prod \ -l siterecovery.trilio.io/protection-group=my-app-group \ --sort-by=.spec.timestamp ``` Each `RPOEvent` is immutable once created and serves as the audit record. The `spec.eventType` field indicates what triggered the event (for example, `RPOViolated`, `ConnectionLost`, `SyncCompleted`). ### Example 3: Watch replication sync progress during initial protection ```bash # On the primary workload cluster kubectl get pr -n dr-prod --watch ``` The printed columns for `ProtectionRequest` are: `VM`, `Namespace`, `Phase`, `Sync`, `Age`. The `Sync` column reflects `status.syncProgress` as a percentage. The phase transitions are: `Pending` → `Validating` → `CreatingDRBD` → `Syncing` → `ReadyToActivate` → `Activating` → `Protected`. ### Example 4: Inspect DRBD connection state on a workload cluster ```bash # List all DRBDResources in the VM's namespace on the primary workload cluster kubectl get dres -n my-vms --context primary-cluster-context ``` The `State` column reflects `status.connectionState`. A value of `Connected` with `Role` of `Primary` and `Phase` of `Consistent` indicates healthy replication. ### Example 5: Check RPO compliance via the TSR web console API ```bash curl -sk -H "Authorization: Bearer $(oc whoami -t)" \ https://trilio-site-recovery./api/protection-zones/prod/rpo-compliance ``` ### Example 6: Retrieve RPO event history for a specific protection group ```bash curl -sk -H "Authorization: Bearer $(oc whoami -t)" \ "https://trilio-site-recovery./api/protection-zones/prod/protection-groups/my-app-group/rpo-history" ``` ### Example 7: Run a health check and view results ```bash # Trigger a health check run curl -sk -X POST -H "Authorization: Bearer $(oc whoami -t)" \ https://trilio-site-recovery./api/protection-zones/prod/health/run # Retrieve health check results curl -sk -H "Authorization: Bearer $(oc whoami -t)" \ https://trilio-site-recovery./api/protection-zones/prod/health/checks ``` ### Example 8: Increase log verbosity on the drbd-node-agent for a specific node The `site-recovery-logging` ConfigMap in `trilio-site-recovery-system` on the workload cluster controls agent log level at runtime: ```bash kubectl patch configmap site-recovery-logging \ -n trilio-site-recovery-system \ --context workload-cluster-context \ --type merge \ -p '{"data":{"level.drbd-node-agent":"debug"}}' ``` To find the agent pod on a specific node: ```bash # Step 1: get the pod name for a specific node AGENT_POD=$(kubectl get pods -n trilio-site-recovery-system \ --context workload-cluster-context \ --field-selector spec.nodeName= \ -l app.kubernetes.io/component=drbd-node-agent \ -o jsonpath='{.items[0].metadata.name}') # Step 2: tail its logs kubectl logs -n trilio-site-recovery-system \ --context workload-cluster-context \ $AGENT_POD --follow ``` ## Troubleshooting ### Symptom: `kubectl get rgs` returns no objects **Likely cause:** You are querying the wrong cluster or namespace. `ReplicationGroupStatus` objects are TSR-Manager-side CRs and live only on the TSR-Manager cluster in the `dr-` namespace. Running `kubectl get rgs` against a workload cluster returns `server doesn't have a resource type`. **Fix:** Confirm you are targeting the TSR-Manager cluster and the correct zone namespace: ```bash kubectl get rgs -n dr- --context tsr-manager-context ``` --- ### Symptom: `ReplicationGroupStatus` shows `rpoStatus: Violated` or `connectionState: Disconnected` **Likely cause:** DRBD replication between the primary and DR worker nodes has been interrupted. Common causes include network connectivity loss on TCP ports 7000–8000, a node reboot, or a DRBD kernel module issue. **Fix:** 1. Check the `DRBDResource` status on both workload clusters: ```bash kubectl get dres -n --context primary-cluster-context kubectl get dres -n --context dr-cluster-context ``` 2. Verify TCP ports 7000–8000 (inclusive) are open between the primary and DR worker nodes. Port 8000 is required; a firewall rule that stops at 7999 will silently prevent replication on the last allocatable volume. 3. Check drbd-node-agent logs on the affected node: ```bash AGENT_POD=$(kubectl get pods -n trilio-site-recovery-system \ --context primary-cluster-context \ --field-selector spec.nodeName= \ -l app.kubernetes.io/component=drbd-node-agent \ -o jsonpath='{.items[0].metadata.name}') kubectl logs -n trilio-site-recovery-system \ --context primary-cluster-context $AGENT_POD ``` --- ### Symptom: Metrics endpoint returns 401 Unauthorized **Likely cause:** The metrics server is running with `metrics.secure=true` (the default), which requires a valid bearer token with RBAC access. **Fix:** Include a valid bearer token in your scrape configuration. On OpenShift: ```bash curl -sk -H "Authorization: Bearer $(oc whoami -t)" \ https://:8443/metrics ``` Alternatively, set `metrics.secure=false` by updating your zone configuration — but this is not recommended for production environments. --- ### Symptom: `RPOEvent` objects are accumulating and cannot be deleted **Likely cause:** `RPOEvent` objects are fully immutable and can only be deleted by the replication-monitor service account. Attempts to delete them with a regular user account or `kubectl delete` will be rejected by the admission webhook. **Fix:** `RPOEvent` retention is managed automatically by the replication monitor. Do not attempt to delete these objects manually. If the namespace is stuck in `Terminating` state, the Kubernetes garbage collector and namespace controller are exempt from this restriction and will clean them up during namespace deletion. --- ### Symptom: `ProtectionZone` shows `CredentialExpiring` condition **Likely cause:** One or more zone credentials are approaching their expiry time. This is an advisory condition that does not affect zone availability. **Fix:** Contact Trilio support to obtain the `tsrctl` CLI, then rotate the expiring credential: ```bash tsrctl zone-credential rotate --refresh-rbac ``` The `--refresh-rbac` flag re-applies the RBAC rules in addition to rotating the token, which is useful if cluster permissions have changed since the credential was first created. --- ### Symptom: `kubectl get dres` against the TSR-Manager cluster returns `server doesn't have a resource type` **Likely cause:** `DRBDResource` CRDs are only registered on workload clusters, not on the TSR-Manager cluster. The TSR-Manager cluster registers 9 CRDs; workload clusters each register 1 (`drbdresources`). **Fix:** Run `kubectl get dres` against the appropriate workload cluster context: ```bash kubectl get dres -n --context ``` --- ### Symptom: drbd-node-agent pod stays in `Pending` and the node has a `siterecovery.trilio.io/not-ready:NoSchedule` taint **Likely cause:** The drbd-node-agent applies this taint to its own node during startup while DRBD converges. This is expected behavior. The taint is removed once DRBD reaches a stable state within the startup deadline (default 120 seconds, configurable via `agent.startupDeadlineSeconds`). **Fix:** Wait for the agent to converge. If the taint persists beyond the startup deadline, check agent logs for DRBD initialization errors: ```bash kubectl logs -n trilio-site-recovery-system \ --context workload-cluster-context \ -l app.kubernetes.io/component=drbd-node-agent \ --field-selector spec.nodeName= ``` --- ### Symptom: `ProtectionRequest` is stuck in `Syncing` phase and sync percentage is not advancing **Likely cause:** DRBD resync may be stalled due to a network issue, insufficient bandwidth, or a disk I/O error on one side. **Fix:** 1. Check `status.slowestVolumeSyncProgress` on the `ProtectionRequest`: ```bash kubectl get pr -n dr- -o jsonpath='{.status.slowestVolumeSyncProgress}' ``` 2. Inspect the corresponding `DRBDResource` on the workload cluster for `status.volumes[].diskState` and `status.volumes[].replicationState`. 3. Review drbd-node-agent logs on both the primary and DR side for the affected node.