Logging

This page explains how Site Recovery components emit logs, how to change log verbosity at runtime without restarting pods, and how to integrate those logs into a centralized log aggregation stack. Site Recovery uses structured JSON logging across all three Go binaries (/tsr-manager, /workload-manager, and the drbd-node-agent) and the Flask UI backend, so every log line carries consistent fields that aggregation tools can index. Understanding the logging subsystem lets you diagnose stuck failovers, trace replication transitions, and satisfy audit requirements without taking a support bundle for every incident.

Before adjusting log levels or configuring aggregation:

• A working Site Recovery deployment with at least one ProtectionZone in Available phase.
• kubectl access to both the TSR-Manager cluster (namespace dr-<zone-name>) and each workload cluster (namespace trilio-site-recovery-system).
• Sufficient RBAC to read and patch ConfigMaps in those namespaces.
• For UI-based log level changes: OpenShift 4.17 or later with the TSR web console accessible via the oauth-proxy Route at https://trilio-site-recovery.<cluster-ingress-domain>.

The logging subsystem requires no installation step. Every Site Recovery component initializes its logger through the shared logging package when it starts. All components emit JSON to stdout from the first log line; no additional configuration is required to receive structured output.

The site-recovery-logging ConfigMap that drives runtime level changes is created automatically by the ProtectionZone controller in each relevant namespace when a zone reaches Available. You do not need to create it manually.

Log format

All Site Recovery Go components (control plane, workload control plane, drbd-node-agent) and the Flask UI backend emit single-line JSON to stdout. Anything written to stderr is abnormal non-JSON output (panics, pre-logger errors). Your log collector should treat stdout and stderr as separate streams.

Every JSON log line carries these fields:

Per-request log lines from the Flask backend also carry a request_id field.

Four-level vocabulary

Site Recovery exposes exactly four log levels:

There is no warn level. The Go logging stack uses a two-class model (error vs. verbose), and warn is not accepted in the ConfigMap.

Startup log level

The initial log level for each component defaults to info. You can override it at pod startup with the --zap-log-level flag (accepted values: debug, info, error, or the integers 1 for debug and 2 for trace). The startup level becomes the fallback: if the site-recovery-logging ConfigMap is later removed or its key cleared, the component reverts to this startup level.

The ProtectionZone spec field spec.logging propagates level choices to the ConfigMap in each namespace:

spec:
  logging:
    level: info              # global fallback for all components in this zone
    controlPlaneLevel: debug # override for the manager-control-plane component
    nodeAgentLevel: info     # override for the drbd-node-agent component

Valid values for all three fields: error, info, debug, trace.

Runtime log level — the site-recovery-logging ConfigMap

Each component runs a LevelWatcher that polls the site-recovery-logging ConfigMap in its own namespace every 15 seconds using an uncached API reader. A level change takes effect within one poll cycle; no pod restart is required.

The ConfigMap supports a global key and per-component override keys:

apiVersion: v1
kind: ConfigMap
metadata:
  name: site-recovery-logging
  namespace: dr-<zone-name>        # TSR-Manager components
data:
  level: info                      # global fallback
  level.control-plane: debug       # manager-control-plane override
  level.protectionzone: info       # protectionzone-controller override

For workload-cluster components, create or patch the ConfigMap in trilio-site-recovery-system:

apiVersion: v1
kind: ConfigMap
metadata:
  name: site-recovery-logging
  namespace: trilio-site-recovery-system
data:
  level: info
  level.drbd-node-agent: debug
  level.control-plane: debug

Valid component override keys:

If the ConfigMap is absent, or the relevant key is absent or empty, the component keeps its startup-flag level. An unrecognized level value (for example, warn) is logged as an error once per distinct bad value, and the component keeps its current level.

Trace level note: The trace level cannot be set via the ProtectionZone spec.logging fields or the TSR web console UI endpoint — those paths accept only error, info, and debug. To enable trace, write the value directly to the ConfigMap level or level.<component> key, or pass --zap-log-level=2 at pod startup (which requires a DaemonSet or Deployment rollout).

UI-based log level changes (OpenShift only)

The TSR web console exposes a logging endpoint for each zone. The Next.js frontend proxies the request to the Flask backend:

PUT /api/protection-zones/<zone_id>/logging

This endpoint updates the site-recovery-logging ConfigMap for the specified zone. Browser access is OpenShift-only via the oauth-proxy Route at https://trilio-site-recovery.<cluster-ingress-domain>. No equivalent endpoint exists for upstream Kubernetes deployments.

Log level chart values

The Helm charts embedded in the ProtectionZone controller binary expose these logging values (set by the controller from the ProtectionZone spec; you do not set chart values directly):

Increasing verbosity to diagnose a stuck failover

When a failover or protection operation stalls and info-level output does not reveal the cause, raise the TSR-Manager control plane to debug. The change takes effect within 15 seconds:

kubectl patch configmap site-recovery-logging \
  -n dr-<zone-name> \
  --type merge \
  -p '{"data":{"level.control-plane":"debug"}}'

To also raise the workload control plane on the primary cluster:

kubectl patch configmap site-recovery-logging \
  -n trilio-site-recovery-system \
  --context <primary-cluster-context> \
  --type merge \
  -p '{"data":{"level.control-plane":"debug"}}'

Restore info once you have captured the relevant logs:

kubectl patch configmap site-recovery-logging \
  -n dr-<zone-name> \
  --type merge \
  -p '{"data":{"level.control-plane":"info"}}'

Raising node-agent verbosity on a specific workload cluster

If a drbd-node-agent pod is not converging, raise its level on the affected workload cluster:

kubectl patch configmap site-recovery-logging \
  -n trilio-site-recovery-system \
  --context <workload-cluster-context> \
  --type merge \
  -p '{"data":{"level.drbd-node-agent":"debug"}}'

Reading logs from the TSR-Manager control plane

The TSR-Manager control plane (failover, protection, replication-monitor, and test-failover reconcilers) runs in dr-<zone-name>:

kubectl logs \
  -n dr-<zone-name> \
  -l app.kubernetes.io/component=manager-control-plane \
  --follow

Reading logs from the ProtectionZone controller

kubectl logs \
  -n dr-<zone-name> \
  -l app.kubernetes.io/component=protectionzone-controller \
  --follow

Reading logs from the workload control plane

kubectl logs \
  -n trilio-site-recovery-system \
  --context <workload-cluster-context> \
  -l app.kubernetes.io/component=workload-control-plane \
  --follow

Reading logs from a specific drbd-node-agent pod

The drbd-node-agent runs as a DaemonSet (one pod per worker node). To read logs for the agent on a specific node, first find the pod name for that node, then stream its logs:

# Step 1: get the pod name for a given node
AGENT_POD=$(kubectl get pods \
  -n trilio-site-recovery-system \
  --context <workload-cluster-context> \
  -l app.kubernetes.io/component=drbd-node-agent \
  --field-selector spec.nodeName=<node-name> \
  -o jsonpath='{.items[0].metadata.name}')

# Step 2: stream the logs
kubectl logs -n trilio-site-recovery-system \
  --context <workload-cluster-context> \
  "$AGENT_POD" --follow

Reading logs from the UI backend

kubectl logs \
  -n dr-<zone-name> \
  -l app.kubernetes.io/part-of=site-recovery \
  -c site-recovery-backend \
  --follow

Log aggregation recommendations

Because all components write JSON to stdout, any OpenShift-native or Kubernetes-compatible log collector (for example, the OpenShift Logging stack using Vector or Fluentd) can ingest Site Recovery logs without custom parsers. Recommended practices:

• Separate stdout from stderr. Normal operational output is always on stdout. Stderr output indicates abnormal conditions (panics, pre-logger errors) and should route to a high-priority alert stream.
• Index the level field. Filtering on level=error in your aggregation tool gives you an immediate operational dashboard without noise.
• Index the logger field. This identifies which component (for example, manager-control-plane, drbd-node-agent) emitted the line, enabling per-component dashboards.
• Preserve RFC3339Nano timestamps. The ts field has sub-second precision. Aggregation pipelines that truncate to seconds lose intra-second ordering, making it impossible to reconstruct the sequence of events during a failover.
• Include the caller field in your index. When a log line from Trilio support is referenced in a diagnostic exchange, the caller field (file:line) identifies the exact emitting location.
• Collect from both clusters. TSR-Manager components in dr-<zone-name> and workload components in trilio-site-recovery-system on each workload cluster form a complete picture. A failover operation spans all three clusters; logs from only one cluster will be incomplete.

Example 1: Set global zone log level to debug via ProtectionZone spec

Edit the zone's logging spec. The ProtectionZone controller propagates the change to the site-recovery-logging ConfigMap in dr-my-zone within one reconcile:

apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionZone
metadata:
  name: my-zone
  namespace: dr-my-zone
spec:
  displayName: "My Zone"
  storageBackendMode: DRBD
  logging:
    level: debug
  clusters:
    - name: primary
      clusterID: cluster1
      kubeconfigSecretRef:
        name: cluster1-kubeconfig
        key: kubeconfig
    - name: dr
      clusterID: cluster2
      kubeconfigSecretRef:
        name: cluster2-kubeconfig
        key: kubeconfig

Example 2: Override only the node agent to debug, keep control plane at info

spec:
  logging:
    level: info
    nodeAgentLevel: debug

Example 3: Patch the ConfigMap directly for an immediate runtime change on TSR-Manager

This takes effect on the next 15-second poll — no pod restart needed:

kubectl patch configmap site-recovery-logging \
  -n dr-my-zone \
  --type merge \
  -p '{"data":{"level.control-plane":"debug","level":"info"}}'

Example 4: Verify the effective ConfigMap contents

kubectl get configmap site-recovery-logging \
  -n dr-my-zone \
  -o yaml

Example 5: Read the ConfigMap from a workload cluster

kubectl get configmap site-recovery-logging \
  -n trilio-site-recovery-system \
  --context <workload-cluster-context> \
  -o yaml

Example 6: Filter aggregated logs for error-level entries from the control plane

If you are using kubectl directly rather than an aggregation stack, combine a label selector with grep:

kubectl logs \
  -n dr-my-zone \
  -l app.kubernetes.io/component=manager-control-plane \
  --since=1h \
| grep '"level":"error"'

Example 7: Check which level the node agent is currently running at

The component logs a line when the level transitions. Look for the runtime log level applied message:

kubectl logs \
  -n trilio-site-recovery-system \
  --context <workload-cluster-context> \
  -l app.kubernetes.io/component=drbd-node-agent \
  --since=30m \
| grep 'runtime log level applied'

ConfigMap change has no effect after 15 seconds

Symptom: You patched site-recovery-logging but the component is still emitting at the old level.

Likely causes:

1. You patched the ConfigMap in the wrong namespace. TSR-Manager components read from dr-<zone-name>; workload components read from trilio-site-recovery-system. Run kubectl get configmap site-recovery-logging -A to confirm which namespaces have the object.
2. The level value is unrecognized. Accepted values are exactly error, info, debug, trace. A value like warn or WARNING is silently ignored (an error is logged once). Check the component logs for runtime log-level ConfigMap has an invalid level.
3. The component cannot reach the API server transiently. In this case it keeps the current level and logs runtime log-level ConfigMap read failed; keeping current level at debug verbosity. Check the pod's connectivity to the API server.

Fix: Confirm the namespace, correct the value to one of error | info | debug | trace, and wait one poll cycle (up to 15 seconds).

Logs are not JSON — they appear as plain text or console output

Symptom: Log lines from a component arrive as human-readable text rather than JSON objects.

Likely cause: The component was started with a --zap-encoder=console flag, or the Flask backend is running with FLASK_ENV=development.

Fix: For Go components, confirm the chart value logging.encoder is json (the default). For the Flask backend, ensure FLASK_ENV is not set to development in the deployment environment. The ProtectionZone controller sets this correctly when it installs the stacks from its embedded charts; a manual override of the environment variable is the most common cause.

Cannot set trace level via the ProtectionZone spec or the web console

Symptom: Setting spec.logging.level: trace in the ProtectionZone CR, or using the PUT /api/protection-zones/<zone_id>/logging endpoint, has no effect; the component stays at debug or info.

Likely cause: The spec.logging fields and the UI logging endpoint accept only error, info, and debug. The trace level corresponds to logr V(2) / zap level -2, which has no named representation in the standard zap level set and is therefore not reachable through these paths.

Fix: Write the value trace directly to the site-recovery-logging ConfigMap using kubectl patch (see Examples section). The LevelWatcher accepts trace from the ConfigMap because it uses the product's own level map, which explicitly includes trace.

Log volume is extremely high and impacting collector performance

Symptom: After enabling debug or trace for a busy zone, log ingestion throughput spikes significantly.

Likely cause: trace is intended for high-frequency per-poll and per-scrape detail and produces a large volume of lines per second per pod. debug is suitable for diagnosing a stuck operation; trace is only warranted when debug is insufficient.

Fix: Lower the level back to debug or info by patching the site-recovery-logging ConfigMap. Use per-component override keys (for example, level.drbd-node-agent) to limit verbosity to only the component under investigation rather than raising the global level key.

kubectl logs returns no output for the drbd-node-agent

Symptom: Running kubectl logs -n trilio-site-recovery-system -l app.kubernetes.io/component=drbd-node-agent returns nothing or an error.

Likely cause: The -l selector matches multiple pods (one per worker node), and kubectl logs requires a single pod when not using --prefix. Alternatively, the DaemonSet pods may not be running yet (the node-agent applies a siterecovery.trilio.io/not-ready:NoSchedule taint to its own node during startup).

Fix: Use the two-step approach shown in the Usage section: first get the pod name for the specific node with --field-selector spec.nodeName=<node>, then call kubectl logs on that pod name. To stream from all node agent pods simultaneously, add --prefix to the label-selector invocation:

kubectl logs \
  -n trilio-site-recovery-system \
  --context <workload-cluster-context> \
  -l app.kubernetes.io/component=drbd-node-agent \
  --prefix --follow

site-recovery-logging ConfigMap does not exist in a namespace

Symptom: kubectl get configmap site-recovery-logging -n dr-<zone-name> returns NotFound.

Likely cause: The ProtectionZone has not yet reached Available phase, or the zone was created before the controller version that auto-creates the ConfigMap.

Fix: Check the ProtectionZone status: kubectl get pz -n dr-<zone-name>. If the zone is Available and the ConfigMap is still absent, the component treats this as "no override" and uses the startup-flag level — this is not an error condition. You can create the ConfigMap manually with the desired level, and the LevelWatcher will pick it up on the next poll. Ensure the ConfigMap name is exactly site-recovery-logging (the LevelWatcher looks up this exact name).