--- title: Logging product: trilio-site-recovery-for-kubernetes doc_type: guide version: 0.2.0 source: git2docs (code-derived, validation-filtered) canonical: https://docs-tsr.trilio.io/trilio-site-recovery-for-kubernetes/site-recovery/logging --- # Logging _Log levels, structured logging output, and log aggregation recommendations_ ## Overview This page explains how Site Recovery components emit logs, how to change log verbosity at runtime without restarting pods, and how to integrate those logs into a centralized log aggregation stack. Site Recovery uses structured JSON logging across all three Go binaries (`/tsr-manager`, `/workload-manager`, and the drbd-node-agent) and the Flask UI backend, so every log line carries consistent fields that aggregation tools can index. Understanding the logging subsystem lets you diagnose stuck failovers, trace replication transitions, and satisfy audit requirements without taking a support bundle for every incident. ## Prerequisites Before adjusting log levels or configuring aggregation: - A working Site Recovery deployment with at least one ProtectionZone in `Available` phase. - `kubectl` access to both the TSR-Manager cluster (namespace `dr-`) and each workload cluster (namespace `trilio-site-recovery-system`). - Sufficient RBAC to read and patch ConfigMaps in those namespaces. - For UI-based log level changes: OpenShift 4.17 or later with the TSR web console accessible via the oauth-proxy Route at `https://trilio-site-recovery.`. ## Installation The logging subsystem requires no installation step. Every Site Recovery component initializes its logger through the shared `logging` package when it starts. All components emit JSON to stdout from the first log line; no additional configuration is required to receive structured output. The `site-recovery-logging` ConfigMap that drives runtime level changes is created automatically by the ProtectionZone controller in each relevant namespace when a zone reaches `Available`. You do not need to create it manually. ## Configuration ### Log format All Site Recovery Go components (control plane, workload control plane, drbd-node-agent) and the Flask UI backend emit single-line JSON to **stdout**. Anything written to **stderr** is abnormal non-JSON output (panics, pre-logger errors). Your log collector should treat stdout and stderr as separate streams. Every JSON log line carries these fields: | Field | Description | |---|---| | `ts` | RFC3339Nano timestamp (sub-second precision) | | `level` | One of `error`, `info`, `debug`, `trace` | | `caller` | Short `file:line` of the emitting call site | | `msg` | Human-readable message | | `logger` | Component name | | `stacktrace` | Stack trace (errors only; key is `stacktrace`, not `exception`) | Per-request log lines from the Flask backend also carry a `request_id` field. ### Four-level vocabulary Site Recovery exposes exactly four log levels: | Level | When to use | |---|---| | `error` | Failures requiring operator attention | | `info` | Normal operational events | | `debug` | Per-phase detail; sufficient to diagnose a stuck failover | | `trace` | High-frequency per-poll and per-scrape detail | There is no `warn` level. The Go logging stack uses a two-class model (error vs. verbose), and `warn` is not accepted in the ConfigMap. ### Startup log level The initial log level for each component defaults to `info`. You can override it at pod startup with the `--zap-log-level` flag (accepted values: `debug`, `info`, `error`, or the integers `1` for debug and `2` for trace). The startup level becomes the **fallback**: if the `site-recovery-logging` ConfigMap is later removed or its key cleared, the component reverts to this startup level. The ProtectionZone spec field `spec.logging` propagates level choices to the ConfigMap in each namespace: ```yaml spec: logging: level: info # global fallback for all components in this zone controlPlaneLevel: debug # override for the manager-control-plane component nodeAgentLevel: info # override for the drbd-node-agent component ``` Valid values for all three fields: `error`, `info`, `debug`, `trace`. ### Runtime log level — the `site-recovery-logging` ConfigMap Each component runs a `LevelWatcher` that polls the `site-recovery-logging` ConfigMap in its own namespace every **15 seconds** using an uncached API reader. A level change takes effect within one poll cycle; no pod restart is required. The ConfigMap supports a global key and per-component override keys: ```yaml apiVersion: v1 kind: ConfigMap metadata: name: site-recovery-logging namespace: dr- # TSR-Manager components data: level: info # global fallback level.control-plane: debug # manager-control-plane override level.protectionzone: info # protectionzone-controller override ``` For workload-cluster components, create or patch the ConfigMap in `trilio-site-recovery-system`: ```yaml apiVersion: v1 kind: ConfigMap metadata: name: site-recovery-logging namespace: trilio-site-recovery-system data: level: info level.drbd-node-agent: debug level.control-plane: debug ``` Valid component override keys: | Key | Component | |---|---| | `level.control-plane` | TSR-Manager control plane or workload control plane | | `level.drbd-node-agent` | drbd-node-agent DaemonSet pods | | `level.protectionzone` | ProtectionZone controller | If the ConfigMap is absent, or the relevant key is absent or empty, the component keeps its startup-flag level. An unrecognized level value (for example, `warn`) is logged as an error once per distinct bad value, and the component keeps its current level. **Trace level note:** The `trace` level cannot be set via the ProtectionZone `spec.logging` fields or the TSR web console UI endpoint — those paths accept only `error`, `info`, and `debug`. To enable `trace`, write the value directly to the ConfigMap `level` or `level.` key, or pass `--zap-log-level=2` at pod startup (which requires a DaemonSet or Deployment rollout). ### UI-based log level changes (OpenShift only) The TSR web console exposes a logging endpoint for each zone. The Next.js frontend proxies the request to the Flask backend: ``` PUT /api/protection-zones//logging ``` This endpoint updates the `site-recovery-logging` ConfigMap for the specified zone. Browser access is OpenShift-only via the oauth-proxy Route at `https://trilio-site-recovery.`. No equivalent endpoint exists for upstream Kubernetes deployments. ### Log level chart values The Helm charts embedded in the ProtectionZone controller binary expose these logging values (set by the controller from the ProtectionZone spec; you do not set chart values directly): | Key | Default | Description | |---|---|---| | `logging.level` | `info` | Startup log level for the control-plane container | | `logging.encoder` | `json` | Log encoder; `json` for production, `console` for development | | `agent.logging.level` | `info` | Startup log level for drbd-node-agent pods | | `agent.logging.encoder` | `json` | Log encoder for drbd-node-agent | ## Usage ### Increasing verbosity to diagnose a stuck failover When a failover or protection operation stalls and `info`-level output does not reveal the cause, raise the TSR-Manager control plane to `debug`. The change takes effect within 15 seconds: ```bash kubectl patch configmap site-recovery-logging \ -n dr- \ --type merge \ -p '{"data":{"level.control-plane":"debug"}}' ``` To also raise the workload control plane on the primary cluster: ```bash kubectl patch configmap site-recovery-logging \ -n trilio-site-recovery-system \ --context \ --type merge \ -p '{"data":{"level.control-plane":"debug"}}' ``` Restore `info` once you have captured the relevant logs: ```bash kubectl patch configmap site-recovery-logging \ -n dr- \ --type merge \ -p '{"data":{"level.control-plane":"info"}}' ``` ### Raising node-agent verbosity on a specific workload cluster If a drbd-node-agent pod is not converging, raise its level on the affected workload cluster: ```bash kubectl patch configmap site-recovery-logging \ -n trilio-site-recovery-system \ --context \ --type merge \ -p '{"data":{"level.drbd-node-agent":"debug"}}' ``` ### Reading logs from the TSR-Manager control plane The TSR-Manager control plane (failover, protection, replication-monitor, and test-failover reconcilers) runs in `dr-`: ```bash kubectl logs \ -n dr- \ -l app.kubernetes.io/component=manager-control-plane \ --follow ``` ### Reading logs from the ProtectionZone controller ```bash kubectl logs \ -n dr- \ -l app.kubernetes.io/component=protectionzone-controller \ --follow ``` ### Reading logs from the workload control plane ```bash kubectl logs \ -n trilio-site-recovery-system \ --context \ -l app.kubernetes.io/component=workload-control-plane \ --follow ``` ### Reading logs from a specific drbd-node-agent pod The drbd-node-agent runs as a DaemonSet (one pod per worker node). To read logs for the agent on a specific node, first find the pod name for that node, then stream its logs: ```bash # Step 1: get the pod name for a given node AGENT_POD=$(kubectl get pods \ -n trilio-site-recovery-system \ --context \ -l app.kubernetes.io/component=drbd-node-agent \ --field-selector spec.nodeName= \ -o jsonpath='{.items[0].metadata.name}') # Step 2: stream the logs kubectl logs -n trilio-site-recovery-system \ --context \ "$AGENT_POD" --follow ``` ### Reading logs from the UI backend ```bash kubectl logs \ -n dr- \ -l app.kubernetes.io/part-of=site-recovery \ -c site-recovery-backend \ --follow ``` ### Log aggregation recommendations Because all components write JSON to stdout, any OpenShift-native or Kubernetes-compatible log collector (for example, the OpenShift Logging stack using Vector or Fluentd) can ingest Site Recovery logs without custom parsers. Recommended practices: - **Separate stdout from stderr.** Normal operational output is always on stdout. Stderr output indicates abnormal conditions (panics, pre-logger errors) and should route to a high-priority alert stream. - **Index the `level` field.** Filtering on `level=error` in your aggregation tool gives you an immediate operational dashboard without noise. - **Index the `logger` field.** This identifies which component (for example, `manager-control-plane`, `drbd-node-agent`) emitted the line, enabling per-component dashboards. - **Preserve RFC3339Nano timestamps.** The `ts` field has sub-second precision. Aggregation pipelines that truncate to seconds lose intra-second ordering, making it impossible to reconstruct the sequence of events during a failover. - **Include the `caller` field in your index.** When a log line from Trilio support is referenced in a diagnostic exchange, the `caller` field (`file:line`) identifies the exact emitting location. - **Collect from both clusters.** TSR-Manager components in `dr-` and workload components in `trilio-site-recovery-system` on each workload cluster form a complete picture. A failover operation spans all three clusters; logs from only one cluster will be incomplete. ## Examples ### Example 1: Set global zone log level to debug via ProtectionZone spec Edit the zone's logging spec. The ProtectionZone controller propagates the change to the `site-recovery-logging` ConfigMap in `dr-my-zone` within one reconcile: ```yaml apiVersion: siterecovery.trilio.io/v1alpha1 kind: ProtectionZone metadata: name: my-zone namespace: dr-my-zone spec: displayName: "My Zone" storageBackendMode: DRBD logging: level: debug clusters: - name: primary clusterID: cluster1 kubeconfigSecretRef: name: cluster1-kubeconfig key: kubeconfig - name: dr clusterID: cluster2 kubeconfigSecretRef: name: cluster2-kubeconfig key: kubeconfig ``` ### Example 2: Override only the node agent to debug, keep control plane at info ```yaml spec: logging: level: info nodeAgentLevel: debug ``` ### Example 3: Patch the ConfigMap directly for an immediate runtime change on TSR-Manager This takes effect on the next 15-second poll — no pod restart needed: ```bash kubectl patch configmap site-recovery-logging \ -n dr-my-zone \ --type merge \ -p '{"data":{"level.control-plane":"debug","level":"info"}}' ``` ### Example 4: Verify the effective ConfigMap contents ```bash kubectl get configmap site-recovery-logging \ -n dr-my-zone \ -o yaml ``` ### Example 5: Read the ConfigMap from a workload cluster ```bash kubectl get configmap site-recovery-logging \ -n trilio-site-recovery-system \ --context \ -o yaml ``` ### Example 6: Filter aggregated logs for error-level entries from the control plane If you are using `kubectl` directly rather than an aggregation stack, combine a label selector with `grep`: ```bash kubectl logs \ -n dr-my-zone \ -l app.kubernetes.io/component=manager-control-plane \ --since=1h \ | grep '"level":"error"' ``` ### Example 7: Check which level the node agent is currently running at The component logs a line when the level transitions. Look for the `runtime log level applied` message: ```bash kubectl logs \ -n trilio-site-recovery-system \ --context \ -l app.kubernetes.io/component=drbd-node-agent \ --since=30m \ | grep 'runtime log level applied' ``` ## Troubleshooting ### ConfigMap change has no effect after 15 seconds **Symptom:** You patched `site-recovery-logging` but the component is still emitting at the old level. **Likely causes:** 1. You patched the ConfigMap in the wrong namespace. TSR-Manager components read from `dr-`; workload components read from `trilio-site-recovery-system`. Run `kubectl get configmap site-recovery-logging -A` to confirm which namespaces have the object. 2. The level value is unrecognized. Accepted values are exactly `error`, `info`, `debug`, `trace`. A value like `warn` or `WARNING` is silently ignored (an error is logged once). Check the component logs for `runtime log-level ConfigMap has an invalid level`. 3. The component cannot reach the API server transiently. In this case it keeps the current level and logs `runtime log-level ConfigMap read failed; keeping current level` at debug verbosity. Check the pod's connectivity to the API server. **Fix:** Confirm the namespace, correct the value to one of `error | info | debug | trace`, and wait one poll cycle (up to 15 seconds). --- ### Logs are not JSON — they appear as plain text or console output **Symptom:** Log lines from a component arrive as human-readable text rather than JSON objects. **Likely cause:** The component was started with a `--zap-encoder=console` flag, or the Flask backend is running with `FLASK_ENV=development`. **Fix:** For Go components, confirm the chart value `logging.encoder` is `json` (the default). For the Flask backend, ensure `FLASK_ENV` is not set to `development` in the deployment environment. The ProtectionZone controller sets this correctly when it installs the stacks from its embedded charts; a manual override of the environment variable is the most common cause. --- ### Cannot set trace level via the ProtectionZone spec or the web console **Symptom:** Setting `spec.logging.level: trace` in the ProtectionZone CR, or using the `PUT /api/protection-zones//logging` endpoint, has no effect; the component stays at `debug` or `info`. **Likely cause:** The `spec.logging` fields and the UI logging endpoint accept only `error`, `info`, and `debug`. The `trace` level corresponds to logr V(2) / zap level -2, which has no named representation in the standard zap level set and is therefore not reachable through these paths. **Fix:** Write the value `trace` directly to the `site-recovery-logging` ConfigMap using `kubectl patch` (see Examples section). The `LevelWatcher` accepts `trace` from the ConfigMap because it uses the product's own level map, which explicitly includes `trace`. --- ### Log volume is extremely high and impacting collector performance **Symptom:** After enabling `debug` or `trace` for a busy zone, log ingestion throughput spikes significantly. **Likely cause:** `trace` is intended for high-frequency per-poll and per-scrape detail and produces a large volume of lines per second per pod. `debug` is suitable for diagnosing a stuck operation; `trace` is only warranted when `debug` is insufficient. **Fix:** Lower the level back to `debug` or `info` by patching the `site-recovery-logging` ConfigMap. Use per-component override keys (for example, `level.drbd-node-agent`) to limit verbosity to only the component under investigation rather than raising the global `level` key. --- ### `kubectl logs` returns no output for the drbd-node-agent **Symptom:** Running `kubectl logs -n trilio-site-recovery-system -l app.kubernetes.io/component=drbd-node-agent` returns nothing or an error. **Likely cause:** The `-l` selector matches multiple pods (one per worker node), and `kubectl logs` requires a single pod when not using `--prefix`. Alternatively, the DaemonSet pods may not be running yet (the node-agent applies a `siterecovery.trilio.io/not-ready:NoSchedule` taint to its own node during startup). **Fix:** Use the two-step approach shown in the Usage section: first get the pod name for the specific node with `--field-selector spec.nodeName=`, then call `kubectl logs` on that pod name. To stream from all node agent pods simultaneously, add `--prefix` to the label-selector invocation: ```bash kubectl logs \ -n trilio-site-recovery-system \ --context \ -l app.kubernetes.io/component=drbd-node-agent \ --prefix --follow ``` --- ### `site-recovery-logging` ConfigMap does not exist in a namespace **Symptom:** `kubectl get configmap site-recovery-logging -n dr-` returns `NotFound`. **Likely cause:** The ProtectionZone has not yet reached `Available` phase, or the zone was created before the controller version that auto-creates the ConfigMap. **Fix:** Check the ProtectionZone status: `kubectl get pz -n dr-`. If the zone is `Available` and the ConfigMap is still absent, the component treats this as "no override" and uses the startup-flag level — this is not an error condition. You can create the ConfigMap manually with the desired level, and the LevelWatcher will pick it up on the next poll. Ensure the ConfigMap name is exactly `site-recovery-logging` (the LevelWatcher looks up this exact name).