--- title: Controller Architecture product: trilio-site-recovery-for-kubernetes doc_type: guide version: 0.2.0 source: git2docs (code-derived, validation-filtered) canonical: https://docs-tsr.trilio.io/trilio-site-recovery-for-kubernetes/site-recovery/controller-architecture --- # Controller Architecture _How controllers watch and reconcile site recovery resources across clusters_ ## Overview This page describes how the Trilio Site Recovery controllers are structured, how they watch and reconcile resources across multiple clusters, and how the admission webhook layer enforces invariants at write time. Understanding the controller architecture helps you interpret status conditions, diagnose stalled operations, and reason about what each component is responsible for—whether that component runs on the TSR-Manager cluster or on a workload cluster. ## Prerequisites Before reading this page you should be familiar with: - Kubernetes controller-runtime concepts (reconcilers, informer caches, field indexes, admission webhooks) - The three-cluster topology: one TSR-Manager cluster plus two workload clusters (primary and DR) - The Site Recovery custom resource hierarchy: `ProtectionZone` → `ProtectionGroup` → `ProtectionRequest` → `DRBDResource` - `kubectl` access to the TSR-Manager cluster (namespace `dr-`) and to both workload clusters (namespace `trilio-site-recovery-system`) Version requirements: - OpenShift 4.17 or later on all three clusters - KubeVirt installed on both workload clusters - DRBD kernel module (`drbd9`) loaded on workload cluster nodes ## Configuration ### ProtectionZone spec The `ProtectionZone` CR drives the entire bootstrap. Key fields: | Field | Default | Effect | |---|---|---| | `spec.storageBackendMode` | — (required) | Must be `DRBD`. The admission webhook rejects `Linstor`. | | `spec.clusters[].clusterID` | — (required) | `cluster1` or `cluster2`. Each zone must have exactly one entry per slot. | | `spec.clusters[].distributionName` | `RHEL-CoreOS-9` | Selects the DRBD kernel module loader image. `RHEL-CoreOS-10` is accepted by the schema but not validated in Tech Preview. | | `spec.clusters[].kubeconfigSecretRef` | — (required) | Name and key of the Secret holding the workload cluster kubeconfig. | | `spec.nodeAgent.startupDeadlineSeconds` | `120` | How long the node agent holds the `siterecovery.trilio.io/not-ready:NoSchedule` taint on startup. | | `spec.logging.level` | `info` | Runtime log level for all components in this zone. Accepted values: `error`, `info`, `debug`, `trace`. | | `spec.hooks.jobTTLSeconds` | `0` (engine default) | How long a succeeded hook Job lingers before TTL cleanup. | | `spec.hooks.failedJobRetentionSeconds` | `0` (engine default) | How long a failed hook Job is retained for log inspection. | Mutable after creation: `spec.displayName`, `spec.description`, each cluster's `spec.clusters[].kubeconfigSecretRef`, and each cluster's `spec.clusters[].distributionName`. All other spec fields are immutable for the zone's lifetime. ### Runtime log levels You can change log levels at runtime without restarting pods by editing the `site-recovery-logging` ConfigMap in the zone namespace: ```bash kubectl edit configmap site-recovery-logging -n dr-production ``` Supported keys: | Key | Scope | |---|---| | `level` | All components | | `level.control-plane` | TSR-Manager control-plane pods | | `level.drbd-node-agent` | Node agent DaemonSet pods | | `level.protectionzone` | ProtectionZone controller pods | Accepted values: `error`, `info`, `debug`, `trace`. Changes take effect within one poll interval (default 15 seconds) without a pod restart. ### Zone-config tunables The `site-recovery-zone-config` ConfigMap in the zone namespace exposes per-zone timing tunables that controllers re-read on every reconcile: | Key | Default | Minimum | Effect | |---|---|---|---| | `drbdMetadataReadyGracePeriod` | `2m` | `30s` | Floor for the metadata provisioning failure deadline. | | `drbdMetadataProvisioningFailureDeadline` | `10m` | `5m` | How long a DRBDResource may report `ConditionPlacementReady=False` before the protection fails. Always ≥ the grace period. | ### DRBDReplicationPolicy The ProtectionZone controller automatically creates one `DRBDReplicationPolicy` named `-default` with `isDefault: true` and Protocol C in the zone namespace. You do not author this object. The only supported replication protocol is Protocol C (synchronous, RPO=0). ### ProtectionRequest replication config When creating a `ProtectionRequest`: | Field | Default | Notes | |---|---|---| | `spec.replicationConfig.replicationProtocol` | `C` | Only `C` is accepted. Protocol `A` is rejected at admission. | | `spec.replicationConfig.rpo` | `0` | Must be `0` with Protocol C. | | `spec.replicationConfig.drbdMetadataConfig.size` | `128Mi` | Minimum 128Mi; enforced by CRD CEL rule. | ## Usage ### Controller placement Site Recovery uses three separate controller binaries, each with a distinct cluster scope: **TSR-Manager cluster** hosts: - **ProtectionZone controller** (`app.kubernetes.io/component: protectionzone-controller`) — watches ProtectionZones cluster-wide, installs the per-zone Helm charts, provisions webhook certificates, and manages the zone lifecycle. - **TSR-Manager control plane** (`app.kubernetes.io/component: manager-control-plane`) — runs in the zone namespace `dr-`; hosts the `protectionrequest`, `failoverrequest`, `testfailover`, `pgsync`, `replicationmonitor`, and `drbdreplicationpolicy` reconcilers. Its informer cache is scoped to the zone namespace. It also reaches both workload clusters through mounted kubeconfig Secrets. - **Admission webhooks** for all TSR-Manager-resident CRs (ProtectionZone, ProtectionGroup, ProtectionRequest, FailoverRequest, TestFailover, Hook, ReplicationGroupStatus, RPOEvent). **Each workload cluster** hosts: - **Workload control plane** (`app.kubernetes.io/component: workload-control-plane`) — runs in `trilio-site-recovery-system`; hosts the DRBDResource placement reconciler and the KubeVirt VM admission guards (live migration deny, disk topology freeze). Its informer cache is **unscoped** (cluster-wide) so it can observe DRBDResources in any VM namespace. - **drbd-node-agent DaemonSet** — one pod per worker node. Manages `drbdadm` configuration and `.res` files, provisions metadata PVCs, holds the startup taint, and is the direct operator of DRBD kernel state. ### How a ProtectionRequest reconciles When you create a `ProtectionRequest`, the TSR-Manager control plane: 1. Validates the zone (`ClustersValidated=True` gate). 2. Creates a `DRBDResource` CR in the VM's namespace on each workload cluster. 3. The workload control plane's placement reconciler binds each `DRBDResource` to a specific node and provisions frontend PVCs. 4. The drbd-node-agent on each bound node configures the DRBD device and starts replication. 5. Status transitions: `Pending` → `Validating` → `CreatingDRBD` → `Syncing` → `Protected`. ```bash # Watch protection progress kubectl get protect -n dr-production -w # Inspect DRBD resource on a workload cluster kubectl get dres -n --context ``` ### How a FailoverRequest reconciles Creating a `FailoverRequest` triggers the failover controller on the TSR-Manager cluster: ```yaml apiVersion: siterecovery.trilio.io/v1alpha1 kind: FailoverRequest metadata: name: failover-to-dr namespace: dr-production spec: protectionGroupRef: name: my-protection-group namespace: dr-production targetCluster: cluster2 failoverType: planned ``` The controller patches each member `ProtectionRequest`'s `spec.sourceCluster` field to `cluster2`, which flips the DRBD Primary role on the target side. Monitor progress: ```bash kubectl get fr -n dr-production -w kubectl get fr failover-to-dr -n dr-production -o jsonpath='{.status.vmStatuses}' ``` ### Zone credential rotation To rotate a zone credential (for example, after a token is about to expire), use `tsrctl` (contact Trilio support to obtain the tool): ```bash # Rotate and refresh RBAC bindings tsrctl zone-credential rotate --refresh-rbac # Rotate and revoke the previous credential immediately tsrctl zone-credential rotate --refresh-rbac --revoke-previous ``` If the `InsufficientClusterPermissions` condition appears on the ProtectionZone, run `tsrctl zone-credential rotate --refresh-rbac` to re-apply the required RBAC. ### Inspecting and deleting zone credentials ```bash # Inspect the current credential for a zone tsrctl zone-credential inspect # Delete a credential (use --force to bypass safety checks) tsrctl zone-credential delete tsrctl zone-credential delete --force ``` ### Checking replication health ```bash # Aggregated replication status for a ProtectionGroup kubectl get rgs -n dr-production # Detailed status kubectl get rgs my-protection-group-replication -n dr-production -o yaml # RPO audit events kubectl get rpoe -n dr-production ``` ### Running a test failover ```yaml apiVersion: siterecovery.trilio.io/v1alpha1 kind: TestFailover metadata: name: dr-drill-q1 namespace: dr-production spec: protectionGroupRef: name: my-protection-group namespace: dr-production cleanupPolicy: Automatic schedulingCheck: Fail ``` ```bash kubectl apply -f testfailover.yaml kubectl get tfo dr-drill-q1 -n dr-production -w # The test namespace is reported in status, never user-provided kubectl get tfo dr-drill-q1 -n dr-production \ -o jsonpath='{.status.testNamespace}' ``` ## Examples ### Example 1 — Minimal ProtectionZone ```yaml apiVersion: siterecovery.trilio.io/v1alpha1 kind: ProtectionZone metadata: name: production namespace: dr-production spec: displayName: Production DR Zone storageBackendMode: DRBD clusters: - name: cluster1 clusterID: cluster1 kubeconfigSecretRef: name: cluster1-kubeconfig key: kubeconfig - name: cluster2 clusterID: cluster2 kubeconfigSecretRef: name: cluster2-kubeconfig key: kubeconfig ``` ### Example 2 — ProtectionRequest for a single VM ```yaml apiVersion: siterecovery.trilio.io/v1alpha1 kind: ProtectionRequest metadata: name: protect-web-vm namespace: dr-production spec: sourceCluster: cluster1 virtualMachine: name: web-vm namespace: production-app replicationConfig: replicationProtocol: C rpo: 0 ``` ### Example 3 — ProtectionGroup with boot ordering and hooks ```yaml apiVersion: siterecovery.trilio.io/v1alpha1 kind: ProtectionGroup metadata: name: app-tier namespace: dr-production spec: displayName: Application Tier sourceCluster: cluster1 virtualMachines: - name: db-vm namespace: production-app bootOrder: 1 - name: web-vm namespace: production-app bootOrder: 2 sla: rpoTargetSeconds: 0 rtoTargetSeconds: 300 hooks: - name: pre-failover-quiesce hookName: db-quiesce type: pre-failover runOn: source onFailure: Abort ``` ### Example 4 — Planned failover ```yaml apiVersion: siterecovery.trilio.io/v1alpha1 kind: FailoverRequest metadata: name: planned-failover-to-dr namespace: dr-production spec: protectionGroupRef: name: app-tier namespace: dr-production targetCluster: cluster2 failoverType: planned ``` ### Example 5 — Unplanned failover ```yaml apiVersion: siterecovery.trilio.io/v1alpha1 kind: FailoverRequest metadata: name: emergency-failover namespace: dr-production spec: protectionGroupRef: name: app-tier namespace: dr-production targetCluster: cluster2 failoverType: unplanned skipHooks: false ``` ### Example 6 — Test failover with reachability verification ```yaml apiVersion: siterecovery.trilio.io/v1alpha1 kind: TestFailover metadata: name: quarterly-drill namespace: dr-production spec: protectionGroupRef: name: app-tier namespace: dr-production cleanupPolicy: Automatic retentionTime: 4h schedulingCheck: Fail verification: reachability: - type: tcp port: 22 timeout: 30s - type: http port: 8080 endpoint: /health expectedStatus: 200 timeout: 30s ``` ### Example 7 — Hook definition ```yaml apiVersion: siterecovery.trilio.io/v1alpha1 kind: Hook metadata: name: db-quiesce namespace: dr-production spec: description: Quiesce the database before failover timeoutSeconds: 120 retries: 1 retryDelaySeconds: 10 jobTemplate: image: quay.io/triliovault/hooks/db-quiesce:1.0.0 command: ["/bin/sh", "-c"] args: ["mysql -u root -p$DB_PASS -e 'FLUSH TABLES WITH READ LOCK;'"] env: - name: DB_PASS valueFrom: secretKeyRef: name: db-credentials key: password ``` ### Example 8 — Listing controller pods by component label ```bash # ProtectionZone controller on TSR-Manager kubectl get pods -n trilio-site-recovery-system \ -l app.kubernetes.io/component=protectionzone-controller # TSR-Manager control plane in the zone namespace kubectl get pods -n dr-production \ -l app.kubernetes.io/component=manager-control-plane # Workload control plane on a workload cluster kubectl get pods -n trilio-site-recovery-system \ -l app.kubernetes.io/component=workload-control-plane \ --context # DRBD node agent DaemonSet pods on a workload cluster kubectl get pods -n trilio-site-recovery-system \ -l app.kubernetes.io/component=drbd-node-agent \ --context ``` ### Example 9 — Reading controller logs ```bash # TSR-Manager control plane logs kubectl logs -n dr-production \ -l app.kubernetes.io/component=manager-control-plane \ --since=1h # Node agent logs for a specific node NODE=worker-1 POD=$(kubectl get pods -n trilio-site-recovery-system \ --field-selector spec.nodeName=${NODE} \ -l app.kubernetes.io/component=drbd-node-agent \ -o jsonpath='{.items[0].metadata.name}' \ --context ) kubectl logs -n trilio-site-recovery-system $POD \ --context ``` ### Example 10 — Checking which VMs are protected ```bash # On a workload cluster — the label is stamped on the KubeVirt VirtualMachine kubectl get vm -A \ -l siterecovery.trilio.io/protected-vm=true \ --context ``` ## Troubleshooting ### ProtectionZone stuck in Provisioning or Unavailable **Symptom:** `kubectl get pz -n dr-` shows `Provisioning` or `Unavailable` long after creation. **Likely causes and fixes:** 1. **ClustersValidated=False / InsufficientClusterPermissions** — The zone credential authenticates but lacks required RBAC verbs. Fix: run `tsrctl zone-credential rotate --refresh-rbac` for the affected workload cluster. 2. **ClustersValidated=False / ClusterIdentityChanged** — The credential now reaches a different cluster (the `kube-system` Namespace UID has changed). There is no override. You must restore the original cluster or re-create the zone. 3. **ClustersValidated=False / ClusterAlreadyClaimed** — The workload cluster is already leased to another zone (the `siterecovery.trilio.io/owner-zone-namespace-uid` annotation on the workload cluster's `trilio-site-recovery-system` Namespace points elsewhere). Each workload cluster can belong to exactly one zone. 4. **SCCPrerequisiteMissing or KubeVirtNotInstalled** — The workload cluster is missing a prerequisite. Check the condition message for details. ```bash kubectl get pz -n dr-production -o jsonpath='{.items[0].status.conditions}' | jq . ``` --- ### ProtectionRequest stuck in Syncing or CreatingDRBD **Symptom:** A `ProtectionRequest` remains in `Syncing` or `CreatingDRBD` phase for an extended period. **Likely causes and fixes:** 1. **DRBD port blocked** — Replication uses TCP ports 7000–8000 inclusive (one port per VM volume). A firewall rule that stops at 7999 silently prevents replication on one volume. Open TCP 7000–8000 on all worker-node pairs between the primary and DR clusters. 2. **ConditionPlacementReady=False on DRBDResource** — The node agent is waiting for metadata PVCs to bind. Check the CSI provisioner in the VM's namespace on the workload cluster. The provisioning failure deadline is configurable via `site-recovery-zone-config` (default 10 minutes). 3. **Node agent startup taint** — The drbd-node-agent holds `siterecovery.trilio.io/not-ready:NoSchedule` on its node during startup. Pods scheduled to that node will not run until DRBD converges. Check agent logs. ```bash # Check DRBDResource conditions on the workload cluster kubectl get dres -n --context -o yaml # Check metadata PVCs kubectl get pvc -n --context ``` --- ### ProtectionRequest in Failed phase **Symptom:** Phase is `Failed`; you want to re-protect the VM. **Cause:** Protection cannot be retried in place. A full unprotect followed by a new `ProtectionRequest` (with a full initial resync) is required. **Fix:** 1. Delete the failed `ProtectionRequest` and wait for the finalizer to complete teardown. 2. Confirm the `DRBDResource` CRs on both workload clusters are gone. 3. Create a new `ProtectionRequest`. --- ### FailoverRequest stuck in WaitingForLocks or Monitoring **Symptom:** A `FailoverRequest` does not advance past `WaitingForLocks`. **Likely cause:** Another `FailoverRequest` or `TestFailover` is already in flight for the same `ProtectionGroup`. The admission webhook prevents creating a second one, but a previously admitted request may still be running. **Fix:** Inspect active operations: ```bash kubectl get fr -n dr-production kubectl get tfo -n dr-production ``` Wait for the in-flight operation to reach a terminal state (`Completed`, `Failed`, or `Cleaned`) before creating a new one. --- ### TestFailover test namespace not visible **Symptom:** You want to inspect VMs in the test namespace but don't know its name. **Cause:** The test namespace is generated by the controller and reported in `status.testNamespace`. It is never user-provided. **Fix:** ```bash kubectl get tfo -n dr-production \ -o jsonpath='{.status.testNamespace}' ``` Then use that namespace to inspect test VMs on the target workload cluster. --- ### VM live migration fails on a protected VM **Symptom:** `VirtualMachineInstanceMigration` is rejected for a protected VM. **Cause:** Live migration of protected VMs is denied by the workload control plane admission webhook. A protected VM is pinned to one node per cluster for the lifetime of the protection; `nodeName` is immutable once set. `oc adm drain` on a node hosting a running protected VM will also not complete. **Fix:** Unprotect the VM first if you need to migrate it. Note that re-protecting requires a full initial resync. --- ### Zone credential about to expire **Symptom:** The `CredentialExpiring` reason appears on the `CredentialsValid` condition of the ProtectionZone. The zone phase is not affected. **Fix:** Rotate the credential before it expires: ```bash tsrctl zone-credential rotate --refresh-rbac ``` --- ### Collecting a support bundle If you need to share diagnostics with Trilio support, collect a must-gather bundle from the TSR-Manager cluster: ```bash oc adm must-gather --image=quay.io/triliovault/site-recovery-must-gather: ``` This collects logs, CRs, and DRBD state from the TSR-Manager cluster and all connected workload clusters into a single archive. --- ### Status patch rejected events **Symptom:** `kubectl describe` on a DRBDResource or ProtectionRequest shows a `Warning StatusPatchRejected` event. **Cause:** The API server rejected a status write from a controller, typically because the stored CRD schema does not include a field the controller tried to set (a CRD drift). The CR's displayed status may be stale. **Fix:** Verify the installed CRDs match the operator version. Check the event message for the specific rejected field. Collect a support bundle and contact Trilio support.