Controller Architecture

This page describes how the Trilio Site Recovery controllers are structured, how they watch and reconcile resources across multiple clusters, and how the admission webhook layer enforces invariants at write time. Understanding the controller architecture helps you interpret status conditions, diagnose stalled operations, and reason about what each component is responsible for—whether that component runs on the TSR-Manager cluster or on a workload cluster.

Before reading this page you should be familiar with:

• Kubernetes controller-runtime concepts (reconcilers, informer caches, field indexes, admission webhooks)
• The three-cluster topology: one TSR-Manager cluster plus two workload clusters (primary and DR)
• The Site Recovery custom resource hierarchy: ProtectionZone → ProtectionGroup → ProtectionRequest → DRBDResource
• kubectl access to the TSR-Manager cluster (namespace dr-<zone-name>) and to both workload clusters (namespace trilio-site-recovery-system)

Version requirements:

• OpenShift 4.17 or later on all three clusters
• KubeVirt installed on both workload clusters
• DRBD kernel module (drbd9) loaded on workload cluster nodes

Site Recovery is installed through OLM (Operator Lifecycle Manager). The operator deploys all control-plane components automatically once you create a ProtectionZone CR; you do not run helm install directly.

Step 1 — Create the operator namespace

kubectl create namespace trilio-site-recovery-system

Step 2 — Create the CatalogSource

apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: site-recovery-catalog
  namespace: trilio-site-recovery-system
spec:
  sourceType: grpc
  image: quay.io/triliodata/site-recovery-catalog:<release-tag>
  displayName: Trilio Site Recovery
  publisher: Trilio

kubectl apply -f catalogsource.yaml

Step 3 — Create the OperatorGroup

apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: site-recovery-og
  namespace: trilio-site-recovery-system
spec: {}

The empty spec: {} is required: the operator supports only the AllNamespaces install mode. Setting spec.targetNamespaces leaves the CSV permanently in Failed with reason UnsupportedOperatorGroup.

kubectl apply -f operatorgroup.yaml

Step 4 — Create the Subscription

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: site-recovery
  namespace: trilio-site-recovery-system
spec:
  source: site-recovery-catalog
  sourceNamespace: trilio-site-recovery-system
  name: site-recovery
  channel: stable
  installPlanApproval: Automatic

kubectl apply -f subscription.yaml

OLM installs the ProtectionZone controller, the TSR web console frontend and backend, and all CRDs owned by the bundle. Wait for the CSV to reach Succeeded:

kubectl get csv -n trilio-site-recovery-system

Step 5 — Provision zone credentials

For each workload cluster, use tsrctl (supplied by Trilio with the release; contact Trilio support to obtain it) to create a zone credential. tsrctl zone-credential create provisions a ServiceAccount in the trilio-site-recovery-access namespace on the workload cluster and stores its kubeconfig Secret on the TSR-Manager cluster.

# Point tsrctl at the workload cluster you want to credential
tsrctl zone-credential create

Repeat for the second workload cluster.

Step 6 — Create the zone namespace and ProtectionZone CR

Create a namespace for the zone on the TSR-Manager cluster:

kubectl create namespace dr-production

Create the kubeconfig Secrets (referencing the outputs from Step 5) and then the ProtectionZone:

apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionZone
metadata:
  name: production
  namespace: dr-production
spec:
  displayName: Production Zone
  storageBackendMode: DRBD
  clusters:
    - name: cluster1
      clusterID: cluster1
      distributionName: RHEL-CoreOS-9
      kubeconfigSecretRef:
        name: cluster1-kubeconfig
        key: kubeconfig
    - name: cluster2
      clusterID: cluster2
      distributionName: RHEL-CoreOS-9
      kubeconfigSecretRef:
        name: cluster2-kubeconfig
        key: kubeconfig

kubectl apply -f protectionzone.yaml

The ProtectionZone controller validates both clusters, installs the TSR-Manager control-plane chart into dr-production, and installs the workload control-plane chart into trilio-site-recovery-system on each workload cluster. Watch the zone reach Available:

kubectl get pz -n dr-production -w

The ClustersValidated condition on the ProtectionZone must be True before you can create ProtectionGroups or ProtectionRequests.

ProtectionZone spec

The ProtectionZone CR drives the entire bootstrap. Key fields:

Mutable after creation: spec.displayName, spec.description, each cluster's spec.clusters[].kubeconfigSecretRef, and each cluster's spec.clusters[].distributionName. All other spec fields are immutable for the zone's lifetime.

Runtime log levels

You can change log levels at runtime without restarting pods by editing the site-recovery-logging ConfigMap in the zone namespace:

kubectl edit configmap site-recovery-logging -n dr-production

Supported keys:

Accepted values: error, info, debug, trace. Changes take effect within one poll interval (default 15 seconds) without a pod restart.

Zone-config tunables

The site-recovery-zone-config ConfigMap in the zone namespace exposes per-zone timing tunables that controllers re-read on every reconcile:

DRBDReplicationPolicy

The ProtectionZone controller automatically creates one DRBDReplicationPolicy named <zoneName>-default with isDefault: true and Protocol C in the zone namespace. You do not author this object. The only supported replication protocol is Protocol C (synchronous, RPO=0).

ProtectionRequest replication config

When creating a ProtectionRequest:

Controller placement

Site Recovery uses three separate controller binaries, each with a distinct cluster scope:

TSR-Manager cluster hosts:

• ProtectionZone controller (app.kubernetes.io/component: protectionzone-controller) — watches ProtectionZones cluster-wide, installs the per-zone Helm charts, provisions webhook certificates, and manages the zone lifecycle.
• TSR-Manager control plane (app.kubernetes.io/component: manager-control-plane) — runs in the zone namespace dr-<name>; hosts the protectionrequest, failoverrequest, testfailover, pgsync, replicationmonitor, and drbdreplicationpolicy reconcilers. Its informer cache is scoped to the zone namespace. It also reaches both workload clusters through mounted kubeconfig Secrets.
• Admission webhooks for all TSR-Manager-resident CRs (ProtectionZone, ProtectionGroup, ProtectionRequest, FailoverRequest, TestFailover, Hook, ReplicationGroupStatus, RPOEvent).

Each workload cluster hosts:

• Workload control plane (app.kubernetes.io/component: workload-control-plane) — runs in trilio-site-recovery-system; hosts the DRBDResource placement reconciler and the KubeVirt VM admission guards (live migration deny, disk topology freeze). Its informer cache is unscoped (cluster-wide) so it can observe DRBDResources in any VM namespace.
• drbd-node-agent DaemonSet — one pod per worker node. Manages drbdadm configuration and .res files, provisions metadata PVCs, holds the startup taint, and is the direct operator of DRBD kernel state.

How a ProtectionRequest reconciles

When you create a ProtectionRequest, the TSR-Manager control plane:

1. Validates the zone (ClustersValidated=True gate).
2. Creates a DRBDResource CR in the VM's namespace on each workload cluster.
3. The workload control plane's placement reconciler binds each DRBDResource to a specific node and provisions frontend PVCs.
4. The drbd-node-agent on each bound node configures the DRBD device and starts replication.
5. Status transitions: Pending → Validating → CreatingDRBD → Syncing → Protected.

# Watch protection progress
kubectl get protect -n dr-production -w

# Inspect DRBD resource on a workload cluster
kubectl get dres -n <vm-namespace> --context <workload-cluster-context>

How a FailoverRequest reconciles

Creating a FailoverRequest triggers the failover controller on the TSR-Manager cluster:

apiVersion: siterecovery.trilio.io/v1alpha1
kind: FailoverRequest
metadata:
  name: failover-to-dr
  namespace: dr-production
spec:
  protectionGroupRef:
    name: my-protection-group
    namespace: dr-production
  targetCluster: cluster2
  failoverType: planned

The controller patches each member ProtectionRequest's spec.sourceCluster field to cluster2, which flips the DRBD Primary role on the target side. Monitor progress:

kubectl get fr -n dr-production -w
kubectl get fr failover-to-dr -n dr-production -o jsonpath='{.status.vmStatuses}'

Zone credential rotation

To rotate a zone credential (for example, after a token is about to expire), use tsrctl (contact Trilio support to obtain the tool):

# Rotate and refresh RBAC bindings
tsrctl zone-credential rotate --refresh-rbac

# Rotate and revoke the previous credential immediately
tsrctl zone-credential rotate --refresh-rbac --revoke-previous

If the InsufficientClusterPermissions condition appears on the ProtectionZone, run tsrctl zone-credential rotate --refresh-rbac to re-apply the required RBAC.

Inspecting and deleting zone credentials

# Inspect the current credential for a zone
tsrctl zone-credential inspect

# Delete a credential (use --force to bypass safety checks)
tsrctl zone-credential delete
tsrctl zone-credential delete --force

Checking replication health

# Aggregated replication status for a ProtectionGroup
kubectl get rgs -n dr-production

# Detailed status
kubectl get rgs my-protection-group-replication -n dr-production -o yaml

# RPO audit events
kubectl get rpoe -n dr-production

Running a test failover

apiVersion: siterecovery.trilio.io/v1alpha1
kind: TestFailover
metadata:
  name: dr-drill-q1
  namespace: dr-production
spec:
  protectionGroupRef:
    name: my-protection-group
    namespace: dr-production
  cleanupPolicy: Automatic
  schedulingCheck: Fail

kubectl apply -f testfailover.yaml
kubectl get tfo dr-drill-q1 -n dr-production -w

# The test namespace is reported in status, never user-provided
kubectl get tfo dr-drill-q1 -n dr-production \
  -o jsonpath='{.status.testNamespace}'

Example 1 — Minimal ProtectionZone

apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionZone
metadata:
  name: production
  namespace: dr-production
spec:
  displayName: Production DR Zone
  storageBackendMode: DRBD
  clusters:
    - name: cluster1
      clusterID: cluster1
      kubeconfigSecretRef:
        name: cluster1-kubeconfig
        key: kubeconfig
    - name: cluster2
      clusterID: cluster2
      kubeconfigSecretRef:
        name: cluster2-kubeconfig
        key: kubeconfig

Example 2 — ProtectionRequest for a single VM

apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionRequest
metadata:
  name: protect-web-vm
  namespace: dr-production
spec:
  sourceCluster: cluster1
  virtualMachine:
    name: web-vm
    namespace: production-app
  replicationConfig:
    replicationProtocol: C
    rpo: 0

Example 3 — ProtectionGroup with boot ordering and hooks

apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionGroup
metadata:
  name: app-tier
  namespace: dr-production
spec:
  displayName: Application Tier
  sourceCluster: cluster1
  virtualMachines:
    - name: db-vm
      namespace: production-app
      bootOrder: 1
    - name: web-vm
      namespace: production-app
      bootOrder: 2
  sla:
    rpoTargetSeconds: 0
    rtoTargetSeconds: 300
  hooks:
    - name: pre-failover-quiesce
      hookName: db-quiesce
      type: pre-failover
      runOn: source
      onFailure: Abort

Example 4 — Planned failover

apiVersion: siterecovery.trilio.io/v1alpha1
kind: FailoverRequest
metadata:
  name: planned-failover-to-dr
  namespace: dr-production
spec:
  protectionGroupRef:
    name: app-tier
    namespace: dr-production
  targetCluster: cluster2
  failoverType: planned

Example 5 — Unplanned failover

apiVersion: siterecovery.trilio.io/v1alpha1
kind: FailoverRequest
metadata:
  name: emergency-failover
  namespace: dr-production
spec:
  protectionGroupRef:
    name: app-tier
    namespace: dr-production
  targetCluster: cluster2
  failoverType: unplanned
  skipHooks: false

Example 6 — Test failover with reachability verification

apiVersion: siterecovery.trilio.io/v1alpha1
kind: TestFailover
metadata:
  name: quarterly-drill
  namespace: dr-production
spec:
  protectionGroupRef:
    name: app-tier
    namespace: dr-production
  cleanupPolicy: Automatic
  retentionTime: 4h
  schedulingCheck: Fail
  verification:
    reachability:
      - type: tcp
        port: 22
        timeout: 30s
      - type: http
        port: 8080
        endpoint: /health
        expectedStatus: 200
        timeout: 30s

Example 7 — Hook definition

apiVersion: siterecovery.trilio.io/v1alpha1
kind: Hook
metadata:
  name: db-quiesce
  namespace: dr-production
spec:
  description: Quiesce the database before failover
  timeoutSeconds: 120
  retries: 1
  retryDelaySeconds: 10
  jobTemplate:
    image: quay.io/triliovault/hooks/db-quiesce:1.0.0
    command: ["/bin/sh", "-c"]
    args: ["mysql -u root -p$DB_PASS -e 'FLUSH TABLES WITH READ LOCK;'"]
    env:
      - name: DB_PASS
        valueFrom:
          secretKeyRef:
            name: db-credentials
            key: password

Example 8 — Listing controller pods by component label

# ProtectionZone controller on TSR-Manager
kubectl get pods -n trilio-site-recovery-system \
  -l app.kubernetes.io/component=protectionzone-controller

# TSR-Manager control plane in the zone namespace
kubectl get pods -n dr-production \
  -l app.kubernetes.io/component=manager-control-plane

# Workload control plane on a workload cluster
kubectl get pods -n trilio-site-recovery-system \
  -l app.kubernetes.io/component=workload-control-plane \
  --context <workload-cluster-context>

# DRBD node agent DaemonSet pods on a workload cluster
kubectl get pods -n trilio-site-recovery-system \
  -l app.kubernetes.io/component=drbd-node-agent \
  --context <workload-cluster-context>

Example 9 — Reading controller logs

# TSR-Manager control plane logs
kubectl logs -n dr-production \
  -l app.kubernetes.io/component=manager-control-plane \
  --since=1h

# Node agent logs for a specific node
NODE=worker-1
POD=$(kubectl get pods -n trilio-site-recovery-system \
  --field-selector spec.nodeName=${NODE} \
  -l app.kubernetes.io/component=drbd-node-agent \
  -o jsonpath='{.items[0].metadata.name}' \
  --context <workload-cluster-context>)
kubectl logs -n trilio-site-recovery-system $POD \
  --context <workload-cluster-context>

Example 10 — Checking which VMs are protected

# On a workload cluster — the label is stamped on the KubeVirt VirtualMachine
kubectl get vm -A \
  -l siterecovery.trilio.io/protected-vm=true \
  --context <workload-cluster-context>

ProtectionZone stuck in Provisioning or Unavailable

Symptom: kubectl get pz -n dr-<zone> shows Provisioning or Unavailable long after creation.

Likely causes and fixes:

1. ClustersValidated=False / InsufficientClusterPermissions — The zone credential authenticates but lacks required RBAC verbs. Fix: run tsrctl zone-credential rotate --refresh-rbac for the affected workload cluster.

2. ClustersValidated=False / ClusterIdentityChanged — The credential now reaches a different cluster (the kube-system Namespace UID has changed). There is no override. You must restore the original cluster or re-create the zone.

3. ClustersValidated=False / ClusterAlreadyClaimed — The workload cluster is already leased to another zone (the siterecovery.trilio.io/owner-zone-namespace-uid annotation on the workload cluster's trilio-site-recovery-system Namespace points elsewhere). Each workload cluster can belong to exactly one zone.

4. SCCPrerequisiteMissing or KubeVirtNotInstalled — The workload cluster is missing a prerequisite. Check the condition message for details.

kubectl get pz -n dr-production -o jsonpath='{.items[0].status.conditions}' | jq .

ProtectionRequest stuck in Syncing or CreatingDRBD

Symptom: A ProtectionRequest remains in Syncing or CreatingDRBD phase for an extended period.

Likely causes and fixes:

1. DRBD port blocked — Replication uses TCP ports 7000–8000 inclusive (one port per VM volume). A firewall rule that stops at 7999 silently prevents replication on one volume. Open TCP 7000–8000 on all worker-node pairs between the primary and DR clusters.

2. ConditionPlacementReady=False on DRBDResource — The node agent is waiting for metadata PVCs to bind. Check the CSI provisioner in the VM's namespace on the workload cluster. The provisioning failure deadline is configurable via site-recovery-zone-config (default 10 minutes).

3. Node agent startup taint — The drbd-node-agent holds siterecovery.trilio.io/not-ready:NoSchedule on its node during startup. Pods scheduled to that node will not run until DRBD converges. Check agent logs.

# Check DRBDResource conditions on the workload cluster
kubectl get dres -n <vm-namespace> --context <workload-cluster-context> -o yaml

# Check metadata PVCs
kubectl get pvc -n <vm-namespace> --context <workload-cluster-context>

ProtectionRequest in Failed phase

Symptom: Phase is Failed; you want to re-protect the VM.

Cause: Protection cannot be retried in place. A full unprotect followed by a new ProtectionRequest (with a full initial resync) is required.

Fix:

1. Delete the failed ProtectionRequest and wait for the finalizer to complete teardown.
2. Confirm the DRBDResource CRs on both workload clusters are gone.
3. Create a new ProtectionRequest.

FailoverRequest stuck in WaitingForLocks or Monitoring

Symptom: A FailoverRequest does not advance past WaitingForLocks.

Likely cause: Another FailoverRequest or TestFailover is already in flight for the same ProtectionGroup. The admission webhook prevents creating a second one, but a previously admitted request may still be running.

Fix: Inspect active operations:

kubectl get fr -n dr-production
kubectl get tfo -n dr-production

Wait for the in-flight operation to reach a terminal state (Completed, Failed, or Cleaned) before creating a new one.

TestFailover test namespace not visible

Symptom: You want to inspect VMs in the test namespace but don't know its name.

Cause: The test namespace is generated by the controller and reported in status.testNamespace. It is never user-provided.

Fix:

kubectl get tfo <name> -n dr-production \
  -o jsonpath='{.status.testNamespace}'

Then use that namespace to inspect test VMs on the target workload cluster.

VM live migration fails on a protected VM

Symptom: VirtualMachineInstanceMigration is rejected for a protected VM.

Cause: Live migration of protected VMs is denied by the workload control plane admission webhook. A protected VM is pinned to one node per cluster for the lifetime of the protection; nodeName is immutable once set. oc adm drain on a node hosting a running protected VM will also not complete.

Fix: Unprotect the VM first if you need to migrate it. Note that re-protecting requires a full initial resync.

Zone credential about to expire

Symptom: The CredentialExpiring reason appears on the CredentialsValid condition of the ProtectionZone. The zone phase is not affected.

Fix: Rotate the credential before it expires:

tsrctl zone-credential rotate --refresh-rbac

Collecting a support bundle

If you need to share diagnostics with Trilio support, collect a must-gather bundle from the TSR-Manager cluster:

oc adm must-gather --image=quay.io/triliovault/site-recovery-must-gather:<release-tag>

This collects logs, CRs, and DRBD state from the TSR-Manager cluster and all connected workload clusters into a single archive.

Status patch rejected events

Symptom: kubectl describe on a DRBDResource or ProtectionRequest shows a Warning StatusPatchRejected event.

Cause: The API server rejected a status write from a controller, typically because the stored CRD schema does not include a field the controller tried to set (a CRD drift). The CR's displayed status may be stale.

Fix: Verify the installed CRDs match the operator version. Check the event message for the specific rejected field. Collect a support bundle and contact Trilio support.