Deployment Guide

This page walks you through deploying Trilio Site Recovery on OpenShift in production: installing the operator via OLM, provisioning zone credentials, creating a ProtectionZone, and verifying that the full control plane stack is healthy across all three clusters. It also covers RBAC setup, resource limits, and the configuration knobs most relevant to a multi-cluster production environment. By the end you will have a running TSR-Manager cluster, two workload clusters joined to a ProtectionZone, and the DRBD node agent running on every worker node — ready for VM protection.

Before you begin, confirm the following are in place:

Clusters

• One dedicated TSR-Manager cluster (OpenShift 4.17 or later) — holds all orchestration state, no VM workloads
• Two workload clusters (OpenShift 4.17 or later) — host KubeVirt VMs and DRBD replication
• Single-Node OpenShift (SNO) is supported for both the TSR-Manager and workload clusters

OpenShift version note: The OLM bundle annotation (com.redhat.openshift.versions) lists v4.14-v4.19 as the installable range. The validated and supported floor for production use is OpenShift 4.17; releases 4.14–4.16 are within the installable range but are not validated for Tech Preview and are not supported.

Software on workload clusters

• OpenShift Virtualization (KubeVirt) installed and serving kubevirt.io/v1 on both workload clusters
• DRBD 9 kernel module capability — the node agent's init container loads it automatically from the drbd9-rhel9 image; no manual installation is needed, but workload nodes must be able to pull from quay.io/triliodata
• RHEL CoreOS 9 (el9-suffix kernels) on all workload nodes — RHEL CoreOS 10 is not validated in Tech Preview

Networking

• TCP ports 7000–8000 inclusive open between primary and DR worker nodes (one port per replicated VM volume; port 8000 is mandatory — a firewall rule ending at 7999 silently prevents replication on one volume)
• Round-trip latency between workload clusters is not required to be under any threshold; Protocol C (synchronous, RPO=0) is the only supported replication protocol

Tooling

• oc CLI (OpenShift CLI) with cluster-admin access to all three clusters
• kubectl with access to the TSR-Manager cluster
• tsrctl — supplied by Trilio with the release; contact Trilio support to obtain it
• OLM (Operator Lifecycle Manager) installed on the TSR-Manager cluster (present by default on OpenShift)

Clock synchronisation

• NTP/chrony running and synchronised on every node of all three clusters — token expiry and cross-cluster timestamp comparisons depend on it

Registry access

• All three clusters must be able to pull images from quay.io/triliodata

Installation has two complementary stages that both must complete:

1. OLM installs the operator on the TSR-Manager cluster — this delivers the ProtectionZone controller, the TSR web console (frontend and backend), and the ProtectionZone CRD.
2. The ProtectionZone controller installs the control plane stacks on the TSR-Manager cluster and on both workload clusters — triggered automatically when you create a ProtectionZone CR with valid zone credentials.

Customers do not run helm install. The three Helm charts (site-recovery-protectionzone-controller, site-recovery-manager-control-plane, site-recovery-workload-control-plane) are embedded in the operator binary and applied by the ProtectionZone controller once the zone reaches Available.

Step 1 — Prepare the zone namespace on the TSR-Manager cluster

All TSR-Manager-side resources for a zone live in a namespace named dr-<zone-name>. Create it before installing the operator so the OLM Subscription and zone CRs land in the right place.

oc --context tsr-manager create namespace trilio-site-recovery-system
oc --context tsr-manager create namespace dr-my-zone

trilio-site-recovery-system is the suggested namespace for the OLM Subscription (the operator itself). The zone namespace dr-my-zone is where you will create the ProtectionZone CR and its kubeconfig Secrets.

Step 2 — Install the operator via OLM

Create a CatalogSource, an OperatorGroup, and a Subscription on the TSR-Manager cluster.

# catalog-source.yaml
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
  name: site-recovery-catalog
  namespace: trilio-site-recovery-system
spec:
  sourceType: grpc
  image: quay.io/triliodata/site-recovery-catalog:<release-tag>
  displayName: Trilio Site Recovery
  publisher: Trilio Data, Inc.

# operator-group.yaml
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: site-recovery-og
  namespace: trilio-site-recovery-system
spec: {}

# subscription.yaml
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: site-recovery
  namespace: trilio-site-recovery-system
spec:
  channel: stable
  name: site-recovery
  source: site-recovery-catalog
  sourceNamespace: trilio-site-recovery-system

oc --context tsr-manager apply -f catalog-source.yaml
oc --context tsr-manager apply -f operator-group.yaml
oc --context tsr-manager apply -f subscription.yaml

Wait for the CSV to reach Succeeded:

oc --context tsr-manager get csv -n trilio-site-recovery-system \
  -w -o custom-columns=NAME:.metadata.name,PHASE:.status.phase

Once the CSV is Succeeded, three Deployments are running in trilio-site-recovery-system:

• protection-zone-controller — the ProtectionZone operator
• site-recovery-frontend — Next.js web console
• site-recovery-backend — Flask API backend

Step 3 — Provision zone credentials with tsrctl

Each workload cluster is reached through a kubeconfig Secret stored on the TSR-Manager cluster. The tsrctl zone-credential subcommand creates the per-zone ServiceAccount (in the trilio-site-recovery-access namespace on each workload cluster) and mints a time-limited token with the cluster CA embedded.

Run tsrctl with cluster-admin access to both workload clusters. Contact Trilio support to obtain the tsrctl binary for your release.

# Create credentials for both workload clusters.
# tsrctl zone-credential create does not take --zone or --kubeconfig flags;
# target each cluster by setting the appropriate kubeconfig context before running.
tsrctl zone-credential create

tsrctl zone-credential create writes a kubeconfig to stdout (or a file, depending on the flags your release supports — consult the built-in help with tsrctl zone-credential create --help). Run it once per workload cluster, using the appropriate kubeconfig context for each cluster.

The credential is bound to the cluster's kube-system Namespace UID. It cannot be redirected to a different cluster — if the UID ever changes, the zone reports ClusterIdentityChanged and must be re-created.

Store each resulting kubeconfig as a Secret in the zone namespace on the TSR-Manager cluster:

# Store cluster1 kubeconfig
oc --context tsr-manager create secret generic cluster1-kubeconfig \
  --from-file=kubeconfig=/path/to/cluster1-kubeconfig \
  -n dr-my-zone

# Store cluster2 kubeconfig
oc --context tsr-manager create secret generic cluster2-kubeconfig \
  --from-file=kubeconfig=/path/to/cluster2-kubeconfig \
  -n dr-my-zone

Step 4 — Create the ProtectionZone CR

The ProtectionZone CR is the top-level configuration object. Creating it triggers cluster validation, RBAC checks, and automated installation of the TSR-Manager and workload control plane stacks on all three clusters.

# protection-zone.yaml
apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionZone
metadata:
  name: my-zone
  namespace: dr-my-zone
spec:
  displayName: "Primary DR Zone"
  description: "Two-cluster DRBD protection zone pairing the primary and DR sites."
  storageBackendMode: DRBD
  clusters:
    - name: primary
      clusterID: cluster1
      distributionName: RHEL-CoreOS-9
      kubeconfigSecretRef:
        name: cluster1-kubeconfig
        key: kubeconfig
    - name: dr
      clusterID: cluster2
      distributionName: RHEL-CoreOS-9
      kubeconfigSecretRef:
        name: cluster2-kubeconfig
        key: kubeconfig

oc --context tsr-manager apply -f protection-zone.yaml

Key field constraints:

• storageBackendMode must be DRBD — the admission webhook rejects Linstor with "Linstor backend is not supported yet; use DRBD"
• clusters must have exactly two entries with distinct clusterID values (cluster1 and cluster2)
• kubeconfigSecretRef is an object with name and key — not a string
• displayName is required (5–253 characters)

Step 5 — Monitor ProtectionZone provisioning

Watch the zone reach Available:

oc --context tsr-manager get pz -n dr-my-zone -w

The ProtectionZone controller runs through several phases:

1. Provisioning — validating cluster reachability, identity (kube-system UID), RBAC, KubeVirt presence, and SCC prerequisites on each workload cluster
2. Available — both clusters validated (ClustersValidated=True), TSR-Manager control plane installed in dr-my-zone, workload control planes and DRBD node agent DaemonSets installed in trilio-site-recovery-system on each workload cluster

Inspect conditions for detail:

oc --context tsr-manager get pz my-zone -n dr-my-zone \
  -o jsonpath='{.status.conditions}' | jq .

The ClustersValidated condition must be True before the TSR-Manager webhooks will accept ProtectionGroup or ProtectionRequest creates.

Step 6 — Verify workload cluster installations

On each workload cluster, confirm the workload control plane and node agent are running:

# Control plane manager (1/1 expected)
oc --context cluster1 get pods -n trilio-site-recovery-system \
  -l app.kubernetes.io/component=workload-control-plane

# DRBD node agent DaemonSet — one pod per schedulable node
oc --context cluster1 get daemonset -n trilio-site-recovery-system \
  -l app.kubernetes.io/component=drbd-node-agent

# Agent pods — confirm all are Running
oc --context cluster1 get pods -n trilio-site-recovery-system \
  -l app.kubernetes.io/component=drbd-node-agent

The DaemonSet object name follows the pattern <release>-site-recovery-workload-control-plane-agent. The label app.kubernetes.io/component=drbd-node-agent is the correct selector — drbd-node-agent is the component label value, not the object name.

On the TSR-Manager cluster, confirm the manager control plane is running in the zone namespace:

oc --context tsr-manager get pods -n dr-my-zone \
  -l app.kubernetes.io/component=manager-control-plane

The manager pod runs a single container named manager and should show 1/1 Ready.

Step 7 — Verify zone status

oc --context tsr-manager describe pz my-zone -n dr-my-zone

A healthy zone shows:

• status.phase: Available
• status.clusters[].status: Reachable for both clusters
• status.clusters[].admittedSCC.nodeAgent: privileged (OpenShift)
• status.clusters[].admittedSCC.controlPlane: restricted-v2 (OpenShift)
• ClustersValidated: True

A default DRBDReplicationPolicy named my-zone-default with isDefault: true and Protocol C is automatically created in dr-my-zone by the ProtectionZone controller. You do not create or modify replication policies.

All configuration is expressed through Kubernetes custom resources and the ProtectionZone spec. The Helm charts are embedded in the operator binary and are not a customer-facing surface — you do not set chart values directly.

ProtectionZone spec fields

Node agent resource limits

The node agent DaemonSet runs on every schedulable worker node. Its default resource allocation is:

These are the values TSR is tested with. If your workload nodes require different limits, contact Trilio support — the values are set by the embedded workload chart and cannot be overridden through the ProtectionZone CR.

DRBD replication policy

The ProtectionZone controller automatically creates a DRBDReplicationPolicy named <zoneName>-default in the zone namespace with isDefault: true. Protocol C (synchronous, RPO=0) is the only supported protocol — Protocol A is rejected at admission. You do not create or modify replication policies.

Key policy defaults (read-only reference):

Runtime log level

You can change the log level of any zone component without a pod restart by editing the ProtectionZone spec:

oc --context tsr-manager patch pz my-zone -n dr-my-zone \
  --type merge -p '{"spec":{"logging":{"level":"debug"}}}'

The change propagates to the site-recovery-logging ConfigMap in each component's namespace and takes effect within approximately 15 seconds. Accepted values: error, info, debug, trace.

You can also edit the site-recovery-logging ConfigMap directly in any component namespace for per-component overrides without going through the ProtectionZone CR:

# On the TSR-Manager cluster, in the zone namespace
oc --context tsr-manager edit configmap site-recovery-logging -n dr-my-zone

The ConfigMap keys are:

• level — global level for all components reading this ConfigMap
• level.control-plane — override for the control-plane manager
• level.drbd-node-agent — override for the node agent
• level.protectionzone — override for the ProtectionZone controller

Zone credential expiry

Credentials minted by tsrctl zone-credential create have a default lifetime of one year. The zone raises a CredentialExpiring advisory condition on CredentialsValid 30 days before the token expires (credentialExpiryWarning defaults to 720h in the ProtectionZone controller). Inspect expiry:

oc --context tsr-manager get pz my-zone -n dr-my-zone \
  -o jsonpath='{.status.clusters[*].credentialExpiresAt}'

Rotate before expiry using tsrctl zone-credential rotate (see the Rotate credentials section under Usage).

Inspect zone and cluster health

# Summary view of all zones
oc --context tsr-manager get pz -A

# Detailed status for one zone
oc --context tsr-manager describe pz my-zone -n dr-my-zone

# Per-cluster reachability and admitted SCCs
oc --context tsr-manager get pz my-zone -n dr-my-zone \
  -o jsonpath='{.status.clusters}' | jq .

Rotate zone credentials

Rotate credentials before they expire. Run tsrctl zone-credential rotate with access to the workload clusters. The --refresh-rbac flag re-applies the ClusterRole behind the zone's ServiceAccount, which also adds any new permission grants introduced in a newer version of tsrctl:

tsrctl zone-credential rotate --refresh-rbac

To also revoke the previous token after the new one is in place:

tsrctl zone-credential rotate --refresh-rbac --revoke-previous

After rotation, update the kubeconfig Secrets in the zone namespace on the TSR-Manager cluster with the new kubeconfig output, then wait for the ProtectionZone controller to detect the changed Secret and roll the manager control plane pod.

Inspect a zone credential

tsrctl zone-credential inspect

This shows the token expiry, the bound ServiceAccount, and the cluster UID the credential is tied to.

List workload control plane components

# TSR-Manager side — all components in the zone namespace
oc --context tsr-manager get all -n dr-my-zone \
  -l app.kubernetes.io/part-of=site-recovery

# Workload side — control plane manager
oc --context cluster1 get deployment -n trilio-site-recovery-system \
  -l app.kubernetes.io/component=workload-control-plane

# Workload side — node agent DaemonSet
oc --context cluster1 get daemonset -n trilio-site-recovery-system \
  -l app.kubernetes.io/component=drbd-node-agent

Check node agent status on a specific node

The node agent applies a siterecovery.trilio.io/not-ready:NoSchedule taint to its node during startup (bounded by spec.nodeAgent.startupDeadlineSeconds, default 120 s). To see which node agent pod is running on a specific node:

# Step 1: find the agent pod on a node
oc --context cluster1 get pods -n trilio-site-recovery-system \
  --field-selector spec.nodeName=<node-name> \
  -l app.kubernetes.io/component=drbd-node-agent \
  -o jsonpath='{.items[0].metadata.name}'

# Step 2: read its logs
oc --context cluster1 logs -n trilio-site-recovery-system <pod-name>

Delete a zone credential

tsrctl zone-credential delete

To force deletion even if the credential cannot be cleanly revoked:

tsrctl zone-credential delete --force

Deleting a ProtectionZone does not automatically revoke credentials — run tsrctl zone-credential delete after zone deletion to clean up the ServiceAccount and token on each workload cluster.

Access the TSR web console

On OpenShift, the ProtectionZone controller creates an OpenShift Route and console link automatically. The console URL is:

https://trilio-site-recovery.<cluster-apps-domain>

Users sign in through OpenShift OAuth. The Route object name is site-recovery in the operator namespace. Web console access is OpenShift-only — no ingress is created on upstream Kubernetes.

Three ClusterRoles control what users can do in the console:

• trilio-site-recovery-admin — full access
• trilio-site-recovery-operator — operational access (failover, protect)
• trilio-site-recovery-viewer — read-only

Bind users or groups to these roles as needed:

oc adm policy add-cluster-role-to-user trilio-site-recovery-operator <username>

Collect a support bundle

Run from the TSR-Manager cluster. The oc adm must-gather command picks up the must-gather image automatically from the CSV annotation on an OLM-installed cluster, or you can name it explicitly:

oc --context tsr-manager adm must-gather \
  --image=quay.io/triliodata/site-recovery-must-gather:<release-tag>

The entrypoint inside the image is /usr/bin/gather. Do not override it. The bundle collects logs, CRs, and DRBD state from the TSR-Manager and all workload clusters into a single archive.

Example 1 — Minimal ProtectionZone manifest

The smallest valid ProtectionZone. storageBackendMode: DRBD and both cluster slots are required.

apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionZone
metadata:
  name: production-zone
  namespace: dr-production-zone
spec:
  displayName: "Production DR Zone"
  storageBackendMode: DRBD
  clusters:
    - name: primary
      clusterID: cluster1
      distributionName: RHEL-CoreOS-9
      kubeconfigSecretRef:
        name: cluster1-kubeconfig
        key: kubeconfig
    - name: dr
      clusterID: cluster2
      distributionName: RHEL-CoreOS-9
      kubeconfigSecretRef:
        name: cluster2-kubeconfig
        key: kubeconfig

Example 2 — ProtectionZone with logging and hook retention tuning

Sets the global runtime log level to debug for all zone components and extends failed hook Job retention to 14 days for post-incident inspection:

apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionZone
metadata:
  name: production-zone
  namespace: dr-production-zone
spec:
  displayName: "Production DR Zone"
  storageBackendMode: DRBD
  clusters:
    - name: primary
      clusterID: cluster1
      distributionName: RHEL-CoreOS-9
      kubeconfigSecretRef:
        name: cluster1-kubeconfig
        key: kubeconfig
    - name: dr
      clusterID: cluster2
      distributionName: RHEL-CoreOS-9
      kubeconfigSecretRef:
        name: cluster2-kubeconfig
        key: kubeconfig
  logging:
    level: debug
  hooks:
    failedJobRetentionSeconds: 1209600
    jobTTLSeconds: 3600

Example 3 — ProtectionZone with extended node agent startup deadline

On nodes with many protected VMs, the default 120-second startup deadline may not be enough for all DRBD resources to converge. Extend it to 300 seconds:

apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionZone
metadata:
  name: production-zone
  namespace: dr-production-zone
spec:
  displayName: "Production DR Zone"
  storageBackendMode: DRBD
  clusters:
    - name: primary
      clusterID: cluster1
      distributionName: RHEL-CoreOS-9
      kubeconfigSecretRef:
        name: cluster1-kubeconfig
        key: kubeconfig
    - name: dr
      clusterID: cluster2
      distributionName: RHEL-CoreOS-9
      kubeconfigSecretRef:
        name: cluster2-kubeconfig
        key: kubeconfig
  nodeAgent:
    startupDeadlineSeconds: 300

Example 4 — OLM Subscription manifest

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: site-recovery
  namespace: trilio-site-recovery-system
spec:
  channel: stable
  name: site-recovery
  source: tsr
  sourceNamespace: openshift-marketplace

Example 5 — Inspect ProtectionZone conditions

oc --context tsr-manager get pz production-zone -n dr-production-zone \
  -o jsonpath='{range .status.conditions[*]}{.type}{"\t"}{.status}{"\t"}{.reason}{"\n"}{end}'

Example 6 — Check the auto-created DRBDReplicationPolicy

The ProtectionZone controller creates this automatically. You should see it after the zone reaches Available:

oc --context tsr-manager get drp -n dr-production-zone

Expected: one policy named production-zone-default with Default: true and Protocol: C.

Example 7 — Support bundle with a 2-hour log window

oc --context tsr-manager adm must-gather \
  --image=quay.io/triliodata/site-recovery-must-gather:<release-tag> \
  -- /usr/bin/gather --since 2h

Example 8 — Rotate credentials and refresh RBAC

tsrctl zone-credential rotate --refresh-rbac --revoke-previous

Then update the Secret in the zone namespace with the new kubeconfig output from tsrctl.

ProtectionZone stuck in Provisioning or Degraded

Symptom: oc get pz shows Status: Provisioning or Status: Degraded for more than a few minutes.

Diagnose:

oc --context tsr-manager get pz <zone-name> -n dr-<zone-name> \
  -o jsonpath='{.status.conditions}' | jq .

Node agent pod stuck in Init:Error or Init:CrashLoopBackOff

Symptom: Agent pods on workload nodes show Init:Error or Init:CrashLoopBackOff.

Diagnose:

# Find the stuck agent pod
oc --context cluster1 get pods -n trilio-site-recovery-system \
  -l app.kubernetes.io/component=drbd-node-agent

# Read the module loader init container logs
oc --context cluster1 logs -n trilio-site-recovery-system <pod-name> \
  -c drbd-module-loader

Likely causes:

• "no matching package / could not find" — the mirrored image has no precompiled kmod for this node's kernel line. Wait for the upstream image to be updated, or contact Trilio support.
• "usermode_helper violation" — a foreign DRBD module is loaded on the node with usermode_helper enabled. Reboot the node after removing whatever loaded it.
• Image pull errors — the node cannot reach quay.io/triliodata. Check network egress and pull secrets.

Check the node kernel version:

oc get node <node-name> -o jsonpath='{.status.nodeInfo.kernelVersion}'

The suffix must be .el9_* for a supported (RHEL CoreOS 9) node.

ClustersValidated=False blocks ProtectionGroup and ProtectionRequest creation

Symptom: Attempting to create a ProtectionGroup or ProtectionRequest results in a webhook denial.

Cause: The ClustersValidated condition on the ProtectionZone is not True. The TSR-Manager webhooks block all ProtectionGroup and ProtectionRequest creates and spec changes until this gate clears.

Fix: Resolve the underlying zone condition (see the table above) and wait for the ProtectionZone to return to Available. Do not attempt to bypass the webhook.

Manager control plane pod not starting in zone namespace

Symptom: No pods in dr-<zone-name> with app.kubernetes.io/component=manager-control-plane.

Diagnose:

oc --context tsr-manager get pods -n dr-<zone-name> -l app.kubernetes.io/part-of=site-recovery
oc --context tsr-manager get events -n dr-<zone-name> --sort-by='.lastTimestamp'

Likely causes:

• The ProtectionZone has not yet reached ClustersValidated=True — the manager control plane chart is installed only after both clusters pass validation.
• The kubeconfig Secrets are missing or malformed — the manager reads both at startup from /etc/kubeconfig/cluster1 and /etc/kubeconfig/cluster2; a missing Secret prevents the Deployment from rolling out.
• Image pull failure — confirm the cluster can reach quay.io/triliodata.

Zone credential expiring warning

Symptom: oc describe pz shows CredentialsValid=True with reason CredentialExpiring, or the TSR web console shows a credential warning.

Cause: The token in a zone credential will expire within 30 days.

Fix:

tsrctl zone-credential rotate --refresh-rbac

Then update the kubeconfig Secret in the zone namespace with the new kubeconfig. The ProtectionZone controller detects the Secret change and restarts the manager control plane pod to pick up the new credential.

Zone stuck with ClusterAlreadyClaimed

Symptom: A zone shows ClustersValidated=False with reason ClusterAlreadyClaimed.

Cause: One of the zone's workload clusters carries a zone-exclusivity lease annotation (siterecovery.trilio.io/owner-zone-namespace-uid on the trilio-site-recovery-system Namespace) belonging to a different zone.

Diagnose:

kubectl --kubeconfig=<workload-kubeconfig> get ns trilio-site-recovery-system \
  -o jsonpath='{.metadata.annotations.siterecovery\.trilio\.io/owner-zone}{"\n"}'

kubectl --kubeconfig=<workload-kubeconfig> get ns trilio-site-recovery-system \
  -o jsonpath='{.metadata.annotations.siterecovery\.trilio\.io/owner-zone-namespace-uid}{"\n"}'

Match the UID against namespaces on your TSR-Manager cluster. Clear the lease only if the owning zone genuinely no longer exists:

kubectl --kubeconfig=<workload-kubeconfig> annotate ns trilio-site-recovery-system \
  siterecovery.trilio.io/owner-zone-namespace-uid- \
  siterecovery.trilio.io/owner-zone-

Warning: Do not clear a lease while the owning zone is still live and using the cluster. Doing so allows two zones to drive one workload cluster simultaneously.

DRBD replication port firewall gap

Symptom: VMs protect successfully but replication shows as degraded or one volume never syncs.

Cause: A firewall rule between primary and DR worker nodes that stops at port 7999 leaves port 8000 blocked. The allocator uses TCP ports 7000–8000 inclusive.

Fix: Open TCP 7000–8000 inclusive (not 7000–7999) between all primary and DR worker node IPs. Port 8000 is allocatable and mandatory.