---
title: Prerequisites
product: trilio-site-recovery-for-kubernetes
doc_type: guide
version: 0.2.0
source: git2docs (code-derived, validation-filtered)
canonical: https://docs-tsr.trilio.io/trilio-site-recovery-for-kubernetes/site-recovery/prerequisites
---

# Prerequisites

_Kubernetes version requirements, required permissions, storage dependencies, and network prerequisites_

## Overview

This page lists every requirement you must satisfy before installing Trilio Site Recovery (Tech Preview). Site Recovery protects KubeVirt VirtualMachines across two workload clusters by replicating their persistent volumes at the block level via DRBD, with all orchestration driven from a dedicated TSR-Manager cluster. Meeting these prerequisites ensures that cluster validation, DRBD module loading, and zone credential provisioning all succeed on the first attempt. Read this page before you proceed to installation.

## Configuration

Site Recovery configuration is split between the ProtectionZone CR (zone-level settings) and the `site-recovery-logging` ConfigMap (runtime log levels). Customers do not author Helm chart values directly; the ProtectionZone controller applies the embedded charts automatically.

---

## ProtectionZone spec fields

All fields below are set on the ProtectionZone CR in the `dr-<name>` namespace on the TSR-Manager cluster.

### Required fields

| Field | Type | Description |
|---|---|---|
| `spec.displayName` | string (5–253 chars) | Human-friendly name shown in the TSR web console |
| `spec.storageBackendMode` | `DRBD` | Storage backend. Must be `DRBD`; `Linstor` is rejected |
| `spec.clusters` | array (exactly 2) | The two workload clusters |
| `spec.clusters[].name` | string (1–63 chars) | Operator-facing identifier for this cluster within the zone |
| `spec.clusters[].clusterID` | `cluster1` or `cluster2` | Stable deployment slot; must be unique across the two clusters |
| `spec.clusters[].kubeconfigSecretRef.name` | string | Name of the Secret holding the kubeconfig |
| `spec.clusters[].kubeconfigSecretRef.key` | string | Key within that Secret |

### Optional fields

| Field | Type | Default | Description |
|---|---|---|---|
| `spec.description` | string | — | Free-form text describing the zone |
| `spec.clusters[].distributionName` | `RHEL-CoreOS-9` or `RHEL-CoreOS-10` | `RHEL-CoreOS-9` | Selects the DRBD kernel-module loader image family. Mutable: update it when an OpenShift upgrade moves RHCOS to a newer RHEL base |
| `spec.nodeAgent.startupDeadlineSeconds` | integer (10–3600) | `120` | How long the DRBD node agent holds its `siterecovery.trilio.io/not-ready:NoSchedule` taint on its node during startup. Increasing this gives more time for DRBD to converge on heavily loaded nodes |
| `spec.hooks.jobTTLSeconds` | integer (≥60) | `3600` | How long a succeeded hook Job is kept on workload clusters |
| `spec.hooks.failedJobRetentionSeconds` | integer (≥60) | `604800` | How long a failed or timed-out hook Job is kept for log inspection |
| `spec.logging.level` | `error`, `info`, `debug`, `trace` | — | Runtime log level for all zone components. Takes effect within ~15 s without pod restarts |
| `spec.logging.controlPlaneLevel` | `error`, `info`, `debug`, `trace` | — | Overrides `level` for the TSR-Manager and workload control plane managers only |
| `spec.logging.nodeAgentLevel` | `error`, `info`, `debug`, `trace` | — | Overrides `level` for DRBD node agents only |

### Immutable fields

Once a ProtectionZone is created, the following fields cannot be changed:

- `spec.storageBackendMode`
- `spec.clusters[].name` (renaming a cluster is rejected)
- `spec.clusters[].clusterID` (slot reassignment is rejected)

Mutable fields (may be changed after creation):

- `spec.displayName`, `spec.description`
- `spec.clusters[].kubeconfigSecretRef` (to rotate credentials)
- `spec.clusters[].distributionName` (to follow an OpenShift RHCOS upgrade)
- `spec.logging.*`, `spec.nodeAgent.*`, `spec.hooks.*`

---

## Runtime log level (without pod restarts)

The `site-recovery-logging` ConfigMap controls runtime log levels for both the control plane manager and the DRBD node agent in its namespace. Both components poll it approximately every 15 seconds.

The ConfigMap is automatically created in each component's namespace when `spec.logging.level` is set on the ProtectionZone. You can also edit it directly:

```bash
# Set all zone components to debug on the TSR-Manager
oc --context tsrmanager patch configmap site-recovery-logging \
  -n dr-prod \
  --type merge \
  -p '{"data":{"level":"debug"}}'

# Set control plane to debug, node agent to info (on a workload cluster)
oc --context cluster1 patch configmap site-recovery-logging \
  -n trilio-site-recovery-system \
  --type merge \
  -p '{"data":{"level":"info","level.control-plane":"debug","level.drbd-node-agent":"info"}}'
```

Accepted values: `error`, `info`, `debug`, `trace`. There is no `warn` level. The `trace` level is available at runtime via the ConfigMap but not as a startup flag.

---

## DRBDReplicationPolicy (managed automatically)

The ProtectionZone controller creates a default `DRBDReplicationPolicy` named `<zoneName>-default` with `isDefault: true` in the zone namespace when the zone is provisioned. You do not create or modify this object.

Key facts:
- Protocol C (synchronous) is the only supported protocol. Protocol A is rejected at the ProtectionRequest admission webhook.
- RPO is always 0 and is not tunable.
- DRBD ports are allocated from the fixed range 7000–8000 inclusive by the allocator in the operator; the `networkConfig.portRange` field in the policy is accepted but not read by the allocator.

---

## ProtectionRequest replication configuration

When you protect a VM, the ProtectionRequest's `spec.replicationConfig` accepts:

| Field | Default | Description |
|---|---|---|
| `replicationProtocol` | `C` | Must be `C`. `A` is rejected |
| `rpo` | `0` | Must be `0` when `replicationProtocol` is `C` |
| `drbdMetadataConfig.size` | `128Mi` | Size of DRBD external-metadata PVCs per protected volume. Minimum 128 Mi |
| `drbdMetadataConfig.storageClassName` | cluster default | StorageClass for metadata PVCs. Must bind immediately; `WaitForFirstConsumer` classes are rejected |

---

## Node agent startup taint

The DRBD node agent applies `siterecovery.trilio.io/not-ready:NoSchedule` to its own node at startup and holds it until all DRBD resources on that node converge (or until `startupDeadlineSeconds` expires). Workloads that do not tolerate this taint — including CSI node plugins and KubeVirt's `virt-handler` — wait at most `startupDeadlineSeconds` on any freshly restarted node.

Adjust the deadline on the ProtectionZone:

```bash
oc --context tsrmanager patch pz prod -n dr-prod \
  --type merge \
  -p '{"spec":{"nodeAgent":{"startupDeadlineSeconds":240}}}'
```

Changing this value triggers a rolling update of the DRBD node agent DaemonSet.

## Usage

All Site Recovery operations are declared through Kubernetes custom resources. The workflow proceeds in this order: install and configure the control plane → provision zone credentials → create a ProtectionZone → protect VMs → monitor replication → run test failovers → execute failovers.

All TSR-Manager-side CRs (`ProtectionZone`, `ProtectionGroup`, `ProtectionRequest`, `FailoverRequest`, `TestFailover`, `Hook`, `ReplicationGroupStatus`, `RPOEvent`) live in the `dr-<name>` namespace on the TSR-Manager cluster. `DRBDResource` CRs live on each workload cluster in the VM's namespace.

---

## Managing zone credentials

Zone credentials are kubeconfig Secrets that grant the TSR-Manager controllers access to workload cluster API servers. They are minted and rotated exclusively with `tsrctl`.

### Inspect a credential

```bash
tsrctl zone-credential inspect
```

### Rotate a credential (reissue token, preserve RBAC)

```bash
tsrctl zone-credential rotate
```

### Rotate and refresh RBAC (after a controller upgrade adds new permissions)

```bash
tsrctl zone-credential rotate --refresh-rbac
```

### Rotate and revoke the previous token

```bash
tsrctl zone-credential rotate --revoke-previous
```

### Delete a credential

```bash
tsrctl zone-credential delete
```

Force deletion (use when the workload cluster is unreachable):

```bash
tsrctl zone-credential delete --force
```

The credential is bound to the kube-system Namespace UID of the workload cluster it was created for. It cannot be redirected to a different cluster. If the zone's `ClustersValidated` condition shows reason `ClusterIdentityChanged`, the credential now reaches a different cluster; the only resolution is to restore the original cluster or re-create the zone.

---

## Checking zone health

```bash
# List all zones
oc --context tsrmanager get pz -A

# Inspect a specific zone
oc --context tsrmanager get pz prod -n dr-prod -o yaml

# Watch conditions
oc --context tsrmanager get pz prod -n dr-prod \
  -o jsonpath='{.status.conditions}' | python3 -m json.tool
```

The `ClustersValidated` condition must be `True` before you can create ProtectionGroup or ProtectionRequest CRs. The TSR-Manager admission webhooks reject those creates while it is not `True`.

---

## Protecting a VM

Create a ProtectionRequest on the TSR-Manager cluster in the zone namespace:

```bash
oc --context tsrmanager apply -f - <<'EOF'
apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionRequest
metadata:
  name: protect-web-vm
  namespace: dr-prod
spec:
  sourceCluster: cluster1
  virtualMachine:
    name: web-vm
    namespace: production
EOF
```

Monitor phase progression:

```bash
oc --context tsrmanager get pr protect-web-vm -n dr-prod -w
```

Phases in order: `Pending` → `Validating` → `CreatingDRBD` → `Syncing` → `ReadyToActivate` → `Activating` → `Protected`.

The VM must be **running** when you protect it. The node it runs on becomes its permanent home on the source cluster for the lifetime of the protection. Live migration and cold migration of protected VMs are denied by admission webhooks.

To check sync progress:

```bash
oc --context tsrmanager get pr protect-web-vm -n dr-prod \
  -o jsonpath='{.status.syncProgress}'
```

---

## Monitoring replication health

The replication monitor auto-creates a `ReplicationGroupStatus` (short name: `rgs`) named `<pg-name>-replication` for each ProtectionGroup:

```bash
oc --context tsrmanager get rgs -n dr-prod

oc --context tsrmanager get rgs my-pg-replication -n dr-prod -o yaml
```

Key status fields:

| Field | Meaning |
|---|---|
| `status.aggregated.connectionState` | Worst-case DRBD connection state across all VMs: `Connected`, `Degraded`, or `Disconnected` |
| `status.aggregated.outOfSyncBytes` | Total bytes not yet replicated across all VMs |
| `status.aggregated.syncedVolumeCount` | Number of VMs with all disks fully in sync |
| `status.rpoStatus` | `Met`, `Degraded`, `Violated`, or `Unknown` |
| `status.lastFullySyncedAt` | When every VM in the group was last fully in sync |

Inspect DRBDResource CRs on the workload clusters for per-VM detail:

```bash
# On the primary workload cluster, in the VM's namespace
oc --context cluster1 get dres -n production
oc --context cluster1 get dres drbd-production-web-vm-<hash8> -n production -o yaml
```

DRBDResource CRs are named `drbd-<namespace>-<vm>-<hash8>`. They exist only on workload clusters; running `kubectl get drbdresource` against the TSR-Manager cluster returns `server doesn't have a resource type`.

---

## Executing a failover

Create a FailoverRequest on the TSR-Manager cluster. `failoverType` is `planned` (graceful, waits for sync) or `unplanned` (force-promote, RTO-optimized):

```bash
oc --context tsrmanager apply -f - <<'EOF'
apiVersion: siterecovery.trilio.io/v1alpha1
kind: FailoverRequest
metadata:
  name: failover-to-dr
  namespace: dr-prod
spec:
  protectionGroupRef:
    name: my-pg
  targetCluster: cluster2
  failoverType: planned
EOF
```

Monitor progress:

```bash
oc --context tsrmanager get fr failover-to-dr -n dr-prod -w
```

Per-VM status is in `status.vmStatuses`. The FailoverRequest has no `message` field and no `failedVMs` field at the top level.

---

## Running a test failover (DR drill)

A TestFailover spins up isolated copies of protected VMs on the standby cluster from DRBD snapshots, inside a controller-generated namespace fenced by a default-deny NetworkPolicy. It does not touch production VMs. Supported in DRBD backend mode only.

```bash
oc --context tsrmanager apply -f - <<'EOF'
apiVersion: siterecovery.trilio.io/v1alpha1
kind: TestFailover
metadata:
  name: drill-my-pg
  namespace: dr-prod
spec:
  protectionGroupRef:
    name: my-pg
  cleanupPolicy: Automatic
  retentionTime: 2h
EOF
```

The test namespace is generated by the controller and reported in `status.testNamespace`. Do not try to predict or construct it.

```bash
oc --context tsrmanager get tfo drill-my-pg -n dr-prod -w
```

Phases: `Queued` → `PreparingEnvironment` → `CreatingSnapshots` → `CreatingVolumes` → `CreatingVMs` → `StartingVMs` → `Verifying` → `Succeeded` or `Failed` → `CleaningUp` → `Cleaned`.

Trigger early cleanup while keeping the record:

```bash
oc --context tsrmanager patch tfo drill-my-pg -n dr-prod \
  --type merge \
  -p '{"spec":{"cleanupRequested":true}}'
```

## Troubleshooting

Use the format below for each issue: **Symptom**, **Likely cause**, **Fix**.

---

### Zone stuck in `Provisioning` phase; `ClustersValidated` is `False`

**Symptom:** `oc get pz <name> -n dr-<name>` shows `Status: Provisioning` and the `ClustersValidated` condition is `False`.

**Likely causes and fixes:**

| `reason` on condition | Cause | Fix |
|---|---|---|
| `KubeVirtNotInstalled` | The workload cluster does not serve `kubevirt.io/v1` | Install OpenShift Virtualization on the workload cluster |
| `InsufficientClusterPermissions` | The zone credential authenticates but lacks required RBAC verbs | Run `tsrctl zone-credential rotate --refresh-rbac` |
| `ClusterIdentityChanged` | The credential now reaches a cluster with a different kube-system UID | Restore the original cluster or re-create the zone |
| `ClusterAlreadyClaimed` | The workload cluster is already registered to another zone | Each workload cluster can belong to only one zone; use a different cluster |
| `SCCPrerequisiteMissing` | The `privileged` SCC or its ClusterRole does not exist | Ensure the cluster is running a supported OpenShift version with standard SCCs |

---

### DRBD node agent pods are not starting; `WorkloadStacksReady` is `False` with reason `SCCAdmissionDenied`

**Symptom:** The zone's `WorkloadStacksReady` condition is `False` with reason `SCCAdmissionDenied`. The node agent DaemonSet shows `FailedCreate` events.

**Likely cause:** The node agent's DaemonSet pod was rejected by OpenShift SCC admission. The node agent requires the `privileged` SCC.

**Fix:** Verify that the ClusterRoleBinding to `system:openshift:scc:privileged` exists for the node agent ServiceAccount:

```bash
oc --context cluster1 get clusterrolebinding -l app.kubernetes.io/part-of=site-recovery
```

If missing, the zone credential may lack the `bind` verb on `system:openshift:scc:privileged`. Rotate with `--refresh-rbac`:

```bash
tsrctl zone-credential rotate --refresh-rbac
```

---

### ProtectionRequest stuck in `Syncing`; sync progress not advancing

**Symptom:** `oc get pr <name> -n dr-<name>` shows `Phase: Syncing` and `Sync` percentage is not increasing.

**Likely causes:**

1. **Network connectivity between worker nodes** — DRBD replication uses TCP ports 7000–8000. A blocked port silently stalls replication.

   **Fix:** Verify firewall rules allow TCP 7000–8000 inclusive in both directions between primary and DR worker nodes. Check that port 8000 is included.

2. **DRBD module not loaded on a node** — the `drbd-module-loader` init container may have failed.

   **Fix:** Check the init container logs on the node agent pod:

   ```bash
   # Find the agent pod on the relevant node
   oc --context cluster1 get pods -n trilio-site-recovery-system \
     -l app.kubernetes.io/component=drbd-node-agent \
     --field-selector spec.nodeName=<node-name> -o name

   # Check the init container
   oc --context cluster1 logs <pod-name> -n trilio-site-recovery-system \
     -c drbd-module-loader
   ```

3. **`usermode_helper` enabled on a pre-loaded module** — the loader exits hard.

   **Fix:** Unload the module and reboot the node.

---

### `oc get dres` against the TSR-Manager cluster returns an error

**Symptom:** `oc --context tsrmanager get dres -A` returns `server doesn't have a resource type 'drbdresources'`.

**Cause:** `DRBDResource` CRDs are installed only on workload clusters. They do not exist on the TSR-Manager cluster by design.

**Fix:** Run DRBDResource queries against the workload clusters:

```bash
oc --context cluster1 get dres -n <vm-namespace>
oc --context cluster2 get dres -n <vm-namespace>
```

---

### ProtectionRequest shows `Ready=False, reason: RoleConflict`

**Symptom:** The ProtectionRequest's `Ready` condition is `False` with reason `RoleConflict`.

**Cause:** A DRBD role conflict was detected — both sides of the replication pair are attempting to be Primary simultaneously.

**Fix:** Inspect the DRBDResource CRs on both workload clusters to identify which side holds an unexpected Primary role:

```bash
oc --context cluster1 get dres -n <vm-namespace> -o wide
oc --context cluster2 get dres -n <vm-namespace> -o wide
```

Do not manually demote DRBD resources. Contact Trilio support with a support bundle.

---

### Zone credential expiry warning

**Symptom:** The ProtectionZone's `CredentialsValid` condition shows reason `CredentialExpiring`.

**Cause:** The zone credential token is approaching expiry (within the configured warning window, default 30 days).

**Effect:** This is advisory only; it does not change the zone phase.

**Fix:** Rotate the credential before it expires:

```bash
tsrctl zone-credential rotate
```

---

### Test failover stuck in `CreatingSnapshots` or `CreatingVolumes`

**Symptom:** A TestFailover stays in `CreatingSnapshots` or `CreatingVolumes` for an extended time.

**Likely cause:** The standby cluster's VolumeSnapshotClass is missing or does not match the StorageClass provisioner of the protected volumes.

**Fix:** On the standby workload cluster, verify that a VolumeSnapshotClass exists whose driver matches the StorageClass used by the protected VMs' PVCs:

```bash
oc --context cluster2 get volumesnapshotclass
```

If missing, create a VolumeSnapshotClass with the appropriate driver.

---

### Node agent holds `siterecovery.trilio.io/not-ready` taint longer than expected

**Symptom:** A workload cluster node carries the `siterecovery.trilio.io/not-ready:NoSchedule` taint beyond the expected startup window. The DRBDResource for a VM shows `NodeStartupGate=False`.

**Cause:** DRBD resources on that node have not converged within `startupDeadlineSeconds`. This can happen on heavily loaded nodes or when the peer site is unreachable.

**Fix:** The condition clears automatically once DRBD converges; no manual action is required. If the condition persists and the out-of-sync count is not decreasing, check network connectivity between worker nodes (TCP 7000–8000) and check the node agent logs:

```bash
# Get pods on the specific node
oc --context cluster1 get pods -n trilio-site-recovery-system \
  -l app.kubernetes.io/component=drbd-node-agent \
  --field-selector spec.nodeName=<node-name> -o name

# Stream logs
oc --context cluster1 logs <pod-name> -n trilio-site-recovery-system -c agent -f
```

If you need more time for DRBD to converge, increase `startupDeadlineSeconds` on the ProtectionZone:

```bash
oc --context tsrmanager patch pz prod -n dr-prod \
  --type merge \
  -p '{"spec":{"nodeAgent":{"startupDeadlineSeconds":300}}}'
```

---

### `oc adm drain` does not complete on a node hosting a protected VM

**Symptom:** `oc adm drain <node>` hangs indefinitely.

**Cause:** Draining a node that hosts a running protected VM is blocked by design. The protected VM is pinned to that node for the lifetime of the protection; live migration and cold migration are denied by the workload admission webhooks.

**Fix:** You have two options:

1. **Fail over the ProtectionGroup** to move the VM to the DR cluster, then drain.
2. **Unprotect the VM** (`ProtectionRequest` deletion), drain the node, then re-protect (which requires a full initial resync).

Do not attempt to bypass the admission webhooks.