Custom Resource Reference
Complete API reference for all site recovery custom resources
This page is the complete API reference for all Trilio Site Recovery custom resources in the siterecovery.trilio.io/v1alpha1 API group. Every DR operation—protecting a VM, triggering a failover, running a test drill, or monitoring replication health—is declared through one of these custom resources and can be driven by kubectl, GitOps tooling, or the TSR web console. Understanding which resource lives on which cluster, what each field does, and which phase values to expect is essential for operating Site Recovery reliably.
Before working with Site Recovery custom resources you need:
- OpenShift 4.17 or later on the TSR-Manager cluster and both workload clusters (OpenShift 4.14–4.16 are within the supported floor but not validated for Tech Preview)
- KubeVirt installed on both workload clusters (the
kubevirt.io/v1API must be served) - DRBD 9 kernel module (
drbd,drbd_transport_tcp,drbd_transport_lb-tcp) loadable on every workload node that will host a protected VM - Three Kubernetes clusters: one TSR-Manager cluster and two workload clusters
- TCP ports 7000–8000 inclusive open between primary and DR worker nodes for DRBD replication (one port per protected VM; port 8000 is mandatory)
kubectlconfigured to reach the TSR-Manager clustertsrctlsupplied by Trilio with the release (contact Trilio support to obtain it)oc(OpenShift CLI) foroc adm must-gathersupport bundle collection- OLM (Operator Lifecycle Manager) installed on the TSR-Manager cluster
- Site Recovery operator installed via OLM (CatalogSource + OperatorGroup + Subscription for package
site-recovery, channeltech-preview, catalogquay.io/triliovault/site-recovery-catalog) intrilio-site-recovery-system - A
ProtectionZoneCR in adr-<name>namespace on the TSR-Manager cluster, which automatically deploys workload control planes and DRBD node agents to both workload clusters
Cluster placement: All TSR-Manager-side CRs (ProtectionZone, ProtectionGroup, ProtectionRequest, DRBDReplicationPolicy, FailoverRequest, TestFailover, Hook, ReplicationGroupStatus, RPOEvent) live in the dr-<zone-deployment-name> namespace on the TSR-Manager cluster. DRBDResource CRs live in the protected VM's namespace on each workload cluster. Running kubectl get drbdresource against the TSR-Manager cluster returns server doesn't have a resource type.
Site Recovery custom resources are installed automatically when you install the operator and create a ProtectionZone. You do not run helm install directly—the Helm charts are embedded in the operator binary and applied by the ProtectionZone controller.
Step 1 — Install the operator via OLM
Create the following objects on the TSR-Manager cluster in order:
apiVersion: operators.coreos.com/v1alpha1
kind: CatalogSource
metadata:
name: site-recovery-catalog
namespace: trilio-site-recovery-system
spec:
sourceType: grpc
image: quay.io/triliodata/site-recovery-catalog:<release-tag>
displayName: Trilio Site Recovery
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
name: site-recovery-og
namespace: trilio-site-recovery-system
spec: {}
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
name: site-recovery
namespace: trilio-site-recovery-system
spec:
channel: stable
name: site-recovery
source: site-recovery-catalog
sourceNamespace: trilio-site-recovery-system
Step 2 — Provision zone credentials
Use tsrctl (supplied by Trilio) to create a per-zone ServiceAccount and kubeconfig Secret on each workload cluster. Run this on a host that can reach both workload clusters:
# Create credentials on each workload cluster
tsrctl zone-credential create \
--namespace trilio-site-recovery-access \
--zone <zone-name> \
--access-namespace trilio-site-recovery-access
This creates a ServiceAccount in trilio-site-recovery-access on each workload cluster and produces kubeconfig Secrets that the TSR-Manager references.
Step 3 — Create the zone namespace and kubeconfig Secrets on TSR-Manager
kubectl create namespace dr-<zone-name>
# Copy or create the kubeconfig Secrets for each workload cluster
kubectl create secret generic cluster1-kubeconfig \
--from-file=kubeconfig=./cluster1.kubeconfig \
-n dr-<zone-name>
kubectl create secret generic cluster2-kubeconfig \
--from-file=kubeconfig=./cluster2.kubeconfig \
-n dr-<zone-name>
Step 4 — Create a ProtectionZone CR
apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionZone
metadata:
name: primary-dr-zone
namespace: dr-primary-dr-zone
spec:
displayName: "Primary DR Zone"
description: "Two-cluster DRBD protection zone."
storageBackendMode: DRBD
clusters:
- name: primary
clusterID: cluster1
distributionName: RHEL-CoreOS-9
kubeconfigSecretRef:
name: cluster1-kubeconfig
key: kubeconfig
- name: dr
clusterID: cluster2
distributionName: RHEL-CoreOS-9
kubeconfigSecretRef:
name: cluster2-kubeconfig
key: kubeconfig
Apply it and wait for the zone to reach Available:
kubectl apply -f protectionzone.yaml
kubectl get pz -n dr-primary-dr-zone --watch
When the zone is Available, the ClustersValidated condition is True and the TSR-Manager webhooks will accept ProtectionGroup and ProtectionRequest creates. The ProtectionZone controller also automatically creates a DRBDReplicationPolicy named <zoneName>-default with isDefault: true and Protocol C—you do not create this yourself.
The following describes the key fields for each custom resource. All resources use apiVersion: siterecovery.trilio.io/v1alpha1. Only spec fields that require explanation beyond their names are described in detail; refer to the field tables in the Usage section for complete field lists.
ProtectionZone
The top-level configuration object. Each zone occupies its own namespace (dr-<name>) on the TSR-Manager cluster, and each workload cluster may belong to only one zone at a time.
| Field | Default | Notes |
|---|---|---|
spec.storageBackendMode | — (required) | Must be DRBD. The schema admits Linstor but the admission webhook rejects it: Linstor backend is not supported yet; use DRBD. |
spec.clusters[].clusterID | — (required) | Must be cluster1 or cluster2. The two entries must use different slot values. |
spec.clusters[].distributionName | RHEL-CoreOS-9 | Selects the DRBD kernel-module loader image. RHEL-CoreOS-10 is accepted by the schema but not validated in Tech Preview. |
spec.clusters[].kubeconfigSecretRef.name | — (required) | Name of the Secret in the zone namespace containing the kubeconfig. Both name and key must be non-empty; the webhook rejects empty values even though the schema permits them. |
spec.logging.level | — (required when logging is set) | One of error, info, debug, trace. Controls runtime log level of all zone components via the site-recovery-logging ConfigMap; takes effect within ~15 s without pod restarts. |
spec.logging.controlPlaneLevel | inherits level | Override for TSR-Manager and workload control-plane pods only. |
spec.logging.nodeAgentLevel | inherits level | Override for DRBD node agent pods only. |
spec.nodeAgent.startupDeadlineSeconds | 120 | How long the node agent keeps the siterecovery.trilio.io/not-ready:NoSchedule taint on its node while DRBD resources converge. Range: 10–3600. Changing this value rolls the agent DaemonSet. |
spec.hooks.jobTTLSeconds | 3600 | How long a succeeded hook Job is retained (seconds). |
spec.hooks.failedJobRetentionSeconds | 604800 | How long a failed or timed-out hook Job is retained (7 days). |
Immutable fields: spec.storageBackendMode and each cluster's name and clusterID are immutable once set. kubeconfigSecretRef and distributionName are mutable to support credential rotation and OpenShift upgrades.
ProtectionGroup
Groups one or more KubeVirt VMs for coordinated failover. Lives on the TSR-Manager cluster in the zone namespace.
| Field | Default | Notes |
|---|---|---|
spec.virtualMachines[].name | — (required) | Name of the KubeVirt VM on the workload cluster. |
spec.virtualMachines[].namespace | CR namespace | Namespace of the VM on the workload cluster. |
spec.virtualMachines[].bootOrder | — | Lower values boot first during failover. VMs sharing a value boot in parallel. VMs without a value boot together in a final batch. |
spec.desiredState | running | Whether VMs in this group should be running or stopped on the active cluster. |
spec.sla.rpoTargetSeconds | 0 | Target RPO in seconds. Feeds the replication monitor objective (ReplicationGroupStatus.spec.rpoObjectiveSeconds). 0 is the Protocol C synchronous guarantee. |
spec.sla.rtoTargetSeconds | — (required) | Target RTO in seconds. Recorded for compliance reporting. |
spec.hooks | — | List of hook bindings. Each entry references a Hook CR by hookName (same namespace). |
spec.hooks[].name | — (required) | Binding identifier, embedded in Job names. |
spec.hooks[].hookName | — (required) | Name of the referenced Hook CR in the same namespace. |
spec.hooks[].type | — (required) | Lifecycle point: pre-failover, post-failover, pre-failback, post-failback, pre-test, post-test, test-verification. |
spec.hooks[].runOn | target | source or target. source is only admitted for pre-failover and pre-failback bindings. |
spec.hooks[].onFailure | Abort | Abort fails the operation; Continue records a warning and proceeds. |
spec.hooks[].order | 0 | Lower runs first among hooks of the same type. |
spec.vmIdentity.macAddresses | Preserve | Preserve copies interface MACs verbatim to the DR copy; Clear lets KubeMacPool assign fresh MACs. |
ProtectionRequest
Declares protection for a single KubeVirt VM. The spec.sourceCluster field is the failover lever—patching it to the DR cluster's slot triggers a role flip.
| Field | Default | Notes |
|---|---|---|
spec.sourceCluster | — (required) | cluster1 or cluster2. The cluster currently running the VM. Patching this field triggers a planned failover for this VM. |
spec.virtualMachine.name | — (required) | Name of the KubeVirt VM. |
spec.virtualMachine.namespace | — (required) | Namespace of the VM on the workload cluster. Must be set explicitly; cannot be inferred from the PR's namespace. |
spec.replicationConfig.replicationProtocol | C | Only C (synchronous, RPO=0) is supported. A is rejected by the admission webhook: ReplicationProtocolA is not supported yet. |
spec.replicationConfig.rpo | 0 | Must be 0 when using Protocol C. A non-zero value is rejected: rpo must be 0 when spec.replicationConfig.replicationProtocol is C. |
spec.replicationConfig.drbdMetadataConfig.size | 128Mi | Size of the DRBD external-metadata PVC created per protected volume. Minimum 128Mi. |
spec.replicationConfig.drbdMetadataConfig.storageClassName | cluster default | StorageClass for metadata PVCs. Must bind immediately (WaitForFirstConsumer classes are rejected). |
spec.standbyPlacement | — | Optional per-key edits to the DR VM's node-level scheduling rules (nodeSelector, nodeAffinity, tolerations). Immutable once set. |
Phase values (in order): Pending → Validating → CreatingDRBD → Syncing → ReadyToActivate → Activating → Protected. On teardown: Unprotecting. On error: Failed.
FailoverRequest
Triggers a planned or unplanned failover of an entire ProtectionGroup. The controller patches each member ProtectionRequest's spec.sourceCluster and waits for the DRBD Primary role to confirm on the target side.
| Field | Default | Notes |
|---|---|---|
spec.protectionGroupRef.name | — (required) | Name of the ProtectionGroup to fail over. |
spec.targetCluster | — (required) | cluster1 or cluster2. The cluster to fail over TO. |
spec.failoverType | planned | planned (graceful, waits for sync) or unplanned (force-promote, RTO-optimized). |
spec.batchBootTimeoutSeconds | 300 | Max seconds to wait for one boot-order batch's VMs to reach Running. |
spec.drainTimeoutSeconds | 300 | Accepted and range-checked (30–3600) but not consumed by the failover controller in Tech Preview. Has no effect on drain or failover behaviour. |
spec.skipHooks | false | When true, all hooks are recorded as Skipped rather than run. Useful for incident bypass. Immutable. |
State values: Pending, InProgress, Completed, Failed, Cancelled.
Phase values: Initializing → Validating → RunningPreHooks → StoppingSource → WaitingForLocks → RemovingTaints → StartingTarget → Monitoring → RunningPostHooks → Completed. On error: Failed.
Note: status.vmStatuses holds per-VM detail. There is no status.message field and no status.failedVMs field on FailoverRequest.
TestFailover
Triggers a non-disruptive DR validation. The controller creates an isolated test namespace (site-recovery-<protection-group>-<random>, reported in status.testNamespace) on the standby cluster, spins up VM copies from DRBD snapshots inside a network-fenced namespace, runs verification, and reports results without touching production VMs.
| Field | Default | Notes |
|---|---|---|
spec.protectionGroupRef.name | — (required) | Name of the ProtectionGroup to test. |
spec.cleanupPolicy | Automatic | Automatic cleans up test resources after retentionTime. Manual keeps resources until cleanupRequested: true or the CR is deleted. |
spec.retentionTime | 2h | How long test resources are kept before automatic cleanup. Pattern: ^[0-9]+(s|m|h|d)$. |
spec.cleanupRequested | false | Set to true to trigger cleanup while keeping the TestFailover CR as a record. Can be set but never unset. |
spec.schedulingCheck | Fail | Fail excludes VMs the DR scheduler cannot place and marks the test Failed. Warn records the verdict and drills everything. |
spec.timeout | — | Optional wall-clock limit (e.g. 2h). No default is applied by design; omit for large protection groups. |
spec.batchBootTimeoutSeconds | 300 | Max seconds to wait for one boot-order batch's test VMs to reach Running. |
spec.skipHooks | false | When true, all hooks (including test-verification hooks) are recorded as Skipped. |
spec.verification.reachability | — | Optional list of network probes to run after automatic boot/agent/filesystem checks. |
spec.verification.reachability[].type | — (required) | tcp or http. |
spec.verification.reachability[].port | — (required) | TCP or HTTP port to probe. |
spec.verification.reachability[].vmName | — | Restricts probe to a named VM. Empty means all test VMs. |
spec.verification.reachability[].endpoint | / | HTTP path. Valid only for type: http. |
spec.verification.reachability[].expectedStatus | 200 | Expected HTTP status code. Valid only for type: http. |
spec.verification.reachability[].timeout | 30s | Probe timeout. Maximum 10m. |
Phase values: Queued → PreparingEnvironment → CreatingSnapshots → CreatingVolumes → CreatingVMs → StartingVMs → Verifying → Succeeded. On error: Failed. Cleanup: CleaningUp → Cleaned.
Note: status.testNamespace is controller-generated and never user-provided. Do not construct, guess, or grep for it.
DRBDResource
A workload-cluster-side CR (one per VM per cluster) representing all of a VM's replicated disks as a single DRBD resource. Named drbd-<namespace>-<vm>-<hash8>. Created and managed by the TSR-Manager-side ProtectionRequest controller; you inspect it to check connection state, phase, and node placement.
| Field | Default | Notes |
|---|---|---|
spec.side | — (required) | Source or Target. Immutable; does not change on failover. |
spec.role | — (required) | Primary or Secondary. Set by the ProtectionRequest controller; changes on role flip. |
spec.port | — (required) | TCP port for DRBD replication. Range: 7000–8000. |
spec.nodeName | — | Node this copy is pinned to. Immutable once set. |
spec.paused | false | Suspends convergence (maintenance knob). |
spec.replicationIntent | Enabled | Suspended renders the resource with no connection mesh (used during unplanned failover recovery). |
spec.volumes[].volumeNumber | — (required) | Stable cross-copy join key (boot disk = 0). Max 15 volumes per resource. |
spec.volumes[].pvcName | — (required) | Local backing PVC name. |
Phase values: Pending, Provisioning, ReadyToActivate, Syncing, Consistent, Degraded, Failed, Deleting.
Note: DRBDResource is only registered on workload clusters. Running kubectl get dres against the TSR-Manager cluster returns server doesn't have a resource type.
DRBDReplicationPolicy
Defines DRBD replication parameters. The ProtectionZone controller automatically creates one policy named <zoneName>-default with isDefault: true and Protocol C in the zone namespace. You do not create this object.
| Field | Default | Notes |
|---|---|---|
spec.drbdProtocol | C | Wire protocol. Only C (synchronous) is supported in Tech Preview. |
spec.replicationMode | Synchronous | Synchronous or Asynchronous. Only Synchronous is functional in Tech Preview. |
spec.isDefault | false | Marks this as the default policy for its namespace. |
spec.clusters[].storageClassMappings | — | Maps source StorageClass names to target StorageClass names. If a source class is absent, it is used as-is. |
spec.networkConfig.portRange | start: 7000, end: 7999 | Accepted by the schema but never read by the port allocator. The allocator always uses 7000–8000 inclusive. |
ReplicationGroupStatus
Auto-created observability CR per ProtectionGroup, named <protection-group-name>-replication. Written by the replication monitor controller; you do not create or update it.
| Status field | Notes |
|---|---|
status.rpoStatus | Met, Degraded, Violated, or Unknown. Driven by spec.rpoObjectiveSeconds (set from ProtectionGroup.spec.sla.rpoTargetSeconds). |
status.aggregated.outOfSyncBytes | Total out-of-sync bytes across all VMs in the group. |
status.aggregated.connectionState | Worst-case connection state: Connected, Degraded, or Disconnected. |
status.volumeReplicationStatuses | One entry per VM (not per disk). A 3-disk VM appears once. |
Filter by label siterecovery.trilio.io/protection-group=<pg-name>. Field selectors on spec.* or status.* do not work on custom resources.
RPOEvent
Immutable audit record emitted by the replication monitor on each replication status transition. All fields are in spec (no status subresource). You do not create or update RPOEvents.
| Field | Notes |
|---|---|
spec.eventType | One of: SyncCompleted, RPODegraded, RPOViolated, ConnectionLost, ConnectionRestored, ResyncStarted, ResyncCompleted, VolumeAdded, VolumeRemoved, MetadataUndersized, ResizePending, ResizeComplete. |
spec.violationReason | One of: ConnectionLost, NetworkTimeout, SyncLagExceeded, ProtocolCOutOfSync, ResyncInProgress, Unknown. |
Hook
Reusable, namespaced definition of a Kubernetes Job to execute at failover or test lifecycle points. Referenced by ProtectionGroup.spec.hooks[].hookName. The admission webhook prevents deletion or spec edits while an operation is using it. Hook has no short name; use kubectl get hooks.
| Field | Default | Notes |
|---|---|---|
spec.jobTemplate.image | — (required) | Container image for the hook Job. Must be pullable from the executing workload cluster. |
spec.timeoutSeconds | 300 | Per-attempt timeout in seconds. |
spec.retries | 0 | Retry count after failure. Maximum 5. |
spec.retryDelaySeconds | 5 | Pause between retry attempts in seconds. |
spec.jobTemplate.env | — | Environment variables. fieldRef and resourceFieldRef sources are rejected at admission. |
spec.jobTemplate.volumes | — | Permitted sources: secret, configMap, emptyDir, projected. hostPath is not representable. |
All Site Recovery operations are performed by creating or patching custom resources on the TSR-Manager cluster, except DRBDResource inspection which is done on workload clusters.
Check zone health
# On the TSR-Manager cluster, in the zone namespace
kubectl get pz -n dr-<zone-name>
kubectl describe pz <zone-name> -n dr-<zone-name>
The ClustersValidated condition must be True before the webhooks will accept ProtectionGroup or ProtectionRequest creates.
Protect a virtual machine
- Create a
ProtectionGroupdeclaring the VM(s):
kubectl apply -f protectiongroup.yaml -n dr-<zone-name>
kubectl get pg -n dr-<zone-name>
- Create a
ProtectionRequestfor each VM:
kubectl apply -f protectionrequest.yaml -n dr-<zone-name>
kubectl get pr -n dr-<zone-name> --watch
Watch the phase progress: Pending → Validating → CreatingDRBD → Syncing → ReadyToActivate → Activating → Protected.
- Check the protected VM label on the workload cluster:
# On the workload cluster
kubectl get vm -A -l siterecovery.trilio.io/protected-vm=true
- Inspect DRBD state on each workload cluster:
# On each workload cluster, in the VM's namespace
kubectl get dres -n <vm-namespace>
kubectl describe dres drbd-<namespace>-<vm>-<hash8> -n <vm-namespace>
Monitor replication health
# ReplicationGroupStatus is auto-created; inspect it by label
kubectl get rgs -n dr-<zone-name> -l siterecovery.trilio.io/protection-group=<pg-name>
kubectl get rgs <pg-name>-replication -n dr-<zone-name>
# View RPO events (audit trail)
kubectl get rpoe -n dr-<zone-name> -l siterecovery.trilio.io/protection-group=<pg-name>
Run a test failover
kubectl apply -f testfailover.yaml -n dr-<zone-name>
kubectl get tfo -n dr-<zone-name> --watch
The test namespace is reported in status.testNamespace. Do not construct or guess this value.
After the test completes, trigger cleanup without losing the record:
kubectl patch tfo <name> -n dr-<zone-name> \
--type merge \
-p '{"spec":{"cleanupRequested":true}}'
Execute a planned failover
kubectl apply -f failoverrequest-planned.yaml -n dr-<zone-name>
kubectl get fr -n dr-<zone-name> --watch
Every failover is initiated by creating a FailoverRequest. Failover does not happen automatically.
Execute an unplanned failover
kubectl apply -f failoverrequest-unplanned.yaml -n dr-<zone-name>
Unprotect a VM
Delete the ProtectionRequest. The controller moves the VM through the Unprotecting phase, removes DRBD resources, and restarts the VM against its original storage.
kubectl delete pr <name> -n dr-<zone-name>
kubectl get pr <name> -n dr-<zone-name> --watch
Rotate zone credentials
# Rotate and refresh RBAC if permissions have changed
tsrctl zone-credential rotate --refresh-rbac
# Rotate and revoke the previous token
tsrctl zone-credential rotate --revoke-previous
Adjust runtime log level
Edit the site-recovery-logging ConfigMap in the zone namespace (TSR-Manager) or workload namespace (trilio-site-recovery-system). Changes take effect within ~15 seconds without pod restarts.
kubectl patch configmap site-recovery-logging \
-n dr-<zone-name> \
--type merge \
-p '{"data":{"level":"debug","level.control-plane":"trace"}}'
Accepted values: error, info, debug, trace. There is no warn level.
ProtectionZone — minimal DRBD zone
apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionZone
metadata:
name: primary-dr-zone
namespace: dr-primary-dr-zone
spec:
displayName: "Primary DR Zone"
description: "DRBD protection zone pairing primary and DR sites."
storageBackendMode: DRBD
clusters:
- name: primary
clusterID: cluster1
distributionName: RHEL-CoreOS-9
kubeconfigSecretRef:
name: cluster1-kubeconfig
key: kubeconfig
- name: dr
clusterID: cluster2
distributionName: RHEL-CoreOS-9
kubeconfigSecretRef:
name: cluster2-kubeconfig
key: kubeconfig
ProtectionZone — with debug logging and custom node-agent deadline
apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionZone
metadata:
name: primary-dr-zone
namespace: dr-primary-dr-zone
spec:
displayName: "Primary DR Zone"
storageBackendMode: DRBD
clusters:
- name: primary
clusterID: cluster1
distributionName: RHEL-CoreOS-9
kubeconfigSecretRef:
name: cluster1-kubeconfig
key: kubeconfig
- name: dr
clusterID: cluster2
distributionName: RHEL-CoreOS-9
kubeconfigSecretRef:
name: cluster2-kubeconfig
key: kubeconfig
logging:
level: info
controlPlaneLevel: debug
nodeAgent:
startupDeadlineSeconds: 180
hooks:
jobTTLSeconds: 7200
failedJobRetentionSeconds: 604800
ProtectionGroup — two VMs with boot ordering and SLA
apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionGroup
metadata:
name: app-tier
namespace: dr-primary-dr-zone
spec:
virtualMachines:
- name: db-vm
namespace: app-production
bootOrder: 1
- name: web-vm
namespace: app-production
bootOrder: 2
desiredState: running
sla:
rpoTargetSeconds: 0
rtoTargetSeconds: 300
vmIdentity:
macAddresses: Preserve
ProtectionRequest — protect a single VM
apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionRequest
metadata:
name: protect-db-vm
namespace: dr-primary-dr-zone
spec:
sourceCluster: cluster1
virtualMachine:
name: db-vm
namespace: app-production
replicationConfig:
replicationProtocol: C
rpo: 0
drbdMetadataConfig:
storageClassName: fast-block
size: 128Mi
ProtectionRequest — with standby placement overrides
apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionRequest
metadata:
name: protect-web-vm
namespace: dr-primary-dr-zone
spec:
sourceCluster: cluster1
virtualMachine:
name: web-vm
namespace: app-production
replicationConfig:
replicationProtocol: C
rpo: 0
standbyPlacement:
nodeSelector:
add:
workload-type: dr-standby
remove:
- zone
FailoverRequest — planned failover to cluster2
apiVersion: siterecovery.trilio.io/v1alpha1
kind: FailoverRequest
metadata:
name: planned-failover-20260801
namespace: dr-primary-dr-zone
spec:
protectionGroupRef:
name: app-tier
namespace: dr-primary-dr-zone
targetCluster: cluster2
failoverType: planned
batchBootTimeoutSeconds: 300
skipHooks: false
FailoverRequest — unplanned failover (skip hooks)
apiVersion: siterecovery.trilio.io/v1alpha1
kind: FailoverRequest
metadata:
name: unplanned-failover-incident
namespace: dr-primary-dr-zone
spec:
protectionGroupRef:
name: app-tier
namespace: dr-primary-dr-zone
targetCluster: cluster2
failoverType: unplanned
skipHooks: true
TestFailover — basic DR drill with automatic cleanup
apiVersion: siterecovery.trilio.io/v1alpha1
kind: TestFailover
metadata:
name: drill-q3-2026
namespace: dr-primary-dr-zone
spec:
protectionGroupRef:
name: app-tier
namespace: dr-primary-dr-zone
cleanupPolicy: Automatic
retentionTime: 2h
schedulingCheck: Fail
TestFailover — with TCP and HTTP reachability probes
apiVersion: siterecovery.trilio.io/v1alpha1
kind: TestFailover
metadata:
name: drill-with-probes
namespace: dr-primary-dr-zone
spec:
protectionGroupRef:
name: app-tier
namespace: dr-primary-dr-zone
cleanupPolicy: Manual
schedulingCheck: Warn
verification:
reachability:
- type: tcp
port: 5432
vmName: db-vm
timeout: 30s
- type: http
port: 8080
vmName: web-vm
endpoint: /healthz
expectedStatus: 200
timeout: 30s
Hook — pre-failover quiesce script
apiVersion: siterecovery.trilio.io/v1alpha1
kind: Hook
metadata:
name: db-quiesce
namespace: dr-primary-dr-zone
spec:
description: "Quiesce the database before failover"
timeoutSeconds: 120
retries: 2
retryDelaySeconds: 10
jobTemplate:
image: quay.io/myorg/db-quiesce-tool:v1.2.3
command: ["/scripts/quiesce.sh"]
args: ["--timeout", "90"]
env:
- name: DB_HOST
value: "db-vm.app-production.svc"
ProtectionGroup — binding a Hook
apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionGroup
metadata:
name: app-tier
namespace: dr-primary-dr-zone
spec:
virtualMachines:
- name: db-vm
namespace: app-production
- name: web-vm
namespace: app-production
desiredState: running
sla:
rpoTargetSeconds: 0
rtoTargetSeconds: 300
hooks:
- name: pre-failover-quiesce
hookName: db-quiesce
type: pre-failover
runOn: source
onFailure: Abort
order: 0
vms:
- name: db-vm
namespace: app-production
Inspect per-VM DRBD state (on the workload cluster)
# List all DRBDResources in the VM's namespace
kubectl get dres -n app-production
# Describe a specific DRBDResource
kubectl describe dres drbd-app-production-db-vm-a1b2c3d4 -n app-production
Check replication group status
# On the TSR-Manager cluster
kubectl get rgs app-tier-replication -n dr-primary-dr-zone
# Watch for RPO events
kubectl get rpoe -n dr-primary-dr-zone \
-l siterecovery.trilio.io/protection-group=app-tier \
--sort-by=.spec.timestamp
View per-VM failover status
kubectl get fr planned-failover-20260801 \
-n dr-primary-dr-zone \
-o jsonpath='{.status.vmStatuses}'
Zone stuck in Provisioning / ClustersValidated=False
Symptom: kubectl get pz -n dr-<zone-name> shows Status: Provisioning and ClustersValidated condition is False.
Likely causes and fixes:
| Reason | Meaning | Fix |
|---|---|---|
ClusterAlreadyClaimed | A workload cluster is leased to another zone (the siterecovery.trilio.io/owner-zone-namespace-uid annotation on its trilio-site-recovery-system namespace belongs to a different zone). | Remove the other zone or use different workload clusters. |
InsufficientClusterPermissions | The zone credential authenticates but fails the RBAC permission sweep. The condition message names the missing verbs. | Run tsrctl zone-credential rotate --refresh-rbac to issue a new credential with current RBAC. |
ClusterIdentityChanged | The credential now reaches a cluster whose kube-system Namespace UID differs from the UID recorded at first claim. No override exists. | Restore the original cluster or re-create the zone. |
KubeVirtNotInstalled | The workload cluster does not serve kubevirt.io/v1. | Install KubeVirt on the workload cluster before creating the zone. |
SCCPrerequisiteMissing | The privileged SCC or its ClusterRole does not exist on the workload cluster. | Verify OpenShift is correctly installed on the workload cluster. |
ProtectionRequest stuck in Syncing
Symptom: A ProtectionRequest has been in Syncing for an unexpectedly long time.
Likely cause: Initial block-level sync is still in progress. Check progress:
# On TSR-Manager
kubectl get pr <name> -n dr-<zone-name> -o jsonpath='{.status.slowestVolumeSyncProgress}'
# On the workload cluster
kubectl get dres drbd-<namespace>-<vm>-<hash8> -n <vm-namespace> \
-o jsonpath='{.status.syncProgress}'
Fix: Wait for sync to complete. If progress is stalled at 0%, check DRBD node agent logs on both workload clusters:
# On the workload cluster — find the node agent pod for the VM's node
POD=$(kubectl get pods -n trilio-site-recovery-system \
--field-selector spec.nodeName=<node-name> \
-l app.kubernetes.io/component=drbd-node-agent \
-o jsonpath='{.items[0].metadata.name}')
kubectl logs -n trilio-site-recovery-system $POD
Verify TCP ports 7000–8000 inclusive are open between primary and DR worker nodes. Port 8000 is mandatory; a firewall stopping at 7999 will silently prevent replication for one volume.
DRBDResource in Degraded phase
Symptom: kubectl get dres -n <vm-namespace> shows Degraded in the Phase column.
Likely cause: The DRBD connection to the peer cluster is lost or the DRBD role is in conflict.
Fix:
- Check the
connectionStatecolumn—DisconnectedorStandAloneindicates a network issue. - Verify network connectivity on TCP 7000–8000 between the two cluster's worker nodes.
- Check if the
ProtectionRequeston TSR-Manager has aReady=Falsecondition with reasonRoleConflict:
kubectl describe pr <name> -n dr-<zone-name> | grep -A5 RoleConflict
- Check the DRBD node agent DaemonSet is healthy on both workload clusters:
kubectl get daemonset -n trilio-site-recovery-system \
-l app.kubernetes.io/component=drbd-node-agent
Node agent applying not-ready taint longer than expected
Symptom: A node carries the siterecovery.trilio.io/not-ready:NoSchedule taint after startup for longer than spec.nodeAgent.startupDeadlineSeconds.
Likely cause: One or more DRBD resources on the node have not converged. After the deadline, the taint is removed anyway and the DRBDResource carries a NodeStartupGate=False condition with reason StartupDeadlineExceeded.
Fix: The condition clears itself when the resource recovers. Check what it is waiting for:
# On the workload cluster
kubectl describe dres <name> -n <vm-namespace> | grep -A5 NodeStartupGate
If the node-agent pod is failing to start, check its logs and verify DRBD module loading:
POD=$(kubectl get pods -n trilio-site-recovery-system \
--field-selector spec.nodeName=<node-name> \
-l app.kubernetes.io/component=drbd-node-agent \
-o jsonpath='{.items[0].metadata.name}')
kubectl logs -n trilio-site-recovery-system $POD -c drbd-module-loader
FailoverRequest stuck in InProgress
Symptom: A FailoverRequest remains InProgress for an extended period.
Likely cause: A hook is running slowly or has failed; VMs on the source cluster are not stopping; the boot-order batch timeout has not elapsed.
Fix: Inspect the hook statuses and per-VM statuses:
kubectl get fr <name> -n dr-<zone-name> -o jsonpath='{.status.phase}'
kubectl get fr <name> -n dr-<zone-name> -o jsonpath='{.status.vmStatuses}'
kubectl get fr <name> -n dr-<zone-name> -o jsonpath='{.status.hookStatuses}'
If a hook is blocking and the incident is time-critical, delete the current FailoverRequest, create a new one with spec.skipHooks: true, and investigate the hook failure afterwards.
TestFailover fails with protectionGroupRef not found
Symptom: Creating a TestFailover returns an error containing protectionGroupRef.
Likely cause: The ProtectionGroup referenced does not exist in the namespace specified in spec.protectionGroupRef.namespace, or spec.protectionGroupRef.namespace was omitted.
Fix: Verify the ProtectionGroup exists on the TSR-Manager cluster and that protectionGroupRef.namespace is set to the correct zone namespace:
kubectl get pg -n dr-<zone-name>
Note that spec.protectionGroupRef.namespace must be set explicitly on a TestFailover; it cannot be inferred.
oc adm must-gather fails to collect
Symptom: oc adm must-gather exits with an error or produces an incomplete bundle.
Fix: Run must-gather from the TSR-Manager cluster with the Site Recovery image. Contact Trilio support for the pinned image reference:
# Run on the TSR-Manager cluster
oc adm must-gather --image=quay.io/triliovault/site-recovery-must-gather:<release-tag>
The entrypoint inside the image is /usr/bin/gather. Do not pass a -- <command> override. The collector reaches through to all zone workload clusters and assembles a single read-only archive.