---
title: Custom Resource Reference
product: trilio-site-recovery-for-kubernetes
doc_type: guide
version: 0.2.0
source: git2docs (code-derived, validation-filtered)
canonical: https://docs-tsr.trilio.io/trilio-site-recovery-for-kubernetes/site-recovery/crd-reference
---

# Custom Resource Reference

_Complete API reference for all site recovery custom resources_

## Overview

This page is the complete API reference for all Trilio Site Recovery custom resources in the `siterecovery.trilio.io/v1alpha1` API group. Every DR operation—protecting a VM, triggering a failover, running a test drill, or monitoring replication health—is declared through one of these custom resources and can be driven by `kubectl`, GitOps tooling, or the TSR web console. Understanding which resource lives on which cluster, what each field does, and which phase values to expect is essential for operating Site Recovery reliably.

## Prerequisites

Before working with Site Recovery custom resources you need:

- **OpenShift 4.17 or later** on the TSR-Manager cluster and both workload clusters (OpenShift 4.14–4.16 are within the supported floor but not validated for Tech Preview)
- **KubeVirt** installed on both workload clusters (the `kubevirt.io/v1` API must be served)
- **DRBD 9 kernel module** (`drbd`, `drbd_transport_tcp`, `drbd_transport_lb-tcp`) loadable on every workload node that will host a protected VM
- **Three Kubernetes clusters**: one TSR-Manager cluster and two workload clusters
- **TCP ports 7000–8000 inclusive** open between primary and DR worker nodes for DRBD replication (one port per protected VM; port 8000 is mandatory)
- **`kubectl`** configured to reach the TSR-Manager cluster
- **`tsrctl`** supplied by Trilio with the release (contact Trilio support to obtain it)
- **`oc`** (OpenShift CLI) for `oc adm must-gather` support bundle collection
- **OLM** (Operator Lifecycle Manager) installed on the TSR-Manager cluster
- Site Recovery operator installed via OLM (CatalogSource + OperatorGroup + Subscription for package `site-recovery`, channel `tech-preview`, catalog `quay.io/triliovault/site-recovery-catalog`) in `trilio-site-recovery-system`
- A `ProtectionZone` CR in a `dr-<name>` namespace on the TSR-Manager cluster, which automatically deploys workload control planes and DRBD node agents to both workload clusters

**Cluster placement**: All TSR-Manager-side CRs (`ProtectionZone`, `ProtectionGroup`, `ProtectionRequest`, `DRBDReplicationPolicy`, `FailoverRequest`, `TestFailover`, `Hook`, `ReplicationGroupStatus`, `RPOEvent`) live in the `dr-<zone-deployment-name>` namespace on the TSR-Manager cluster. `DRBDResource` CRs live in the protected VM's namespace on each workload cluster. Running `kubectl get drbdresource` against the TSR-Manager cluster returns `server doesn't have a resource type`.

## Configuration

The following describes the key fields for each custom resource. All resources use `apiVersion: siterecovery.trilio.io/v1alpha1`. Only `spec` fields that require explanation beyond their names are described in detail; refer to the field tables in the Usage section for complete field lists.

### ProtectionZone

The top-level configuration object. Each zone occupies its own namespace (`dr-<name>`) on the TSR-Manager cluster, and each workload cluster may belong to only one zone at a time.

| Field | Default | Notes |
|---|---|---|
| `spec.storageBackendMode` | — (required) | Must be `DRBD`. The schema admits `Linstor` but the admission webhook rejects it: `Linstor backend is not supported yet; use DRBD`. |
| `spec.clusters[].clusterID` | — (required) | Must be `cluster1` or `cluster2`. The two entries must use different slot values. |
| `spec.clusters[].distributionName` | `RHEL-CoreOS-9` | Selects the DRBD kernel-module loader image. `RHEL-CoreOS-10` is accepted by the schema but not validated in Tech Preview. |
| `spec.clusters[].kubeconfigSecretRef.name` | — (required) | Name of the Secret in the zone namespace containing the kubeconfig. Both `name` and `key` must be non-empty; the webhook rejects empty values even though the schema permits them. |
| `spec.logging.level` | — (required when `logging` is set) | One of `error`, `info`, `debug`, `trace`. Controls runtime log level of all zone components via the `site-recovery-logging` ConfigMap; takes effect within ~15 s without pod restarts. |
| `spec.logging.controlPlaneLevel` | inherits `level` | Override for TSR-Manager and workload control-plane pods only. |
| `spec.logging.nodeAgentLevel` | inherits `level` | Override for DRBD node agent pods only. |
| `spec.nodeAgent.startupDeadlineSeconds` | `120` | How long the node agent keeps the `siterecovery.trilio.io/not-ready:NoSchedule` taint on its node while DRBD resources converge. Range: 10–3600. Changing this value rolls the agent DaemonSet. |
| `spec.hooks.jobTTLSeconds` | `3600` | How long a succeeded hook Job is retained (seconds). |
| `spec.hooks.failedJobRetentionSeconds` | `604800` | How long a failed or timed-out hook Job is retained (7 days). |

**Immutable fields**: `spec.storageBackendMode` and each cluster's `name` and `clusterID` are immutable once set. `kubeconfigSecretRef` and `distributionName` are mutable to support credential rotation and OpenShift upgrades.

### ProtectionGroup

Groups one or more KubeVirt VMs for coordinated failover. Lives on the TSR-Manager cluster in the zone namespace.

| Field | Default | Notes |
|---|---|---|
| `spec.virtualMachines[].name` | — (required) | Name of the KubeVirt VM on the workload cluster. |
| `spec.virtualMachines[].namespace` | CR namespace | Namespace of the VM on the workload cluster. |
| `spec.virtualMachines[].bootOrder` | — | Lower values boot first during failover. VMs sharing a value boot in parallel. VMs without a value boot together in a final batch. |
| `spec.desiredState` | `running` | Whether VMs in this group should be `running` or `stopped` on the active cluster. |
| `spec.sla.rpoTargetSeconds` | `0` | Target RPO in seconds. Feeds the replication monitor objective (`ReplicationGroupStatus.spec.rpoObjectiveSeconds`). `0` is the Protocol C synchronous guarantee. |
| `spec.sla.rtoTargetSeconds` | — (required) | Target RTO in seconds. Recorded for compliance reporting. |
| `spec.hooks` | — | List of hook bindings. Each entry references a `Hook` CR by `hookName` (same namespace). |
| `spec.hooks[].name` | — (required) | Binding identifier, embedded in Job names. |
| `spec.hooks[].hookName` | — (required) | Name of the referenced `Hook` CR in the same namespace. |
| `spec.hooks[].type` | — (required) | Lifecycle point: `pre-failover`, `post-failover`, `pre-failback`, `post-failback`, `pre-test`, `post-test`, `test-verification`. |
| `spec.hooks[].runOn` | `target` | `source` or `target`. `source` is only admitted for `pre-failover` and `pre-failback` bindings. |
| `spec.hooks[].onFailure` | `Abort` | `Abort` fails the operation; `Continue` records a warning and proceeds. |
| `spec.hooks[].order` | `0` | Lower runs first among hooks of the same type. |
| `spec.vmIdentity.macAddresses` | `Preserve` | `Preserve` copies interface MACs verbatim to the DR copy; `Clear` lets KubeMacPool assign fresh MACs. |

### ProtectionRequest

Declares protection for a single KubeVirt VM. The `spec.sourceCluster` field is the failover lever—patching it to the DR cluster's slot triggers a role flip.

| Field | Default | Notes |
|---|---|---|
| `spec.sourceCluster` | — (required) | `cluster1` or `cluster2`. The cluster currently running the VM. Patching this field triggers a planned failover for this VM. |
| `spec.virtualMachine.name` | — (required) | Name of the KubeVirt VM. |
| `spec.virtualMachine.namespace` | — (required) | Namespace of the VM on the workload cluster. Must be set explicitly; cannot be inferred from the PR's namespace. |
| `spec.replicationConfig.replicationProtocol` | `C` | Only `C` (synchronous, RPO=0) is supported. `A` is rejected by the admission webhook: `ReplicationProtocolA is not supported yet`. |
| `spec.replicationConfig.rpo` | `0` | Must be `0` when using Protocol C. A non-zero value is rejected: `rpo must be 0 when spec.replicationConfig.replicationProtocol is C`. |
| `spec.replicationConfig.drbdMetadataConfig.size` | `128Mi` | Size of the DRBD external-metadata PVC created per protected volume. Minimum 128Mi. |
| `spec.replicationConfig.drbdMetadataConfig.storageClassName` | cluster default | StorageClass for metadata PVCs. Must bind immediately (`WaitForFirstConsumer` classes are rejected). |
| `spec.standbyPlacement` | — | Optional per-key edits to the DR VM's node-level scheduling rules (nodeSelector, nodeAffinity, tolerations). Immutable once set. |

**Phase values** (in order): `Pending` → `Validating` → `CreatingDRBD` → `Syncing` → `ReadyToActivate` → `Activating` → `Protected`. On teardown: `Unprotecting`. On error: `Failed`.

### FailoverRequest

Triggers a planned or unplanned failover of an entire ProtectionGroup. The controller patches each member ProtectionRequest's `spec.sourceCluster` and waits for the DRBD Primary role to confirm on the target side.

| Field | Default | Notes |
|---|---|---|
| `spec.protectionGroupRef.name` | — (required) | Name of the ProtectionGroup to fail over. |
| `spec.targetCluster` | — (required) | `cluster1` or `cluster2`. The cluster to fail over TO. |
| `spec.failoverType` | `planned` | `planned` (graceful, waits for sync) or `unplanned` (force-promote, RTO-optimized). |
| `spec.batchBootTimeoutSeconds` | `300` | Max seconds to wait for one boot-order batch's VMs to reach Running. |
| `spec.drainTimeoutSeconds` | `300` | Accepted and range-checked (30–3600) but not consumed by the failover controller in Tech Preview. Has no effect on drain or failover behaviour. |
| `spec.skipHooks` | `false` | When `true`, all hooks are recorded as Skipped rather than run. Useful for incident bypass. Immutable. |

**State values**: `Pending`, `InProgress`, `Completed`, `Failed`, `Cancelled`.

**Phase values**: `Initializing` → `Validating` → `RunningPreHooks` → `StoppingSource` → `WaitingForLocks` → `RemovingTaints` → `StartingTarget` → `Monitoring` → `RunningPostHooks` → `Completed`. On error: `Failed`.

**Note**: `status.vmStatuses` holds per-VM detail. There is no `status.message` field and no `status.failedVMs` field on `FailoverRequest`.

### TestFailover

Triggers a non-disruptive DR validation. The controller creates an isolated test namespace (`site-recovery-<protection-group>-<random>`, reported in `status.testNamespace`) on the standby cluster, spins up VM copies from DRBD snapshots inside a network-fenced namespace, runs verification, and reports results without touching production VMs.

| Field | Default | Notes |
|---|---|---|
| `spec.protectionGroupRef.name` | — (required) | Name of the ProtectionGroup to test. |
| `spec.cleanupPolicy` | `Automatic` | `Automatic` cleans up test resources after `retentionTime`. `Manual` keeps resources until `cleanupRequested: true` or the CR is deleted. |
| `spec.retentionTime` | `2h` | How long test resources are kept before automatic cleanup. Pattern: `^[0-9]+(s\|m\|h\|d)$`. |
| `spec.cleanupRequested` | `false` | Set to `true` to trigger cleanup while keeping the TestFailover CR as a record. Can be set but never unset. |
| `spec.schedulingCheck` | `Fail` | `Fail` excludes VMs the DR scheduler cannot place and marks the test Failed. `Warn` records the verdict and drills everything. |
| `spec.timeout` | — | Optional wall-clock limit (e.g. `2h`). No default is applied by design; omit for large protection groups. |
| `spec.batchBootTimeoutSeconds` | `300` | Max seconds to wait for one boot-order batch's test VMs to reach Running. |
| `spec.skipHooks` | `false` | When `true`, all hooks (including test-verification hooks) are recorded as Skipped. |
| `spec.verification.reachability` | — | Optional list of network probes to run after automatic boot/agent/filesystem checks. |
| `spec.verification.reachability[].type` | — (required) | `tcp` or `http`. |
| `spec.verification.reachability[].port` | — (required) | TCP or HTTP port to probe. |
| `spec.verification.reachability[].vmName` | — | Restricts probe to a named VM. Empty means all test VMs. |
| `spec.verification.reachability[].endpoint` | `/` | HTTP path. Valid only for `type: http`. |
| `spec.verification.reachability[].expectedStatus` | `200` | Expected HTTP status code. Valid only for `type: http`. |
| `spec.verification.reachability[].timeout` | `30s` | Probe timeout. Maximum `10m`. |

**Phase values**: `Queued` → `PreparingEnvironment` → `CreatingSnapshots` → `CreatingVolumes` → `CreatingVMs` → `StartingVMs` → `Verifying` → `Succeeded`. On error: `Failed`. Cleanup: `CleaningUp` → `Cleaned`.

**Note**: `status.testNamespace` is controller-generated and never user-provided. Do not construct, guess, or grep for it.

### DRBDResource

A workload-cluster-side CR (one per VM per cluster) representing all of a VM's replicated disks as a single DRBD resource. Named `drbd-<namespace>-<vm>-<hash8>`. Created and managed by the TSR-Manager-side ProtectionRequest controller; you inspect it to check connection state, phase, and node placement.

| Field | Default | Notes |
|---|---|---|
| `spec.side` | — (required) | `Source` or `Target`. Immutable; does not change on failover. |
| `spec.role` | — (required) | `Primary` or `Secondary`. Set by the ProtectionRequest controller; changes on role flip. |
| `spec.port` | — (required) | TCP port for DRBD replication. Range: 7000–8000. |
| `spec.nodeName` | — | Node this copy is pinned to. Immutable once set. |
| `spec.paused` | `false` | Suspends convergence (maintenance knob). |
| `spec.replicationIntent` | `Enabled` | `Suspended` renders the resource with no connection mesh (used during unplanned failover recovery). |
| `spec.volumes[].volumeNumber` | — (required) | Stable cross-copy join key (boot disk = 0). Max 15 volumes per resource. |
| `spec.volumes[].pvcName` | — (required) | Local backing PVC name. |

**Phase values**: `Pending`, `Provisioning`, `ReadyToActivate`, `Syncing`, `Consistent`, `Degraded`, `Failed`, `Deleting`.

**Note**: `DRBDResource` is only registered on workload clusters. Running `kubectl get dres` against the TSR-Manager cluster returns `server doesn't have a resource type`.

### DRBDReplicationPolicy

Defines DRBD replication parameters. The ProtectionZone controller automatically creates one policy named `<zoneName>-default` with `isDefault: true` and Protocol C in the zone namespace. You do not create this object.

| Field | Default | Notes |
|---|---|---|
| `spec.drbdProtocol` | `C` | Wire protocol. Only `C` (synchronous) is supported in Tech Preview. |
| `spec.replicationMode` | `Synchronous` | `Synchronous` or `Asynchronous`. Only `Synchronous` is functional in Tech Preview. |
| `spec.isDefault` | `false` | Marks this as the default policy for its namespace. |
| `spec.clusters[].storageClassMappings` | — | Maps source StorageClass names to target StorageClass names. If a source class is absent, it is used as-is. |
| `spec.networkConfig.portRange` | start: 7000, end: 7999 | Accepted by the schema but never read by the port allocator. The allocator always uses 7000–8000 inclusive. |

### ReplicationGroupStatus

Auto-created observability CR per ProtectionGroup, named `<protection-group-name>-replication`. Written by the replication monitor controller; you do not create or update it.

| Status field | Notes |
|---|---|
| `status.rpoStatus` | `Met`, `Degraded`, `Violated`, or `Unknown`. Driven by `spec.rpoObjectiveSeconds` (set from `ProtectionGroup.spec.sla.rpoTargetSeconds`). |
| `status.aggregated.outOfSyncBytes` | Total out-of-sync bytes across all VMs in the group. |
| `status.aggregated.connectionState` | Worst-case connection state: `Connected`, `Degraded`, or `Disconnected`. |
| `status.volumeReplicationStatuses` | One entry per VM (not per disk). A 3-disk VM appears once. |

Filter by label `siterecovery.trilio.io/protection-group=<pg-name>`. Field selectors on `spec.*` or `status.*` do not work on custom resources.

### RPOEvent

Immutable audit record emitted by the replication monitor on each replication status transition. All fields are in `spec` (no status subresource). You do not create or update RPOEvents.

| Field | Notes |
|---|---|
| `spec.eventType` | One of: `SyncCompleted`, `RPODegraded`, `RPOViolated`, `ConnectionLost`, `ConnectionRestored`, `ResyncStarted`, `ResyncCompleted`, `VolumeAdded`, `VolumeRemoved`, `MetadataUndersized`, `ResizePending`, `ResizeComplete`. |
| `spec.violationReason` | One of: `ConnectionLost`, `NetworkTimeout`, `SyncLagExceeded`, `ProtocolCOutOfSync`, `ResyncInProgress`, `Unknown`. |

### Hook

Reusable, namespaced definition of a Kubernetes Job to execute at failover or test lifecycle points. Referenced by `ProtectionGroup.spec.hooks[].hookName`. The admission webhook prevents deletion or spec edits while an operation is using it. `Hook` has no short name; use `kubectl get hooks`.

| Field | Default | Notes |
|---|---|---|
| `spec.jobTemplate.image` | — (required) | Container image for the hook Job. Must be pullable from the executing workload cluster. |
| `spec.timeoutSeconds` | `300` | Per-attempt timeout in seconds. |
| `spec.retries` | `0` | Retry count after failure. Maximum 5. |
| `spec.retryDelaySeconds` | `5` | Pause between retry attempts in seconds. |
| `spec.jobTemplate.env` | — | Environment variables. `fieldRef` and `resourceFieldRef` sources are rejected at admission. |
| `spec.jobTemplate.volumes` | — | Permitted sources: `secret`, `configMap`, `emptyDir`, `projected`. `hostPath` is not representable. |

## Usage

All Site Recovery operations are performed by creating or patching custom resources on the TSR-Manager cluster, except `DRBDResource` inspection which is done on workload clusters.

### Check zone health

```bash
# On the TSR-Manager cluster, in the zone namespace
kubectl get pz -n dr-<zone-name>
kubectl describe pz <zone-name> -n dr-<zone-name>
```

The `ClustersValidated` condition must be `True` before the webhooks will accept `ProtectionGroup` or `ProtectionRequest` creates.

### Protect a virtual machine

1. Create a `ProtectionGroup` declaring the VM(s):

```bash
kubectl apply -f protectiongroup.yaml -n dr-<zone-name>
kubectl get pg -n dr-<zone-name>
```

2. Create a `ProtectionRequest` for each VM:

```bash
kubectl apply -f protectionrequest.yaml -n dr-<zone-name>
kubectl get pr -n dr-<zone-name> --watch
```

Watch the phase progress: `Pending` → `Validating` → `CreatingDRBD` → `Syncing` → `ReadyToActivate` → `Activating` → `Protected`.

3. Check the protected VM label on the workload cluster:

```bash
# On the workload cluster
kubectl get vm -A -l siterecovery.trilio.io/protected-vm=true
```

4. Inspect DRBD state on each workload cluster:

```bash
# On each workload cluster, in the VM's namespace
kubectl get dres -n <vm-namespace>
kubectl describe dres drbd-<namespace>-<vm>-<hash8> -n <vm-namespace>
```

### Monitor replication health

```bash
# ReplicationGroupStatus is auto-created; inspect it by label
kubectl get rgs -n dr-<zone-name> -l siterecovery.trilio.io/protection-group=<pg-name>
kubectl get rgs <pg-name>-replication -n dr-<zone-name>

# View RPO events (audit trail)
kubectl get rpoe -n dr-<zone-name> -l siterecovery.trilio.io/protection-group=<pg-name>
```

### Run a test failover

```bash
kubectl apply -f testfailover.yaml -n dr-<zone-name>
kubectl get tfo -n dr-<zone-name> --watch
```

The test namespace is reported in `status.testNamespace`. Do not construct or guess this value.

After the test completes, trigger cleanup without losing the record:

```bash
kubectl patch tfo <name> -n dr-<zone-name> \
  --type merge \
  -p '{"spec":{"cleanupRequested":true}}'
```

### Execute a planned failover

```bash
kubectl apply -f failoverrequest-planned.yaml -n dr-<zone-name>
kubectl get fr -n dr-<zone-name> --watch
```

Every failover is initiated by creating a `FailoverRequest`. Failover does not happen automatically.

### Execute an unplanned failover

```bash
kubectl apply -f failoverrequest-unplanned.yaml -n dr-<zone-name>
```

### Unprotect a VM

Delete the `ProtectionRequest`. The controller moves the VM through the `Unprotecting` phase, removes DRBD resources, and restarts the VM against its original storage.

```bash
kubectl delete pr <name> -n dr-<zone-name>
kubectl get pr <name> -n dr-<zone-name> --watch
```

### Rotate zone credentials

```bash
# Rotate and refresh RBAC if permissions have changed
tsrctl zone-credential rotate --refresh-rbac

# Rotate and revoke the previous token
tsrctl zone-credential rotate --revoke-previous
```

### Adjust runtime log level

Edit the `site-recovery-logging` ConfigMap in the zone namespace (TSR-Manager) or workload namespace (`trilio-site-recovery-system`). Changes take effect within ~15 seconds without pod restarts.

```bash
kubectl patch configmap site-recovery-logging \
  -n dr-<zone-name> \
  --type merge \
  -p '{"data":{"level":"debug","level.control-plane":"trace"}}'
```

Accepted values: `error`, `info`, `debug`, `trace`. There is no `warn` level.

## Examples

### ProtectionZone — minimal DRBD zone

```yaml
apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionZone
metadata:
  name: primary-dr-zone
  namespace: dr-primary-dr-zone
spec:
  displayName: "Primary DR Zone"
  description: "DRBD protection zone pairing primary and DR sites."
  storageBackendMode: DRBD
  clusters:
    - name: primary
      clusterID: cluster1
      distributionName: RHEL-CoreOS-9
      kubeconfigSecretRef:
        name: cluster1-kubeconfig
        key: kubeconfig
    - name: dr
      clusterID: cluster2
      distributionName: RHEL-CoreOS-9
      kubeconfigSecretRef:
        name: cluster2-kubeconfig
        key: kubeconfig
```

### ProtectionZone — with debug logging and custom node-agent deadline

```yaml
apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionZone
metadata:
  name: primary-dr-zone
  namespace: dr-primary-dr-zone
spec:
  displayName: "Primary DR Zone"
  storageBackendMode: DRBD
  clusters:
    - name: primary
      clusterID: cluster1
      distributionName: RHEL-CoreOS-9
      kubeconfigSecretRef:
        name: cluster1-kubeconfig
        key: kubeconfig
    - name: dr
      clusterID: cluster2
      distributionName: RHEL-CoreOS-9
      kubeconfigSecretRef:
        name: cluster2-kubeconfig
        key: kubeconfig
  logging:
    level: info
    controlPlaneLevel: debug
  nodeAgent:
    startupDeadlineSeconds: 180
  hooks:
    jobTTLSeconds: 7200
    failedJobRetentionSeconds: 604800
```

### ProtectionGroup — two VMs with boot ordering and SLA

```yaml
apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionGroup
metadata:
  name: app-tier
  namespace: dr-primary-dr-zone
spec:
  virtualMachines:
    - name: db-vm
      namespace: app-production
      bootOrder: 1
    - name: web-vm
      namespace: app-production
      bootOrder: 2
  desiredState: running
  sla:
    rpoTargetSeconds: 0
    rtoTargetSeconds: 300
  vmIdentity:
    macAddresses: Preserve
```

### ProtectionRequest — protect a single VM

```yaml
apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionRequest
metadata:
  name: protect-db-vm
  namespace: dr-primary-dr-zone
spec:
  sourceCluster: cluster1
  virtualMachine:
    name: db-vm
    namespace: app-production
  replicationConfig:
    replicationProtocol: C
    rpo: 0
    drbdMetadataConfig:
      storageClassName: fast-block
      size: 128Mi
```

### ProtectionRequest — with standby placement overrides

```yaml
apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionRequest
metadata:
  name: protect-web-vm
  namespace: dr-primary-dr-zone
spec:
  sourceCluster: cluster1
  virtualMachine:
    name: web-vm
    namespace: app-production
  replicationConfig:
    replicationProtocol: C
    rpo: 0
  standbyPlacement:
    nodeSelector:
      add:
        workload-type: dr-standby
      remove:
        - zone
```

### FailoverRequest — planned failover to cluster2

```yaml
apiVersion: siterecovery.trilio.io/v1alpha1
kind: FailoverRequest
metadata:
  name: planned-failover-20260801
  namespace: dr-primary-dr-zone
spec:
  protectionGroupRef:
    name: app-tier
    namespace: dr-primary-dr-zone
  targetCluster: cluster2
  failoverType: planned
  batchBootTimeoutSeconds: 300
  skipHooks: false
```

### FailoverRequest — unplanned failover (skip hooks)

```yaml
apiVersion: siterecovery.trilio.io/v1alpha1
kind: FailoverRequest
metadata:
  name: unplanned-failover-incident
  namespace: dr-primary-dr-zone
spec:
  protectionGroupRef:
    name: app-tier
    namespace: dr-primary-dr-zone
  targetCluster: cluster2
  failoverType: unplanned
  skipHooks: true
```

### TestFailover — basic DR drill with automatic cleanup

```yaml
apiVersion: siterecovery.trilio.io/v1alpha1
kind: TestFailover
metadata:
  name: drill-q3-2026
  namespace: dr-primary-dr-zone
spec:
  protectionGroupRef:
    name: app-tier
    namespace: dr-primary-dr-zone
  cleanupPolicy: Automatic
  retentionTime: 2h
  schedulingCheck: Fail
```

### TestFailover — with TCP and HTTP reachability probes

```yaml
apiVersion: siterecovery.trilio.io/v1alpha1
kind: TestFailover
metadata:
  name: drill-with-probes
  namespace: dr-primary-dr-zone
spec:
  protectionGroupRef:
    name: app-tier
    namespace: dr-primary-dr-zone
  cleanupPolicy: Manual
  schedulingCheck: Warn
  verification:
    reachability:
      - type: tcp
        port: 5432
        vmName: db-vm
        timeout: 30s
      - type: http
        port: 8080
        vmName: web-vm
        endpoint: /healthz
        expectedStatus: 200
        timeout: 30s
```

### Hook — pre-failover quiesce script

```yaml
apiVersion: siterecovery.trilio.io/v1alpha1
kind: Hook
metadata:
  name: db-quiesce
  namespace: dr-primary-dr-zone
spec:
  description: "Quiesce the database before failover"
  timeoutSeconds: 120
  retries: 2
  retryDelaySeconds: 10
  jobTemplate:
    image: quay.io/myorg/db-quiesce-tool:v1.2.3
    command: ["/scripts/quiesce.sh"]
    args: ["--timeout", "90"]
    env:
      - name: DB_HOST
        value: "db-vm.app-production.svc"
```

### ProtectionGroup — binding a Hook

```yaml
apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionGroup
metadata:
  name: app-tier
  namespace: dr-primary-dr-zone
spec:
  virtualMachines:
    - name: db-vm
      namespace: app-production
    - name: web-vm
      namespace: app-production
  desiredState: running
  sla:
    rpoTargetSeconds: 0
    rtoTargetSeconds: 300
  hooks:
    - name: pre-failover-quiesce
      hookName: db-quiesce
      type: pre-failover
      runOn: source
      onFailure: Abort
      order: 0
      vms:
        - name: db-vm
          namespace: app-production
```

### Inspect per-VM DRBD state (on the workload cluster)

```bash
# List all DRBDResources in the VM's namespace
kubectl get dres -n app-production

# Describe a specific DRBDResource
kubectl describe dres drbd-app-production-db-vm-a1b2c3d4 -n app-production
```

### Check replication group status

```bash
# On the TSR-Manager cluster
kubectl get rgs app-tier-replication -n dr-primary-dr-zone

# Watch for RPO events
kubectl get rpoe -n dr-primary-dr-zone \
  -l siterecovery.trilio.io/protection-group=app-tier \
  --sort-by=.spec.timestamp
```

### View per-VM failover status

```bash
kubectl get fr planned-failover-20260801 \
  -n dr-primary-dr-zone \
  -o jsonpath='{.status.vmStatuses}'
```

## Troubleshooting

### Zone stuck in Provisioning / ClustersValidated=False

**Symptom**: `kubectl get pz -n dr-<zone-name>` shows `Status: Provisioning` and `ClustersValidated` condition is `False`.

**Likely causes and fixes**:

| Reason | Meaning | Fix |
|---|---|---|
| `ClusterAlreadyClaimed` | A workload cluster is leased to another zone (the `siterecovery.trilio.io/owner-zone-namespace-uid` annotation on its `trilio-site-recovery-system` namespace belongs to a different zone). | Remove the other zone or use different workload clusters. |
| `InsufficientClusterPermissions` | The zone credential authenticates but fails the RBAC permission sweep. The condition message names the missing verbs. | Run `tsrctl zone-credential rotate --refresh-rbac` to issue a new credential with current RBAC. |
| `ClusterIdentityChanged` | The credential now reaches a cluster whose `kube-system` Namespace UID differs from the UID recorded at first claim. No override exists. | Restore the original cluster or re-create the zone. |
| `KubeVirtNotInstalled` | The workload cluster does not serve `kubevirt.io/v1`. | Install KubeVirt on the workload cluster before creating the zone. |
| `SCCPrerequisiteMissing` | The `privileged` SCC or its ClusterRole does not exist on the workload cluster. | Verify OpenShift is correctly installed on the workload cluster. |

---

### ProtectionRequest stuck in Syncing

**Symptom**: A `ProtectionRequest` has been in `Syncing` for an unexpectedly long time.

**Likely cause**: Initial block-level sync is still in progress. Check progress:

```bash
# On TSR-Manager
kubectl get pr <name> -n dr-<zone-name> -o jsonpath='{.status.slowestVolumeSyncProgress}'

# On the workload cluster
kubectl get dres drbd-<namespace>-<vm>-<hash8> -n <vm-namespace> \
  -o jsonpath='{.status.syncProgress}'
```

**Fix**: Wait for sync to complete. If progress is stalled at 0%, check DRBD node agent logs on both workload clusters:

```bash
# On the workload cluster — find the node agent pod for the VM's node
POD=$(kubectl get pods -n trilio-site-recovery-system \
  --field-selector spec.nodeName=<node-name> \
  -l app.kubernetes.io/component=drbd-node-agent \
  -o jsonpath='{.items[0].metadata.name}')
kubectl logs -n trilio-site-recovery-system $POD
```

Verify TCP ports 7000–8000 inclusive are open between primary and DR worker nodes. Port 8000 is mandatory; a firewall stopping at 7999 will silently prevent replication for one volume.

---

### DRBDResource in Degraded phase

**Symptom**: `kubectl get dres -n <vm-namespace>` shows `Degraded` in the Phase column.

**Likely cause**: The DRBD connection to the peer cluster is lost or the DRBD role is in conflict.

**Fix**:

1. Check the `connectionState` column—`Disconnected` or `StandAlone` indicates a network issue.
2. Verify network connectivity on TCP 7000–8000 between the two cluster's worker nodes.
3. Check if the `ProtectionRequest` on TSR-Manager has a `Ready=False` condition with reason `RoleConflict`:

```bash
kubectl describe pr <name> -n dr-<zone-name> | grep -A5 RoleConflict
```

4. Check the DRBD node agent DaemonSet is healthy on both workload clusters:

```bash
kubectl get daemonset -n trilio-site-recovery-system \
  -l app.kubernetes.io/component=drbd-node-agent
```

---

### Node agent applying not-ready taint longer than expected

**Symptom**: A node carries the `siterecovery.trilio.io/not-ready:NoSchedule` taint after startup for longer than `spec.nodeAgent.startupDeadlineSeconds`.

**Likely cause**: One or more DRBD resources on the node have not converged. After the deadline, the taint is removed anyway and the `DRBDResource` carries a `NodeStartupGate=False` condition with reason `StartupDeadlineExceeded`.

**Fix**: The condition clears itself when the resource recovers. Check what it is waiting for:

```bash
# On the workload cluster
kubectl describe dres <name> -n <vm-namespace> | grep -A5 NodeStartupGate
```

If the node-agent pod is failing to start, check its logs and verify DRBD module loading:

```bash
POD=$(kubectl get pods -n trilio-site-recovery-system \
  --field-selector spec.nodeName=<node-name> \
  -l app.kubernetes.io/component=drbd-node-agent \
  -o jsonpath='{.items[0].metadata.name}')
kubectl logs -n trilio-site-recovery-system $POD -c drbd-module-loader
```

---

### FailoverRequest stuck in InProgress

**Symptom**: A `FailoverRequest` remains `InProgress` for an extended period.

**Likely cause**: A hook is running slowly or has failed; VMs on the source cluster are not stopping; the boot-order batch timeout has not elapsed.

**Fix**: Inspect the hook statuses and per-VM statuses:

```bash
kubectl get fr <name> -n dr-<zone-name> -o jsonpath='{.status.phase}'
kubectl get fr <name> -n dr-<zone-name> -o jsonpath='{.status.vmStatuses}'
kubectl get fr <name> -n dr-<zone-name> -o jsonpath='{.status.hookStatuses}'
```

If a hook is blocking and the incident is time-critical, delete the current `FailoverRequest`, create a new one with `spec.skipHooks: true`, and investigate the hook failure afterwards.

---

### TestFailover fails with protectionGroupRef not found

**Symptom**: Creating a `TestFailover` returns an error containing `protectionGroupRef`.

**Likely cause**: The `ProtectionGroup` referenced does not exist in the namespace specified in `spec.protectionGroupRef.namespace`, or `spec.protectionGroupRef.namespace` was omitted.

**Fix**: Verify the `ProtectionGroup` exists on the TSR-Manager cluster and that `protectionGroupRef.namespace` is set to the correct zone namespace:

```bash
kubectl get pg -n dr-<zone-name>
```

Note that `spec.protectionGroupRef.namespace` must be set explicitly on a `TestFailover`; it cannot be inferred.

---

### oc adm must-gather fails to collect

**Symptom**: `oc adm must-gather` exits with an error or produces an incomplete bundle.

**Fix**: Run must-gather from the TSR-Manager cluster with the Site Recovery image. Contact Trilio support for the pinned image reference:

```bash
# Run on the TSR-Manager cluster
oc adm must-gather --image=quay.io/triliovault/site-recovery-must-gather:<release-tag>
```

The entrypoint inside the image is `/usr/bin/gather`. Do not pass a `-- <command>` override. The collector reaches through to all zone workload clusters and assembles a single read-only archive.