--- title: Creating Your First Resource product: trilio-site-recovery-for-kubernetes doc_type: guide version: 0.2.0 source: git2docs (code-derived, validation-filtered) canonical: https://docs-tsr.trilio.io/trilio-site-recovery-for-kubernetes/site-recovery/first-resource --- # Creating Your First Resource _Example walkthrough of defining a recovery plan and triggering a test recovery_ ## Overview This page walks you through defining a complete disaster-recovery plan for a KubeVirt VirtualMachine and then validating that plan with a non-disruptive test recovery. By the end you will have a ProtectionGroup grouping your VM, a ProtectionRequest driving block-level DRBD replication to your DR cluster, and a TestFailover that proves your VM can boot from a snapshot on the standby side — without touching production workloads. All operations are declared through Kubernetes custom resources in the `siterecovery.trilio.io/v1alpha1` API group and applied against the TSR-Manager cluster. ## Prerequisites Before starting this walkthrough, confirm the following are in place: - **Trilio Site Recovery installed**: The OLM-based operator is running on your TSR-Manager cluster and a ProtectionZone CR exists in a `dr-` namespace with `status.phase: Available` and the `ClustersValidated` condition set to `True`. Operator workload components (`trilio-site-recovery-system` namespace) must be healthy on both workload clusters. - **OpenShift 4.17 or later** on the TSR-Manager cluster and both workload clusters. Upstream Kubernetes workload clusters are supported on versions corresponding to OpenShift 4.17 and later. - **Two registered workload clusters** (`cluster1` and `cluster2`) in the ProtectionZone, each reachable by the zone credential created with `tsrctl zone-credential`. - **A KubeVirt VirtualMachine** (`kubevirt.io/v1`) running on the cluster you designate as `cluster1`, with at least one bound PVC backed by a StorageClass that supports block-mode volumes. - **TCP ports 7000–8000 inclusive** open between primary and DR worker nodes. Each replicated VM requires one port from this range; port 8000 is mandatory — a firewall rule that stops at 7999 will silently prevent replication on one volume. - **`kubectl`** configured with a context pointing to the TSR-Manager cluster for all TSR-Manager CR operations, and separate contexts for each workload cluster when inspecting workload-side resources. - **`oc`** (OpenShift CLI) available for must-gather support bundle collection if needed. - **`tsrctl`** available — supplied by Trilio with the release. Contact Trilio support to obtain it. - **The VM is not currently live-migrating.** Protected VMs are pinned to one node per cluster for the lifetime of protection; live migration and cold migration are denied once protection is active. ## Installation There is no additional installation step for this walkthrough — the operator and workload stacks are already deployed by the ProtectionZone controller. The steps below create the application-layer custom resources that define your recovery plan. **Step 1 — Confirm zone readiness** Before creating any protection resources, verify that the ProtectionZone has validated both clusters. All CRs in this guide are created in the zone's namespace on the TSR-Manager cluster (e.g. `dr-prod`). Substitute your actual zone namespace throughout. ```bash # Run against the TSR-Manager cluster kubectl get pz -n dr-prod ``` The `STATUS` column must show `Available`. If it shows `Degraded` or `Unavailable`, inspect the zone conditions before continuing: ```bash kubectl describe pz -n dr-prod ``` Look for the `ClustersValidated` condition. It must be `True` — the TSR-Manager webhooks will reject ProtectionGroup and ProtectionRequest creates until it is. **Step 2 — Create a ProtectionGroup** A ProtectionGroup bundles one or more KubeVirt VMs for coordinated failover. Create the following manifest and apply it to the TSR-Manager cluster in your zone namespace. ```yaml apiVersion: siterecovery.trilio.io/v1alpha1 kind: ProtectionGroup metadata: name: my-app-pg namespace: dr-prod spec: displayName: "My Application Protection Group" virtualMachines: - name: my-app-vm namespace: my-app-namespace bootOrder: 1 desiredState: running sourceCluster: cluster1 sla: rtoTargetSeconds: 300 rpoTargetSeconds: 0 ``` ```bash kubectl apply -f protectiongroup.yaml ``` Key fields: - `spec.virtualMachines[].namespace` — the namespace where the VM lives on the workload cluster. - `spec.sourceCluster` — the cluster ID (`cluster1` or `cluster2`) where the VM currently runs. - `spec.sla.rtoTargetSeconds` — required field; sets your recovery time target for compliance reporting. - `spec.sla.rpoTargetSeconds: 0` — correct for synchronous Protocol C replication (RPO=0 is the only supported mode). **Step 3 — Create a ProtectionRequest** A ProtectionRequest declares protection for a single VM. The controller provisions a DRBD resource pair (one CR per workload cluster), wires the VM to DRBD-backed frontend PVCs, and drives the VM through to `Protected` phase. ```yaml apiVersion: siterecovery.trilio.io/v1alpha1 kind: ProtectionRequest metadata: name: protect-my-app-vm namespace: dr-prod spec: sourceCluster: cluster1 virtualMachine: name: my-app-vm namespace: my-app-namespace replicationConfig: replicationProtocol: C rpo: 0 drbdMetadataConfig: storageClassName: ocs-storagecluster-ceph-rbd size: "128Mi" ``` ```bash kubectl apply -f protectionrequest.yaml ``` Key fields: - `spec.sourceCluster` — must match the cluster where the VM runs; also acts as the failover lever (patching it triggers a role flip). - `spec.replicationConfig.replicationProtocol: C` — Protocol C synchronous replication is the only supported mode. Protocol A is rejected by the admission webhook. - `spec.replicationConfig.drbdMetadataConfig.storageClassName` — the StorageClass used for per-volume DRBD external-metadata PVCs. Must bind immediately; `WaitForFirstConsumer` classes are rejected during validation. Watch the ProtectionRequest progress through its phases: ```bash kubectl get pr protect-my-app-vm -n dr-prod -w ``` The phase sequence is: `Pending` → `Validating` → `CreatingDRBD` → `Syncing` → `ReadyToActivate` → `Activating` → `Protected`. Initial sync duration depends on the volume size. The `Sync` column shows percentage completion. ## Configuration The custom resources in this walkthrough expose the following key configuration fields. All fields below are in the `siterecovery.trilio.io/v1alpha1` API group. ### ProtectionGroup | Field | Type | Default | Effect | |---|---|---|---| | `spec.virtualMachines[].bootOrder` | integer | — | Lower values boot first during failover. VMs sharing a value boot in parallel. VMs without a value boot together in a final batch. | | `spec.desiredState` | `running` \| `stopped` | `running` | Declares whether VMs should be running or stopped on the active cluster. | | `spec.sourceCluster` | `cluster1` \| `cluster2` | — | Records the protect-time home cluster; the failover controller reads this to classify a failover direction (failover vs. failback) for hook selection. | | `spec.sla.rtoTargetSeconds` | integer | required | Recovery time objective in seconds, recorded for compliance reporting. | | `spec.sla.rpoTargetSeconds` | integer | `0` | Feeds the replication monitor's RPO objective (`ReplicationGroupStatus.spec.rpoObjectiveSeconds`). Must be `0` for Protocol C synchronous replication. | | `spec.hooks` | array | — | Binds reusable Hook definitions to lifecycle points (`pre-failover`, `post-failover`, `pre-test`, `post-test`, `test-verification`, etc.). Up to 32 hooks. | | `spec.vmIdentity.macAddresses` | `Preserve` \| `Clear` | `Preserve` | Controls whether interface MAC addresses are preserved or cleared on the DR copy. | ### ProtectionRequest | Field | Type | Default | Effect | |---|---|---|---| | `spec.sourceCluster` | `cluster1` \| `cluster2` | required | The cluster where the VM currently runs. **This is the failover lever**: patching it to the DR cluster triggers the DRBD role flip that constitutes a failover for this VM. | | `spec.replicationConfig.replicationProtocol` | `C` | `C` | DRBD wire protocol. Only `C` (synchronous, RPO=0) is supported. `A` is rejected at admission. | | `spec.replicationConfig.drbdMetadataConfig.size` | quantity | `128Mi` | Size of the per-volume DRBD external-metadata PVC. 128 Mi accommodates backing volumes up to 4 TiB without resizing. | | `spec.standbyPlacement` | object | — | Overrides node-level scheduling rules on the DR copy only (node selector, node affinity, tolerations). Immutable once set; unprotect and re-protect to change. | ### TestFailover | Field | Type | Default | Effect | |---|---|---|---| | `spec.cleanupPolicy` | `Manual` \| `Automatic` | `Automatic` | `Automatic` removes test resources after `retentionTime` elapses. `Manual` keeps them until `cleanupRequested: true` is set or the CR is deleted. | | `spec.retentionTime` | duration string | `2h` | How long test resources are kept before automatic cleanup (e.g. `30m`, `4h`). | | `spec.timeout` | duration string | — | Maximum wall-clock duration before the test is marked `Failed`. Leave unset to avoid spurious failures on large protection groups. | | `spec.verification.reachability` | array | — | Optional per-VM TCP or HTTP reachability probes run after the automatic boot/agent/filesystem checks. | | `spec.schedulingCheck` | `Fail` \| `Warn` | `Fail` | What to do when the DR scheduler cannot place a member VM: `Fail` marks the test failed; `Warn` continues and records the verdict. | | `spec.skipHooks` | boolean | `false` | When `true`, all hooks (including test-verification hooks) are recorded as Skipped rather than executed. | ### DRBDReplicationPolicy (auto-managed) The ProtectionZone controller automatically creates one default policy named `-default` with `isDefault: true` in the zone namespace. You do not create or modify this object. The controller selects it for all ProtectionRequests in the namespace. Protocol C synchronous replication is always used. ## Usage ### Monitoring protection status After applying a ProtectionRequest, track progress on the TSR-Manager cluster: ```bash # Summary view — Phase and Sync columns kubectl get pr -n dr-prod # Detailed status including conditions and sync progress kubectl describe pr protect-my-app-vm -n dr-prod ``` The `status.slowestVolumeSyncProgress.percentage` field shows the percentage of initial sync complete for the slowest volume. Once all volumes reach 100% sync, the phase advances to `Protected`. To inspect the DRBD resource on a workload cluster, note that `DRBDResource` CRs live on the workload clusters (not on the TSR-Manager), in the VM's namespace: ```bash # Run against the primary workload cluster kubectl get dres -n my-app-namespace # Detailed view — connection state, phase, sync progress kubectl describe dres -n my-app-namespace ``` DRBD resource names follow the pattern `drbd---`. To identify which VMs are protected on a workload cluster: ```bash kubectl get vm -A -l siterecovery.trilio.io/protected-vm=true ``` ### Monitoring replication health The replication monitor auto-creates a `ReplicationGroupStatus` CR per ProtectionGroup, named `-replication`: ```bash kubectl get rgs my-app-pg-replication -n dr-prod kubectl describe rgs my-app-pg-replication -n dr-prod ``` Key status fields: - `status.rpoStatus` — `Met`, `Degraded`, `Violated`, or `Unknown`. - `status.aggregated.connectionState` — worst-case DRBD connection state across all VMs in the group. - `status.aggregated.outOfSyncBytes` — total unsynced bytes; should be `0` in steady state with Protocol C. - `status.lastFullySyncedAt` — when every VM in the group was last fully in sync. RPO events (status transitions such as sync completed, connection lost, RPO violated) are recorded as immutable `RPOEvent` CRs: ```bash kubectl get rpoe -n dr-prod -l siterecovery.trilio.io/protection-group=my-app-pg ``` ### Running a test failover A TestFailover validates DR readiness without disrupting production. The controller spins up isolated VM copies on the standby cluster from DRBD snapshots, inside a controller-generated namespace fenced by a default-deny NetworkPolicy. The test namespace name is reported in `status.testNamespace` after the CR is created — it is generated by the controller and never user-provided. Create a TestFailover CR on the TSR-Manager cluster: ```yaml apiVersion: siterecovery.trilio.io/v1alpha1 kind: TestFailover metadata: name: my-app-pg-test-01 namespace: dr-prod spec: protectionGroupRef: name: my-app-pg cleanupPolicy: Automatic retentionTime: 2h schedulingCheck: Fail ``` ```bash kubectl apply -f testfailover.yaml ``` Monitor progress: ```bash kubectl get tfo my-app-pg-test-01 -n dr-prod -w ``` The phase sequence is: `Queued` → `PreparingEnvironment` → `CreatingSnapshots` → `CreatingVolumes` → `CreatingVMs` → `StartingVMs` → `Verifying` → `Succeeded` (or `Failed`). The controller selects the target cluster automatically (the cluster the production VMs are **not** running on) and records its reasoning in `status.targetClusterReason`. After the test completes, inspect the result: ```bash kubectl describe tfo my-app-pg-test-01 -n dr-prod ``` Check `status.result` (`Succeeded` or `Failed`), `status.verificationResults.passed`, and `status.schedulingCheck.verdict`. With `cleanupPolicy: Automatic`, test resources are removed after the `retentionTime` window. To trigger cleanup immediately without deleting the record: ```bash kubectl patch tfo my-app-pg-test-01 -n dr-prod \ --type merge -p '{"spec":{"cleanupRequested":true}}' ``` To delete the CR and all associated test resources: ```bash kubectl delete tfo my-app-pg-test-01 -n dr-prod ``` ## Examples ### Example 1 — Minimal ProtectionGroup for a single VM ```yaml apiVersion: siterecovery.trilio.io/v1alpha1 kind: ProtectionGroup metadata: name: my-app-pg namespace: dr-prod spec: displayName: "My Application" virtualMachines: - name: my-app-vm namespace: my-app-namespace desiredState: running sourceCluster: cluster1 sla: rtoTargetSeconds: 300 rpoTargetSeconds: 0 ``` ### Example 2 — ProtectionRequest with custom metadata storage This example specifies a dedicated StorageClass for DRBD external-metadata PVCs and a larger metadata volume to accommodate a large boot disk. ```yaml apiVersion: siterecovery.trilio.io/v1alpha1 kind: ProtectionRequest metadata: name: protect-my-app-vm namespace: dr-prod spec: sourceCluster: cluster1 virtualMachine: name: my-app-vm namespace: my-app-namespace replicationConfig: replicationProtocol: C rpo: 0 drbdMetadataConfig: storageClassName: ocs-storagecluster-ceph-rbd size: "256Mi" ``` ### Example 3 — ProtectionGroup with boot ordering for a multi-tier application Here the database VM boots first (order 1), then the application server (order 2), allowing you to control failover sequencing. ```yaml apiVersion: siterecovery.trilio.io/v1alpha1 kind: ProtectionGroup metadata: name: my-app-tiered-pg namespace: dr-prod spec: displayName: "Tiered App — DB first" virtualMachines: - name: my-db-vm namespace: my-app-namespace bootOrder: 1 - name: my-app-vm namespace: my-app-namespace bootOrder: 2 desiredState: running sourceCluster: cluster1 sla: rtoTargetSeconds: 600 rpoTargetSeconds: 0 ``` ### Example 4 — TestFailover with reachability probes and manual cleanup This test adds HTTP reachability verification against port 8080 of a specific VM, and keeps test resources available for manual inspection until you explicitly request cleanup. ```yaml apiVersion: siterecovery.trilio.io/v1alpha1 kind: TestFailover metadata: name: my-app-pg-drill-01 namespace: dr-prod spec: protectionGroupRef: name: my-app-tiered-pg cleanupPolicy: Manual retentionTime: 4h schedulingCheck: Fail verification: reachability: - type: http vmName: my-app-vm port: 8080 endpoint: /healthz expectedStatus: 200 timeout: 30s - type: tcp vmName: my-db-vm port: 5432 timeout: 15s ``` ```bash # Apply the test kubectl apply -f testfailover-drill.yaml # Watch phases kubectl get tfo my-app-pg-drill-01 -n dr-prod -w # After Succeeded — find the generated test namespace kubectl get tfo my-app-pg-drill-01 -n dr-prod \ -o jsonpath='{.status.testNamespace}' # Check verification results kubectl get tfo my-app-pg-drill-01 -n dr-prod \ -o jsonpath='{.status.verificationResults}' # When done inspecting, request cleanup without deleting the record kubectl patch tfo my-app-pg-drill-01 -n dr-prod \ --type merge -p '{"spec":{"cleanupRequested":true}}' ``` ### Example 5 — Checking replication health for a ProtectionGroup ```bash # List the auto-created ReplicationGroupStatus (short name: rgs) kubectl get rgs -n dr-prod # Detailed metrics — out-of-sync bytes, connection state, RPO status kubectl describe rgs my-app-pg-replication -n dr-prod # RPO audit events for this group (filter by label) kubectl get rpoe -n dr-prod \ -l siterecovery.trilio.io/protection-group=my-app-pg ``` ### Example 6 — Inspect DRBD resources on a workload cluster ```bash # Run against the primary workload cluster context kubectl get dres -n my-app-namespace \ --context # Run against the DR workload cluster context kubectl get dres -n my-app-namespace \ --context ``` Each DRBDResource CR is named `drbd---`. The `kubectl get` printer columns show `Side`, `Role`, `Node`, `State`, and `Phase`. ### Example 7 — Adjusting the runtime log level without restarting pods Edit the `site-recovery-logging` ConfigMap in the zone namespace on the TSR-Manager cluster to increase verbosity for the control plane: ```bash kubectl patch configmap site-recovery-logging -n dr-prod \ --type merge \ -p '{"data":{"level":"info","level.control-plane":"debug"}}' ``` Accepted values: `error`, `info`, `debug`, `trace`. The change takes effect within approximately 15 seconds without pod restarts. ## Troubleshooting Use the following as a reference for common issues you may encounter when creating protection resources or running a test recovery. --- ### ProtectionRequest stuck in `Validating` phase **Symptom:** `kubectl get pr` shows `Phase: Validating` for more than a few minutes. **Likely causes:** 1. The `ClustersValidated` condition on the ProtectionZone is not `True`. 2. The VM does not exist in the namespace specified, or the namespace does not exist on the workload cluster. 3. The VM's PVCs use a `WaitForFirstConsumer` StorageClass for metadata — this is rejected during validation. **Fix:** ```bash # Check zone condition kubectl describe pz -n dr-prod | grep -A5 ClustersValidated # Check ProtectionRequest conditions for a specific error message kubectl describe pr protect-my-app-vm -n dr-prod # Verify the VM exists on the primary workload cluster kubectl get vm my-app-vm -n my-app-namespace --context ``` --- ### ProtectionRequest reaches `Failed` phase **Symptom:** `kubectl get pr` shows `Phase: Failed`. **Likely causes:** 1. DRBD port exhaustion — the zone already has 1,000 protected VMs (one port per VM, range 7000–8000). 2. The specified metadata StorageClass does not exist or cannot provision immediately. 3. The DR cluster's drbd-node-agent DaemonSet is not healthy. **Fix:** ```bash # Check ProtectionRequest conditions for the failure reason kubectl describe pr protect-my-app-vm -n dr-prod # Check the drbd-node-agent DaemonSet on the DR workload cluster kubectl get daemonset -n trilio-site-recovery-system \ -l app.kubernetes.io/component=drbd-node-agent \ --context # Check node agent pod logs on the DR cluster (two-step: get pod name, then logs) kubectl get pods -n trilio-site-recovery-system \ --field-selector spec.nodeName= \ -l app.kubernetes.io/component=drbd-node-agent \ -o jsonpath='{.items[0].metadata.name}' \ --context kubectl logs -n trilio-site-recovery-system \ --context ``` --- ### ProtectionGroup webhooks reject ProtectionRequest create **Symptom:** `kubectl apply` returns an admission error such as "ClustersValidated is not True". **Likely cause:** The ProtectionZone's `ClustersValidated` condition is `False` or `Unknown`. This gates all ProtectionGroup and ProtectionRequest creates and spec changes. **Fix:** Resolve the zone condition first. ```bash kubectl describe pz -n dr-prod ``` Common reasons and their resolutions: - `InsufficientClusterPermissions` — rotate the zone credential with `tsrctl zone-credential rotate --refresh-rbac`. - `ClusterAlreadyClaimed` — the workload cluster is already registered to another zone. - `ClusterIdentityChanged` — the credential now reaches a different cluster (different `kube-system` UID). There is no override; restore the original cluster or re-create the zone. - `KubeVirtNotInstalled` — install KubeVirt on the workload cluster before proceeding. --- ### TestFailover stuck in `PreparingEnvironment` **Symptom:** `kubectl get tfo` shows `Phase: PreparingEnvironment` for an extended period. **Likely cause:** The controller could not create the fenced test namespace or apply the default-deny NetworkPolicy on the target workload cluster. This may indicate a permissions issue or an unhealthy workload control plane. **Fix:** ```bash # Check TestFailover conditions kubectl describe tfo my-app-pg-test-01 -n dr-prod # Check the workload control plane manager on the DR cluster kubectl get pods -n trilio-site-recovery-system \ -l app.kubernetes.io/component=workload-control-plane \ --context ``` --- ### TestFailover shows `Succeeded` but `verificationResults.passed` is `false` **Symptom:** The test phase is `Succeeded` but `status.verificationResults.passed` is `false`. **Likely cause:** One or more reachability probes failed (the VM booted but was not reachable on the probed port within the timeout), or a `test-verification` hook returned a non-zero exit code. **Fix:** ```bash # Inspect verification detail kubectl get tfo my-app-pg-test-01 -n dr-prod \ -o jsonpath='{.status.verificationResults.checks}' # Find the generated test namespace (controller-assigned) kubectl get tfo my-app-pg-test-01 -n dr-prod \ -o jsonpath='{.status.testNamespace}' # List test VMs in the fenced namespace (on DR workload cluster) kubectl get vm -n --context ``` If the VMs booted but the reachability probe timed out, check that the probed service is running inside the guest and that the port is not blocked by a NetworkPolicy inside the test namespace. --- ### DRBD replication shows `status.aggregated.connectionState: Disconnected` **Symptom:** `kubectl get rgs my-app-pg-replication -n dr-prod` shows `Connection: Disconnected`. **Likely cause:** Network connectivity between primary and DR worker nodes is disrupted on TCP ports 7000–8000. A firewall rule that stops at 7999 will silently leave one VM unable to replicate. **Fix:** ```bash # Check which VM's DRBD resource is disconnected kubectl describe rgs my-app-pg-replication -n dr-prod # Inspect the DRBDResource on the primary workload cluster kubectl get dres -n my-app-namespace --context kubectl describe dres -n my-app-namespace \ --context # Check recent RPO events for connection loss kubectl get rpoe -n dr-prod \ -l siterecovery.trilio.io/protection-group=my-app-pg ``` Verify that TCP ports 7000–8000 inclusive are open in both directions between the worker nodes. Port 8000 is not optional. --- ### Collecting a support bundle If you cannot resolve an issue, collect a diagnostic bundle from the TSR-Manager cluster (it automatically reaches through to all zone workload clusters): ```bash oc adm must-gather \ --image=quay.io/triliovault/site-recovery-must-gather: ``` For a narrower collection window: ```bash oc adm must-gather \ --image=quay.io/triliovault/site-recovery-must-gather: \ -- /usr/bin/gather --since 2h ``` For a single zone only: ```bash oc adm must-gather \ --image=quay.io/triliovault/site-recovery-must-gather: \ -- /usr/bin/gather --zone dr-prod ``` Replace `` with the pinned image tag supplied with your release. The bundle includes logs, CRs, and DRBD state from all clusters. Secret data and kubeconfigs are never written into the archive.