--- title: Webhook Issues product: trilio-site-recovery-for-kubernetes doc_type: guide version: 0.2.0 source: git2docs (code-derived, validation-filtered) canonical: https://docs-tsr.trilio.io/trilio-site-recovery-for-kubernetes/site-recovery/troubleshooting-webhooks --- # Webhook Issues _Admission webhook failures and validation errors on recovery resource creation_ ## Overview This page explains the admission webhook failures and validation errors you may encounter when creating or updating Site Recovery custom resources. Trilio Site Recovery runs validating and mutating webhooks on the TSR-Manager cluster for every resource in the `siterecovery.trilio.io/v1alpha1` API group; a rejected request returns an immediate error with a structured field path that identifies the exact problem. Understanding the webhook architecture and the rules each webhook enforces lets you resolve admission errors without trial-and-error. ## Prerequisites Before using this guide, ensure you have: - A working TSR-Manager cluster with the Site Recovery operator installed via OLM (package `site-recovery`, channel `tech-preview`, catalog `quay.io/triliovault/site-recovery-catalog`) - `kubectl` (or `oc`) configured to target the TSR-Manager cluster - A `ProtectionZone` CR in the `dr-` namespace with `status.phase: Available` — the `ClustersValidated` condition on the ProtectionZone must be `True` before the webhooks will accept new `ProtectionGroup` or `ProtectionRequest` creates or spec changes - OpenShift 4.17 or later on both the TSR-Manager cluster and all workload clusters - Basic familiarity with `kubectl` and Kubernetes admission webhooks ## Installation The admission webhooks are installed automatically. You do not create, rotate, or modify any webhook serving certificate yourself. **How webhook certificates are provisioned** The ProtectionZone controller provisions the `webhook-server-cert` Secret before deploying each control plane stack. It re-issues the Secret whenever the certificate is unparsable, missing a required Subject Alternative Name, or approaching expiry. No manual certificate rotation is needed and no `openssl` procedure applies here. **Webhook serving configuration** | Parameter | Value | |---|---| | `webhook.port` | `443` | | `webhook.containerPort` | `9443` | | `webhook.certSecretName` | `webhook-server-cert` | | `webhook.enabled` | `true` (set `enableWebhooks: false` only for local debugging) | **Verifying the webhooks are registered** Run the following against the TSR-Manager cluster to confirm all webhook configurations are present: ```bash kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations \ -l app.kubernetes.io/part-of=site-recovery ``` The manager control-plane pod (running in the `dr-` namespace) hosts all TSR-Manager-side webhooks. Verify it is running: ```bash kubectl get pods -n dr- \ -l app.kubernetes.io/component=manager-control-plane ``` Expected output shows one pod with `1/1` ready containers, because the manager Deployment runs a single container named `manager`. The workload control-plane pod (running in `trilio-site-recovery-system` on each workload cluster) hosts the DRBDResource guard and VM admission webhooks. Verify it on each workload cluster: ```bash kubectl get pods -n trilio-site-recovery-system \ -l app.kubernetes.io/component=workload-control-plane ``` ## Configuration The webhooks enforce rules at two levels: the CRD schema (enforced by the API server before the webhook runs) and cross-field or cross-resource rules (enforced by the webhook itself). You cannot configure which rules are active; all webhooks run with `failurePolicy: Fail`, so a webhook server that is unreachable causes the request to be rejected. ### ProtectionZone webhook **Mutating webhook path:** `/mutate-siterecovery-trilio-io-v1alpha1-protectionzone` **Validating webhook path:** `/validate-siterecovery-trilio-io-v1alpha1-protectionzone` **Verbs covered:** `create`, `update` The mutating webhook defaults `spec.storageBackendMode` to `DRBD` when the field is omitted, because the field is schema-required with no schema-level default. The validating webhook enforces rules the schema cannot express: | Rule | Error field | Error message excerpt | |---|---|---| | `storageBackendMode: Linstor` is rejected | `spec.storageBackendMode` | `Linstor backend is not supported yet; use DRBD` | | `kubeconfigSecretRef.name` must not be empty | `spec.clusters[i].kubeconfigSecretRef.name` | `must name the Secret holding this cluster's kubeconfig` | | `kubeconfigSecretRef.key` must not be empty | `spec.clusters[i].kubeconfigSecretRef.key` | `must name the key in the Secret that holds the kubeconfig` | | Both clusters must use different `clusterID` slots | `spec.clusters[i].clusterID` | `Duplicate value` | | On UPDATE: `storageBackendMode`, cluster `name`, and cluster `clusterID` are immutable | varies | `field is immutable` | | On UPDATE: mutable fields are `displayName`, `description`, each cluster's `kubeconfigSecretRef`, and each cluster's `distributionName` | — | — | **`distributionName` enum and default:** | Value | Meaning | |---|---| | `RHEL-CoreOS-9` (default) | RHEL CoreOS 9 nodes; selects the `drbd9-rhel9` module-loader image | | `RHEL-CoreOS-10` | RHEL CoreOS 10 nodes; not validated in Tech Preview | **`clusterID` enum:** `cluster1` or `cluster2` — both entries must occupy different slots. --- ### ProtectionRequest webhook **Validating webhook path:** `/validate-siterecovery-trilio-io-v1alpha1-protectionrequest` **Verbs covered:** `create`, `update`, `delete` There is no mutating webhook for ProtectionRequest. The `spec.replicationConfig.replicationProtocol` field defaults to `C` and `spec.replicationConfig.rpo` defaults to `0` via CRD schema markers. | Rule | Error field | Error message excerpt | |---|---|---| | Zone must be Available (`ClustersValidated: True`) before create or spec-change | — | admission refused until zone is ready | | `spec.virtualMachine.namespace` must be set explicitly | `spec.virtualMachine.namespace` | `must be set explicitly` | | `replicationProtocol: A` is rejected (Tech Preview) | `spec.replicationConfig.replicationProtocol` | `ReplicationProtocolA is not supported yet` | | `replicationProtocol: C` with `rpo != 0` is rejected | `spec.replicationConfig.rpo` | `rpo must be 0 when spec.replicationConfig.replicationProtocol is C` | | `drbdMetadataConfig.size` below 128Mi is rejected | `spec.replicationConfig.drbdMetadataConfig.size` | `size must be at least 128Mi` | | One ProtectionRequest per VM per tenant namespace (including Terminating ones) | — | duplicate VM reference | | Delete blocked while a FailoverRequest or TestFailover is in-flight for a PG that contains this VM | — | in-flight operation reference | **Key defaults:** | Field | Default | |---|---| | `spec.replicationConfig.replicationProtocol` | `C` | | `spec.replicationConfig.rpo` | `0` | | `spec.replicationConfig.drbdMetadataConfig.size` | `128Mi` | --- ### ProtectionGroup webhook **Mutating webhook path:** (ProtectionGroup mutating — derives `spec.sourceCluster`) **Verbs covered:** `create`, `update` The mutating webhook derives `spec.sourceCluster` from the member VMs' Protected ProtectionRequests when the field is omitted. If an explicit value is provided and disagrees with the members' ProtectionRequests, the create is denied. | Rule | Error field | Error message excerpt | |---|---|---| | Zone must be Available before create or spec-change | — | admission refused until zone is ready | | `spec.sourceCluster` must agree with members' ProtectionRequests | `spec.sourceCluster` | `disagrees with the source cluster recorded on the members' ProtectionRequests` | | Members' ProtectionRequests must not span both cluster sides | `spec.sourceCluster` | `cannot span both sides` | | `spec.sourceCluster` cannot be derived when no member has a Protected ProtectionRequest | `spec.sourceCluster` | `no member VM is backed by a Protected ProtectionRequest` | | Each VM in `spec.hooks[].vms` must reference a member of the ProtectionGroup | varies | — | --- ### TestFailover webhook **Mutating webhook path:** `/mutate-siterecovery-trilio-io-v1alpha1-testfailover` **Validating webhook path:** `/validate-siterecovery-trilio-io-v1alpha1-testfailover` **Verbs covered:** `create`, `update`, `delete` The mutating webhook defaults `spec.protectionGroupRef.namespace` from the TestFailover's own namespace when omitted. | Rule | Error field | Error message excerpt | |---|---|---| | `spec.protectionGroupRef.namespace` must be set | `spec.protectionGroupRef.namespace` | `must be set explicitly: cannot be empty` | | Referenced ProtectionGroup must exist | `spec.protectionGroupRef` | `not found` | | `spec.cleanupPolicy` must be `Manual` or `Automatic` | `spec.cleanupPolicy` | schema enum rejection | | `spec.schedulingCheck` defaults to `Fail` | — | — | | A Pending or InProgress FailoverRequest for the same PG blocks TestFailover create | — | `FailoverRequest` | | TCP reachability probes must not include `endpoint` or `expectedStatus` | `spec.verification.reachability[i].endpoint` or `.expectedStatus` | `must not be provided for tcp reachability probes` | | Reachability probe `timeout` must not exceed 10 minutes | `spec.verification.reachability[i].timeout` | `probe timeout must not exceed 10m` | | `spec.verification.reachability[i].vmName` must reference a member VM of the ProtectionGroup | `spec.verification.reachability[i].vmName` | `is not a virtualMachine of ProtectionGroup` | --- ### RPOEvent webhook **Mutating webhook path:** `/mutate-siterecovery-trilio-io-v1alpha1-rpoevent` **Validating webhook path:** `/validate-siterecovery-trilio-io-v1alpha1-rpoevent` **Verbs covered:** `create`, `update`, `delete` RPOEvent is an immutable audit record. Only the replication-monitor service account may create or delete RPOEvents when the operator is configured with `monitorServiceAccountName`. | Rule | Error field | Error message excerpt | |---|---|---| | `spec.protectionGroupRef.namespace` must be set | `spec.protectionGroupRef.namespace` | `must be set explicitly` | | Referenced ProtectionGroup must exist | `spec.protectionGroupRef` | `not found` | | All updates are rejected | — | `RPOEvent is immutable; updates are not permitted` | | Create and delete restricted to the monitor service account (when configured) | — | `only the replication-monitor service account may create RPOEvents` | --- ### ReplicationGroupStatus webhook **Mutating webhook path:** `/mutate-siterecovery-trilio-io-v1alpha1-replicationgroupstatus` **Validating webhook path:** `/validate-siterecovery-trilio-io-v1alpha1-replicationgroupstatus` **Verbs covered:** `create`, `update`, `delete` ReplicationGroupStatus objects are auto-created by the replication monitor controller and are named `-replication`. You do not create these manually. | Rule | Error field | Error message excerpt | |---|---|---| | `spec.protectionGroupRef.namespace` must be set | `spec.protectionGroupRef.namespace` | `must be set explicitly` | | Referenced ProtectionGroup must exist | `spec.protectionGroupRef` | `not found` | | `spec.protectionGroupRef` is immutable once set | `spec.protectionGroupRef` | `protectionGroupRef is immutable once set` | | `spec.replicationProtocol` is immutable once set | `spec.replicationProtocol` | `replicationProtocol is immutable once set` | | `spec.pollingIntervalSeconds` must be between 5 and 300 | `spec.pollingIntervalSeconds` | schema bounds rejection | | Create, update, and delete restricted to the monitor service account (when configured) | — | `only service account ... may manage ReplicationGroupStatus objects` | ## Usage ### Checking zone readiness before creating protected resources The `ClustersValidated` condition on the ProtectionZone must be `True` before the webhooks will admit new `ProtectionGroup` or `ProtectionRequest` creates or spec changes. Check zone status first: ```bash kubectl get pz -n dr- ``` Inspect the conditions when the zone is not `Available`: ```bash kubectl describe pz -n dr- ``` Look for the `ClustersValidated` condition in the output. Common reasons that gate admission: | Condition reason | Meaning | |---|---| | `ClusterAlreadyClaimed` | The workload cluster is leased to another zone | | `InsufficientClusterPermissions` | The zone credential lacks required RBAC; rotate with `tsrctl zone-credential rotate --refresh-rbac` | | `ClusterIdentityChanged` | The credential now reaches a different cluster than the one originally registered; no override exists — restore the original cluster or re-create the zone | | `KubeVirtNotInstalled` | KubeVirt is not present on the workload cluster | | `SCCPrerequisiteMissing` | Required SCC is absent | --- ### Creating a ProtectionRequest The ProtectionRequest webhook enforces that `spec.virtualMachine.namespace` is always set explicitly, because the VM lives on a workload cluster in its application namespace — that namespace is unrelated to the TSR-Manager tenant namespace and cannot be inferred. ```yaml apiVersion: siterecovery.trilio.io/v1alpha1 kind: ProtectionRequest metadata: name: vm-web-protect namespace: dr- spec: sourceCluster: cluster1 virtualMachine: name: web-vm namespace: production # required; cannot be omitted replicationConfig: replicationProtocol: C # only supported value rpo: 0 # must be 0 with Protocol C ``` Apply it: ```bash kubectl apply -f protectionrequest.yaml ``` The webhook rejects the request immediately with a structured error if any field violates the rules. For example, supplying `replicationProtocol: A` produces: ``` Error from server: admission webhook "vprotectionrequest-v1alpha1.siterecovery.trilio.io" denied the request: spec.replicationConfig.replicationProtocol: Invalid value: "A": ReplicationProtocolA is not supported yet ``` --- ### Creating a ProtectionZone The validating webhook rejects `storageBackendMode: Linstor` at admission. Always use `DRBD`: ```yaml apiVersion: siterecovery.trilio.io/v1alpha1 kind: ProtectionZone metadata: name: prod-dr namespace: dr-prod-dr spec: displayName: Production DR Zone storageBackendMode: DRBD clusters: - name: cluster1 clusterID: cluster1 kubeconfigSecretRef: name: cluster1-kubeconfig key: kubeconfig - name: cluster2 clusterID: cluster2 kubeconfigSecretRef: name: cluster2-kubeconfig key: kubeconfig ``` If you omit `storageBackendMode`, the mutating webhook defaults it to `DRBD` before validation runs. --- ### Creating a TestFailover The mutating webhook defaults `spec.protectionGroupRef.namespace` from the TestFailover's own namespace when omitted. The validating webhook then checks that the referenced ProtectionGroup actually exists and that no Pending or InProgress FailoverRequest is already running for the same PG. ```yaml apiVersion: siterecovery.trilio.io/v1alpha1 kind: TestFailover metadata: name: quarterly-drill namespace: dr- spec: protectionGroupRef: name: pg-web-tier namespace: dr- # can be omitted; defaulted from TFO namespace cleanupPolicy: Automatic schedulingCheck: Fail ``` To add reachability verification probes, note that `endpoint` and `expectedStatus` are only valid for `http` probes, not `tcp`: ```yaml verification: reachability: - type: tcp port: 22 timeout: 30s - type: http port: 8080 endpoint: /health expectedStatus: 200 timeout: 60s ``` A `timeout` value greater than `10m` is rejected at admission. --- ### Checking ProtectionRequest phase after admission Once a ProtectionRequest is admitted, follow its phase through the controller lifecycle: ```bash kubectl get pr -n dr- -w ``` Valid phases in order: `Pending` → `Validating` → `CreatingDRBD` → `Syncing` → `ReadyToActivate` → `Activating` → `Protected`. If the request reaches `Failed`, inspect conditions: ```bash kubectl describe pr -n dr- ``` Look for `Ready=False` with reason `RoleConflict` in the conditions list, which indicates a DRBD Primary role conflict on the workload cluster. ## Examples ### Example 1 — ProtectionZone rejected for Linstor backend ```bash kubectl apply -f - <<'EOF' apiVersion: siterecovery.trilio.io/v1alpha1 kind: ProtectionZone metadata: name: bad-zone namespace: dr-bad-zone spec: displayName: Bad Zone storageBackendMode: Linstor clusters: - name: cluster1 clusterID: cluster1 kubeconfigSecretRef: name: cluster1-kubeconfig key: kubeconfig - name: cluster2 clusterID: cluster2 kubeconfigSecretRef: name: cluster2-kubeconfig key: kubeconfig EOF ``` --- ### Example 2 — ProtectionZone rejected for duplicate clusterID slot ```bash kubectl apply -f - <<'EOF' apiVersion: siterecovery.trilio.io/v1alpha1 kind: ProtectionZone metadata: name: dup-slot-zone namespace: dr-dup-slot-zone spec: displayName: Duplicate Slot Zone storageBackendMode: DRBD clusters: - name: cluster1 clusterID: cluster1 kubeconfigSecretRef: name: cluster1-kubeconfig key: kubeconfig - name: cluster2 clusterID: cluster1 # duplicate — both claim cluster1 kubeconfigSecretRef: name: cluster2-kubeconfig key: kubeconfig EOF ``` --- ### Example 3 — ProtectionRequest rejected for missing VM namespace ```bash kubectl apply -f - <<'EOF' apiVersion: siterecovery.trilio.io/v1alpha1 kind: ProtectionRequest metadata: name: pr-no-ns namespace: dr- spec: sourceCluster: cluster1 virtualMachine: name: web-vm # namespace intentionally omitted — webhook will reject EOF ``` --- ### Example 4 — ProtectionRequest rejected for Protocol A ```bash kubectl apply -f - <<'EOF' apiVersion: siterecovery.trilio.io/v1alpha1 kind: ProtectionRequest metadata: name: pr-proto-a namespace: dr- spec: sourceCluster: cluster1 virtualMachine: name: web-vm namespace: production replicationConfig: replicationProtocol: A # rejected by webhook rpo: 30 EOF ``` --- ### Example 5 — ProtectionRequest rejected for non-zero RPO with Protocol C ```bash kubectl apply -f - <<'EOF' apiVersion: siterecovery.trilio.io/v1alpha1 kind: ProtectionRequest metadata: name: pr-nonzero-rpo namespace: dr- spec: sourceCluster: cluster1 virtualMachine: name: web-vm namespace: production replicationConfig: replicationProtocol: C rpo: 5 # rejected: must be 0 with Protocol C EOF ``` --- ### Example 6 — TestFailover rejected for concurrent FailoverRequest ```bash # First, check whether a FailoverRequest is already Pending or InProgress kubectl get fr -n dr- # Attempting to create a TestFailover while a FailoverRequest is active kubectl apply -f - <<'EOF' apiVersion: siterecovery.trilio.io/v1alpha1 kind: TestFailover metadata: name: blocked-drill namespace: dr- spec: protectionGroupRef: name: pg-web-tier namespace: dr- cleanupPolicy: Automatic EOF ``` --- ### Example 7 — TestFailover rejected for tcp probe with HTTP-only fields ```bash kubectl apply -f - <<'EOF' apiVersion: siterecovery.trilio.io/v1alpha1 kind: TestFailover metadata: name: drill-bad-probe namespace: dr- spec: protectionGroupRef: name: pg-web-tier namespace: dr- cleanupPolicy: Automatic verification: reachability: - type: tcp port: 22 endpoint: /health # invalid on tcp probe — webhook rejects expectedStatus: 200 EOF ``` --- ### Example 8 — ProtectionGroup sourceCluster mismatch ```bash # If vm-a's ProtectionRequest is Protected on cluster1, the following is rejected: kubectl apply -f - <<'EOF' apiVersion: siterecovery.trilio.io/v1alpha1 kind: ProtectionGroup metadata: name: pg-mismatch namespace: dr- spec: sourceCluster: cluster2 # disagrees with vm-a's ProtectionRequest virtualMachines: - name: vm-a namespace: production EOF ``` ## Troubleshooting Use the following patterns to diagnose webhook failures. Every admission rejection includes the webhook name (e.g. `vprotectionrequest-v1alpha1.siterecovery.trilio.io`) and a structured field path in the error message. --- ### Symptom: `admission webhook ... denied the request: Linstor backend is not supported yet; use DRBD` **Cause:** `spec.storageBackendMode` was set to `Linstor` on a ProtectionZone. **Fix:** Change `spec.storageBackendMode` to `DRBD`. This is the only supported backend in Tech Preview. If you omit the field entirely, the mutating webhook defaults it to `DRBD`. ```bash kubectl patch pz -n dr- \ --type merge -p '{"spec":{"storageBackendMode":"DRBD"}}' ``` --- ### Symptom: `admission webhook ... denied the request: ReplicationProtocolA is not supported yet` **Cause:** `spec.replicationConfig.replicationProtocol` was set to `A` on a ProtectionRequest. Protocol A asynchronous replication is not available in Tech Preview; the webhook rejects it. **Fix:** Remove the field or set it to `C`. Protocol C synchronous replication (RPO=0) is the only supported mode. ```bash kubectl patch pr -n dr- \ --type merge -p '{"spec":{"replicationConfig":{"replicationProtocol":"C","rpo":0}}}' ``` --- ### Symptom: `rpo must be 0 when spec.replicationConfig.replicationProtocol is C` **Cause:** A ProtectionRequest specifies `replicationProtocol: C` (synchronous) with a non-zero `rpo` value. Protocol C delivers RPO=0 by definition; a non-zero RPO is contradictory and is rejected. **Fix:** Set `spec.replicationConfig.rpo` to `0`. --- ### Symptom: `spec.virtualMachine.namespace: Required value: must be set explicitly` **Cause:** A ProtectionRequest was created without `spec.virtualMachine.namespace`. The VM lives on a workload cluster in its application namespace; that namespace is unrelated to the TSR-Manager tenant namespace and cannot be inferred. **Fix:** Always supply `spec.virtualMachine.namespace` with the namespace the VM occupies on the workload cluster. --- ### Symptom: ProtectionGroup or ProtectionRequest create is rejected before any field error is shown **Cause:** The `ClustersValidated` condition on the ProtectionZone is not `True`. The TSR-Manager webhooks refuse all new ProtectionGroup and ProtectionRequest creates (and spec changes) until the zone is fully validated. **Diagnosis:** ```bash kubectl describe pz -n dr- | grep -A5 ClustersValidated ``` **Fix:** Resolve the condition reason shown. Common reasons and fixes: | Reason | Fix | |---|---| | `InsufficientClusterPermissions` | Run `tsrctl zone-credential rotate --refresh-rbac` for the affected cluster | | `ClusterAlreadyClaimed` | The cluster is registered in another zone; each cluster can belong to only one zone at a time | | `ClusterIdentityChanged` | The credential now points to a different cluster; restore the original cluster or delete and re-create the zone | | `KubeVirtNotInstalled` | Install KubeVirt on the workload cluster | | `SCCPrerequisiteMissing` | Ensure the required SCC is present on the workload cluster | --- ### Symptom: `TestFailover` create blocked with `FailoverRequest` in the error **Cause:** A FailoverRequest for the same ProtectionGroup is currently Pending or InProgress. The webhook prevents running a test failover concurrently with a production failover. **Diagnosis:** ```bash kubectl get fr -n dr- ``` **Fix:** Wait for the FailoverRequest to reach `Completed` or `Failed` state before creating the TestFailover. --- ### Symptom: `spec.protectionGroupRef.namespace: Required value: must be set explicitly: cannot be empty` **Cause (TestFailover):** The mutating webhook should have defaulted `spec.protectionGroupRef.namespace` from the TestFailover's own namespace, but the webhook server was not reachable and the field was left empty, or the field was explicitly cleared. **Cause (RPOEvent / ReplicationGroupStatus):** These objects require an explicit `spec.protectionGroupRef.namespace` that is set to the workload-cluster application namespace — it cannot be inferred. **Fix:** Supply `spec.protectionGroupRef.namespace` explicitly in your manifest. --- ### Symptom: `RPOEvent is immutable; updates are not permitted` **Cause:** You attempted to update an RPOEvent. RPOEvents are immutable audit records; the webhook rejects all updates. **Fix:** RPOEvents cannot be modified. If you need to remove a record, and the operator is not configured with a `monitorServiceAccountName` restriction, delete the object: ```bash kubectl delete rpoe -n dr- ``` If `monitorServiceAccountName` is configured, only the replication-monitor service account may delete RPOEvents. --- ### Symptom: Webhook server unreachable — `no endpoints available` or TLS timeout **Cause:** The manager control-plane pod in `dr-` is not running, or the `webhook-server-cert` Secret has not yet been provisioned. **Diagnosis:** ```bash # Check manager pod status kubectl get pods -n dr- \ -l app.kubernetes.io/component=manager-control-plane # Check webhook cert secret kubectl get secret webhook-server-cert -n dr- # Check manager pod logs kubectl logs -n dr- \ -l app.kubernetes.io/component=manager-control-plane ``` **Fix:** The ProtectionZone controller provisions the `webhook-server-cert` Secret automatically. If the manager pod is crash-looping, inspect the logs for the root cause. Do not manually create or modify the `webhook-server-cert` Secret. --- ### Symptom: `spec.sourceCluster: Required value: no member VM is backed by a Protected ProtectionRequest` **Cause:** You created a ProtectionGroup, but none of the VMs listed in `spec.virtualMachines` has a corresponding ProtectionRequest in the `Protected` phase. The mutating webhook cannot derive `spec.sourceCluster` without at least one Protected member. **Fix:** Wait for all member VMs' ProtectionRequests to reach the `Protected` phase before creating the ProtectionGroup, or supply `spec.sourceCluster` explicitly if you know the correct cluster: ```bash kubectl get pr -n dr- ``` All listed PRs should show `Protected` in the `PHASE` column before you create the ProtectionGroup. --- ### Symptom: `probe timeout must not exceed 10m` on a TestFailover **Cause:** A reachability probe in `spec.verification.reachability` specifies a `timeout` value greater than 10 minutes (e.g. `30m` or `2h`). **Fix:** Reduce the probe `timeout` to 10 minutes or less. The maximum is `10m`. --- ### Symptom: ProtectionZone update rejected with `field is immutable` **Cause:** You attempted to change a field that is frozen for the lifetime of the zone. Immutable fields are `spec.storageBackendMode`, each cluster's `name`, and each cluster's `clusterID`. **Mutable fields** (safe to update): `spec.displayName`, `spec.description`, each cluster's `kubeconfigSecretRef`, and each cluster's `distributionName`. **Fix:** If you need to change an immutable field, you must delete and re-create the ProtectionZone. Be aware that deleting a zone removes all its orchestration state.