Webhook Issues

This page explains the admission webhook failures and validation errors you may encounter when creating or updating Site Recovery custom resources. Trilio Site Recovery runs validating and mutating webhooks on the TSR-Manager cluster for every resource in the siterecovery.trilio.io/v1alpha1 API group; a rejected request returns an immediate error with a structured field path that identifies the exact problem. Understanding the webhook architecture and the rules each webhook enforces lets you resolve admission errors without trial-and-error.

Before using this guide, ensure you have:

• A working TSR-Manager cluster with the Site Recovery operator installed via OLM (package site-recovery, channel tech-preview, catalog quay.io/triliovault/site-recovery-catalog)
• kubectl (or oc) configured to target the TSR-Manager cluster
• A ProtectionZone CR in the dr-<deployment-name> namespace with status.phase: Available — the ClustersValidated condition on the ProtectionZone must be True before the webhooks will accept new ProtectionGroup or ProtectionRequest creates or spec changes
• OpenShift 4.17 or later on both the TSR-Manager cluster and all workload clusters
• Basic familiarity with kubectl and Kubernetes admission webhooks

The admission webhooks are installed automatically. You do not create, rotate, or modify any webhook serving certificate yourself.

How webhook certificates are provisioned

The ProtectionZone controller provisions the webhook-server-cert Secret before deploying each control plane stack. It re-issues the Secret whenever the certificate is unparsable, missing a required Subject Alternative Name, or approaching expiry. No manual certificate rotation is needed and no openssl procedure applies here.

Webhook serving configuration

Verifying the webhooks are registered

Run the following against the TSR-Manager cluster to confirm all webhook configurations are present:

kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations \
  -l app.kubernetes.io/part-of=site-recovery

The manager control-plane pod (running in the dr-<deployment-name> namespace) hosts all TSR-Manager-side webhooks. Verify it is running:

kubectl get pods -n dr-<deployment-name> \
  -l app.kubernetes.io/component=manager-control-plane

Expected output shows one pod with 1/1 ready containers, because the manager Deployment runs a single container named manager.

The workload control-plane pod (running in trilio-site-recovery-system on each workload cluster) hosts the DRBDResource guard and VM admission webhooks. Verify it on each workload cluster:

kubectl get pods -n trilio-site-recovery-system \
  -l app.kubernetes.io/component=workload-control-plane

The webhooks enforce rules at two levels: the CRD schema (enforced by the API server before the webhook runs) and cross-field or cross-resource rules (enforced by the webhook itself). You cannot configure which rules are active; all webhooks run with failurePolicy: Fail, so a webhook server that is unreachable causes the request to be rejected.

ProtectionZone webhook

Mutating webhook path: /mutate-siterecovery-trilio-io-v1alpha1-protectionzone
Validating webhook path: /validate-siterecovery-trilio-io-v1alpha1-protectionzone
Verbs covered: create, update

The mutating webhook defaults spec.storageBackendMode to DRBD when the field is omitted, because the field is schema-required with no schema-level default.

The validating webhook enforces rules the schema cannot express:

distributionName enum and default:

clusterID enum: cluster1 or cluster2 — both entries must occupy different slots.

ProtectionRequest webhook

Validating webhook path: /validate-siterecovery-trilio-io-v1alpha1-protectionrequest
Verbs covered: create, update, delete

There is no mutating webhook for ProtectionRequest. The spec.replicationConfig.replicationProtocol field defaults to C and spec.replicationConfig.rpo defaults to 0 via CRD schema markers.

Key defaults:

ProtectionGroup webhook

Mutating webhook path: (ProtectionGroup mutating — derives spec.sourceCluster)
Verbs covered: create, update

The mutating webhook derives spec.sourceCluster from the member VMs' Protected ProtectionRequests when the field is omitted. If an explicit value is provided and disagrees with the members' ProtectionRequests, the create is denied.

TestFailover webhook

Mutating webhook path: /mutate-siterecovery-trilio-io-v1alpha1-testfailover
Validating webhook path: /validate-siterecovery-trilio-io-v1alpha1-testfailover
Verbs covered: create, update, delete

The mutating webhook defaults spec.protectionGroupRef.namespace from the TestFailover's own namespace when omitted.

RPOEvent webhook

Mutating webhook path: /mutate-siterecovery-trilio-io-v1alpha1-rpoevent
Validating webhook path: /validate-siterecovery-trilio-io-v1alpha1-rpoevent
Verbs covered: create, update, delete

RPOEvent is an immutable audit record. Only the replication-monitor service account may create or delete RPOEvents when the operator is configured with monitorServiceAccountName.

ReplicationGroupStatus webhook

Mutating webhook path: /mutate-siterecovery-trilio-io-v1alpha1-replicationgroupstatus
Validating webhook path: /validate-siterecovery-trilio-io-v1alpha1-replicationgroupstatus
Verbs covered: create, update, delete

ReplicationGroupStatus objects are auto-created by the replication monitor controller and are named <protection-group-name>-replication. You do not create these manually.

Checking zone readiness before creating protected resources

The ClustersValidated condition on the ProtectionZone must be True before the webhooks will admit new ProtectionGroup or ProtectionRequest creates or spec changes. Check zone status first:

kubectl get pz -n dr-<deployment-name>

Inspect the conditions when the zone is not Available:

kubectl describe pz <zone-name> -n dr-<deployment-name>

Look for the ClustersValidated condition in the output. Common reasons that gate admission:

Creating a ProtectionRequest

The ProtectionRequest webhook enforces that spec.virtualMachine.namespace is always set explicitly, because the VM lives on a workload cluster in its application namespace — that namespace is unrelated to the TSR-Manager tenant namespace and cannot be inferred.

apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionRequest
metadata:
  name: vm-web-protect
  namespace: dr-<deployment-name>
spec:
  sourceCluster: cluster1
  virtualMachine:
    name: web-vm
    namespace: production          # required; cannot be omitted
  replicationConfig:
    replicationProtocol: C        # only supported value
    rpo: 0                         # must be 0 with Protocol C

Apply it:

kubectl apply -f protectionrequest.yaml

The webhook rejects the request immediately with a structured error if any field violates the rules. For example, supplying replicationProtocol: A produces:

Error from server: admission webhook "vprotectionrequest-v1alpha1.siterecovery.trilio.io" denied the request:
spec.replicationConfig.replicationProtocol: Invalid value: "A": ReplicationProtocolA is not supported yet

Creating a ProtectionZone

The validating webhook rejects storageBackendMode: Linstor at admission. Always use DRBD:

apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionZone
metadata:
  name: prod-dr
  namespace: dr-prod-dr
spec:
  displayName: Production DR Zone
  storageBackendMode: DRBD
  clusters:
    - name: cluster1
      clusterID: cluster1
      kubeconfigSecretRef:
        name: cluster1-kubeconfig
        key: kubeconfig
    - name: cluster2
      clusterID: cluster2
      kubeconfigSecretRef:
        name: cluster2-kubeconfig
        key: kubeconfig

If you omit storageBackendMode, the mutating webhook defaults it to DRBD before validation runs.

Creating a TestFailover

The mutating webhook defaults spec.protectionGroupRef.namespace from the TestFailover's own namespace when omitted. The validating webhook then checks that the referenced ProtectionGroup actually exists and that no Pending or InProgress FailoverRequest is already running for the same PG.

apiVersion: siterecovery.trilio.io/v1alpha1
kind: TestFailover
metadata:
  name: quarterly-drill
  namespace: dr-<deployment-name>
spec:
  protectionGroupRef:
    name: pg-web-tier
    namespace: dr-<deployment-name>   # can be omitted; defaulted from TFO namespace
  cleanupPolicy: Automatic
  schedulingCheck: Fail

To add reachability verification probes, note that endpoint and expectedStatus are only valid for http probes, not tcp:

  verification:
    reachability:
      - type: tcp
        port: 22
        timeout: 30s
      - type: http
        port: 8080
        endpoint: /health
        expectedStatus: 200
        timeout: 60s

A timeout value greater than 10m is rejected at admission.

Checking ProtectionRequest phase after admission

Once a ProtectionRequest is admitted, follow its phase through the controller lifecycle:

kubectl get pr -n dr-<deployment-name> -w

Valid phases in order: Pending → Validating → CreatingDRBD → Syncing → ReadyToActivate → Activating → Protected.

If the request reaches Failed, inspect conditions:

kubectl describe pr <name> -n dr-<deployment-name>

Look for Ready=False with reason RoleConflict in the conditions list, which indicates a DRBD Primary role conflict on the workload cluster.

Example 1 — ProtectionZone rejected for Linstor backend

kubectl apply -f - <<'EOF'
apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionZone
metadata:
  name: bad-zone
  namespace: dr-bad-zone
spec:
  displayName: Bad Zone
  storageBackendMode: Linstor
  clusters:
    - name: cluster1
      clusterID: cluster1
      kubeconfigSecretRef:
        name: cluster1-kubeconfig
        key: kubeconfig
    - name: cluster2
      clusterID: cluster2
      kubeconfigSecretRef:
        name: cluster2-kubeconfig
        key: kubeconfig
EOF

Example 2 — ProtectionZone rejected for duplicate clusterID slot

kubectl apply -f - <<'EOF'
apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionZone
metadata:
  name: dup-slot-zone
  namespace: dr-dup-slot-zone
spec:
  displayName: Duplicate Slot Zone
  storageBackendMode: DRBD
  clusters:
    - name: cluster1
      clusterID: cluster1
      kubeconfigSecretRef:
        name: cluster1-kubeconfig
        key: kubeconfig
    - name: cluster2
      clusterID: cluster1          # duplicate — both claim cluster1
      kubeconfigSecretRef:
        name: cluster2-kubeconfig
        key: kubeconfig
EOF

Example 3 — ProtectionRequest rejected for missing VM namespace

kubectl apply -f - <<'EOF'
apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionRequest
metadata:
  name: pr-no-ns
  namespace: dr-<deployment-name>
spec:
  sourceCluster: cluster1
  virtualMachine:
    name: web-vm
    # namespace intentionally omitted — webhook will reject
EOF

Example 4 — ProtectionRequest rejected for Protocol A

kubectl apply -f - <<'EOF'
apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionRequest
metadata:
  name: pr-proto-a
  namespace: dr-<deployment-name>
spec:
  sourceCluster: cluster1
  virtualMachine:
    name: web-vm
    namespace: production
  replicationConfig:
    replicationProtocol: A     # rejected by webhook
    rpo: 30
EOF

Example 5 — ProtectionRequest rejected for non-zero RPO with Protocol C

kubectl apply -f - <<'EOF'
apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionRequest
metadata:
  name: pr-nonzero-rpo
  namespace: dr-<deployment-name>
spec:
  sourceCluster: cluster1
  virtualMachine:
    name: web-vm
    namespace: production
  replicationConfig:
    replicationProtocol: C
    rpo: 5                      # rejected: must be 0 with Protocol C
EOF

Example 6 — TestFailover rejected for concurrent FailoverRequest

# First, check whether a FailoverRequest is already Pending or InProgress
kubectl get fr -n dr-<deployment-name>

# Attempting to create a TestFailover while a FailoverRequest is active
kubectl apply -f - <<'EOF'
apiVersion: siterecovery.trilio.io/v1alpha1
kind: TestFailover
metadata:
  name: blocked-drill
  namespace: dr-<deployment-name>
spec:
  protectionGroupRef:
    name: pg-web-tier
    namespace: dr-<deployment-name>
  cleanupPolicy: Automatic
EOF

Example 7 — TestFailover rejected for tcp probe with HTTP-only fields

kubectl apply -f - <<'EOF'
apiVersion: siterecovery.trilio.io/v1alpha1
kind: TestFailover
metadata:
  name: drill-bad-probe
  namespace: dr-<deployment-name>
spec:
  protectionGroupRef:
    name: pg-web-tier
    namespace: dr-<deployment-name>
  cleanupPolicy: Automatic
  verification:
    reachability:
      - type: tcp
        port: 22
        endpoint: /health          # invalid on tcp probe — webhook rejects
        expectedStatus: 200
EOF

Example 8 — ProtectionGroup sourceCluster mismatch

# If vm-a's ProtectionRequest is Protected on cluster1, the following is rejected:
kubectl apply -f - <<'EOF'
apiVersion: siterecovery.trilio.io/v1alpha1
kind: ProtectionGroup
metadata:
  name: pg-mismatch
  namespace: dr-<deployment-name>
spec:
  sourceCluster: cluster2           # disagrees with vm-a's ProtectionRequest
  virtualMachines:
    - name: vm-a
      namespace: production
EOF

Use the following patterns to diagnose webhook failures. Every admission rejection includes the webhook name (e.g. vprotectionrequest-v1alpha1.siterecovery.trilio.io) and a structured field path in the error message.

Symptom: admission webhook ... denied the request: Linstor backend is not supported yet; use DRBD

Cause: spec.storageBackendMode was set to Linstor on a ProtectionZone.

Fix: Change spec.storageBackendMode to DRBD. This is the only supported backend in Tech Preview. If you omit the field entirely, the mutating webhook defaults it to DRBD.

kubectl patch pz <zone-name> -n dr-<deployment-name> \
  --type merge -p '{"spec":{"storageBackendMode":"DRBD"}}'

Symptom: admission webhook ... denied the request: ReplicationProtocolA is not supported yet

Cause: spec.replicationConfig.replicationProtocol was set to A on a ProtectionRequest. Protocol A asynchronous replication is not available in Tech Preview; the webhook rejects it.

Fix: Remove the field or set it to C. Protocol C synchronous replication (RPO=0) is the only supported mode.

kubectl patch pr <pr-name> -n dr-<deployment-name> \
  --type merge -p '{"spec":{"replicationConfig":{"replicationProtocol":"C","rpo":0}}}'

Symptom: rpo must be 0 when spec.replicationConfig.replicationProtocol is C

Cause: A ProtectionRequest specifies replicationProtocol: C (synchronous) with a non-zero rpo value. Protocol C delivers RPO=0 by definition; a non-zero RPO is contradictory and is rejected.

Fix: Set spec.replicationConfig.rpo to 0.

Symptom: spec.virtualMachine.namespace: Required value: must be set explicitly

Cause: A ProtectionRequest was created without spec.virtualMachine.namespace. The VM lives on a workload cluster in its application namespace; that namespace is unrelated to the TSR-Manager tenant namespace and cannot be inferred.

Fix: Always supply spec.virtualMachine.namespace with the namespace the VM occupies on the workload cluster.

Symptom: ProtectionGroup or ProtectionRequest create is rejected before any field error is shown

Cause: The ClustersValidated condition on the ProtectionZone is not True. The TSR-Manager webhooks refuse all new ProtectionGroup and ProtectionRequest creates (and spec changes) until the zone is fully validated.

Diagnosis:

kubectl describe pz <zone-name> -n dr-<deployment-name> | grep -A5 ClustersValidated

Fix: Resolve the condition reason shown. Common reasons and fixes:

Symptom: TestFailover create blocked with FailoverRequest in the error

Cause: A FailoverRequest for the same ProtectionGroup is currently Pending or InProgress. The webhook prevents running a test failover concurrently with a production failover.

Diagnosis:

kubectl get fr -n dr-<deployment-name>

Fix: Wait for the FailoverRequest to reach Completed or Failed state before creating the TestFailover.

Symptom: spec.protectionGroupRef.namespace: Required value: must be set explicitly: cannot be empty

Cause (TestFailover): The mutating webhook should have defaulted spec.protectionGroupRef.namespace from the TestFailover's own namespace, but the webhook server was not reachable and the field was left empty, or the field was explicitly cleared.

Cause (RPOEvent / ReplicationGroupStatus): These objects require an explicit spec.protectionGroupRef.namespace that is set to the workload-cluster application namespace — it cannot be inferred.

Fix: Supply spec.protectionGroupRef.namespace explicitly in your manifest.

Symptom: RPOEvent is immutable; updates are not permitted

Cause: You attempted to update an RPOEvent. RPOEvents are immutable audit records; the webhook rejects all updates.

Fix: RPOEvents cannot be modified. If you need to remove a record, and the operator is not configured with a monitorServiceAccountName restriction, delete the object:

kubectl delete rpoe <name> -n dr-<deployment-name>

If monitorServiceAccountName is configured, only the replication-monitor service account may delete RPOEvents.

Symptom: Webhook server unreachable — no endpoints available or TLS timeout

Cause: The manager control-plane pod in dr-<deployment-name> is not running, or the webhook-server-cert Secret has not yet been provisioned.

Diagnosis:

# Check manager pod status
kubectl get pods -n dr-<deployment-name> \
  -l app.kubernetes.io/component=manager-control-plane

# Check webhook cert secret
kubectl get secret webhook-server-cert -n dr-<deployment-name>

# Check manager pod logs
kubectl logs -n dr-<deployment-name> \
  -l app.kubernetes.io/component=manager-control-plane

Fix: The ProtectionZone controller provisions the webhook-server-cert Secret automatically. If the manager pod is crash-looping, inspect the logs for the root cause. Do not manually create or modify the webhook-server-cert Secret.

Symptom: spec.sourceCluster: Required value: no member VM is backed by a Protected ProtectionRequest

Cause: You created a ProtectionGroup, but none of the VMs listed in spec.virtualMachines has a corresponding ProtectionRequest in the Protected phase. The mutating webhook cannot derive spec.sourceCluster without at least one Protected member.

Fix: Wait for all member VMs' ProtectionRequests to reach the Protected phase before creating the ProtectionGroup, or supply spec.sourceCluster explicitly if you know the correct cluster:

kubectl get pr -n dr-<deployment-name>

All listed PRs should show Protected in the PHASE column before you create the ProtectionGroup.

Symptom: probe timeout must not exceed 10m on a TestFailover

Cause: A reachability probe in spec.verification.reachability specifies a timeout value greater than 10 minutes (e.g. 30m or 2h).

Fix: Reduce the probe timeout to 10 minutes or less. The maximum is 10m.

Symptom: ProtectionZone update rejected with field is immutable

Cause: You attempted to change a field that is frozen for the lifetime of the zone. Immutable fields are spec.storageBackendMode, each cluster's name, and each cluster's clusterID.

Mutable fields (safe to update): spec.displayName, spec.description, each cluster's kubeconfigSecretRef, and each cluster's distributionName.

Fix: If you need to change an immutable field, you must delete and re-create the ProtectionZone. Be aware that deleting a zone removes all its orchestration state.