RBAC Issues

This page covers RBAC-related errors in Trilio Site Recovery: permission denied failures, service account problems, and cluster-scoped access issues. Understanding how Site Recovery uses RBAC is essential for diagnosing why zone validation stalls, why controllers cannot reach workload clusters, and why admission webhooks reject or restrict certain operations. All DR operations are CRD-driven and the system's authorization model spans two distinct layers — Kubernetes RBAC on each cluster, and zone kubeconfig credentials managed by tsrctl — so most permission failures fall into one of those two categories.

Before working through these procedures, you need:

• Access to the TSR-Manager cluster with cluster-admin or equivalent rights
• kubectl configured to reach the TSR-Manager cluster and both workload clusters
• oc (OpenShift CLI) — required for SCC inspection and oc adm must-gather
• tsrctl — supplied by Trilio with the release; contact Trilio support to obtain it
• The zone name (used to identify the zone namespace dr-<zone-name>) and the workload cluster kubeconfig secrets
• OpenShift 4.17 or later on all three clusters (TSR-Manager, primary workload, DR workload)

No additional installation is required to diagnose RBAC issues. All the tools you need (kubectl, oc, tsrctl) must already be present as part of the Site Recovery installation. If tsrctl is not available, contact Trilio support.

Verify your cluster access before starting:

# Confirm you can reach the TSR-Manager cluster
kubectl get protectionzones -A

# Confirm you can reach each workload cluster (substitute your kubeconfig paths)
kubectl --kubeconfig /path/to/cluster1.kubeconfig get nodes
kubectl --kubeconfig /path/to/cluster2.kubeconfig get nodes

Verify Site Recovery components are running on the TSR-Manager cluster:

# Check the ProtectionZone controller
kubectl get pods -n dr-<zone-name> \
  -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=protectionzone-controller

# Check the TSR-Manager control plane
kubectl get pods -n dr-<zone-name> \
  -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=manager-control-plane

Verify Site Recovery components are running on each workload cluster:

kubectl --kubeconfig /path/to/cluster1.kubeconfig get pods -n trilio-site-recovery-system \
  -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=workload-control-plane

kubectl --kubeconfig /path/to/cluster1.kubeconfig get pods -n trilio-site-recovery-system \
  -l app.kubernetes.io/component=drbd-node-agent

Site Recovery's RBAC model has three distinct surfaces. Understanding which surface applies to your error is the first step in diagnosis.

1. Zone kubeconfig credentials (tsrctl zone-credential)

The TSR-Manager controllers reach workload clusters exclusively through kubeconfig Secrets stored in the zone namespace (dr-<zone-name>). These credentials are bound to a specific cluster by its kube-system Namespace UID and cannot be redirected to a different cluster. They are created and rotated only by tsrctl zone-credential.

Each credential is associated with a ServiceAccount in the trilio-site-recovery-access namespace on the workload cluster. The tsrctl zone-credential command manages these ServiceAccounts and the RBAC rules that grant them the verbs the Site Recovery controllers require.

tsrctl zone-credential flags (applies to all subcommands unless noted):

2. Kubernetes RBAC on the TSR-Manager cluster

The OLM bundle ships three ClusterRoles for the TSR web console personas:

Bind these to users or groups with a ClusterRoleBinding. The TSR web console evaluates each action at runtime via SelfSubjectAccessReview using the caller's own token, so the role binding determines what a user can do in the UI.

3. OpenShift SCC for the workload node agent

On OpenShift workload clusters, the workload control plane chart binds the node agent's ServiceAccount to the privileged SCC. This is performed automatically by the ProtectionZone controller when it installs the workload stack. If the SCC binding is missing or denied, the node agent DaemonSet will not start.

4. Admission webhook guards

Several Site Recovery admission webhooks enforce authorization rules beyond standard Kubernetes RBAC:

• ReplicationGroupStatus and RPOEvent objects may only be created, updated, or deleted by the replication-monitor ServiceAccount. Any other principal is rejected with a Forbidden error.
• ProtectionGroup and ProtectionRequest creates and spec changes are blocked by the TSR-Manager webhooks while the ProtectionZone's ClustersValidated condition is not True.
• ProtectionRequest with replicationProtocol: A is rejected at admission (ReplicationProtocolA is not supported yet).

Diagnosing a zone credential failure

When the ProtectionZone controller cannot reach a workload cluster, it sets a condition on the ProtectionZone CR. Inspect the conditions first:

kubectl get pz <zone-name> -n dr-<zone-name> -o yaml | \
  kubectl neat | grep -A 30 'conditions:'

The ClustersValidated condition reason tells you which failure class you are dealing with:

Rotating a zone credential with RBAC refresh

When ClustersValidated shows InsufficientClusterPermissions, the ServiceAccount exists but is missing required RBAC verbs. Use --refresh-rbac when rotating:

# Rotate the credential and re-apply RBAC rules for cluster1
tsrctl zone-credential rotate --refresh-rbac

After rotation, verify the zone recovers:

kubectl get pz <zone-name> -n dr-<zone-name>
# STATUS column should return to Available

Inspecting the current credential state

# Inspect the credential for a workload cluster
tsrctl zone-credential inspect

Checking if the ClustersValidated gate is blocking ProtectionGroup or ProtectionRequest admission

If creating a ProtectionGroup or ProtectionRequest returns an error, the zone may not have completed validation:

kubectl get pz <zone-name> -n dr-<zone-name> -o jsonpath='{.status.conditions}' | python3 -m json.tool

Look for ClustersValidated with status: "False". Until this condition is True, the TSR-Manager webhooks reject all ProtectionGroup and ProtectionRequest creates and spec changes.

Diagnosing ReplicationGroupStatus or RPOEvent write rejections

These objects are restricted to the replication-monitor ServiceAccount. If you see a Forbidden error when trying to create or modify them manually, that is expected — these objects are system-managed. Do not create, update, or delete them directly.

If the replication monitor itself cannot write them (visible as stalled status or missing ReplicationGroupStatus objects for a ProtectionGroup), check that the replication-monitor pod is running:

kubectl get pods -n dr-<zone-name> \
  -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=manager-control-plane

Checking SCC admission for the node agent on OpenShift

If the ProtectionZone shows SCCAdmissionDenied on its WorkloadStacksReady condition, the node agent DaemonSet was rejected by OpenShift SCC admission after the chart was installed:

# Check node agent pods on the affected workload cluster
kubectl --kubeconfig /path/to/workload.kubeconfig get pods -n trilio-site-recovery-system \
  -l app.kubernetes.io/component=drbd-node-agent

# Describe a failing pod
kubectl --kubeconfig /path/to/workload.kubeconfig describe pod <node-agent-pod> \
  -n trilio-site-recovery-system

The privileged SCC binding is applied automatically by the ProtectionZone controller. If it is absent, re-trigger zone reconciliation by making a benign update to the ProtectionZone CR (for example, updating spec.description).

Granting a user access to the TSR web console

Bind one of the three persona ClusterRoles to the user:

# Grant admin access
kubectl create clusterrolebinding <binding-name> \
  --clusterrole=trilio-site-recovery-admin \
  --user=<username>

# Grant operator access
kubectl create clusterrolebinding <binding-name> \
  --clusterrole=trilio-site-recovery-operator \
  --user=<username>

# Grant viewer access
kubectl create clusterrolebinding <binding-name> \
  --clusterrole=trilio-site-recovery-viewer \
  --user=<username>

Example 1 — Rotate a zone credential and refresh RBAC

Use this when ClustersValidated shows InsufficientClusterPermissions.

tsrctl zone-credential rotate --refresh-rbac

Example 2 — Rotate a credential and revoke the previous one

tsrctl zone-credential rotate --refresh-rbac --revoke-previous

Example 3 — Inspect the current credential

tsrctl zone-credential inspect

Example 4 — Create a new zone credential

tsrctl zone-credential create

Example 5 — Delete a zone credential

tsrctl zone-credential delete

To force deletion when the credential is in use:

tsrctl zone-credential delete --force

Example 6 — Check ProtectionZone conditions for RBAC failures

kubectl get pz <zone-name> -n dr-<zone-name> -o yaml

Look for the ClustersValidated condition under status.conditions and read the reason and message fields.

Example 7 — Verify that the trilio-site-recovery-access namespace and ServiceAccount exist on a workload cluster

kubectl --kubeconfig /path/to/workload.kubeconfig \
  get namespace trilio-site-recovery-access

kubectl --kubeconfig /path/to/workload.kubeconfig \
  get serviceaccounts -n trilio-site-recovery-access

Example 8 — Bind the admin ClusterRole to a user

kubectl create clusterrolebinding site-recovery-admin-alice \
  --clusterrole=trilio-site-recovery-admin \
  --user=alice

Example 9 — Check whether a user can create a ProtectionRequest

kubectl auth can-i create protectionrequests.siterecovery.trilio.io \
  -n dr-<zone-name> --as <username>

Example 10 — Collect a support bundle after an RBAC failure

If the problem is not resolved by the steps above, collect a diagnostic bundle and send it to Trilio support:

oc adm must-gather \
  --image=quay.io/triliovault/site-recovery-must-gather:<release-tag> \
  -- /usr/bin/gather

Use the following table to match a symptom to its most likely cause and fix.

Symptom: ProtectionZone stuck in Unavailable or Degraded; ClustersValidated condition is False with reason InsufficientClusterPermissions

Cause: The zone kubeconfig credential authenticates successfully, but the bound ServiceAccount is missing one or more RBAC verbs that the Site Recovery controllers require.

Fix: Rotate the credential with --refresh-rbac to re-apply the RBAC rules:

tsrctl zone-credential rotate --refresh-rbac

After rotation, watch the ProtectionZone until ClustersValidated becomes True:

kubectl get pz <zone-name> -n dr-<zone-name> -w

Symptom: ClustersValidated is False with reason ClusterIdentityChanged

Cause: The zone credential now reaches a cluster whose kube-system Namespace UID differs from the UID recorded when the zone first claimed the cluster. This happens when the credential is repointed to a different cluster.

Fix: There is no override. You must either restore the original cluster or re-create the ProtectionZone. Contact Trilio support for guidance.

Symptom: ClustersValidated is False with reason ClusterAlreadyClaimed

Cause: The workload cluster is already leased to another ProtectionZone. Each workload cluster can belong to only one zone at a time.

Fix: Identify the zone that holds the lease by inspecting the siterecovery.trilio.io/owner-zone annotation on the trilio-site-recovery-system Namespace of the workload cluster:

kubectl --kubeconfig /path/to/workload.kubeconfig \
  get namespace trilio-site-recovery-system \
  -o jsonpath='{.metadata.annotations}'

Release the cluster from the other zone before adding it to this one.

Symptom: Creating a ProtectionGroup or ProtectionRequest returns a webhook rejection

Cause: The ProtectionZone's ClustersValidated condition is not yet True. The TSR-Manager admission webhooks block all ProtectionGroup and ProtectionRequest creates and spec changes until validation passes.

Fix: Wait for or resolve the ClustersValidated condition. Check zone status:

kubectl get pz <zone-name> -n dr-<zone-name> \
  -o jsonpath='{.status.conditions[?(@.type=="ClustersValidated")]}

Symptom: Creating a ProtectionRequest returns ReplicationProtocolA is not supported yet

Cause: spec.replicationConfig.replicationProtocol was set to A. Protocol A (asynchronous) is rejected by the admission webhook in Tech Preview. Only Protocol C (synchronous) is supported.

Fix: Remove the replicationProtocol field or set it to C. RPO must be 0 when Protocol C is used:

spec:
  replicationConfig:
    replicationProtocol: C
    rpo: 0

Symptom: Manually creating or updating a ReplicationGroupStatus or RPOEvent returns Forbidden

Cause: These objects are restricted to the replication-monitor ServiceAccount by the admission webhook. The error message will reference replication-monitor.

Fix: Do not create or modify these objects manually. They are system-managed by the replication-monitor controller. If they are missing, the controller will recreate them on its next polling interval. If the controller itself is not running, check the manager-control-plane pod:

kubectl get pods -n dr-<zone-name> \
  -l app.kubernetes.io/component=manager-control-plane

Symptom: Node agent DaemonSet pods are not starting on a workload cluster; ProtectionZone shows SCCAdmissionDenied on the WorkloadStacksReady condition

Cause: The node agent DaemonSet requires the privileged SCC on OpenShift. The ProtectionZone controller applies this binding automatically, but it may have been removed or the initial application may have failed.

Fix: Verify the SCC binding exists on the workload cluster:

kubectl --kubeconfig /path/to/workload.kubeconfig \
  get clusterrolebindings | grep site-recovery

If missing, re-trigger zone reconciliation by making a benign change to the ProtectionZone CR (for example, update spec.description). The controller will re-apply the SCC binding on the next reconcile.

Symptom: A user can authenticate to the TSR web console but cannot perform operations (buttons are disabled or actions are rejected)

Cause: The user's account is not bound to a Site Recovery persona ClusterRole, or is bound to a lower-privilege role (viewer) than required for the action.

Fix: Bind the appropriate ClusterRole:

# For full operational access
kubectl create clusterrolebinding <binding-name> \
  --clusterrole=trilio-site-recovery-operator \
  --user=<username>

The three roles are trilio-site-recovery-admin, trilio-site-recovery-operator, and trilio-site-recovery-viewer. The UI evaluates each action at runtime via SelfSubjectAccessReview using the caller's token.

Symptom: kubectl get protectionzones (or other Site Recovery CRs) against a workload cluster returns server doesn't have a resource type

Cause: The TSR-Manager-side CRDs (ProtectionZone, ProtectionGroup, ProtectionRequest, FailoverRequest, TestFailover, Hook, DRBDReplicationPolicy, ReplicationGroupStatus, RPOEvent) are only registered on the TSR-Manager cluster. The workload cluster only has the DRBDResource CRD.

Fix: Run TSR-Manager-side CR queries against the TSR-Manager cluster. Run DRBDResource queries against the workload cluster:

# Correct — TSR-Manager cluster
kubectl get protectionzones -n dr-<zone-name>

# Correct — workload cluster
kubectl --kubeconfig /path/to/workload.kubeconfig \
  get drbdresources -n <vm-namespace>

Symptom: oc adm drain on a workload node does not complete

Cause: A running protected VM is pinned to that node. Live migration and cold migration of protected VMs are denied by the admission webhook. The drain cannot evict the VM.

Fix: Before draining a node that hosts a protected VM, you must either complete an unprotect operation for the VM, or perform a planned failover of its ProtectionGroup to move the active workload to the DR cluster. Do not attempt to force-delete the VM or its DRBDResource without contacting Trilio support.