--- title: RBAC Issues product: trilio-site-recovery-for-kubernetes doc_type: guide version: 0.2.0 source: git2docs (code-derived, validation-filtered) canonical: https://docs-tsr.trilio.io/trilio-site-recovery-for-kubernetes/site-recovery/troubleshooting-rbac --- # RBAC Issues _Permission denied errors, service account problems, cluster-scoped access issues_ ## Overview This page covers RBAC-related errors in Trilio Site Recovery: permission denied failures, service account problems, and cluster-scoped access issues. Understanding how Site Recovery uses RBAC is essential for diagnosing why zone validation stalls, why controllers cannot reach workload clusters, and why admission webhooks reject or restrict certain operations. All DR operations are CRD-driven and the system's authorization model spans two distinct layers — Kubernetes RBAC on each cluster, and zone kubeconfig credentials managed by `tsrctl` — so most permission failures fall into one of those two categories. ## Prerequisites Before working through these procedures, you need: - Access to the TSR-Manager cluster with `cluster-admin` or equivalent rights - `kubectl` configured to reach the TSR-Manager cluster and both workload clusters - `oc` (OpenShift CLI) — required for SCC inspection and `oc adm must-gather` - `tsrctl` — supplied by Trilio with the release; contact Trilio support to obtain it - The zone name (used to identify the zone namespace `dr-`) and the workload cluster kubeconfig secrets - OpenShift 4.17 or later on all three clusters (TSR-Manager, primary workload, DR workload) ## Installation No additional installation is required to diagnose RBAC issues. All the tools you need (`kubectl`, `oc`, `tsrctl`) must already be present as part of the Site Recovery installation. If `tsrctl` is not available, contact Trilio support. **Verify your cluster access before starting:** ```bash # Confirm you can reach the TSR-Manager cluster kubectl get protectionzones -A # Confirm you can reach each workload cluster (substitute your kubeconfig paths) kubectl --kubeconfig /path/to/cluster1.kubeconfig get nodes kubectl --kubeconfig /path/to/cluster2.kubeconfig get nodes ``` **Verify Site Recovery components are running on the TSR-Manager cluster:** ```bash # Check the ProtectionZone controller kubectl get pods -n dr- \ -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=protectionzone-controller # Check the TSR-Manager control plane kubectl get pods -n dr- \ -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=manager-control-plane ``` **Verify Site Recovery components are running on each workload cluster:** ```bash kubectl --kubeconfig /path/to/cluster1.kubeconfig get pods -n trilio-site-recovery-system \ -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=workload-control-plane kubectl --kubeconfig /path/to/cluster1.kubeconfig get pods -n trilio-site-recovery-system \ -l app.kubernetes.io/component=drbd-node-agent ``` ## Configuration Site Recovery's RBAC model has three distinct surfaces. Understanding which surface applies to your error is the first step in diagnosis. **1. Zone kubeconfig credentials (`tsrctl zone-credential`)** The TSR-Manager controllers reach workload clusters exclusively through kubeconfig Secrets stored in the zone namespace (`dr-`). These credentials are bound to a specific cluster by its `kube-system` Namespace UID and cannot be redirected to a different cluster. They are created and rotated only by `tsrctl zone-credential`. Each credential is associated with a ServiceAccount in the `trilio-site-recovery-access` namespace on the workload cluster. The `tsrctl zone-credential` command manages these ServiceAccounts and the RBAC rules that grant them the verbs the Site Recovery controllers require. `tsrctl zone-credential` flags (applies to all subcommands unless noted): | Flag | Applies to | Purpose | |------|-----------|--------| | `--kubeconfig ` | all | Path to kubeconfig for the workload cluster | | `--context ` | all | Context within the kubeconfig | | `--namespace ` | all | Namespace override | | `--zone ` | all | Zone name | | `--access-namespace ` | all | Override for the `trilio-site-recovery-access` namespace | | `--refresh-rbac` | `rotate` only | Re-applies the RBAC rules alongside the credential rotation | | `--revoke-previous` | `rotate` only | Revokes the previous credential after rotation | | `--force` | `delete` only | Forces deletion even when the credential is in use | **2. Kubernetes RBAC on the TSR-Manager cluster** The OLM bundle ships three ClusterRoles for the TSR web console personas: | ClusterRole | Purpose | |-------------|--------| | `trilio-site-recovery-admin` | Full read/write access to all Site Recovery CRs | | `trilio-site-recovery-operator` | Operational access — can trigger failovers and test failovers | | `trilio-site-recovery-viewer` | Read-only access to all Site Recovery CRs | Bind these to users or groups with a `ClusterRoleBinding`. The TSR web console evaluates each action at runtime via `SelfSubjectAccessReview` using the caller's own token, so the role binding determines what a user can do in the UI. **3. OpenShift SCC for the workload node agent** On OpenShift workload clusters, the workload control plane chart binds the node agent's ServiceAccount to the `privileged` SCC. This is performed automatically by the ProtectionZone controller when it installs the workload stack. If the SCC binding is missing or denied, the node agent DaemonSet will not start. **4. Admission webhook guards** Several Site Recovery admission webhooks enforce authorization rules beyond standard Kubernetes RBAC: - `ReplicationGroupStatus` and `RPOEvent` objects may only be created, updated, or deleted by the replication-monitor ServiceAccount. Any other principal is rejected with a `Forbidden` error. - `ProtectionGroup` and `ProtectionRequest` creates and spec changes are blocked by the TSR-Manager webhooks while the ProtectionZone's `ClustersValidated` condition is not `True`. - `ProtectionRequest` with `replicationProtocol: A` is rejected at admission (`ReplicationProtocolA is not supported yet`). ## Usage ### Diagnosing a zone credential failure When the ProtectionZone controller cannot reach a workload cluster, it sets a condition on the ProtectionZone CR. Inspect the conditions first: ```bash kubectl get pz -n dr- -o yaml | \ kubectl neat | grep -A 30 'conditions:' ``` The `ClustersValidated` condition reason tells you which failure class you are dealing with: | Reason | Meaning | Fix | |--------|---------|-----| | `InsufficientClusterPermissions` | Credential authenticates but RBAC checks fail | Rotate with `--refresh-rbac` | | `ClusterIdentityChanged` | Credential now reaches a different cluster (different `kube-system` UID) | Restore original cluster or re-create zone | | `ClusterAlreadyClaimed` | Workload cluster is leased to another zone | Release from other zone first | | `SCCPrerequisiteMissing` | Required SCC prerequisites not met on workload cluster | Contact Trilio support | | `KubeVirtNotInstalled` | KubeVirt not present on workload cluster | Install KubeVirt | ### Rotating a zone credential with RBAC refresh When `ClustersValidated` shows `InsufficientClusterPermissions`, the ServiceAccount exists but is missing required RBAC verbs. Use `--refresh-rbac` when rotating: ```bash # Rotate the credential and re-apply RBAC rules for cluster1 tsrctl zone-credential rotate --refresh-rbac ``` After rotation, verify the zone recovers: ```bash kubectl get pz -n dr- # STATUS column should return to Available ``` ### Inspecting the current credential state ```bash # Inspect the credential for a workload cluster tsrctl zone-credential inspect ``` ### Checking if the ClustersValidated gate is blocking ProtectionGroup or ProtectionRequest admission If creating a ProtectionGroup or ProtectionRequest returns an error, the zone may not have completed validation: ```bash kubectl get pz -n dr- -o jsonpath='{.status.conditions}' | python3 -m json.tool ``` Look for `ClustersValidated` with `status: "False"`. Until this condition is `True`, the TSR-Manager webhooks reject all ProtectionGroup and ProtectionRequest creates and spec changes. ### Diagnosing ReplicationGroupStatus or RPOEvent write rejections These objects are restricted to the replication-monitor ServiceAccount. If you see a `Forbidden` error when trying to create or modify them manually, that is expected — these objects are system-managed. Do not create, update, or delete them directly. If the replication monitor itself cannot write them (visible as stalled status or missing `ReplicationGroupStatus` objects for a ProtectionGroup), check that the replication-monitor pod is running: ```bash kubectl get pods -n dr- \ -l app.kubernetes.io/part-of=site-recovery,app.kubernetes.io/component=manager-control-plane ``` ### Checking SCC admission for the node agent on OpenShift If the ProtectionZone shows `SCCAdmissionDenied` on its `WorkloadStacksReady` condition, the node agent DaemonSet was rejected by OpenShift SCC admission after the chart was installed: ```bash # Check node agent pods on the affected workload cluster kubectl --kubeconfig /path/to/workload.kubeconfig get pods -n trilio-site-recovery-system \ -l app.kubernetes.io/component=drbd-node-agent # Describe a failing pod kubectl --kubeconfig /path/to/workload.kubeconfig describe pod \ -n trilio-site-recovery-system ``` The `privileged` SCC binding is applied automatically by the ProtectionZone controller. If it is absent, re-trigger zone reconciliation by making a benign update to the ProtectionZone CR (for example, updating `spec.description`). ### Granting a user access to the TSR web console Bind one of the three persona ClusterRoles to the user: ```bash # Grant admin access kubectl create clusterrolebinding \ --clusterrole=trilio-site-recovery-admin \ --user= # Grant operator access kubectl create clusterrolebinding \ --clusterrole=trilio-site-recovery-operator \ --user= # Grant viewer access kubectl create clusterrolebinding \ --clusterrole=trilio-site-recovery-viewer \ --user= ``` ## Examples ### Example 1 — Rotate a zone credential and refresh RBAC Use this when `ClustersValidated` shows `InsufficientClusterPermissions`. ```bash tsrctl zone-credential rotate --refresh-rbac ``` ### Example 2 — Rotate a credential and revoke the previous one ```bash tsrctl zone-credential rotate --refresh-rbac --revoke-previous ``` ### Example 3 — Inspect the current credential ```bash tsrctl zone-credential inspect ``` ### Example 4 — Create a new zone credential ```bash tsrctl zone-credential create ``` ### Example 5 — Delete a zone credential ```bash tsrctl zone-credential delete ``` To force deletion when the credential is in use: ```bash tsrctl zone-credential delete --force ``` ### Example 6 — Check ProtectionZone conditions for RBAC failures ```bash kubectl get pz -n dr- -o yaml ``` Look for the `ClustersValidated` condition under `status.conditions` and read the `reason` and `message` fields. ### Example 7 — Verify that the `trilio-site-recovery-access` namespace and ServiceAccount exist on a workload cluster ```bash kubectl --kubeconfig /path/to/workload.kubeconfig \ get namespace trilio-site-recovery-access kubectl --kubeconfig /path/to/workload.kubeconfig \ get serviceaccounts -n trilio-site-recovery-access ``` ### Example 8 — Bind the admin ClusterRole to a user ```bash kubectl create clusterrolebinding site-recovery-admin-alice \ --clusterrole=trilio-site-recovery-admin \ --user=alice ``` ### Example 9 — Check whether a user can create a ProtectionRequest ```bash kubectl auth can-i create protectionrequests.siterecovery.trilio.io \ -n dr- --as ``` ### Example 10 — Collect a support bundle after an RBAC failure If the problem is not resolved by the steps above, collect a diagnostic bundle and send it to Trilio support: ```bash oc adm must-gather \ --image=quay.io/triliovault/site-recovery-must-gather: \ -- /usr/bin/gather ``` ## Troubleshooting Use the following table to match a symptom to its most likely cause and fix. --- **Symptom:** `ProtectionZone` stuck in `Unavailable` or `Degraded`; `ClustersValidated` condition is `False` with reason `InsufficientClusterPermissions` **Cause:** The zone kubeconfig credential authenticates successfully, but the bound ServiceAccount is missing one or more RBAC verbs that the Site Recovery controllers require. **Fix:** Rotate the credential with `--refresh-rbac` to re-apply the RBAC rules: ```bash tsrctl zone-credential rotate --refresh-rbac ``` After rotation, watch the ProtectionZone until `ClustersValidated` becomes `True`: ```bash kubectl get pz -n dr- -w ``` --- **Symptom:** `ClustersValidated` is `False` with reason `ClusterIdentityChanged` **Cause:** The zone credential now reaches a cluster whose `kube-system` Namespace UID differs from the UID recorded when the zone first claimed the cluster. This happens when the credential is repointed to a different cluster. **Fix:** There is no override. You must either restore the original cluster or re-create the ProtectionZone. Contact Trilio support for guidance. --- **Symptom:** `ClustersValidated` is `False` with reason `ClusterAlreadyClaimed` **Cause:** The workload cluster is already leased to another ProtectionZone. Each workload cluster can belong to only one zone at a time. **Fix:** Identify the zone that holds the lease by inspecting the `siterecovery.trilio.io/owner-zone` annotation on the `trilio-site-recovery-system` Namespace of the workload cluster: ```bash kubectl --kubeconfig /path/to/workload.kubeconfig \ get namespace trilio-site-recovery-system \ -o jsonpath='{.metadata.annotations}' ``` Release the cluster from the other zone before adding it to this one. --- **Symptom:** Creating a `ProtectionGroup` or `ProtectionRequest` returns a webhook rejection **Cause:** The ProtectionZone's `ClustersValidated` condition is not yet `True`. The TSR-Manager admission webhooks block all ProtectionGroup and ProtectionRequest creates and spec changes until validation passes. **Fix:** Wait for or resolve the `ClustersValidated` condition. Check zone status: ```bash kubectl get pz -n dr- \ -o jsonpath='{.status.conditions[?(@.type=="ClustersValidated")]} ``` --- **Symptom:** Creating a `ProtectionRequest` returns `ReplicationProtocolA is not supported yet` **Cause:** `spec.replicationConfig.replicationProtocol` was set to `A`. Protocol A (asynchronous) is rejected by the admission webhook in Tech Preview. Only Protocol C (synchronous) is supported. **Fix:** Remove the `replicationProtocol` field or set it to `C`. RPO must be `0` when Protocol C is used: ```yaml spec: replicationConfig: replicationProtocol: C rpo: 0 ``` --- **Symptom:** Manually creating or updating a `ReplicationGroupStatus` or `RPOEvent` returns `Forbidden` **Cause:** These objects are restricted to the replication-monitor ServiceAccount by the admission webhook. The error message will reference `replication-monitor`. **Fix:** Do not create or modify these objects manually. They are system-managed by the replication-monitor controller. If they are missing, the controller will recreate them on its next polling interval. If the controller itself is not running, check the manager-control-plane pod: ```bash kubectl get pods -n dr- \ -l app.kubernetes.io/component=manager-control-plane ``` --- **Symptom:** Node agent DaemonSet pods are not starting on a workload cluster; ProtectionZone shows `SCCAdmissionDenied` on the `WorkloadStacksReady` condition **Cause:** The node agent DaemonSet requires the `privileged` SCC on OpenShift. The ProtectionZone controller applies this binding automatically, but it may have been removed or the initial application may have failed. **Fix:** Verify the SCC binding exists on the workload cluster: ```bash kubectl --kubeconfig /path/to/workload.kubeconfig \ get clusterrolebindings | grep site-recovery ``` If missing, re-trigger zone reconciliation by making a benign change to the ProtectionZone CR (for example, update `spec.description`). The controller will re-apply the SCC binding on the next reconcile. --- **Symptom:** A user can authenticate to the TSR web console but cannot perform operations (buttons are disabled or actions are rejected) **Cause:** The user's account is not bound to a Site Recovery persona ClusterRole, or is bound to a lower-privilege role (viewer) than required for the action. **Fix:** Bind the appropriate ClusterRole: ```bash # For full operational access kubectl create clusterrolebinding \ --clusterrole=trilio-site-recovery-operator \ --user= ``` The three roles are `trilio-site-recovery-admin`, `trilio-site-recovery-operator`, and `trilio-site-recovery-viewer`. The UI evaluates each action at runtime via `SelfSubjectAccessReview` using the caller's token. --- **Symptom:** `kubectl get protectionzones` (or other Site Recovery CRs) against a workload cluster returns `server doesn't have a resource type` **Cause:** The TSR-Manager-side CRDs (ProtectionZone, ProtectionGroup, ProtectionRequest, FailoverRequest, TestFailover, Hook, DRBDReplicationPolicy, ReplicationGroupStatus, RPOEvent) are only registered on the TSR-Manager cluster. The workload cluster only has the `DRBDResource` CRD. **Fix:** Run TSR-Manager-side CR queries against the TSR-Manager cluster. Run `DRBDResource` queries against the workload cluster: ```bash # Correct — TSR-Manager cluster kubectl get protectionzones -n dr- # Correct — workload cluster kubectl --kubeconfig /path/to/workload.kubeconfig \ get drbdresources -n ``` --- **Symptom:** `oc adm drain` on a workload node does not complete **Cause:** A running protected VM is pinned to that node. Live migration and cold migration of protected VMs are denied by the admission webhook. The drain cannot evict the VM. **Fix:** Before draining a node that hosts a protected VM, you must either complete an unprotect operation for the VM, or perform a planned failover of its ProtectionGroup to move the active workload to the DR cluster. Do not attempt to force-delete the VM or its DRBDResource without contacting Trilio support.