Keystone Registration — Service, Endpoints, Account, Roles

This page walks you through registering Trilio Site Recovery as a first-class OpenStack service: creating the Keystone service entry and catalog endpoints, provisioning the service account, and assigning the roles the service and its operators require. Because the service integrates with OpenStack via standard Keystone mechanisms — token validation through keystonemiddleware, catalog discovery by clients, and trust delegation to tenant projects — the registration step must be completed on both sites before any DR operation can proceed. Skipping or misconfiguring any part of this workflow causes authentication failures, missing catalog entries, or failed trust creation that surface only during a live failover.

Before registering the service, confirm the following on each site:

• OpenStack ≥ Victoria with Keystone v3 available on both sites.
• Admin credentials for the target cloud. The registration commands require the ability to create services, endpoints, users, and roles. Use the admin OpenRC file, not the DR operator credentials.
• Network connectivity between both sites:
  • Port 5000 — Keystone
  • Port 8774 — Nova
  • Port 8776 — Cinder
  • Port 8788 — protector-api
• The protector-api service deployed and reachable on port 8788. The endpoint URLs you register must resolve from both sites.
• python3-triliositerecovery installed on the machine running the openstack dr CLI, so the OpenStackClient plugin is available for verification steps.
• If you are using a Juju-based deployment, the trilio-site-recovery charm must already be related to Keystone, MySQL, and RabbitMQ. The charm's register-service action automates this entire page — read the Juju deployment path section before proceeding manually.

Juju — automated registration

If you deployed using the trilio-site-recovery-for-openstack Juju charm, Keystone registration is handled automatically when the identity-credentials relation is established, and can be re-run at any time with the register-service charm action:

juju run trilio-site-recovery/leader register-service

The charm creates the service, endpoints, roles, and service account using the credentials supplied through the identity-credentials relation. Proceed to the Verification section to confirm the results.

Kolla-Ansible and bare-metal — manual registration

For Kolla-Ansible or any deployment where the charm is not managing the Keystone relation, run the following steps on each site using admin credentials.

Step 1 — Source admin credentials

source /path/to/admin-openrc.sh

Step 2 — Create the service account

The service runs as the protector user in the service project. Create the user and assign it a role that allows keystonemiddleware to validate incoming tokens. Grant the service role if it exists on your cloud; otherwise fall back to admin:

# Create the protector service user
openstack user create \
  --domain Default \
  --password "<strong-password>" \
  protector

# Grant a role on the service project
# Prefer 'service' if the role exists; fall back to 'admin'
if openstack role show service >/dev/null 2>&1; then
  openstack role add --project service --user protector service
else
  openstack role add --project service --user protector admin
fi

Record the password — you will need it in protector.conf under [service_credentials].

Step 3 — Create DR roles

Both roles must exist in Keystone before the service starts processing requests. Trust creation names each role explicitly; a missing role causes the first DR operation to fail.

for ROLE in dr_operator dr_site_admin; do
  openstack role show "$ROLE" >/dev/null 2>&1 \
    || openstack role create "$ROLE"
done

Repeat this step on the peer site.

Step 4 — Register the service

Create the primary service entry with type disaster-recovery. The charm also registers a secondary entry with type protector for compatibility; create both:

# Primary service type (used by catalog discovery)
openstack service create \
  --name protector \
  --description "Trilio Site Recovery for OpenStack" \
  disaster-recovery

# Secondary / compatibility service type
openstack service create \
  --name protector \
  --description "Trilio Site Recovery for OpenStack" \
  protector

Step 5 — Register endpoints

Replace <PUBLIC_HOST>, <INTERNAL_HOST>, <REGION>, and <TENANT_ID_PLACEHOLDER> with values appropriate for your site. The API listens on port 8788 and the URL path includes the tenant ID template:

REGION="RegionOne"
PUBLIC_URL="http://<PUBLIC_HOST>:8788/v1/%(tenant_id)s"
INTERNAL_URL="http://<INTERNAL_HOST>:8788/v1/%(tenant_id)s"

for SERVICE_TYPE in disaster-recovery protector; do
  for IFACE in public internal admin; do
    # Pick the right base URL for each interface
    if [ "$IFACE" = "public" ]; then
      URL="$PUBLIC_URL"
    else
      URL="$INTERNAL_URL"
    fi

    # Upsert: update if the endpoint already exists, create otherwise
    EP_ID=$(openstack endpoint list \
      --service "$SERVICE_TYPE" \
      --interface "$IFACE" \
      -f value -c ID 2>/dev/null | head -1)

    if [ -n "$EP_ID" ]; then
      openstack endpoint set --url "$URL" "$EP_ID"
    else
      openstack endpoint create \
        --region "$REGION" \
        "$SERVICE_TYPE" "$IFACE" "$URL"
    fi
  done
done

Repeat Steps 4 and 5 on the peer site using that site's host addresses.

Step 6 — Run the keystone_register.py helper (optional)

The charm ships deploy/charms/trilio-site-recovery/files/keystone_register.py, an idempotent registration script that accepts a JSON request on stdin and writes a JSON result on stdout. You can use it instead of the manual CLI steps above:

python3 keystone_register.py <<'EOF'
{
  "auth_url": "https://<KEYSTONE_HOST>:35357/v3",
  "username": "admin",
  "password": "<admin-password>",
  "project_name": "admin",
  "user_domain_name": "Default",
  "project_domain_name": "Default",
  "region": "RegionOne",
  "service_name": "protector",
  "service_type": "disaster-recovery",
  "extra_service_types": ["protector"],
  "description": "Trilio Site Recovery for OpenStack",
  "roles": ["dr_site_admin", "dr_operator"],
  "endpoints": {
    "public":   "http://<PUBLIC_HOST>:8788/v1/%(tenant_id)s",
    "internal": "http://<INTERNAL_HOST>:8788/v1/%(tenant_id)s",
    "admin":    "http://<INTERNAL_HOST>:8788/v1/%(tenant_id)s"
  },
  "action": "register"
}
EOF

The script is idempotent — running it again updates stale URLs or descriptions without duplicating entries.

OpenShift (RHOSO18) — installer-managed registration

On OpenShift, Keystone registration is performed by the trilio-site-recovery-keystone-init Job, which is applied automatically by the installer script. Do not apply the job manifest in isolation:

bash deploy/openshift/scripts/install.sh

The installer applies all twelve manifests in order — namespace, service account, ConfigMap, secrets, init Jobs (db-init, rabbitmq-init, keystone-init, db-sync), Service, and Deployments — gating each Job on oc wait --for=condition=complete. Running only the deployment manifests leaves the service without a registered catalog entry, database schema, or RabbitMQ credentials.

If you need to re-run the keystone-init Job after a failed install, delete the completed Job first so Kubernetes will recreate it:

oc -n trilio-site-recovery delete job trilio-site-recovery-keystone-init
bash deploy/openshift/scripts/install.sh

After registration, the service account credentials and Keystone middleware settings must be reflected in /etc/protector/protector.conf. The options below are the ones directly relevant to Keystone integration.

[keystone_authtoken]

This section configures keystonemiddleware, which validates incoming API tokens.

[keystone_authtoken]
auth_url = https://<KEYSTONE_HOST>:5000/v3
www_authenticate_uri = https://<KEYSTONE_HOST>:5000/v3

[service_credentials]

This section holds the credentials the protector-engine uses when making cross-site OpenStack API calls and when creating per-tenant Keystone trusts. All options here affect how the engine authenticates to remote sites.

[service_credentials]
username = protector
project_name = service
default_service_project = service
default_service_user_domain = Default
default_service_project_domain = Default
# default_trust_roles defaults to member,dr_operator — omit unless you need to change it
# cafile = /etc/protector/ca.pem

[DEFAULT] — site identity

The local_site option tells the service which registered DR site this instance belongs to. It must be set before the service can participate in any DR operation. Without it the engine cannot determine which storage arrays and OpenStack endpoints are local versus remote.

[DEFAULT]
local_site = site-a

On a Juju deployment, set this via the charm option rather than editing the file directly (the charm rewrites protector.conf on every hook):

juju config trilio-site-recovery local-site=site-a

Role assignment for operators

Two roles gate access to DR operations:

Grant roles to operators after both roles exist in Keystone:

# Cloud operator — cloud-wide DR administration
openstack role add --project admin --user <admin-user> dr_site_admin

# Tenant operator — DR operations within a specific project
openstack role add --project <project> --user <operator-user> dr_operator

Important: dr_site_admin is cloud-wide regardless of which project it is granted on. Do not grant it to tenant users on DR projects expecting project-scoped access.

Verifying catalog registration

After registration, confirm that both service types are present and all three endpoint interfaces resolve correctly:

# List the registered services
openstack service list | grep -E 'disaster-recovery|protector'

# List all endpoints for the disaster-recovery service type
openstack endpoint list --service disaster-recovery

# List all endpoints for the protector service type
openstack endpoint list --service protector

Expect three endpoints per service type (public, internal, admin), each pointing to port 8788 on the appropriate host.

Verifying the service account

# Confirm the protector user exists
openstack user show protector

# Confirm its role assignment on the service project
openstack role assignment list --user protector --project service

Verifying DR roles exist on both sites

Run this command on each site using that site's admin credentials:

for ROLE in dr_operator dr_site_admin; do
  openstack role show "$ROLE" && echo "$ROLE OK" || echo "$ROLE MISSING"
done

A missing role on either site causes trust creation to fail at the first DR operation. The error message names the missing role.

Registering the peer site

Once the service is running and registered on both sites, use the openstack dr site create command (requires dr_site_admin) to register each site's API endpoint with the other:

# On site-a: register site-b
openstack dr site create site-b \
  --auth-url https://<SITE_B_KEYSTONE>:5000/v3 \
  --region RegionOne \
  --service-username protector \
  --service-password "<site-b-service-password>" \
  --service-project-name service \
  --protector-api-url http://<SITE_B_HOST>:8788

# On site-b: register site-a
openstack dr site create site-a \
  --auth-url https://<SITE_A_KEYSTONE>:5000/v3 \
  --region RegionOne \
  --service-username protector \
  --service-password "<site-a-service-password>" \
  --service-project-name service \
  --protector-api-url http://<SITE_A_HOST>:8788

Deregistering the service

To remove catalog entries without deleting the service account or roles (which Keystone owns through the identity relation), use the keystone_register.py script with "action": "deregister":

python3 keystone_register.py <<'EOF'
{
  "auth_url": "https://<KEYSTONE_HOST>:35357/v3",
  "username": "admin",
  "password": "<admin-password>",
  "project_name": "admin",
  "user_domain_name": "Default",
  "project_domain_name": "Default",
  "region": "RegionOne",
  "service_name": "protector",
  "service_type": "disaster-recovery",
  "extra_service_types": ["protector"],
  "action": "deregister"
}
EOF

The deregister action removes only the endpoints and service entries this deployment created. It does not touch the protector user, dr_site_admin, or dr_operator roles, because those may be granted to real operators.

On a Juju deployment, the charm handles deregistration automatically when the application is removed if deregister-on-remove is set to true:

juju config trilio-site-recovery deregister-on-remove=true

Example 1 — Create the DR roles and service account (Kolla / bare-metal)

source /path/to/admin-openrc.sh

# Create roles
openstack role create dr_operator
openstack role create dr_site_admin

# Create service user
openstack user create \
  --domain Default \
  --password "SecureServicePass!" \
  protector

# Grant role on service project
openstack role add --project service --user protector admin

Example 2 — Register the disaster-recovery service and endpoints

# Create service entries
openstack service create \
  --name protector \
  --description "Trilio Site Recovery for OpenStack" \
  disaster-recovery

openstack service create \
  --name protector \
  --description "Trilio Site Recovery for OpenStack" \
  protector

# Register endpoints for both service types
for SERVICE_TYPE in disaster-recovery protector; do
  openstack endpoint create --region RegionOne \
    "$SERVICE_TYPE" public \
    "http://10.0.1.10:8788/v1/%(tenant_id)s"

  openstack endpoint create --region RegionOne \
    "$SERVICE_TYPE" internal \
    "http://10.0.0.10:8788/v1/%(tenant_id)s"

  openstack endpoint create --region RegionOne \
    "$SERVICE_TYPE" admin \
    "http://10.0.0.10:8788/v1/%(tenant_id)s"
done

Example 3 — Idempotent registration with keystone_register.py

python3 deploy/charms/trilio-site-recovery/files/keystone_register.py <<'EOF'
{
  "auth_url": "https://10.0.0.1:35357/v3",
  "username": "admin",
  "password": "admin-secret",
  "project_name": "admin",
  "user_domain_name": "Default",
  "project_domain_name": "Default",
  "region": "RegionOne",
  "service_name": "protector",
  "service_type": "disaster-recovery",
  "extra_service_types": ["protector"],
  "description": "Trilio Site Recovery for OpenStack",
  "roles": ["dr_site_admin", "dr_operator"],
  "endpoints": {
    "public":   "http://10.0.1.10:8788/v1/%(tenant_id)s",
    "internal": "http://10.0.0.10:8788/v1/%(tenant_id)s",
    "admin":    "http://10.0.0.10:8788/v1/%(tenant_id)s"
  },
  "action": "register"
}
EOF

Example 4 — Juju charm registration

# Deploy the charm (assumes MySQL, RabbitMQ, Keystone already deployed)
juju deploy trilio-site-recovery-for-openstack trilio-site-recovery

# Add relations — exact relation endpoint names are required
juju add-relation trilio-site-recovery:shared-db mysql:shared-db
juju add-relation trilio-site-recovery:amqp rabbitmq:amqp
juju add-relation trilio-site-recovery:identity-credentials keystone:identity-credentials

# Wait for the unit to reach active/idle, then set the site name
juju config trilio-site-recovery local-site=site-a

# Registration happens automatically via the identity-credentials relation.
# You can re-trigger it explicitly:
juju run trilio-site-recovery/leader register-service

Example 5 — Grant dr_operator to a tenant user

# Grant dr_operator on the tenant's DR project
openstack role add \
  --project my-dr-project \
  --user alice \
  dr_operator

# Verify the assignment
openstack role assignment list \
  --user alice \
  --project my-dr-project \
  --names

Example 6 — Verify the catalog from a client node

source /path/to/dr-operator-openrc.sh

# Confirm the service is discoverable
openstack catalog show disaster-recovery

# Confirm the DR CLI can reach the API
openstack dr site list

Authentication fails at the first DR operation: "Trust creation failed for role dr_operator"

Symptom: The first openstack dr protection group command or failover attempt returns an authentication error mentioning a missing role.

Cause: The dr_operator role does not exist in Keystone on one or both sites. Trust creation names the role explicitly and fails if it is absent rather than silently skipping it. The default_trust_roles configuration (member,dr_operator) requires both roles to exist in every Keystone that the service delegates trusts to.

Fix: Run the following on each site that reports the error:

source /path/to/admin-openrc.sh
for ROLE in dr_operator dr_site_admin; do
  openstack role show "$ROLE" >/dev/null 2>&1 \
    || openstack role create "$ROLE"
done

Then retry the operation.

openstack dr site list returns "Unauthorized" or a catalog lookup failure

Symptom: The openstack dr CLI cannot contact the API and reports either an authentication error or "endpoint not found in catalog."

Cause (endpoint missing): The disaster-recovery service type was not registered, or only the protector service type was registered. The CLI discovers the endpoint by service type.

Fix — verify the catalog:

openstack service list | grep -E 'disaster-recovery|protector'
openstack endpoint list --service disaster-recovery

If entries are missing, re-run the registration steps or the register-service charm action.

Cause (wrong credentials): The user is not authenticated against the site where the protector-api is registered.

Fix: Confirm that your OpenRC file points to the correct Keystone endpoint and that the user has the dr_operator or dr_site_admin role.

Juju unit is BLOCKED after upgrade: "set local-site, pure-primary-site-name, pure-secondary-site-name"

Symptom: After upgrading the trilio-site-recovery-for-openstack charm, the unit reports a BLOCKED status with a message listing unset config keys.

Cause: This is expected behavior in charm version 1.0.20 and later. The charm requires local-site to be explicitly configured and will not infer it. If a Pure Storage array is configured, the two site name options must also be set.

Fix:

juju config trilio-site-recovery local-site=<this-site-name>

# If using Pure Storage:
juju config trilio-site-recovery \
  pure-primary-site-name=<primary-site-name> \
  pure-secondary-site-name=<secondary-site-name>

The unit returns to active once all required options are set.

Token validation fails: "401 Unauthorized" on API requests that use valid tokens

Symptom: Requests to the protector-api return 401 even though the token is valid for other OpenStack services.

Cause: The [keystone_authtoken] section in protector.conf has an incorrect auth_url or www_authenticate_uri, or the protector service account does not have a role that allows keystonemiddleware to validate tokens (requires service or admin role on the service project).

Fix: Confirm the service account role assignment:

openstack role assignment list --user protector --project service --names

Expect either the service or admin role to appear. If missing:

openstack role add --project service --user protector admin

Then confirm auth_url in protector.conf resolves to the correct Keystone v3 endpoint and restart the protector-api.

Failover fails mid-operation: "Policy doesn't allow volume_extension:volume_manage to be performed (HTTP 403)"

Symptom: Failover succeeds through storage replication but fails when importing volumes on the recovery site.

Cause: The Cinder policy on the recovery site restricts volume_extension:volume_manage to admins. The service runs as a tenant through a Keystone trust and needs this policy relaxed on both sites. Five Cinder policy rules must be relaxed, not three: volume_extension:volume_manage, volume_extension:volume_unmanage, volume_extension:services:index, volume_extension:type_get, and volume_extension:type_get_all.

Fix: On the recovery site's Cinder, add or update the policy file:

{
  "volume_extension:volume_manage": "rule:admin_or_owner",
  "volume_extension:volume_unmanage": "rule:admin_or_owner",
  "volume_extension:services:index": "rule:admin_or_owner",
  "volume_extension:type_get": "rule:admin_or_owner",
  "volume_extension:type_get_all": "rule:admin_or_owner"
}

On a Juju deployment, attach this as a policyd override to the Cinder charm using the resource name policyd-override.

Tenant mapping fails: cross-site trust scoped to wrong project

Symptom: Operations succeed on the primary site but cross-site calls return "Project not found" or "Unauthorized" on the peer site.

Cause: No tenant mapping exists for the project. Without a mapping, the engine assumes the project UUID is identical on both sites (_translate_tenant_id returns the input unchanged). This assumption holds for shared or federated Keystones but almost never for two independent clusters.

Fix: Create an explicit tenant mapping:

openstack dr tenant mapping create \
  --local-tenant <local-project-id> \
  --remote-site <peer-site-name-or-id> \
  --remote-tenant <remote-project-id>

Then validate the mapping:

openstack dr tenant mapping validate <mapping-id>