Operations
Running the service: process management, scaling, and HA
This page explains how to run Trilio Site Recovery for OpenStack day-to-day: starting, stopping, and restarting the two services (protector-api and protector-engine), scaling the API tier, verifying service health, reading logs, and understanding the operational differences between Juju-managed, Kolla-Ansible, and OpenShift deployments. It also covers routine tasks such as running database migrations manually, upgrading packages, and purging the database during decommissioning. Because Trilio Site Recovery is registered in the Keystone catalogue and coordinates across two independent OpenStack clusters, keeping both services running and in sync is essential — a failed engine silently stops all DR workflows while the API continues to accept requests.
Before performing any operational task on this page:
- Trilio Site Recovery is fully installed on both sites (see the Installation guide).
- Both
protector-apiandprotector-engineservices are managed by exactly one of the supported deployment tools: Juju (Charmed OpenStack), Kolla-Ansible, or OpenShift (RHOSO18). Commands differ by deployment model — verify which applies to your environment before running anything. - The
local_siteoption in the[DEFAULT]section of/etc/protector/protector.confis set to the name of the DR site this instance belongs to. The service cannot participate in any DR operation without it. - For Juju deployments: Juju 2.9+ client with model access, and the
trilio-site-recoveryapplication deployed and related to MySQL, RabbitMQ, and Keystone. - For Kolla deployments: SSH access to the OpenStack nodes running the
protector_apiandprotector_enginecontainers. - For OpenShift deployments:
ocCLI authenticated to the RHOSO18 cluster with access to thetrilio-site-recoverynamespace. - The
python3-triliositerecoverypackage installed on any client node where you runopenstack drcommands. Machines not managed by the Juju charm must upgrade this package separately viaapt-get. - The
dr_operatoranddr_site_adminKeystone roles must exist on both sites. Trust creation fails at the first DR operation if either role is missing.
Installation is handled during initial deployment and is not repeated here. This section covers package upgrades and the operational steps that follow a service package update.
Upgrading service packages — Juju
The Juju charm only touches apt when a source list changes, a package is missing, or the charm is refreshed. A package-only release therefore requires an explicit upgrade action.
1. Run the upgrade on each unit. Order matters: schema migrations must run before the new code starts. The action handles this automatically.
# Identify unit numbers
juju status trilio-site-recovery
# Upgrade the leader unit first (runs db-sync before restarting)
juju run-action trilio-site-recovery/leader upgrade-packages --wait
# Then upgrade each remaining unit
juju run-action trilio-site-recovery/1 upgrade-packages --wait
The action reports package versions before and after. When nothing changed it makes no other change — no restart, no migration.
2. Upgrade the CLI on client nodes not managed by the charm.
sudo apt-get update
sudo apt-get install --only-upgrade python3-triliositerecovery
3. To pin a specific version (for rollback or staged rollout):
juju run-action trilio-site-recovery/leader upgrade-packages version=1.0.23 --wait
4. Stage the migration without restarting services (advanced):
juju run-action trilio-site-recovery/leader upgrade-packages restart=false --wait
# Review, then restart when ready
juju run-action trilio-site-recovery/leader restart-services --wait
The restart is refused if the schema is behind the installed packages. Upgrade the leader and let its db-sync complete before restarting non-leader units.
Pre-upgrade preflight check (migration 022 and later)
If your upgrade crosses migration 022_add_reconcile_operation_type, the dr_operations.operation_type ENUM must be validated before the migration runs. Under Juju this happens automatically inside the db-sync and upgrade-packages actions. Under Kolla or bare-metal deployments you must run it manually:
protector-manage db preflight
protector-manage db sync
The upgrade_preflight.py script (if provided by support) must be obtained from the target release tag — it is not shipped in any Debian package.
Upgrading service packages — Kolla
# Pull the new image on each node
docker pull protector:latest
# Re-run the installer to restart containers with the new image
bash deploy/kolla/install.sh
Run protector-manage db preflight and protector-manage db sync before restarting containers when crossing migration 022.
Upgrading service packages — OpenShift
Update the image tag in your deployment or re-run the installer:
bash deploy/openshift/scripts/install.sh
The installer applies init Jobs in the correct order, including the trilio-site-recovery-db-sync Job that runs protector-manage db sync inside the container.
Configuration is managed by oslo.config. The charm renders /etc/protector/protector.conf from relation data; do not edit this file by hand on Juju-managed nodes because the charm overwrites it on every hook. Use juju config for options the charm models directly, and config-flags for anything else.
Key options by section
[DEFAULT]
| Option | Type | Default | Purpose |
|---|---|---|---|
local_site | string | (none) | Required. The name of the DR site this instance belongs to. Must match a registered site name exactly. The service cannot participate in any DR operation without this set. |
host | string | localhost | Hostname used to identify this service node in RPC messages. |
use_mock_storage | boolean | false | Use the in-memory mock storage driver instead of Pure Storage. For evaluation and lab use only. |
use_mock_cinder | boolean | false | Use a mock Cinder client. For evaluation only. |
[api]
| Option | Type | Default | Purpose |
|---|---|---|---|
bind_host | string | 0.0.0.0 | IP address the API listens on. |
bind_port | integer | 8788 | TCP port for the API. Both sites must be reachable on this port. |
workers | integer | 4 | Number of gunicorn worker processes. Scale this to match available CPU cores on the API node. |
[database]
| Option | Type | Default | Purpose |
|---|---|---|---|
connection | string | (none) | SQLAlchemy connection URL for the protector MariaDB/MySQL database, e.g. mysql+pymysql://protector:PASSWORD@HOST/protector. |
[engine]
| Option | Type | Default | Purpose |
|---|---|---|---|
topic | string | protector-engine | RabbitMQ topic the engine listens on. |
recovery_point_interval_minutes | integer | 0 | How often the engine creates automatic recovery points. 0 disables scheduled recovery points. |
recovery_point_max_count | integer | 20 | Maximum number of recovery points retained per protection group. |
recovery_point_ttl_hours | integer | 24 | Retention time for recovery points in hours. |
auto_create_security_groups | boolean | false | Whether the engine creates missing security groups on the DR site during failover. |
cross_site_timeout | integer | 30 | Timeout in seconds for cross-site API calls. Increase on high-latency links. |
[pure_storage]
Required when storage_driver = pure. Each primary_site_name / secondary_site_name must name the site whose FlashArray is at the matching primary_management_ip / secondary_management_ip. The pairing is an invariant: an identical section works unchanged on both nodes.
| Option | Type | Default | Purpose |
|---|---|---|---|
primary_management_ip | string | (none) | Management IP of the primary FlashArray. |
primary_api_token | string | (none) | API token for the primary array. |
secondary_management_ip | string | (none) | Management IP of the secondary FlashArray. |
secondary_api_token | string | (none) | API token for the secondary array. |
primary_site_name | string | (none) | OpenStack site name that uses the primary FlashArray. |
secondary_site_name | string | (none) | OpenStack site name that uses the secondary FlashArray. |
default_rpo_seconds | integer | 900 | Default RPO for async replication. |
snapshot_interval_seconds | integer | 300 | Interval between automatic snapshots for async replication. |
verify_ssl | boolean | false | Verify TLS certificates when connecting to FlashArrays. |
additional_arrays | list | (empty) | Names of additional array-pair sections, e.g. array-pair-2. Each named section [pure_array:array-pair-2] carries its own management IPs, API tokens, site names, and cinder_backend_names. |
[storage]
| Option | Type | Default | Purpose |
|---|---|---|---|
storage_driver | string | mock | Active storage backend. Valid short names: mock, pure. drbd is in development and not for production. A full class path is also accepted. |
[service_credentials]
| Option | Type | Default | Purpose |
|---|---|---|---|
default_trust_roles | string | member,dr_operator | Roles delegated by every per-tenant Keystone trust. Both roles must exist in Keystone before any DR operation. Do not remove member — it carries the Nova/Cinder access the trust needs. |
default_service_project | string | service | Default project for the service account. |
[keystone_authtoken]
| Option | Type | Purpose |
|---|---|
| auth_url | string | Keystone authentication URL for this site. |
| www_authenticate_uri | string | Public Keystone URI returned in WWW-Authenticate headers. |
Setting options on Juju
For options the charm models directly:
juju config trilio-site-recovery local-site=site-a
juju config trilio-site-recovery pure-primary-site-name=site-a
juju config trilio-site-recovery pure-secondary-site-name=site-b
For any configuration key the charm does not model, use config-flags. The format is section:key=value:
juju config trilio-site-recovery config-flags="engine:cross_site_timeout=60"
Warning: Do not use
config-flagsfor a key the charm already renders. The charm template will produce a duplicate section header, which tools other than oslo.config (such as Python'sconfigparser) will reject withDuplicateSectionError. The keyengine:cross_site_timeoutis safe because the charm template does not render it.
After a Juju upgrade to 1.0.20 or later, the unit may land in BLOCKED with the message set local-site, pure-primary-site-name, pure-secondary-site-name. This is expected — set those three config options and the unit will proceed.
Starting and stopping services
Juju
The charm manages both protector-api and protector-engine as systemd services. Use charm actions rather than systemd directly:
# Restart both services on a unit
juju run-action trilio-site-recovery/0 restart-services --wait
# Pause (stop) services without removing them
juju run-action trilio-site-recovery/0 pause --wait
# Resume paused services
juju run-action trilio-site-recovery/0 resume --wait
Kolla
Services run as Docker containers named protector_api and protector_engine:
# Restart the API container
docker restart protector_api
# Restart the engine container
docker restart protector_engine
# Stop a container
docker stop protector_engine
# Start a stopped container
docker start protector_engine
OpenShift
Services run as Kubernetes Deployments in the trilio-site-recovery namespace. Restart by rolling the Deployment:
# Restart the API deployment
oc rollout restart deployment/trilio-site-recovery-api -n trilio-site-recovery
# Restart the engine deployment
oc rollout restart deployment/trilio-site-recovery-engine -n trilio-site-recovery
# Check rollout status
oc rollout status deployment/trilio-site-recovery-api -n trilio-site-recovery
Checking service health
The API exposes an unauthenticated health endpoint at /healthz. It returns HTTP 200 when the API can serve requests and HTTP 503 when it cannot.
# From any host with network access to port 8788
curl -s -o /dev/null -w "%{http_code}" http://<api-host>:8788/healthz
The Kolla container HEALTHCHECK polls /healthz every 30 seconds automatically.
For Juju, the health-check action checks the local API port and reports the systemd state of both services:
juju run-action trilio-site-recovery/0 health-check --wait
Reading logs
Juju and Kolla (bare-metal)
Logs are written to /var/log/protector/:
# Follow the API log
tail -f /var/log/protector/protector-api.log
# Follow the engine log
tail -f /var/log/protector/protector-engine.log
OpenShift
On OpenShift, services log to stdout only — there is no /var/log/protector/ directory. The oc logs command is the only way to read service output. Both label selectors are required: app.kubernetes.io/name identifies the product and app.kubernetes.io/component distinguishes api from engine.
# API logs
oc logs -n trilio-site-recovery \
-l app.kubernetes.io/name=trilio-site-recovery,app.kubernetes.io/component=api \
--tail=100 -f
# Engine logs
oc logs -n trilio-site-recovery \
-l app.kubernetes.io/name=trilio-site-recovery,app.kubernetes.io/component=engine \
--tail=100 -f
Important: Do not use
-l app=trilio-site-recovery-apior similar bareapplabels — those labels do not exist on these pods and the command returns "No resources found".
Running database migrations
The supported command for running migrations is protector-manage db sync. It reads /etc/protector/protector.conf by default.
protector-manage db sync
# To use a different config file
protector-manage db sync --config-file /path/to/protector.conf
Under Juju, run the charm action instead — the charm passes the correct arguments automatically:
juju run-action trilio-site-recovery/leader db-sync --wait
Under OpenShift, the trilio-site-recovery-db-sync Job handles this. Under Kolla, the install script's db-sync entrypoint resolves to the same command.
Never use
python3 -m protector.db.migration upgrade headoralembic --config /etc/protector/alembic.ini upgrade head. Neither path exists. Thealembic.inifile is installed at/usr/share/trilio-site-recovery/alembic.ini, not under/etc/protector.
Re-registering the Keystone service
If Keystone endpoints are lost or need updating:
# Juju
juju run-action trilio-site-recovery/leader register-service --wait
This action is idempotent and safe to re-run.
Scaling the API tier
The protector-api service is stateless — it validates tokens, enforces RBAC, and dispatches work to the engine over RabbitMQ. You can run multiple API workers or multiple API units.
Workers per unit — increase workers in [api]:
# Juju
juju config trilio-site-recovery api-workers=8
Multiple Juju units — add units to the application:
juju add-unit trilio-site-recovery
The engine is a singleton RPC consumer per unit. Running multiple engine units increases throughput for concurrent DR operations but each consumes messages from the same protector-engine RabbitMQ topic, so operations are distributed across whichever engines are available.
Purging the database
The purge-database action is destructive and irreversible. Use it only for decommissioning or rebuilding a lab. Stop the services before the purge, and do not restart them after — the action leaves them stopped.
# Juju — confirmation required (pass the application name)
juju run-action trilio-site-recovery/leader purge-database confirm=trilio-site-recovery --wait
# Then remove the application
juju remove-application trilio-site-recovery
To drop the database itself rather than just its contents:
juju run-action trilio-site-recovery/leader purge-database confirm=trilio-site-recovery drop-schema=true --wait
On OpenShift, apply the purge Job:
oc apply -f deploy/openshift/manifests/job-purge.yaml
oc wait --for=condition=complete job/trilio-site-recovery-purge \
-n trilio-site-recovery --timeout=120s
Monitoring DR operations
The service emits no oslo.messaging notifications. The observability surface is entirely poll-based:
# Check replication health for a protection group
openstack dr replication health <protection-group>
# Check with a live probe against the recovery site's resources
openstack dr replication health <protection-group> --probe
# Run a full replication audit across all sites
openstack dr replication audit
# List recent recovery points
openstack dr recovery point list <protection-group>
# List recent operations
openstack dr operation list
# Show a specific operation
openstack dr operation show <operation-id>
The --probe flag asks the recovery site whether the protection group's networks, security groups, and flavors resolve there in the project it recovers into. It does not test the storage backend or replication lag. Because it makes live calls to the remote site through the same trust-scoped session a recovery uses, it correctly identifies resources that exist on the DR site but under a different project — which a plain health check cannot detect. Run it before any planned failover window.
Operation records include a progress field (0–100) and display source_site and target_site by name. There is no estimated completion time — the service does not compute one.
Check service health from the CLI
curl -s -o /dev/null -w "%{http_code}" http://192.0.2.10:8788/healthz
Restart services on a Juju unit
juju run-action trilio-site-recovery/0 restart-services --wait
Run a health check action on a Juju unit
juju run-action trilio-site-recovery/0 health-check --wait
Upgrade packages on all Juju units
# Leader first (runs schema migration)
juju run-action trilio-site-recovery/leader upgrade-packages --wait
# Then remaining units
juju run-action trilio-site-recovery/1 upgrade-packages --wait
juju run-action trilio-site-recovery/2 upgrade-packages --wait
Upgrade CLI on a client node not managed by the charm
sudo apt-get update
sudo apt-get install --only-upgrade python3-triliositerecovery
Pin a specific package version for rollback
juju run-action trilio-site-recovery/leader upgrade-packages version=1.0.22 --wait
Run database migrations manually (Kolla or bare-metal)
protector-manage db preflight
protector-manage db sync
Run the db-sync charm action
juju run-action trilio-site-recovery/leader db-sync --wait
View the rendered configuration (secrets redacted)
juju run-action trilio-site-recovery/0 show-config --wait
Follow API logs on a Kolla node
tail -f /var/log/protector/protector-api.log
Follow engine logs on OpenShift
oc logs -n trilio-site-recovery \
-l app.kubernetes.io/name=trilio-site-recovery,app.kubernetes.io/component=engine \
-f
Restart Kolla containers
docker restart protector_api
docker restart protector_engine
Roll out a Kolla deployment update
docker pull protector:latest
bash deploy/kolla/install.sh
Roll out an OpenShift deployment update
bash deploy/openshift/scripts/install.sh
Restart OpenShift Deployments
oc rollout restart deployment/trilio-site-recovery-api -n trilio-site-recovery
oc rollout restart deployment/trilio-site-recovery-engine -n trilio-site-recovery
oc rollout status deployment/trilio-site-recovery-api -n trilio-site-recovery
Set a timeout for cross-site calls via config-flags (Juju)
juju config trilio-site-recovery config-flags="engine:cross_site_timeout=60"
Poll operation status
openstack dr operation list
openstack dr operation show <operation-id>
Check replication health with a live probe
openstack dr replication health my-protection-group --probe
Purge the database before decommissioning (Juju)
juju run-action trilio-site-recovery/leader purge-database confirm=trilio-site-recovery --wait
juju remove-application trilio-site-recovery
Unit lands in BLOCKED after upgrade with message about local-site
Symptom: After upgrading the Juju charm to 1.0.20 or later, the unit status shows BLOCKED: set local-site, pure-primary-site-name, pure-secondary-site-name.
Cause: This is expected behaviour. The charm requires these three options to be explicitly set after an upgrade — it does not infer which site it is on.
Fix:
juju config trilio-site-recovery local-site=<site-name>
juju config trilio-site-recovery pure-primary-site-name=<primary-site-name>
juju config trilio-site-recovery pure-secondary-site-name=<secondary-site-name>
API returns HTTP 503 — database temporarily unavailable
Symptom: Requests return 503 Service Unavailable with the message Database temporarily unavailable.
Cause: The API cannot reach the MariaDB/MySQL protector database. This is a local connectivity issue, unrelated to cross-site reachability.
Fix:
- Verify the database server is running and accessible from the API node.
- Check the
[database] connectionoption in/etc/protector/protector.conf. - On Juju, verify the
shared-dbrelation is healthy:juju status. - Check API logs for the underlying exception:
tail -f /var/log/protector/protector-api.log.
DR operations stop immediately with Policy doesn't allow volume_extension:volume_manage
Symptom: The first failover attempt fails partway through with an HTTP 403 from Cinder: Policy doesn't allow volume_extension:volume_manage to be performed.
Cause: Cinder restricts volume manage/unmanage to admin by default. The service runs as a tenant through a Keystone trust and needs relaxed Cinder policy on both sites. Five rules are required: volume_manage, volume_unmanage, services:index, type_get, and type_get_all.
Fix: On both sites, add or update the Cinder policy to allow these operations for the admin_or_owner rule, then restart Cinder. See the deployment prerequisites documentation for the exact policy lines.
Trust creation fails with a missing role error
Symptom: The first DR operation fails with an error indicating a Keystone role cannot be found during trust creation.
Cause: The dr_operator or dr_site_admin role does not exist in Keystone on one or both sites. Trust creation names the missing role rather than skipping it.
Fix: Create both roles on both sites before any DR operation:
openstack role create dr_operator
openstack role create dr_site_admin
openstack dr command fails with unrecognized arguments: --site-type
Symptom: Running openstack dr site create with --site-type fails.
Cause: The --site-type flag does not exist. Primary and secondary are roles a protection group assigns to a pair of sites, not properties of the site itself.
Fix: Remove --site-type from the command. The site name is the only positional argument; --auth-url is the only required flag.
oc logs returns "No resources found" on OpenShift
Symptom: Attempting to read logs with -l app=trilio-site-recovery-api or -l app=trilio-site-recovery-engine returns nothing.
Cause: The OpenShift manifests do not define a bare app label. The correct label selector uses both app.kubernetes.io/name and app.kubernetes.io/component.
Fix:
# API
oc logs -n trilio-site-recovery \
-l app.kubernetes.io/name=trilio-site-recovery,app.kubernetes.io/component=api -f
# Engine
oc logs -n trilio-site-recovery \
-l app.kubernetes.io/name=trilio-site-recovery,app.kubernetes.io/component=engine -f
Note: On OpenShift, oc logs is the only way to read service output — there is no /var/log/protector/ directory on these pods.
juju config unknown option error for cross-site-timeout or engine-cross-site-timeout
Symptom: Setting a timeout via juju config trilio-site-recovery cross-site-timeout=60 fails with an unknown option error.
Cause: Neither cross-site-timeout nor engine-cross-site-timeout is a modelled charm option. The charm exposes a fixed set of options and does not include a timeout option.
Fix: Use config-flags to reach configuration keys the charm does not model directly:
juju config trilio-site-recovery config-flags="engine:cross_site_timeout=60"
Migration fails with ENUM validation error when crossing migration 022
Symptom: protector-manage db sync fails when upgrading across migration 022_add_reconcile_operation_type with an error about the operation_type ENUM column.
Cause: Migration 022 requires the ENUM's existing member names to be in uppercase. The preflight check is not run automatically by the bare protector-manage db sync command — only the Juju charm's db-sync and upgrade-packages actions run it automatically.
Fix: On Kolla and bare-metal deployments, run the preflight check manually before syncing:
protector-manage db preflight
protector-manage db sync
Operation source_site and target_site columns are blank
Symptom: openstack dr operation show <id> displays empty source_site and target_site columns.
Cause: You are running a version of the service prior to the fix that added site name resolution to these columns. The underlying API emits only source_site_id and target_site_id on old versions; name resolution was added in a later release.
Fix: Upgrade to the current release. The columns show the site name, falling back to the site ID.
--network-mapping or --flavor-mapping silently does not apply
Symptom: A failover or failback completes but the mapping you specified was not used — the wrong network or flavor was selected.
Cause: You used : as the separator instead of =. argparse accepts the colon-separated value as a single opaque string; no error is raised at parse time and the mapping is never applied.
Fix: Use = as the separator:
openstack dr failover my-pg \
--network-mapping source-network-name=target-network-id
The left side of --network-mapping must be the source network name as it appears in openstack dr operation show output, not its UUID.
Failback appears to hang on volume detach
Symptom: Failback is running but progress stalls for up to 120 seconds per volume.
Cause: This is expected behaviour. Failback waits up to 120 seconds per volume for Nova to release the Cinder attachment after the DR instance is deleted. Nova releases the attachment asynchronously after the server is gone. The wait is best-effort: if a volume is still attached at the deadline it is unmanaged anyway and logged, but failback is not aborted. On a site where Nova detaches slowly this appears as a visibly slower failback — the wait is working as intended.
Fix: No fix required. If the delay is consistently longer than expected, check Nova and Cinder logs on the DR site for attachment release errors.