Secrets rotation runbook¶

This runbook documents the rotation procedures for all Key Vault-backed runtime secrets used in the low-side and high-side deployments.

Scope¶

All secrets are stored in Azure Key Vault and consumed at runtime by Azure Container Apps workloads or VM-hosted containers. Rotation happens operator-side using protected local input files with az keyvault secret set --file so secret values are not exposed in shell history, process arguments, or terminal scrollback. Runtime workloads have read-only access (Key Vault Secrets User RBAC role) and cannot rotate secrets themselves.

Secret inventory and ownership¶

Secret Name	Consumer	Owner	Rotation Trigger
`pulp-secret-key`	Pulp Django application secret	Platform team	Annual or on suspected compromise
`pulp-admin-password`	Pulp admin account authentication	Operations team	Quarterly or on personnel change
`pulp-db-password`	PostgreSQL Pulp user password	Platform team	Quarterly or on suspected compromise
`pulp-db-symmetric-key`	Database field encryption (Fernet)	Platform team	Annual or on suspected compromise
`pulp-storage-account-key`	Azure Blob Storage access	Platform team	Quarterly or when Azure regenerates primary key
`rhsm-username`	Red Hat Subscription Manager auth	Operations team	Per RHSM policy or personnel change
`rhsm-password`	Red Hat Subscription Manager auth	Operations team	Per RHSM policy or personnel change
`pulp-tls-cert`	HTTPS ingress certificate	Platform team	90 days before expiration
`pulp-tls-key`	HTTPS ingress private key	Platform team	When certificate is renewed

Prerequisites¶

Before rotating any secret:

Confirm you have Azure Key Vault Secrets Officer RBAC access to the target Key Vault
Identify all runtime consumers of the secret (ACA apps/jobs or VM-hosted containers)
Plan maintenance window for workload restarts
Prepare rollback plan (previous secret version remains in Key Vault)

Standard rotation procedure¶

Step 1: Generate new secret value¶

For random secrets, write the value directly to a protected local file instead of printing it to the terminal:

umask 077
SECRET_FILE="artifacts/rotation/<secret-name>-$(date +%Y%m%d).secret"
mkdir -p "$(dirname "${SECRET_FILE}")"

# Django secret key (50 chars)
python3 -c 'import os, base64, pathlib; pathlib.Path("'"${SECRET_FILE}"'").write_text(base64.urlsafe_b64encode(os.urandom(50)).decode().rstrip("="))'

# High-entropy password (32 bytes = 43 chars base64)
python3 -c 'import os, base64, pathlib; pathlib.Path("'"${SECRET_FILE}"'").write_text(base64.urlsafe_b64encode(os.urandom(32)).decode().rstrip("=_-"))'

# Database symmetric key (32 bytes for Fernet)
python3 -c 'import os, base64, pathlib; pathlib.Path("'"${SECRET_FILE}"'").write_text(base64.urlsafe_b64encode(os.urandom(32)).decode())'

Run only the generator matching the secret being rotated. For credential-based secrets (RHSM, TLS), obtain new values from the authoritative source and save them to a protected file with umask 077; do not paste secret values into shell commands.

Step 2: Update Key Vault secret¶

az keyvault secret set \
  --vault-name <key-vault-name> \
  --name <secret-name> \
  --file "${SECRET_FILE}"

Key Vault automatically versions secrets. The previous value remains available via version ID for rollback.

After verification and evidence capture, securely remove any local staging file:

shred -u "${SECRET_FILE}" 2>/dev/null || rm -f "${SECRET_FILE}"

Step 3: Restart runtime consumers¶

For Azure Container Apps deployments:

# Restart ACA apps
az containerapp revision restart \
  --resource-group <rg> \
  --name <app-name> \
  --revision <current-revision>

# Restart ACA jobs on next execution (jobs fetch secrets at start)
# No explicit restart needed; next scheduled execution picks up new secret

For VM-hosted compose deployments:

# SSH to VM
ssh -i ~/.ssh/<key> azureuser@<vm-host>

# Restart affected services
cd /opt/pulp-runtime
docker compose restart pulp-api pulp-content pulp-worker

Step 4: Verify operation¶

After restart, confirm runtime services are healthy:

# Check API status
curl https://<api-host>/pulp/api/v3/status/

# Check container logs for secret fetch errors
az containerapp logs show \
  --resource-group <rg> \
  --name <app-name> \
  --tail 50

# Or for VM-hosted:
docker compose logs pulp-api | tail -50

Step 5: Document rotation¶

Log rotation evidence:

Date and time
Secret name
Operator identity
Reason for rotation
Verification result

Recommended evidence path: artifacts/rotation/<secret-name>-<YYYYMMDD>.log

Secret-specific procedures¶

pulp-secret-key¶

Impact: Invalidates all Django sessions and CSRF tokens. Users must re-authenticate.

Procedure: 1. Generate new 50-char random string 2. Update Key Vault secret pulp-secret-key 3. Restart Pulp API and worker containers 4. Notify operators that re-authentication is required

Rollback: Restore previous Key Vault secret version and restart containers.

pulp-admin-password¶

Impact: Invalidates current admin credentials. Automation and operators must use new password.

Procedure: 1. Generate new high-entropy password 2. Update Key Vault secret pulp-admin-password 3. Restart Pulp API containers (password is read from env on startup) 4. Update any automation that uses hardcoded admin password

Rollback: Restore previous Key Vault secret version and restart API containers.

Note: For production, consider migrating to token-based authentication rather than password auth.

pulp-db-password¶

Impact: Database connection fails until all Pulp services restart with new password.

Procedure: 1. Generate new high-entropy password 2. Connect to PostgreSQL and update the Pulp user password from the protected password file without placing the value in shell history, SQL history, or process arguments:

export SECRET_FILE="artifacts/rotation/pulp-db-password-$(date +%Y%m%d).secret"
python3 - <<'PY' | PSQL_HISTORY=/dev/null psql "host=<pg-host> dbname=postgres user=<admin-user> sslmode=require" --set=ON_ERROR_STOP=1
import os
from pathlib import Path
password = Path(os.environ["SECRET_FILE"]).read_text()
if "\x00" in password:
    raise SystemExit("password contains NUL")
escaped = password.replace("'", "''")
print(f"ALTER USER pulp WITH PASSWORD '{escaped}';")
PY

3. Update Key Vault secret pulp-db-password 4. Restart all Pulp services (API, content, worker) simultaneously to minimize downtime

Rollback: Revert PostgreSQL user password and Key Vault secret, then restart services.

pulp-db-symmetric-key¶

Impact: Existing encrypted database fields become unreadable if key is lost. Rotation requires re-encryption.

Procedure: 1. WARNING: This rotation requires a maintenance window and data migration 2. Generate new 32-byte Fernet key 3. Export all encrypted data (Pulp remote credentials) 4. Update Key Vault secret pulp-db-symmetric-key 5. Restart Pulp services 6. Re-import encrypted data using Pulp API 7. Verify all remotes and credentials are accessible

Rollback: Restore previous Key Vault secret version. If data was re-encrypted, restore from backup.

Note: Defer this rotation unless compromise is suspected. Coordinate with database team.

pulp-storage-account-key¶

Impact: Pulp cannot read or write artifacts until services restart.

Procedure: 1. Rotate Azure Storage Account key using Azure Portal or CLI:

az storage account keys renew \
  --resource-group <rg> \
  --account-name <storage-account> \
  --key primary

2. Fetch new primary key:

umask 077
SECRET_FILE="artifacts/rotation/pulp-storage-account-key-$(date +%Y%m%d).secret"
mkdir -p "$(dirname "${SECRET_FILE}")"
az storage account keys list \
  --resource-group <rg> \
  --account-name <storage-account> \
  --query '[0].value' -o tsv | python3 -c 'import pathlib, sys; pathlib.Path("'"${SECRET_FILE}"'").write_text(sys.stdin.read().strip())'

3. Update Key Vault secret:

az keyvault secret set \
  --vault-name <kv-name> \
  --name pulp-storage-account-key \
  --file "${SECRET_FILE}"

4. Restart Pulp services

Rollback: Regenerate previous key slot (secondary) and update Key Vault + restart.

rhsm-username and rhsm-password¶

Impact: Red Hat content sync fails until credentials are updated.

Procedure: 1. Obtain new RHSM credentials from Red Hat Customer Portal or account owner 2. Update Key Vault secrets rhsm-username and rhsm-password 3. Restart Pulp worker containers 4. Trigger manual sync to validate:

python3 automation/bootstrap/reconcile_pulp.py --sync --repo rhel-8-baseos

Rollback: Restore previous Key Vault secret versions and restart workers.

pulp-tls-cert and pulp-tls-key¶

Impact: HTTPS ingress will fail if certificate is invalid or mismatched with private key.

Procedure: 1. Obtain new certificate and private key from certificate authority or ACME service 2. Verify certificate matches private key:

openssl x509 -noout -modulus -in new-cert.pem | openssl md5
openssl rsa -noout -modulus -in new-key.pem | openssl md5
# MD5 hashes must match

3. Update Key Vault secrets (certificate first, then key):

az keyvault secret set --vault-name <kv> --name pulp-tls-cert --file new-cert.pem
az keyvault secret set --vault-name <kv> --name pulp-tls-key --file new-key.pem

4. Restart ingress/reverse proxy containers or ACA ingress configuration 5. Validate HTTPS:

curl -v https://<api-host>/pulp/api/v3/status/
openssl s_client -connect <api-host>:443 -servername <api-host>

Rollback: Restore previous certificate and key versions from Key Vault.

Emergency rotation (suspected compromise)¶

If a secret is suspected to be compromised:

Immediate action: Rotate the secret following standard procedure

Audit: Review Key Vault access logs to identify unauthorized access:

az monitor activity-log list \
  --resource-group <rg> \
  --offset 7d \
  --query "[?contains(operationName.value, 'KeyVault')]"

Notify: Alert security team and document incident
Verify: Check for anomalous runtime behavior or unauthorized API access
Post-incident: Update threat model and review access controls

Automation considerations¶

Future rotation automation should: - Use Azure Key Vault secret expiration dates to trigger rotation - Integrate with Azure Container Apps revision lifecycle for zero-downtime rotation - Emit rotation events to Azure Monitor for audit trail - Support dry-run mode for validation before production rotation

Automation is tracked in Milestone 3 (issue S-M2-03).