Secret inventory and lifecycle¶

This document provides the authoritative inventory of all secrets used in the linux-update-cds system, their consumers, storage locations, and lifecycle management.

Overview¶

All production secrets are stored in Azure Key Vault with RBAC-based access control. Runtime workloads consume secrets via Key Vault references (Azure Container Apps) or secret fetching at container startup (VM-hosted deployments). No production secrets are committed to source control.

Secret storage architecture¶

Production storage: Azure Key Vault (one per environment: low-side, high-side)
Development storage: Local .env files (gitignored, never committed)
Access control: Azure RBAC
Operators: Key Vault Secrets Officer (read + write)
Runtime workloads: Key Vault Secrets User (read-only)
Workload identity: Azure Container Apps managed identity or VM system-assigned managed identity

Complete secret inventory¶

Core Pulp runtime secrets¶

Secret Name	Storage Location	Runtime Binding	Consumer	Purpose	Generation Method	Rotation Owner	Default Rotation Interval
`pulp-secret-key`	Azure Key Vault	`PULP_SECRET_KEY`	Pulp API, workers	Django application secret for session signing, CSRF tokens	50-byte random string (base64)	Operator	Annual
`pulp-admin-password`	Azure Key Vault	`PULP_ADMIN_PASSWORD`	Pulp API, automation scripts	Admin account password for Pulp API authentication	32-byte random password (base64)	Operator	Quarterly
`pulp-db-password`	Azure Key Vault	`POSTGRES_PASSWORD`	Pulp API, workers, content	PostgreSQL `pulp` user password	32-byte random password (base64)	Operator	Quarterly
`pulp-db-symmetric-key`	Azure Key Vault	(file-backed via secret volume)	Pulp API, workers	Database field encryption using Fernet	32-byte Fernet key (base64)	Operator	Annual

Azure infrastructure secrets¶

Secret Name	Storage Location	Runtime Binding	Consumer	Purpose	Generation Method	Rotation Owner	Default Rotation Interval
`pulp-storage-account-key`	Azure Key Vault	`AZURE_STORAGE_KEY`	Pulp API, workers	Azure Blob Storage artifact backend access	Azure-managed primary key	Operator	Quarterly
`pulp-redis-url`	ACA inline secret (not Key Vault)	`PULP_REDIS_URL`	Pulp API, workers, content	Redis task queue connection string with embedded TLS primary key (`rediss://:KEY@HOST:PORT/0`)	Derived from `redis.listKeys().primaryKey` at Bicep deploy time (`infra/low-side/containerapps.bicep`)	Operator	When Azure Cache for Redis primary key is regenerated

External integration secrets (future)¶

Secret Name	Storage Location	Runtime Binding	Consumer	Purpose	Generation Method	Rotation Owner	Default Rotation Interval
`rhsm-username`	Azure Key Vault	`RHSM_USERNAME`	Pulp workers	Red Hat Subscription Manager username	Operator-provided RHSM credential	Operator	Per RHSM policy
`rhsm-password`	Azure Key Vault	`RHSM_PASSWORD`	Pulp workers	Red Hat Subscription Manager password	Operator-provided RHSM credential	Operator	Per RHSM policy

TLS/HTTPS secrets¶

Secret Name	Storage Location	Runtime Binding	Consumer	Purpose	Generation Method	Rotation Owner	Default Rotation Interval
`pulp-tls-cert`	Azure Key Vault	`TLS_CERT_PATH`	ACA ingress or reverse proxy	HTTPS certificate (PEM)	Certificate authority or ACME service	Operator/CA	90 days before expiration
`pulp-tls-key`	Azure Key Vault	`TLS_KEY_PATH`	ACA ingress or reverse proxy	HTTPS private key (PEM)	Generated with certificate request	Operator/CA	When certificate is renewed

Secret lifecycle stages¶

1. Generation¶

Secrets are generated using one of these methods:

Random high-entropy: Python os.urandom() with base64 encoding (default for most secrets)
Azure-managed: Storage account keys regenerated via Azure CLI (az storage account keys renew)
Operator-provided: RHSM credentials, TLS certificates from external sources
Key Vault automatic: Secrets Officer can create directly in Key Vault using Azure Portal or CLI

Primary generation script: automation/bootstrap/prepare_container_apps.py

This script implements "generate-if-absent" logic: 1. Check if secret exists in Key Vault 2. If missing, generate using appropriate method 3. Store in Key Vault with current timestamp metadata 4. Return secret value for runtime use

2. Storage¶

All production secrets are stored in Azure Key Vault with these controls:

Purge protection: Enabled (secrets cannot be permanently deleted for 90 days after soft-delete)
RBAC authorization: Enabled (no legacy access policies)
Network ACLs: Allow for Milestone 1 bootstrap; future milestone may restrict to private endpoint
Audit logging: Enabled via Azure Monitor (all secret read/write operations logged)
Versioning: Automatic (Key Vault maintains previous secret versions for rollback)

Key Vault naming: pulp<namePrefix>kv (e.g., pulpm1lowkv for low-side, pulpm1highkv for high-side)

Bicep configuration: infra/_shared/keyvault.bicep, infra/high-side/keyvault.bicep

3. Distribution¶

Secrets are distributed to runtime workloads via two paths:

Azure Container Apps (production path): - Secrets stored as ACA secret references in infra/low-side/containerapps.bicep - Key Vault reference format: @Microsoft.KeyVault(SecretUri=https://<vault>.vault.azure.net/secrets/<name>) - ACA managed identity fetches secrets at container startup - Environment variables injected into container runtime

VM-hosted Compose (development and transition path): - Operator runs prepare_container_apps.py to fetch secrets from Key Vault - Secrets written to local .env file (gitignored) - Docker Compose mounts .env as environment variables - Alternative: SSH to VM and use Azure CLI with VM managed identity

4. Consumption¶

Runtime workloads consume secrets as environment variables:

Pulp API: PULP_SECRET_KEY, PULP_ADMIN_PASSWORD, POSTGRES_PASSWORD, AZURE_STORAGE_KEY
Pulp workers: Same as API, plus future RHSM_USERNAME, RHSM_PASSWORD
Pulp content: POSTGRES_PASSWORD, AZURE_STORAGE_KEY
Bootstrap/reconcile jobs: PULP_ADMIN_PASSWORD for API authentication

Settings configuration: runtime/compose/pulp/settings.py

Fail-closed behavior: Missing required secrets cause runtime startup failure (no unsafe fallbacks in production)

5. Rotation¶

Secrets are rotated according to schedule or trigger events (compromise, personnel change, expiration).

Rotation procedure: See ../runbooks/secrets-rotation.md

Rotation triggers: - Scheduled: Annual (secret key, symmetric key), Quarterly (passwords, storage key), 90-day (TLS certificates) - Event-driven: Personnel change (admin password, RHSM), suspected compromise (any secret), Azure key regeneration (storage key) - Automated (future): Key Vault expiration dates + Azure Functions or Logic Apps

Evidence collection: Rotation events logged to artifacts/rotation/<secret-name>-<YYYYMMDD>.log (gitignored)

6. Revocation and expiration¶

Revocation process: 1. Rotate secret immediately (generates new value) 2. Remove compromised value from Key Vault (soft-delete, purge after 90 days) 3. Audit access logs to identify unauthorized access 4. Document incident for security review

Expiration policy (future Milestone 3): - Set Key Vault secret expiration dates at creation time - Configure Azure Monitor alerts for secrets nearing expiration - Automate rotation workflow triggered by expiration dates

Secret leakage prevention¶

Committed source control guardrails¶

.gitignore protections:

runtime/compose/.env
runtime/compose/.env*.local
runtime/compose/pulp/certs/
infra/low-side/*.local.bicepparam
artifacts/

CI checks (planned for Epic 4: Supply-chain security): - Detect committed .local.bicepparam files - Scan for real-looking secrets (high-entropy base64 strings, Azure connection strings) - Validate .env.example contains only placeholder values

Runtime leakage prevention¶

Container logs: Pulp settings.py does not log secret values
Error messages: Django and Pulp redact sensitive fields in stack traces
Environment variable inspection: Operators with SSH/exec access can view env vars; limit access via RBAC
Memory dumps: Not captured in normal operation; require explicit operator action

Development workflow guardrails¶

Local .env files: - Must never be committed (enforced by .gitignore) - Should be regenerated per-developer, not shared via Slack/email - Contain placeholder values in .env.example template

Azure parameter files: - .local.bicepparam files are gitignored - .example.bicepparam files contain placeholders only - Real postgresAdminPassword and other secrets provided at az deployment group create time

Evidence and audit¶

Secret access audit¶

Query Key Vault secret access logs:

az monitor activity-log list \
  --resource-group <rg> \
  --offset 7d \
  --query "[?contains(operationName.value, 'KeyVault')]" \
  -o table

Secret inventory validation¶

Validate Key Vault contains required secrets:

az keyvault secret list \
  --vault-name <vault-name> \
  --query "[].name" \
  -o tsv

Expected secrets (Key Vault): - pulp-secret-key - pulp-admin-password - pulp-db-password - pulp-db-symmetric-key - pulp-storage-account-key

Note: pulp-redis-url is an ACA inline secret constructed from redis.listKeys().primaryKey at Bicep deploy time. It is not stored in Key Vault and will not appear in the list above. Validate it via az containerapp show ... --query properties.configuration.secrets.

Runtime consumption validation¶

Check ACA secret references:

az containerapp show \
  --resource-group <rg> \
  --name <app-name> \
  --query properties.configuration.secrets \
  -o table

Check VM-hosted container environment:

ssh azureuser@<vm-host>
docker exec <container-name> sh -c '
  if [ -n "${PULP_SECRET_KEY:-}" ] && [ "${PULP_SECRET_KEY}" != "unsafe-local-dev-key" ]; then
    echo "PULP_SECRET_KEY present and not the development fallback"
  else
    echo "PULP_SECRET_KEY missing or set to the development fallback" >&2
    exit 1
  fi
'

Development vs. production mode¶

Development mode (PULP_DEV_MODE=true): - Allows unsafe-local-dev-key fallback for PULP_SECRET_KEY - Accepts placeholder passwords and localhost-only URLs - MUST NOT be enabled in Azure production deployments

Production mode (default, PULP_DEV_MODE unset or false): - Requires PULP_SECRET_KEY from Key Vault; fails closed if missing - All passwords and keys must be high-entropy random values - Fail-closed behavior prevents accidental unsafe deployments

Settings enforcement: runtime/compose/pulp/settings.py lines 9-19

Compliance mappings¶

Control ID	Framework	Requirement	Implementation
IA-5	NIST SP 800-53	Authenticator management	Key Vault RBAC, rotation procedures
SC-28	NIST SP 800-53	Protection of information at rest	Key Vault encryption, symmetric key for DB fields
AC-6(1)	NIST SP 800-53	Least privilege	Read-only runtime identities, Secrets Officer for operators
AU-2	NIST SP 800-53	Audit events	Key Vault access logs via Azure Monitor
IA-5(1)	DoD IL5+	Password-based authentication	High-entropy generated passwords, quarterly rotation

Milestone roadmap¶

Milestone 1 (current): - ✅ Key Vault storage with RBAC - ✅ Generate-if-absent secret creation - ✅ Fail-closed production settings - ✅ Rotation runbook documentation

Milestone 2 (in progress): - [ ] CI secret leakage detection - [ ] Secret inventory validation tests - [ ] High-side secret replication strategy - [ ] RHSM credential integration

Milestone 3 (future): - [ ] Automated rotation workflows - [ ] Key Vault expiration monitoring - [ ] Zero-downtime rotation for ACA - [ ] Secret usage telemetry and alerting