Skip to content

Low-Side E2E Troubleshooting Runbook

How to use this runbook

Encounter an error during run_e2e.sh, prepare_container_apps.py, or deploy_container_apps.py? Match the symptom to the sections below. The current low-side happy path uses Azure Container Apps, not VM staging.

Diagnostic commands

Run these to inspect the deployed state:

# Show the latest successful infra deployment outputs
az deployment group list \
  --resource-group <rg> \
  --query "[?properties.provisioningState=='Succeeded'] | sort_by(@, &properties.timestamp)[-1].name"

# Review ACA app status
az containerapp show --resource-group <rg> --name <resource-suffix>-pulp-api
az containerapp show --resource-group <rg> --name <resource-suffix>-pulp-content
az containerapp show --resource-group <rg> --name <resource-suffix>-pulp-worker

# Review ACA job execution state
az containerapp job execution show \
  --resource-group <rg> \
  --name <resource-suffix>-pulp-db-init \
  --job-execution-name <execution-name>

# Tail ACA logs
az containerapp logs show \
  --resource-group <rg> \
  --name <resource-suffix>-pulp-api \
  --follow false

# List Key Vault secrets
az keyvault secret list --vault-name <kv-name> --query "[].name"

R-10: Python dependency import failure

Symptom:

ModuleNotFoundError: No module named 'yaml'

Root cause: the local Python dependencies required by the bootstrap helpers are not installed.

Fix:

python3 -m pip install 'PyYAML>=6.0,<7.0' 'requests>=2.31,<3.0'

Prevention: install the dependencies before running the bootstrap helpers on a new workstation.

R-16: Key Vault secret operations return 403 Forbidden

Symptom:

(AuthorizationFailed) The user, group or application does not have the right permissions on the resource.

Root cause: the operator identity does not have enough Key Vault secret permissions to create or update the runtime secrets.

Fix: grant the operator identity Key Vault Secrets Officer on the deployed Key Vault, then rerun prepare_container_apps.py or run_e2e.sh.

Prevention: keep keyVaultAccessObjectIds populated with the operator object IDs before the initial deployment.

R-20: Pulp API returns 401 Invalid username/password

Symptom:

{"detail":"Invalid username/password"}

Root cause: the admin password in Key Vault does not match the password applied during the pulp-db-init job, or the initialization job did not complete successfully.

Fix:

  1. inspect artifacts/e2e/<timestamp>/db-init-execution.json
  2. verify the pulp-admin-password secret value in Key Vault
  3. rerun the pulp-db-init job after correcting the secret

Prevention: let prepare_container_apps.py own the runtime secret creation path and avoid manual drift between Key Vault and the database-init job.

R-23: Operator cannot reach the ACA ingress

Symptom:

Expected 200 from https://<host>/pulp/api/v3/status/, got 000

or:

run_e2e.sh currently requires enablePublicValidation=true

Root cause: either enablePublicValidation is disabled, or repoClientAddressPrefix does not include the operator's egress CIDR.

Fix:

  1. set enablePublicValidation=true in the low-side parameter file
  2. set repoClientAddressPrefix to the operator CIDR in CIDR notation
  3. redeploy infra/low-side/main.bicep
  4. rerun run_e2e.sh

Prevention: treat the operator CIDR and public-validation flag as required inputs for the current ACA E2E workflow.

R-24: ACA build or job execution fails

Symptom:

Job <name> execution <execution> failed with status Failed

or:

ERROR: (ContainerAppOperationError) ...

Root cause: the derived image build failed, the ACA deployment failed, or one of the bootstrap jobs exited non-zero.

Fix:

  1. inspect artifacts/e2e/<timestamp>/run_e2e.log
  2. inspect artifacts/e2e/<timestamp>/db-init-execution.json or reconcile-execution.json
  3. review ACA logs with az containerapp logs show
  4. fix the underlying image, secret, or runtime configuration error and rerun the deployment step

Prevention: validate the Bicep and Python surfaces locally before deployment:

az bicep build --file infra/low-side/main.bicep
az bicep build --file infra/low-side/containerapps.bicep
python3 -m py_compile automation/bootstrap/prepare_container_apps.py automation/bootstrap/deploy_container_apps.py automation/bootstrap/reconcile_pulp.py automation/bootstrap/validate_upstream.py
bash -n automation/bootstrap/run_e2e.sh

How to access the Pulp admin console

Overview

Pulp does not expose a traditional web UI at /admin/. The operator interface is the DRF browsable API at /pulp/api/v3/ and raw JSON API calls authenticated with HTTP Basic Auth.

  • Admin username: admin (set in infra/low-side/containerapps.bicep)
  • Admin password: stored as secret pulp-admin-password in the deployed Key Vault

Step 1 — retrieve the admin password from AKV

az keyvault secret show \
  --vault-name <kv-name> \
  --name pulp-admin-password \
  --query value -o tsv

Replace <kv-name> with the deployed Key Vault name (visible in Bicep outputs or the Azure portal). The operator identity requires at minimum Key Vault Secrets User on the vault.

Step 2 — verify API access

curl -u admin:<password> \
  https://<api-host>/pulp/api/v3/status/

Expected: HTTP 200 with a JSON body containing online_workers.

Step 3 — browse the API (optional)

The DRF browsable HTML interface is available when you request text/html:

https://<api-host>/pulp/api/v3/

Open this URL in a browser and authenticate with admin / <password> when prompted.

Note: /admin/ returns 404. The Django admin interface is not enabled in the Pulp ACA deployment.

Common API operations

# List repositories
curl -u admin:<password> https://<api-host>/pulp/api/v3/repositories/

# List distributions
curl -u admin:<password> https://<api-host>/pulp/api/v3/distributions/

# List users
curl -u admin:<password> https://<api-host>/pulp/api/v3/users/

# View sync task status
curl -u admin:<password> https://<api-host>/pulp/api/v3/tasks/

Troubleshooting

Symptom Likely cause Fix
HTTP 401 Invalid username/password Password drift between AKV and DB init See R-20 above
HTTP 403 on AKV secret show Missing Secrets User/Officer role See R-16 above
curl: (6) Could not resolve host ACA ingress not reachable See R-23 above

See also