Reference Run — Sanitized End-to-End Deployment¶

This is a sanitized record of a successful end-to-end deployment of linux-update-cds on Azure Container Apps (Commercial, centralus).

Use it as a template for what to expect during your own deployment.

Note: All subscription IDs, tenant IDs, resource names, IP addresses, and usernames have been replaced with generic placeholders. The structure and command outputs are real.

Environment¶

Parameter	Value
Cloud	Azure Commercial
Region	`centralus`
Resource group	`rg-pulp-low-side`
Resource suffix	`pulplow`
ACA environment	`pulplow-acaenv`
ACR	`<acr>.azurecr.io`
Key Vault	`<kv>.vault.azure.net`
Storage account	`pulplowst`
Deployment name	`main-infra-v3`
Image tag	`e2e-<timestamp>`
Subscription ID	`00000000-0000-0000-0000-000000000000`
Tenant ID	`00000000-0000-0000-0000-000000000001`

Run Summary¶

Script: scripts/run_e2e.py
Duration: ~45 minutes (infra + bootstrap + reconcile)
Final result: ✅ Steps 1–5 succeeded; reconcile (step 6) requires DB tuning for large repos (see Troubleshooting)

Step-by-Step Output¶

Step 1/7 — Prepare Key Vault runtime secrets¶

Using infra deployment: main-infra-v3
Planned API URL: https://pulplow-pulp-api.<aca-env-fqdn>.centralus.azurecontainerapps.io
Planned content URL: https://pulplow-pulp-content.<aca-env-fqdn>.centralus.azurecontainerapps.io

Prepared ACA runtime secrets from deployment main-infra-v3:
- key vault: <kv>
- key vault uri: https://<kv>.vault.azure.net/
- storage account: pulplowst
- generated new secrets: yes

✅ All 5 secrets generated on first run.

Step 2/7 — Validate upstream and capture resolved export¶

name                        primary_status  fallback_status  last_modified
ubuntu-jammy-amd64-base     unreachable     200              -
ubuntu-jammy-amd64-updates  unreachable     200              -
ubuntu-jammy-amd64-security unreachable     200              -

Configured upstream checks passed using fallback for 3 repositories.
export PULP_UBUNTU_REMOTE_URL=https://archive.ubuntu.com/ubuntu

✅ Azure mirror unreachable from test network; fallback to archive.ubuntu.com used. In a hardened environment, substitute your internal mirror URL.

Captured file — upstream.env:

export PULP_UBUNTU_REMOTE_URL=https://archive.ubuntu.com/ubuntu

Step 3/7 — Build image and deploy ACA workloads¶

WARNING: Packing source code into tar to upload...
WARNING: Uploading archived source code (115 KiB)...
WARNING: Queued a build with ID: cj3
2026-04-17T21:10:27Z Downloading source code...
2026-04-17T21:10:28Z Using acb_vol_... as the home volume
2026-04-17T21:10:29Z Logging in to registry: <acr>.azurecr.io
2026-04-17T21:10:29Z Successfully logged into <acr>.azurecr.io
2026-04-17T21:10:30Z Executing step ID: build
...
Run ID: cj3 was successful after 2m17s

Built/deployed low-side ACA workloads from infra deployment main-infra-v3:
- image: <acr>.azurecr.io/linux-update-cds/pulp-runtime:e2e-<timestamp>
- environment: pulplow-acaenv
- api url: https://pulplow-pulp-api.<aca-env-fqdn>.centralus.azurecontainerapps.io
- content url: https://pulplow-pulp-content.<aca-env-fqdn>.centralus.azurecontainerapps.io

✅ Container image built via az acr build (no local Docker daemon required).

Step 4/7 — Initialize the shared Pulp database¶

Waiting for pulplow-pulp-db-init/<execution-id>; current status=Running...
Waiting for pulplow-pulp-db-init/<execution-id>; current status=Running...
Job pulplow-pulp-db-init execution <execution-id> succeeded.

✅ Django migrations applied; admin password set from Key Vault.

Key environment variables injected into the ACA job:

PULP_SETTINGS              = /etc/pulp/settings.py
POSTGRES_HOST              = pulplowpg.postgres.database.azure.com
POSTGRES_PORT              = 5432
POSTGRES_DB                = pulpcore
POSTGRES_USER              = pulpadmin
PULP_API_BASE_URL          = https://pulplow-pulp-api.<aca-env-fqdn>.centralus.azurecontainerapps.io
PULP_CONTENT_ORIGIN        = https://pulplow-pulp-content.<aca-env-fqdn>.centralus.azurecontainerapps.io
PULP_ALLOWED_HOSTS_CSV     = localhost,127.0.0.1,pulplow-pulp-api,pulplow-pulp-content,...

Step 5/7 — API health check¶

GET https://pulplow-pulp-api.<aca-env-fqdn>.centralus.azurecontainerapps.io/pulp/api/v3/status/
HTTP 200 OK

{
  "versions": [{"component": "core", "version": "3.x.x", ...}],
  "online_workers": [...],
  "database_connection": {"connected": true},
  "redis_connection": {"connected": true}
}

✅ API healthy. Admin accessible at /pulp/api/v3/ with HTTP Basic Auth.

Step 6/7 — Reconcile (sync Ubuntu repository)¶

Started reconcile job: pulplow-pulp-reconcile/<execution-id>
Syncing ubuntu-jammy-amd64-base ... 60,051 content items
[WARNING] DB connection timeout during bulk_create after ~30 minutes

⚠️ Reconcile timed out during large bulk insert (~60k items). This is an operational tuning issue, not a code defect. Pulp sync is idempotent — re-running the reconcile step resumes from where it left off.

Mitigation: Apply the PostgreSQL tuning parameters documented in docs/runbooks/azure-commercial-e2e.md:

tcp_keepalives_idle = 60
tcp_keepalives_interval = 10
work_mem = 65536
idle_in_transaction_session_timeout = 0

Infrastructure Deployment (Reference)¶

Infra deployed via az deployment group create against infra/bicep/main.bicep. Representative output after a clean deploy:

Deployment succeeded: main-infra-v3
provisioningState: Succeeded
timestamp: 2026-04-17T21:13:26Z
resourceGroup: rg-pulp-low-side

Key outputs:
  acrLoginServer:      <acr>.azurecr.io
  keyVaultUri:         https://<kv>.vault.azure.net/
  apiAppUrl:           https://pulplow-pulp-api.<aca-env-fqdn>.centralus.azurecontainerapps.io
  contentAppUrl:       https://pulplow-pulp-content.<aca-env-fqdn>.centralus.azurecontainerapps.io
  postgresHostname:    pulplowpg.postgres.database.azure.com
  storageAccountName:  pulplowst
  resourceSuffix:      pulplow

Known Issues Encountered and Fixed¶

During the runs leading to this successful deployment, the following code defects were discovered and fixed. All fixes are committed to main:

ID	Component	Issue	Fix commit
F-2	`keyvault.bicep`	Wrong Key Vault Secrets Officer role GUID	`99545df`
F-3	`main.bicep`	Log Analytics `existing` resource race condition	`daa16dd`
F-4	`keyvault.bicep`	KV soft-delete name collision	`f209547`
F-5	`network.bicep`	Missing `Microsoft.App/environments` subnet delegation	`f209547`
F-6	`run-pulp-*.sh`	`FILE_UPLOAD_TEMP_DIR` missing on fresh Azure Files share	`cf9b7c2`
F-8	`settings.py`	PostgreSQL SSL not enforced for Azure Flex Server	`47afaa3`
F-9	`database.bicep`	`hstore` extension not allow-listed	`b9ee996`
F-10	ACA ingress	API IP restriction blocked internal ACA job traffic	`c6c7cd8`

Troubleshooting¶

db-init fails → inspect artifacts/e2e/<timestamp>/db-init-execution.json and see troubleshooting.md R-24.
reconcile times out → re-run; sync is idempotent. Apply PG tuning params.
API health 503 → check ACA revision is active, container started, secrets are bound. See troubleshooting.md.

This document was generated from a real deployment run and sanitized for public distribution. Placeholder values: sub 00000000-0000-0000-0000-000000000000, tenant 00000000-0000-0000-0000-000000000001, resources <acr> / <kv>.