Operational Playbooks

This page contains step-by-step procedures for common operational tasks, troubleshooting paths, and recovery work. Use it when something has gone wrong, when a routine maintenance action must be performed, or when you need a tested procedure instead of starting from scratch.

Quick Reference

Playbook: Terraform State Lock Stuck

Diagnosis

1. Check if another operation is running

   Look at GitHub Actions for any in-progress Terraform jobs.

2. Check the lock info

   az storage blob show \
       --account-name <storage_account> \
       --container-name tfstate \
       --name terraform.tfstate \
       --query "properties.lease"

Resolution

Option 1: Wait and Retry (Safest)

Locks typically auto-release after 15-60 minutes. Wait and retry.

Option 2: Break the Blob Lease

# List current leases
az storage blob lease show \
    --account-name <storage_account> \
    --container-name tfstate \
    --blob-name terraform.tfstate

# Break the lease (requires Storage Blob Data Owner role)
az storage blob lease break \
    --account-name <storage_account> \
    --container-name tfstate \
    --blob-name terraform.tfstate

Option 3: Force Unlock via Terraform

# Get the lock ID from the error message, then:
terraform force-unlock <LOCK_ID>

# Example:
terraform force-unlock 12345678-1234-1234-1234-123456789012

Prevention

• Never cancel Terraform runs mid-operation
• Use workflow concurrency controls (already configured)
• Ensure proper timeout settings on pipeline jobs

Playbook: OIDC Authentication Failing

Diagnosis by Error Code

Resolution Steps

1. Verify GitHub Secrets

# These secrets must be set in GitHub repository settings:
AZURE_CLIENT_ID      # Managed Identity Client ID
AZURE_TENANT_ID      # Azure AD Tenant ID
AZURE_SUBSCRIPTION_ID # Target Subscription ID

2. Verify Federated Credential Configuration

# Check federated credentials on the managed identity
az ad app federated-credential list \
    --id <APP_OBJECT_ID> \
    --query "[].{name:name, subject:subject, issuer:issuer}"

The subject must match exactly:

• For environment: repo:bcgov/ai-hub-tracking:environment:dev
• For branch: repo:bcgov/ai-hub-tracking:ref:refs/heads/main

3. Check Token Claims

Add this step to your workflow to debug the token:

- name: Debug OIDC Token
  run: |
    TOKEN=$(curl -s -H "Authorization: bearer $ACTIONS_ID_TOKEN_REQUEST_TOKEN" \
      "$ACTIONS_ID_TOKEN_REQUEST_URL&audience=api://AzureADTokenExchange" | jq -r '.value')
    echo "Token claims:"
    echo $TOKEN | cut -d. -f2 | base64 -d 2>/dev/null | jq .

4. Recreate Federated Credential

# Delete and recreate if misconfigured
az ad app federated-credential delete \
    --id <APP_OBJECT_ID> \
    --federated-credential-id <CREDENTIAL_ID>

# Recreate with correct subject
./initial-setup/initial-azure-setup.sh \
    -g "your-rg" -n "your-identity" \
    -r "bcgov/ai-hub-tracking" -e "dev"

Playbook: Rollback Failed Deployment

Option 1: Git Revert (Recommended)

# Find the last good commit
git log --oneline -10

# Revert the problematic commit(s)
git revert <BAD_COMMIT_SHA>

# Push to trigger pipeline with reverted code
git push origin main

Option 2: Restore from State Backup

Azure Blob Storage keeps versions of the state file:

# List state file versions
az storage blob list \
    --account-name <storage_account> \
    --container-name tfstate \
    --include v \
    --query "[?name=='terraform.tfstate'].{version:versionId, modified:properties.lastModified}"

# Download a previous version
az storage blob download \
    --account-name <storage_account> \
    --container-name tfstate \
    --name terraform.tfstate \
    --version-id <VERSION_ID> \
    --file terraform.tfstate.backup

Option 3: Targeted Destroy and Recreate

# Destroy specific problematic resources
terraform destroy -target=azurerm_virtual_machine.jumpbox

# Reapply to recreate with correct config
terraform apply

Playbook: Rotate Federated Credentials

To Update the Trust Relationship

1. Delete Existing Federated Credential

az ad app federated-credential list --id <APP_OBJECT_ID>
az ad app federated-credential delete \
    --id <APP_OBJECT_ID> \
    --federated-credential-id <CREDENTIAL_ID>

2. Create New Federated Credential

./initial-setup/initial-azure-setup.sh \
    -g "your-rg" \
    -n "your-identity" \
    -r "bcgov/new-repo-name" \
    -e "dev"

3. Update GitHub Secrets (if Client ID changed)

1. Go to Repository Settings → Secrets and variables → Actions
2. Update AZURE_CLIENT_ID with new Managed Identity Client ID

4. Verify New Configuration

# Trigger a test workflow run
gh workflow run deploy.yml

Playbook: Bastion Access Issues

Diagnosis

1. Check if Bastion is Deployed

az network bastion list \
    --resource-group <RG_NAME> \
    --query "[].{name:name, state:provisioningState}"

If empty, Bastion may be disabled (cost-saving). Deploy it:

# Via GitHub Actions
gh workflow run add-or-remove-module.yml -f action=add

# Or via manual trigger in GitHub UI

2. Check NSG Rules

# Bastion subnet requires specific NSG rules
az network nsg rule list \
    --resource-group <RG_NAME> \
    --nsg-name <BASTION_NSG> \
    --query "[].{name:name, access:access, direction:direction, port:destinationPortRange}"

Required inbound rules:

• HTTPS (443) from Internet
• Gateway Manager from GatewayManager service tag
• Azure Load Balancer from AzureLoadBalancer service tag

3. Check VM Status

az vm get-instance-view \
    --resource-group <RG_NAME> \
    --name <VM_NAME> \
    --query "instanceView.statuses[1].displayStatus"

VM must be in "VM running" state.

Playbook: GitHub Actions Pipeline Stuck

Resolution

1. Check for Pending Approvals

Production environments require approval. Check the workflow run for pending reviews.

2. Cancel and Retry

# Cancel via CLI
gh run cancel <RUN_ID>

# Retry
gh workflow run <WORKFLOW_NAME>

3. Check for State Lock

If Terraform is waiting on state lock, see State Lock Playbook.

4. Check Runner Health

# View recent workflow runs
gh run list --limit 10

# Check specific run logs
gh run view <RUN_ID> --log

5. GitHub Status

Check GitHub Status Page for platform issues.

Playbook: Bastion Tunnel for Local Development

Prerequisites

• Azure CLI (az) installed and signed in (az login)
• Bastion + ssh CLI extensions — the script installs them on first use
• "Virtual Machine Administrator Login" RBAC on the jumpbox (Entra ID auth) — request from the platform team
• The shared Azure Bastion + jumpbox deployed by the bcgov/action-deployer-vm-bastion-alz action (resource group ai-hub-bastion-tools, tools subscription)

Step 1: Open the Bastion SOCKS5 tunnel

The tunnel is opened with Azure Bastion native client tunnelling — no credentials to copy, auth is Entra ID + RBAC. The tunnel script is maintained upstream by the bcgov/action-deployer-vm-bastion-alz action, so fetch it on demand (we don't vendor it):

curl -fsSL \
  https://raw.githubusercontent.com/bcgov/action-deployer-vm-bastion-alz/v1.0.0/bastion-consumer-scripts/bastion-proxy.sh \
  -o bastion-proxy.sh && chmod +x bastion-proxy.sh

./bastion-proxy.sh -g ai-hub-bastion-tools -b ai-hub-bastion -v ai-hub-jumpbox \
  -s <tools-subscription-id> -t <tenant-id> -p 8228

# Prints: SOCKS5 proxy ready on localhost:8228   (leave it running)

See initial-setup/infra/scripts/bastion-proxy.md for the PowerShell variant and full details.

Step 2: Connect to a Private Database (SOCKS5)

A single SOCKS5 port reaches any hostname the jumpbox can resolve. Route a client through it — for example PostgreSQL via proxychains:

# proxychains4 configured with: socks5 127.0.0.1 8228
proxychains4 psql \
  -h <postgres-server>.postgres.database.azure.com \
  -U <username> -d <database>

Step 3: Browse Private Endpoints (Firefox + SmartProxy)

1. Install SmartProxy extension
2. Add a proxy server:
   • Type: SOCKS5
   • Address: localhost
   • Port: 8228 (or the port the script printed)
   • Enable "Proxy DNS when using SOCKS5" (remote DNS for private endpoints)
3. Add rules for Azure private endpoints:
   • *.vault.azure.net
   • *.postgres.database.azure.com
   • *.blob.core.windows.net

Step 4: Connect to Key Vault / use the Azure CLI

For HTTP-proxy clients (Azure CLI, Terraform, Postman), bridge the SOCKS5 proxy to an HTTP proxy with privoxy (see azure-proxy/privoxy or docker compose up -d):

# privoxy exposes http://127.0.0.1:8118 → Bastion SOCKS5
HTTPS_PROXY=http://127.0.0.1:8118 az keyvault secret list --vault-name <keyvault-name>

Troubleshooting

Security Notes

Related Resources

• ADR-011: Control Plane vs Data Plane Access & Bastion Tunnel
• bcgov/action-deployer-vm-bastion-alz
• Azure Bastion native client support