Technical Deep Dive: Management Access, Data Access, and the Bastion Tunnel

This document explains one of the most important ideas in the whole platform: the difference between managing an Azure resource and accessing the private data inside that resource. It then shows why some Terraform operations fail from a public automation runner and how the Azure Bastion native tunnel solves that problem.

Control Plane vs Data Plane: The Fundamental Divide
Key Vault Operations: What Requires Data Plane Access
Terraform + Key Vault: When Operations Fail
Azure Bastion: How the Tunnel Works
Deployment Scenarios: Who Needs What

1. Control Plane vs Data Plane: The Fundamental Divide

What is the Control Plane?

The control plane is Azure's management layer. It is the part of the platform that lets you create, update, delete, and describe resources:

Creating, updating, deleting resources
Reading metadata and configuration
Managing RBAC and policies

Endpoint: management.azure.com

Authentication: OpenID Connect, often shortened to OIDC, works well here because these calls go to Azure's public management endpoint.

What is the Data Plane?

The data plane is where real data access happens. This is the layer you touch when you read, write, upload, download, or query actual tenant or platform data:

Reading/writing secrets from Key Vault
Accessing storage blobs
Querying databases
Any operation that touches actual data

Endpoint: Service-specific (e.g., *.vault.azure.net, *.blob.core.windows.net)

Authentication: Authentication can still succeed, but the request often fails when private endpoints are enabled because the caller may not have a network path to the service.

Why the Divide?

Azure separates these two layers for security and architectural reasons:

Control plane can use service principals and managed identities with OIDC
Data plane requires direct network access to the service endpoint
Private endpoints block public access to data plane endpoints

Critical Insight: When you enable private endpoints, you are closing off public routes to the data-access side of a service. A tool can still obtain a valid identity token and yet fail because it has no network route to the private endpoint.

2. Key Vault Operations: What Requires Data Plane Access

Key Vault Endpoints

Control Plane: management.azure.com, used to create the vault, assign permissions, and read configuration details.
Data Plane: *.vault.azure.net, used to read and write secrets, keys, and certificates.

Operations That Require Data Plane Access

Reading Secrets

// This REQUIRES data plane access
data "azurerm_key_vault_secret" "example" {
  name     = "my-secret"
  vault_id = "/subscriptions/.../keyvaults/myvault"
}

// When private endpoints are enabled:
// ✗ FAILS: Cannot reach myvault.vault.azure.net from GitHub runner
// ✓ WORKS: Can reach myvault.vault.azure.net from within VNet or via the Bastion tunnel

Writing Secrets

// This REQUIRES data plane access
resource "azurerm_key_vault_secret" "example" {
  name         = "my-secret"
  value        = "secret-value"
  key_vault_id = azurerm_key_vault.main.id

  // Terraform must connect to: myvault.vault.azure.net
  // When private endpoints are enabled:
  // ✗ FAILS: GitHub runner cannot reach myvault.vault.azure.net
  // ✓ WORKS: Bastion tunnel provides access to myvault.vault.azure.net
}

Why These Fail Even When OpenID Connect Works

GitHub Runner (OIDC Authentication)
        ↓
Azure AD (Token: aud=management.azure.com)
        ↓
Azure ARM API (management.azure.com) ✓ WORKS
        ↓
Terraform creates resource
        ↓
Terraform tries to write secret
        ↓
needs to access: myvault.vault.azure.net ✗ BLOCKED
                ↑
        Private endpoint blocks public access
        GitHub runner has no route to VNet

3. Terraform + Key Vault: When Operations Fail

The following examples show how Terraform interacts with Key Vault. In this repository, the real implementation is spread across infra-ai-hub/stacks/ rather than one single main.tf file, but the same networking rule still applies.

Creating Key Vault (Control Plane - Works)

resource "azurerm_key_vault" "main" {
  name                = var.key_vault_name
  location            = var.location
  resource_group_name = azurerm_resource_group.main.name
}

// ✓ WORKS with OIDC
// Uses: management.azure.com
// No private endpoint yet, so no blocking

Creating Private Endpoint (Control Plane - Works)

resource "azurerm_private_endpoint" "key_vault_pe" {
  name                = "${var.app_name}-kv-pe"
  location            = var.location
  resource_group_name = azurerm_resource_group.main.name
  subnet_id           = module.network.private_endpoint_subnet_id

  private_service_connection {
    name                           = "${var.app_name}-kv-psc"
    private_connection_resource_id = azurerm_key_vault.main.id
    is_manual_connection           = false
    subresource_names              = ["vault"]
  }
}

// ✓ WORKS with OIDC
// Uses: management.azure.com
// This BLOCKS public access to *.vault.azure.net

Waiting for DNS (DNS Propagation Workaround)

resource "null_resource" "wait_for_key_vault_private_dns" {
  triggers = {
    private_endpoint_id = azurerm_private_endpoint.key_vault_pe.id
    // ... other values
  }

  provisioner "local-exec" {
    command = <<EOT
      # Wait for DNS to be ready before trying data plane operations
      # Azure creates private DNS records asynchronously
      # We poll until the private endpoint can resolve its DNS name
    EOT
  }

  depends_on = [azurerm_private_endpoint.key_vault_pe]
}

// Why necessary?
// After private endpoint creation, Azure asynchronously:
// 1. Creates private DNS zone: privatelink.vaultcore.azure.net
// 2. Links it to the VNet
// 3. Creates A records for the private endpoint
// 4. This can take 30-120 seconds
//
// If we try to create secrets immediately:
// ✗ FAILS: Cannot resolve myvault.vault.azure.net
// ✓ WORKS: After DNS is ready, myvault.vault.azure.net resolves to private IP

Creating Secrets (Data Plane - FAILS without the Bastion tunnel)

resource "azurerm_key_vault_secret" "secret_one" {
  name            = "example-secret-test-one"
  value           = random_password.secret_one.result
  key_vault_id    = azurerm_key_vault.main.id
  expiration_date = "2025-12-31T23:59:59Z"

  depends_on = [null_resource.wait_for_key_vault_private_dns]
}

resource "azurerm_key_vault_secret" "secret_two" {
  name            = "example-secret-test-two"
  value           = random_password.secret_two.result
  key_vault_id    = azurerm_key_vault.main.id
  expiration_date = "2025-12-31T23:59:59Z"

  depends_on = [null_resource.wait_for_key_vault_private_dns]
}

// ✗ FAILS with OIDC + Private Endpoints
// Terraform tries to connect to: myvault.vault.azure.net
// GitHub runner has no route to the private IP
//
// ✓ WORKS with:
//   1. Self-hosted runner inside VNet
//   2. Bastion native tunnel → jumpbox (SOCKS5 into the VNet → *.vault.azure.net)
//   3. Azure Bastion tunnel from a developer laptop (same path, ad-hoc)

4. Azure Bastion: How the Tunnel Works

The platform reaches private-endpoint data planes through Azure Bastion native client tunnelling (Standard SKU) to a small Linux jumpbox VM that lives inside the VNet. There is no public proxy server and no shared tunnel password — access is authenticated with Entra ID and RBAC over the same OIDC login the pipeline already uses. The Bastion + jumpbox are provisioned by the bcgov/action-deployer-vm-bastion-alz action in the tools subscription.

The Problem Bastion Solves

BEFORE the tunnel:
┌──────────────┐         ┌──────────────┐
│ GitHub Runner │ ──OIDC──> │ Azure ARM API │
│ (Public)      │          │ mgmt.azure.com│
└──────────────┘          └──────┬───────┘
                                │
                                │ Creates resources
                                ↓
┌──────────────┐         ┌──────▼───────┐
│ GitHub Runner │         │ Private      │
│ ✗ No route   │         │ Endpoint     │
│ to VNet      │         │ *.vault.azure.net │
└──────────────┘         │ BLOCKED      │
                          └──────────────┘

AFTER the tunnel:
┌──────────────┐         ┌──────────────┐
│ GitHub Runner │ ──OIDC──> │ Azure ARM API │
│ (Public)      │          │ mgmt.azure.com│
└──────┬───────┘          └──────┬───────┘
       │                          │
       │ az network bastion ssh -D (SOCKS5)
       ↓                          │
┌──────▼──────────┐         ┌─────▼──────┐
│ Azure Bastion   │         │ Private    │
│ → jumpbox VM    │────────> │ Endpoint   │
│ (in the VNet)   │         │ *.vault.azure.net │
└──────┬──────────┘         │ ACCESSIBLE │
       │                          │
       │ SOCKS5 (remote DNS)      │
       ↓                          │
┌──────▼──────────┐              │
│ Runner / Laptop │ ←────────────┘
│ privoxy → SOCKS │
└─────────────────┘

How the Bastion Tunnel Works (Technical Deep Dive)

Step 1: Bastion + jumpbox deployment (Control Plane - Works with OIDC)

The deploy-bastion-in-tools job calls the BC Gov action (.github/workflows/.deployer.yml):

- uses: bcgov/action-deployer-vm-bastion-alz@v1.0.0
  with:
    app_name: ai-hub
    app_env: tools
    bastion_sku: Standard                 # Standard SKU = native client tunnelling
    enable_bastion: "true"
    enable_jumpbox: "true"
    # CI service principal must be listed here so AAD bastion SSH works:
    vm_admin_login_principal_ids: ${{ secrets.VM_ADMIN_LOGIN_PRINCIPAL_IDS }}

# ✓ This WORKS with OIDC because every step is control plane (management.azure.com):
# 1. Create the Bastion host + public IP
# 2. Create the jumpbox VM (no public IP) inside the jumpbox subnet
# 3. Grant "Virtual Machine Administrator Login" RBAC for Entra ID SSH

Step 2: Open a SOCKS5 tunnel through Bastion

The deploy pipeline (.github/workflows/.deployer-using-secure-tunnel.yml) opens a SOCKS5 proxy by SSH dynamic port-forwarding (-D) through Bastion native tunnelling to the jumpbox:

# Native bastion tunnelling — no public proxy, no shared password.
az network bastion ssh \
  --name "$BASTION_NAME" -g ai-hub-bastion-tools --subscription "$TOOLS_SUBSCRIPTION_ID" \
  --target-resource-id "$VM_ID" --auth-type AAD \
  -- -D 1080 -N -q &

# -D 1080  SOCKS5 dynamic port-forward on localhost:1080
# -N       no remote command (keep the tunnel open)
# AAD      authenticate with the OIDC service principal (Entra ID + RBAC)

Step 3: privoxy bridges HTTP(S)_PROXY → SOCKS5

Terraform and the Azure CLI speak to an HTTP proxy; privoxy forwards that to the Bastion SOCKS5 proxy with forward-socks5t (remote DNS, so private-endpoint hostnames resolve on the jumpbox side):

docker run -d --name privoxy --network host \
  -e SOCKS_HOST=127.0.0.1 -e SOCKS_PORT=1080 \
  ghcr.io/<org>/ai-hub-tracking/azure-proxy/privoxy:latest

# Terraform / az then use:
export HTTP_PROXY=http://127.0.0.1:8118
export HTTPS_PROXY=http://127.0.0.1:8118

Step 4: Traffic Flow with Private Endpoints

┌──────────────────┐
│ Runner / Laptop  │
│ Terraform / az   │
│ HTTP_PROXY :8118 │
└──────┬───────────┘
       │
       │ 1. privoxy → SOCKS5 (localhost:1080)
       ↓
┌──────▼──────────┐
│ Azure Bastion   │
│ native tunnel   │
│ (Entra ID auth) │
└──────┬──────────┘
       │
       │ 2. SSH -D to jumpbox (inside VNet)
       ↓
┌──────▼──────────────────┐
│ Private Endpoint        │
│ Subnet                  │
│ Routes to:              │
│ myvault.vault.azure.net │
└──────┬──────────────────┘
       │
       │ 3. Data plane access
       │ *.vault.azure.net
       ↓
┌──────▼──────────┐
│ Key Vault       │
│ Data operations │
│ - Get secrets   │
│ - Set secrets   │
│ - List secrets  │
└────────────────┘

✓ SUCCESS: Terraform can read/write secrets!

Why Bastion native tunnelling vs Alternatives?

Method	Cost	Setup	Use Case	Limitations
Bastion + jumpbox (native tunnel)	~$140/mo (auto-deleted off-hours)	Medium	CI/CD + local dev, Entra ID auth, no shared secret	Standard SKU required; jumpbox/bastion warm-up on first use
Self-hosted Runner	$100/mo	Medium	CI/CD with secrets	Always running, needs management
Public runner, no tunnel	Free	Low	Control-plane-only deploys	Cannot reach private data planes at all

5. Deployment Scenarios: Who Needs What

Scenario 1: Platform Team (Deploys Landing Zone)

Who: Platform Services Team, Infrastructure Admins

What they deploy:

✓ Network (VNets, subnets, NSGs)
✓ Key Vault + Private Endpoints
✓ Azure Bastion + jumpbox (native tunnel, via the bcgov action)
✓ Self-hosted runners (optional)
✓ API Management, App Gateway
✓ Azure AI Foundry

When they need data plane access:

Creating Key Vault secrets (initial setup)
Testing private endpoints
Debugging network connectivity
Manual secret rotation

Recommended access method:

Option 1: Bastion native tunnel → jumpbox — for most developers and CI/CD (use .deployer-using-secure-tunnel)
Option 2: SSH/RDP to the jumpbox via Bastion — for interactive admin work inside the VNet

Scenario 2: Project Teams (Deploy Applications)

Who: Ministry Application Teams

What they deploy:

✓ App Service Plans (their apps)
✓ Container Apps (their workloads)
✓ Storage Accounts (their data)
✓ Application Code

What they DON'T deploy:

✗ Network infrastructure (provided by platform)
✗ Bastion/jumpbox (platform team maintains)
✗ Key Vault (may use shared or create their own)
✗ Private endpoints (configured by platform)

When they need data plane access:

Reading secrets from Key Vault (their app needs secrets)
Writing secrets during deployment
Testing their apps against private endpoints

Recommended access method:

Use the platform's Bastion tunnel - provided by platform team
Public GitHub runners - for control plane operations
No self-hosted needed - platform provides infrastructure

Scenario 3: Solo Developer

Who: Single developer, proof-of-concept work

What they deploy:

✓ Their application code
✓ Minimal infrastructure if needed

Recommended setup:

Bastion native tunnel - connect laptop to the VNet via the jumpbox
Public GitHub runners (free) - for infrastructure deploys

Skip:

✗ A dedicated Bastion + jumpbox - reuse the shared tools Bastion the platform already runs
✗ Self-hosted runners ($100/mo) - not needed for solo work

6. Portal → Hub Key Vault Integration

The Tenant Onboarding Portal reads APIM subscription keys from the hub Key Vault at request time, rather than storing them in its own data store. This section describes the end-to-end flow and access model.

Access Flow

An authenticated tenant-admin user requests credentials in the portal UI (env tab selected).
The portal backend (AppController.getCredentials) verifies the session and confirms the user is in the tenant's admin_users list.
HubKeyVaultService selects the SecretClient for the requested environment and calls getSecret for {tenant}-apim-primary-key, {tenant}-apim-secondary-key, and {tenant}-apim-rotation-metadata.
The portal's system-assigned Managed Identity authenticates to the hub Key Vault via DefaultAzureCredential. No connection strings or stored credentials are used.
The response is returned over the authenticated session — keys are immediately written to the clipboard client-side and never stored in React state or rendered as DOM text.

RBAC Model

The portal MI is granted Key Vault Secrets User on each hub Key Vault (one azurerm_role_assignment per environment in tenant-onboarding-portal/infra/main.tf, gated on var.hub_keyvault_id_{env} != ""). This is a read-only RBAC role — the portal can retrieve secret values but cannot create, update, or delete secrets.

APIM Tenant Info

The GET /tenants/:name/tenant-info endpoint proxies the APIM /{tenant}/internal/tenant-info policy (which already existed — no APIM changes were required). The portal uses the per-env apimGatewayUrl setting and the tenant's primary key to authenticate the internal call.

Configuration

Six environment variables wire the hub Key Vault URIs and APIM gateway URLs into the portal backend:

PORTAL_HUB_KEYVAULT_URL_{DEV,TEST,PROD} — hub Key Vault URI per environment
PORTAL_APIM_GATEWAY_URL_{DEV,TEST,PROD} — APIM (or App Gateway) public URL per environment

These are populated by the CI/CD pipeline from hub Terraform remote state (via the collect-hub-outputs matrix job — see ADR on hub output collection).

Summary

Why We Need Key Vault Access

Security: Private endpoints prevent public access to secrets
Compliance: BC Gov requires zero-trust, no public data plane access
Operations: Applications need to read secrets at runtime
Automation: Terraform needs to write secrets during deployment

Why OIDC Alone Isn't Enough

OIDC provides access to management.azure.com (control plane)
Private endpoints block *.vault.azure.net (data plane)
GitHub runners have no route to VNet private IPs
Even with valid tokens, network access is required

How the Bastion Tunnel Solves This

A jumpbox VM runs inside the VNet (no public IP)
Azure Bastion (Standard SKU) provides Entra ID-authenticated native tunnelling to it
The runner/laptop opens a SOCKS5 tunnel: az network bastion ssh -- -D 1080
Traffic flows: runner → Bastion → jumpbox → VNet → private endpoints
Result: data plane access with no public proxy and no shared password

When to Use Each Method

Platform Team: Deploy all, use the Bastion tunnel for access
Project Teams: Deploy apps only, use the platform's Bastion tunnel
Solo Dev: Deploy minimal, reuse the shared Bastion tunnel
CI/CD with secrets: Use GitHub-hosted runners + the Bastion tunnel (.deployer-using-secure-tunnel workflow)
Admin work: SSH/RDP to the jumpbox via Bastion

Technical Deep Dive: Management Access, Data Access, and the Bastion Tunnel

Table of Contents

1. Control Plane vs Data Plane: The Fundamental Divide

What is the Control Plane?

What is the Data Plane?

Why the Divide?

2. Key Vault Operations: What Requires Data Plane Access

Key Vault Endpoints

Operations That Require Data Plane Access

Reading Secrets

Writing Secrets

Why These Fail Even When OpenID Connect Works

3. Terraform + Key Vault: When Operations Fail

Creating Key Vault (Control Plane - Works)

Creating Private Endpoint (Control Plane - Works)

Waiting for DNS (DNS Propagation Workaround)

Creating Secrets (Data Plane - FAILS without the Bastion tunnel)

4. Azure Bastion: How the Tunnel Works

The Problem Bastion Solves

How the Bastion Tunnel Works (Technical Deep Dive)

Step 1: Bastion + jumpbox deployment (Control Plane - Works with OIDC)

Step 2: Open a SOCKS5 tunnel through Bastion

Step 3: privoxy bridges HTTP(S)_PROXY → SOCKS5

Step 4: Traffic Flow with Private Endpoints

Why Bastion native tunnelling vs Alternatives?

5. Deployment Scenarios: Who Needs What

Scenario 1: Platform Team (Deploys Landing Zone)

Scenario 2: Project Teams (Deploy Applications)

Scenario 3: Solo Developer

6. Portal → Hub Key Vault Integration

Access Flow

RBAC Model

APIM Tenant Info

Configuration

Summary

Why We Need Key Vault Access

Why OIDC Alone Isn't Enough

How the Bastion Tunnel Solves This

When to Use Each Method