Architecture Decision Records

Context

The Problem: Why Can't GitHub Just Run Terraform Directly?

The Solution: Landing Zone Architecture

We need infrastructure inside the private network that can:

• Receive commands from GitHub (via OIDC tokens)
• Run Terraform against private endpoints
• Allow humans to access resources for debugging

How Terraform Actually Runs

┌─────────────────────────────────────────────────────────────────────────┐
│                         DEPLOYMENT FLOW                                 │
├─────────────────────────────────────────────────────────────────────────┤
│                                                                         │
│   GitHub Actions          │ Azure Landing Zone (Private Network)        │
│   (Public Internet)       │                                             │
│                           │                                             │
│   ┌──────────────┐  OIDC  │    ┌──────────────┐    ┌─────────────────┐  │
│   │ GitHub-Hosted│───────▶│    │ Azure Proxy  │───▶│ Private         │  │
│   │ Runner       │        │    │ (Chisel App  │    │ Endpoints       │  │
│   │ ubuntu-24.04 │◀SOCKS5─┘    │  Service)    │    │ (Storage, KV)   │  │
│   └──────────────┘        │    └──────────────┘    └─────────────────┘  │
│          │                │                                             │
│          ▼                │                                             │
│   ┌──────────────┐        │                                             │
│   │ terraform    │        │                                             │
│   │ plan/apply   │        │                                             │
│   └──────────────┘        │                                             │
│   (via SOCKS tunnel)      │                                             │
└─────────────────────────────────────────────────────────────────────────┘

Two deployment options:

What About Other Repositories?

What This Repo Provides (Set Up Once)

How Access Actually Works (Public vs Private)

┌─────────────────────────────────────────────────────────────────────────────┐
│                           BC Gov Network                                    │
│  ┌──────────────┐                                                           │
│  │ Ministry     │                                                           │
│  │ Applications │──┐                                                        │
│  └──────────────┘  │                                                        │
│                    │                                                        │
│  ┌──────────────┐  │    ┌──────────────────────────────────────────────────┐│
│  │ Ministry     │──┼────│         ExpressRoute (Private Circuit)           ││
│  │ Users        │  │    │         NOT public internet!                     ││
│  └──────────────┘  │    └──────────────────────────────────────────────────┘│
└────────────────────┼────────────────────────────────────────────────────────┘
                     │ Firewall Rules (Allowed Traffic)
                     ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│                           Azure (Private VNet)                              │
│                                                                             │
│  ┌────────────────┐    ┌────────────────┐    ┌────────────────────────────┐ │
│  │  App Gateway   │───▶│     APIM       │───▶│   Private Endpoints        │ │
│  │  (Public IP)   │    │  (Internal IP) │    │   (Storage, OpenAI, etc.)  │ │
│  └────────────────┘    └────────────────┘    └────────────────────────────┘ │
│         ▲                                                                   │
│         │                                                                   │
│         │ All private IPs - reachable via ExpressRoute                      │
│                                                                             │
│  ════════════════════════════════════════════════════════════════════════   │
│                                                                             │
│  ┌────────────────┐    ┌────────────────┐                                   │
│  │    BASTION     │─▶ │   Jumpbox VM   │    ◀── Only Bastion has public IP  │
│  │  (PUBLIC IP)   │    │  (Private IP)  │        (for admin access)         │
│  └────────────────┘    └────────────────┘                                   │
│         ▲                                                                   │
└─────────┼───────────────────────────────────────────────────────────────────┘
          │
          │ Admin accesses via Azure Portal (browser)
          │
┌─────────┴────────┐
│    INTERNET      │
│  (Platform Team) │
└──────────────────┘
    

Why Not Just Open Public Endpoints Temporarily?

Consequences

Positive

• Fully policy compliant - No public endpoints ever
• Shared infrastructure - One-time setup, many projects benefit
• Consistent security - All projects inherit the same secure baseline
• Cost efficient - Single Bastion (~$140/mo) serves all projects

Negative

• Initial complexity - Landing Zone must be built first
• Proxy dependency - Chisel App Service must be healthy before Terraform workflows run
• VNet planning - Must allocate IP ranges carefully

References

• Azure Landing Zone Concepts
• Chisel - Fast TCP/UDP tunnel over HTTP
• Network Architecture Diagram
• Deployment Pipeline Diagram

Context

GitHub Actions workflows need to authenticate with Azure to deploy infrastructure via Terraform. We evaluated three options for credential management:

Decision

We chose Option C: OIDC Federated Credentials.

While the platform team's rotating key solution (Option B) is policy compliant, it requires maintaining a cron job to continuously fetch and update credentials. OIDC eliminates this operational burden - the bearer token is obtained directly within the GitHub Actions workflow at runtime, with no external synchronization required.

Rationale

Consequences

Positive

• Zero secrets to rotate - Eliminates credential management overhead
• Reduced blast radius - Tokens valid for minutes, not years
• Fine-grained access - Can restrict to specific branches/environments
• Better audit trail - Every token exchange is logged with JWT claims
• No secret sprawl - Secrets don't end up in logs, config files, or developer machines

Negative

• More complex initial setup - Federated credential configuration is more involved
• Newer technology - Less documentation and community examples available
• GitHub dependency - Tightly coupled to GitHub's OIDC provider

Neutral

• Requires understanding of JWT claims and subject matching
• Debugging auth failures requires knowledge of OIDC flow

References

• GitHub OIDC Documentation
• Azure Workload Identity Federation
• Token Exchange Flow Diagram

Context

Public Endpoints in the Architecture

Operators need secure access to jumpbox VMs for debugging and administration. The VMs cannot have public IPs per policy. We evaluated:

Decision

We chose Option C: Azure Bastion.

Rationale

Consequences

Positive

• No public IPs on VMs - VMs only have private IPs
• No client software - Works from any browser
• Azure AD integration - Uses existing identity
• Session recording - Can enable for audit
• Cost control - Can deploy/destroy on demand via workflow

Negative

• Azure Portal dependency - Must use Azure UI or CLI
• File transfer limitations - No native SCP/SFTP
• Latency - Browser-based adds some lag

References

• Azure Bastion Documentation
• Connect to Linux VM via Bastion
• Network Environments Diagram

Context

Azure PaaS services (Storage Accounts, Key Vaults, Databases, etc.) by default have public endpoints accessible from the internet. BC Gov policy requires these to be locked down.

Policy Requirements

Implementation

Client Connectivity Model

Consequences

Challenges

• GitHub Actions cannot reach private endpoints directly - Requires the Chisel SOCKS tunnel or VNet-internal access (see ADR-001)
• Local development complexity - Developers cannot access resources without VPN/Bastion
• DNS resolution - Must configure private DNS zones correctly
• Debugging difficulty - Cannot easily test from outside the network

Workarounds

• Terraform State: Use the Chisel SOCKS tunnel (GitHub-hosted runner + Azure Proxy) to access the private storage endpoint. The use_azuread_auth = true setting enables Azure AD authentication for state access.
• Development: Use Bastion + Jumpbox for all Azure resource access (see ADR-003)
• CI/CD: Use the Chisel SOCKS tunnel for full private-endpoint access during Terraform runs (see ADR-001)

References

• Azure Private Endpoint Documentation
• Network Architecture Diagram

Context

We needed a documentation site for the project. Options considered:

Decision

We built a custom Bash-based static site generator.

Rationale

• Portability: Runs anywhere with Bash (Linux, Mac, WSL, CI)
• No dependency management: No npm, gem, pip, go install required
• Simplicity: Anyone can understand the 60-line build script
• Speed: Builds in milliseconds
• GitHub Pages native: No special plugins or build configurations
• AI-friendly: HTML generation is trivial for AI assistants

Consequences

Positive

• Zero build dependencies to maintain or update
• Works in any environment without setup
• Easy to understand and modify
• No security vulnerabilities from npm packages

Negative

• No built-in Markdown support (write HTML directly)
• No automatic table of contents generation
• No built-in search (added custom client-side solution)

Mitigations

• Created template page with all components for easy copy-paste
• AI assistants generate HTML as easily as Markdown
• Added custom SVG viewer for diagrams

Context

This repo needs a repeatable, auditable way to provision and update Azure infrastructure (networking, private endpoints, RBAC, diagnostics, and PaaS services) under BC Government policy constraints.

Given the Landing Zone design (private endpoints, limited portal use, and CI/CD execution from within the VNet), we need Infrastructure as Code that supports:

• Idempotent, reviewable changes (pull requests as the change record)
• Policy-driven patterns (private endpoints, NSGs, diagnostics settings)
• Composable modules (prefer Azure Verified Modules where possible)
• Automation via GitHub Actions using OIDC and Chisel SOCKS tunnel

Options Considered

Decision

We use Terraform as the primary Infrastructure as Code (IaC) tool for this repo.

Rationale

• Fits Landing Zone operations: Terraform runs cleanly on GitHub-hosted runners via the Chisel SOCKS tunnel, providing data-plane access to private endpoints when required.
• Standardization via modules: We can prefer Azure Verified Modules (AVM) for consistent, policy-aligned deployments.
• Auditable change control: Plans and applies can be gated by pull request review and CI checks.
• Multi-environment support: Variables and modules make it straightforward to deploy dev/test/prod consistently.
• Sustainability: Terraform's widespread adoption ensures long-term community and vendor support. Team members working across AWS, Azure, and OpenShift can use one IaC tool consistently across the stack

Consequences

Positive

• Repeatable deployments - Same inputs produce the same infrastructure
• Versioned infrastructure - Git history becomes the change log
• Policy-aligned defaults - Modules can encode private endpoint and logging patterns

Negative

• Learning curve - Contributors must understand Terraform workflows
• State management - Requires careful backend configuration and access controls
• Upgrades - Provider/module version bumps require ongoing maintenance

Mitigations

• Use pinned module versions and keep provider versions explicit
• Use CI to run terraform fmt, terraform validate, and plans
• Prefer AVM modules over custom resources where feasible

References

• Terraform Reference - Complete module, variable, and output docs
• Terraform Documentation
• Azure Verified Modules (Terraform index)

Context

BC Government Platform Services has established ExpressRoute connectivity between on-premises networks and Azure. For this AI Hub Landing Zone, the question arose: should ministry applications connect directly to AI services via ExpressRoute, or through a managed ingress layer?

Options Considered

Decision

For a clear multi-tenant platform, we recommend all clients connect through App Gateway → APIM → Private Endpoints.

ExpressRoute connectivity is provided by Platform Services and is available. However, to support consistent security controls, audit logging, and fair resource allocation across multiple ministries, we recommend the App Gateway + APIM path as the standard connectivity model.

Traffic Flow

┌─────────────────────────────────────────────────────────────────────────────┐
│                        CLIENT CONNECTIVITY MODEL                             │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                              │
│   On-Premises                    │         Azure Landing Zone               │
│   (Ministry Apps)                │                                          │
│                                  │                                          │
│   ┌──────────────┐              │    ┌─────────────────────────────────┐   │
│   │ Ministry App │              │    │         App Gateway              │   │
│   │ (Health)     │──────────────────▶│  • SSL Termination              │   │
│   └──────────────┘   Express    │    │  • WAF Protection               │   │
│                      Route      │    │  • DDoS Mitigation              │   │
│   ┌──────────────┐              │    └──────────────┬──────────────────┘   │
│   │ Ministry App │              │                   │                       │
│   │ (SDPR)       │──────────────────▶               ▼                       │
│   └──────────────┘              │    ┌─────────────────────────────────┐   │
│                                 │    │            APIM                  │   │
│   ┌──────────────┐              │    │  • Authentication (API Keys)    │   │
│   │ Ministry App │              │    │  • Rate Limiting                │   │
│   │ (Justice)    │──────────────────▶│  • Request Validation           │   │
│   └──────────────┘              │    │  • Ministry Routing             │   │
│                                 │    │  • Audit Logging                │   │
│                                 │    └──────────────┬──────────────────┘   │
│                                 │                   │                       │
│                                 │                   ▼                       │
│                                 │    ┌─────────────────────────────────┐   │
│                                 │    │      Private Endpoints          │   │
│                                 │    │  • AI Foundry                   │   │
│                                 │    │  • Azure OpenAI                 │   │
│                                 │    │  • AI Search                    │   │
│                                 │    │  • Storage (RAG docs)           │   │
│                                 │    └─────────────────────────────────┘   │
│                                 │                                          │
└─────────────────────────────────────────────────────────────────────────────┘

Security Controls at Each Layer

Consequences

Positive

• Defense in depth - Multiple security layers before reaching backends
• Centralized policy - All clients subject to same rules
• Audit compliance - Every request logged with full context
• Flexibility - Can update policies without changing backends
• Cost attribution - Can track usage per ministry via APIM metrics

Negative

• Added latency - Two extra hops (App Gateway + APIM)
• Cost - App Gateway and APIM have significant monthly costs
• Complexity - More components to configure and maintain

Mitigations

• Latency is typically <10ms additional per hop
• Costs are shared across all ministries (per ADR-010)
• Infrastructure-as-code manages complexity

References

• AI Request Data Flow Diagram
• Full Stack Architecture Diagram
• Azure API Management Overview
• Azure Application Gateway Overview

Context

A valid question arises: If we're customizing everything for BC Gov requirements anyway, why use Microsoft's AI Landing Zone at all? Why not just build our own custom solution from scratch?

This ADR explains what value the AI Landing Zone and Azure Verified Modules (AVM) actually provide, even when we can't use Microsoft's default assumptions.

What AI Landing Zone Actually Provides

The Real Value: AVM Modules

What We Get From AI Landing Zone Reference

AVM Module Maturity - Honest Assessment

What This Means

Concrete Example: Storage Account

# ~200 lines of Terraform to handle:
resource "azurerm_storage_account" "main" { ... }
resource "azurerm_storage_account_network_rules" "main" { ... }
resource "azurerm_private_endpoint" "blob" { ... }
resource "azurerm_private_endpoint" "file" { ... }
resource "azurerm_private_endpoint" "queue" { ... }
resource "azurerm_private_dns_zone_virtual_network_link" "blob" { ... }
resource "azurerm_role_assignment" "contributor" { ... }
resource "azurerm_role_assignment" "reader" { ... }
resource "azurerm_monitor_diagnostic_setting" "main" { ... }
# Plus encryption, lifecycle policies, soft delete, versioning...
# Plus handling for every Azure API version change...
    

module "storage" {
  source  = "Azure/avm-res-storage-storageaccount/azurerm"
  version = "0.6.7"

  name                = "ministryhealthstorage"
  resource_group_name = azurerm_resource_group.ministry.name
  location            = "canadacentral"

  # Private endpoints - one line enables the pattern
  private_endpoints = {
    blob = { subnet_id = module.network.subnet_ids["private-endpoints"] }
  }

  # All the security, RBAC, monitoring handled by module
  tags = var.common_tags
}
    

Why Not 100% Custom?

What We're Actually Doing

┌─────────────────────────────────────────────────────────────────────┐
│                    BC Gov AI Hub Architecture                        │
├─────────────────────────────────────────────────────────────────────┤
│                                                                      │
│   ┌─────────────────────────────────────────────────────────────┐   │
│   │  BC Gov Customization Layer (Our Code)                      │   │
│   │  • Multi-tenant resource group structure                    │   │
│   │  • APIM subscription per ministry                           │   │
│   │  • IP allocation policies                                   │   │
│   │  • Platform Services DNS integration                        │   │
│   │  • Canada data residency enforcement                        │   │
│   └─────────────────────────────────────────────────────────────┘   │
│                              │                                       │
│                              ▼                                       │
│   ┌─────────────────────────────────────────────────────────────┐   │
│   │  Azure Verified Modules (Microsoft Maintained)              │   │
│   │  • avm-res-cognitiveservices-account                        │   │
│   │  • avm-res-machinelearningservices-workspace                │   │
│   │  • avm-res-storage-storageaccount                           │   │
│   │  • avm-res-network-virtualnetwork                           │   │
│   │  • avm-res-apimanagement-service                            │   │
│   │  • ... 100+ more modules                                    │   │
│   └─────────────────────────────────────────────────────────────┘   │
│                              │                                       │
│                              ▼                                       │
│   ┌─────────────────────────────────────────────────────────────┐   │
│   │  Azure Resource Manager APIs                                │   │
│   └─────────────────────────────────────────────────────────────┘   │
│                                                                      │
└─────────────────────────────────────────────────────────────────────┘
    

Decision

We use AI Landing Zone reference architecture and AVM modules as our foundation, with a BC Gov customization layer on top.

We do NOT use Microsoft's default configuration. We use their:

• Reference patterns - How to wire services together
• AVM modules - Tested, maintained infrastructure components
• Best practices - Security, networking, RBAC patterns

Consequences

Positive

• Reduced maintenance - Microsoft maintains 90% of the code
• Faster development - Use proven patterns instead of inventing
• Security updates - Get patches via module version bumps
• Community support - Issues/bugs reported and fixed by others
• Audit trail - Using "official" modules helps with compliance

Negative

• Module constraints - Can only do what AVM modules support
• Version management - Must track and update module versions
• Abstraction leakage - Sometimes need to understand module internals

References

• Azure Verified Modules (AVM)
• AI/ML Landing Zone Pattern Module
• Azure Landing Zone Concepts
• Architecture Layers Diagram

Context

The AI Hub Landing Zone is designed to serve multiple BC Government ministries from a single shared infrastructure deployment. This creates a multi-tenancy challenge: how do we provide cost-efficient shared services while ensuring strict data isolation between ministries?

The Problem: Shared Infrastructure, Isolated Data

Decision

We implement a four-layer isolation model:

Implementation Architecture

┌─────────────────────────────────────────────────────────────────────────────┐
│                      MULTI-TENANT ISOLATION MODEL                            │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                              │
│   Ministry Health        Ministry SDPR         Ministry Justice             │
│   ┌─────────────┐       ┌─────────────┐       ┌─────────────┐              │
│   │ APIM Sub: H │       │ APIM Sub: S │       │ APIM Sub: J │              │
│   └──────┬──────┘       └──────┬──────┘       └──────┬──────┘              │
│          │                     │                     │                      │
│          ▼                     ▼                     ▼                      │
│   ┌─────────────────────────────────────────────────────────────┐          │
│   │              Shared APIM (Policy Enforcement)                │          │
│   │         (validates subscription → routes to ministry)        │          │
│   └─────────────────────────────────────────────────────────────┘          │
│          │                     │                     │                      │
│          ▼                     ▼                     ▼                      │
│   ┌─────────────┐       ┌─────────────┐       ┌─────────────┐              │
│   │ Storage: H  │       │ Storage: S  │       │ Storage: J  │              │
│   │ Index: H    │       │ Index: S    │       │ Index: J    │              │
│   └─────────────┘       └─────────────┘       └─────────────┘              │
│                                                                              │
│   ════════════════════════════════════════════════════════════════          │
│                    Shared Infrastructure Layer                               │
│        (VNet, Bastion, AI Foundry Compute, Monitoring)                      │
│   ════════════════════════════════════════════════════════════════          │
│                                                                              │
└─────────────────────────────────────────────────────────────────────────────┘

Consequences

Positive

• Strong data isolation - Ministry data never co-mingled
• Cost efficiency - Shared compute and network infrastructure
• Audit compliance - Clear ministry attribution in all logs
• Scalable onboarding - New ministries get isolated resources automatically
• Flexible isolation levels - Can increase isolation (dedicated compute) if needed

Negative

• Resource multiplication - Each ministry needs separate storage/indexes
• Complexity - More resources to manage and monitor
• Cost per ministry - Base cost increases with each ministry onboarded

Cost Tracking

Multi-tenant isolation also enables per-tenant cost tracking and chargeback. See the detailed cost allocation strategy:

Click diagram to view full Cost Tracking documentation

References

• Cost Tracking Documentation - Full cost allocation strategy
• Multi-Tenant Isolation Diagram
• Azure Multi-Tenant Architecture Guide
• AI Request Data Flow Diagram

Context

Azure services have two distinct access paths that are often confused:

The Problem: Private Endpoints Block Data Plane

┌─────────────────────────────────────────────────────────────────────────────┐
│                    WHAT WORKS vs WHAT'S BLOCKED                              │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                              │
│   From Public Internet (GitHub Actions, Azure Portal, Your Laptop):          │
│                                                                              │
│   ✅ CONTROL PLANE (management.azure.com)                                    │
│      • Create Key Vault                    → ARM API → Works                 │
│      • Create Storage Account              → ARM API → Works                 │
│      • View resource properties in Portal  → ARM API → Works                 │
│      • OIDC authentication                 → Azure AD → Works                │
│                                                                              │
│   ❌ DATA PLANE (*.vault.azure.net, *.blob.core.windows.net)                 │
│      • Read Key Vault secret               → Private Endpoint → BLOCKED      │
│      • Write Storage blob                  → Private Endpoint → BLOCKED      │
│      • "View Secret Value" in Portal       → Private Endpoint → BLOCKED      │
│      • Terraform state read/write          → Private Endpoint → BLOCKED      │
│                                                                              │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                              │
│   From Inside VNet (Chisel Tunnel, Jumpbox, or Optional Self-hosted Runners):│
│                                                                              │
│   ✅ CONTROL PLANE → Still works (ARM is public)                             │
│   ✅ DATA PLANE    → Works via Private Endpoints (network path exists)       │
│                                                                              │
└─────────────────────────────────────────────────────────────────────────────┘

Why Azure Portal Shows Resources But Not Data

The Azure Portal is a Control Plane UI. When you browse to a Key Vault in the portal:

• ✅ You can see the vault exists (control plane: list resources)
• ✅ You can see its configuration (control plane: read properties)
• ❌ You CANNOT click "Show Secret Value" (data plane: blocked by private endpoint)

This is why ADR-008 states "No Azure Portal or Foundry Studio UI Access" for data operations - the portal physically cannot reach private data plane endpoints from your browser.

Why OIDC Is Used for Control Plane

OIDC (OpenID Connect) federation provides passwordless authentication from GitHub Actions to Azure:

┌─────────────────────────────────────────────────────────────────────────────┐
│                        OIDC AUTHENTICATION FLOW                              │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                              │
│   1. GitHub Actions workflow starts                                          │
│           ↓                                                                  │
│   2. GitHub OIDC Provider issues JWT token                                   │
│      (claims: repo, branch, environment, workflow)                           │
│           ↓                                                                  │
│   3. Token sent to Azure AD                                                  │
│           ↓                                                                  │
│   4. Azure AD validates against Federated Credential                         │
│      (checks: issuer=GitHub, subject=repo:bcgov/ai-hub-tracking:...)        │
│           ↓                                                                  │
│   5. Azure AD issues Access Token (valid ~1 hour)                            │
│           ↓                                                                  │
│   6. Access Token used for ARM API calls (Control Plane)                     │
│           ↓                                                                  │
│   ✅ terraform plan/apply (resource management)                              │
│   ✅ az cli commands (control plane operations)                              │
│   ✅ No secrets stored in GitHub!                                            │
│                                                                              │
│   BUT: OIDC gives credentials, NOT network access.                           │
│        Data plane still blocked without VNet connectivity.                   │
│                                                                              │
└─────────────────────────────────────────────────────────────────────────────┘

Decision: Multiple Access Methods for Different Needs

We provide four toggleable access mechanisms via Terraform, each serving a different use case:

Chisel Tunnel: Data Plane Access for Platform Maintainers

What is Chisel? A fast TCP/UDP tunnel over HTTP that creates a secure reverse proxy from your local machine into the Azure VNet.

┌─────────────────────────────────────────────────────────────────────────────┐
│                        CHISEL TUNNEL ARCHITECTURE                            │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                              │
│   Platform Maintainer's Laptop                                               │
│   ┌─────────────────────────────┐                                            │
│   │  Chisel Client (Docker)     │                                            │
│   │  Listens on localhost:5432  │                                            │
│   └──────────────┬──────────────┘                                            │
│                  │ HTTPS (encrypted)                                         │
│                  ▼                                                            │
│   ┌─────────────────────────────────────────────────────────────────────┐    │
│   │                    Azure App Service (Chisel Server)                │    │
│   │                    Inside VNet (app-service-subnet)                 │    │
│   │                    Random auth: tunnel:XXXXXXXX                     │    │
│   └──────────────┬──────────────────────────────────────────────────────┘    │
│                  │ VNet Integration (Private Network)                        │
│                  ▼                                                            │
│   ┌─────────────────────────────────────────────────────────────────────┐    │
│   │                    Private Endpoints                                │    │
│   │  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐              │    │
│   │  │  PostgreSQL  │  │   Key Vault  │  │   CosmosDB   │              │    │
│   │  │  :5432       │  │   :443       │  │   :443       │              │    │
│   │  └──────────────┘  └──────────────┘  └──────────────┘              │    │
│   └─────────────────────────────────────────────────────────────────────┘    │
│                                                                              │
│   Result: psql -h localhost -p 5432 → tunnels to private PostgreSQL          │
│                                                                              │
└─────────────────────────────────────────────────────────────────────────────┘

What Terraform Operations Need Data Plane?

Most Terraform operations are control plane only. Data plane is only needed when:

Access Model Summary

┌─────────────────────────────────────────────────────────────────────────────┐
│                        COMPLETE ACCESS MODEL                                 │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                              │
│  TENANT DEVELOPERS (Ministry Teams)                                          │
│  ┌─────────────┐      ┌──────────────┐      ┌─────────────────┐             │
│  │ Their Apps  │─────▶│ App Gateway  │─────▶│ APIM (rate      │─────▶ AI    │
│  │             │      │ + WAF        │      │ limited, metered)│      APIs   │
│  └─────────────┘      └──────────────┘      └─────────────────┘             │
│                              ▲                                               │
│                              │ Public endpoint (by design)                   │
│                              │ No direct private endpoint access             │
│                                                                              │
├─────────────────────────────────────────────────────────────────────────────┤
│                                                                              │
│  PLATFORM MAINTAINERS (This Team)                                            │
│                                                                              │
│  ┌────────────────────────────────────────────────────────────────────────┐ │
│  │ Method              │ Toggle Variable              │ Use Case          │ │
│  ├────────────────────────────────────────────────────────────────────────┤ │
│  │ Self-hosted Runners │ github_runners_aca_enabled   │ Optional tenant CI/CD │ │
│  │ Bastion + Jumpbox   │ enable_bastion/jumpbox       │ Admin access      │ │
│  │ Chisel Tunnel       │ enable_azure_proxy           │ Local dev access  │ │
│  └────────────────────────────────────────────────────────────────────────┘ │
│                                                                              │
│  All three provide: Control Plane + Data Plane access                        │
│                                                                              │
└─────────────────────────────────────────────────────────────────────────────┘

Consequences

Positive

• Clear mental model - Understanding control vs data plane explains "why" behind many decisions
• Flexible access - Enable only what you need (cost optimization)
• Policy compliant - No public data plane endpoints ever
• Secure tenant isolation - Tenants use APIM, never touch private endpoints directly

Negative

• Complexity - Must understand two planes, not just "Azure access"
• Chisel proxy dependency - Terraform workflows require the Azure Proxy App Service to be healthy before running
• Portal limitations - Can't "see" data in Portal even with permissions

References

• Playbook: Using Chisel Tunnel for Local Development
• Terraform Reference: azure-proxy module (Chisel)
• ADR-001: Shared Landing Zone
• ADR-002: OIDC Authentication
• ADR-008: No Azure Portal UI Access
• Microsoft Docs: Control Plane and Data Plane

Context

The AI Services Hub operates as a multi-tenant platform serving multiple ministries. This shared infrastructure creates a financial governance challenge: we must accurately allocate costs to specific tenants to ensure accountability and cost recovery.

The architecture includes two types of resources:

• Tenant-Dedicated: Resources used exclusively by one tenant (e.g., Azure OpenAI instances, Document Intelligence, Cosmos DB).
• Shared Infrastructure: Resources serving all tenants simultaneously (e.g., APIM, App Gateway, Application Insights).

We need a standardized strategy to track consumption and generate accurate chargeback invoices that account for both resource types across our dual-region deployment (Canada Central & East).

Decision

We will adopt a Hybrid Cost Allocation Model that combines native Azure billing with custom usage tracking:

1. Direct Attribution (90% of costs): We will isolate high-cost resources (AI Foundry Projects, Document Intelligence) into tenant-specific resource groups. These will be billed directly to tenants using Azure Cost Management tags (tenant-id), requiring no manual calculation.
2. Proportional Allocation (10% of costs): We will split the cost of shared infrastructure (APIM, App Gateway) based on actual usage metrics.
   • APIM & Gateway costs split by API Request Volume.
   • Monitoring costs split by Log Ingestion Volume.
3. Centralized Tracking via APIM: Azure API Management will serve as the single source of truth for usage metrics. All traffic must flow through APIM to ensure consistent tenant identification and logging.
4. Custom Egress Calculation: We will implement a custom calculation pipeline using App Gateway logs to track and charge for cross-region network egress (between Canada Central and Canada East).

Rationale

• Minimizes Operational Overhead: By using Direct Attribution for the most expensive resources (OpenAI tokens, Search), we rely on Azure's native billing engine for the vast majority of chargebacks. We only maintain custom logic for the shared platform components.
• Fairness: A flat fee for shared infrastructure would be unfair to smaller tenants. Proportional allocation based on request volume ensures ministries only pay for the platform capacity they actually consume.
• Granular Visibility: Using APIM as the central logging point allows us to capture operational metrics (latency, errors, token usage) alongside billing data without adding sidecar proxies to every service.

Consequences

Positive

• Transparency: Tenants can verify their direct Azure costs in the portal using their tenant tag.
• Scalability: New tenants can be onboarded simply by adding tags; the cost model adjusts automatically.
• Cost Recovery: Ensures the Platform Team fully recovers infrastructure costs rather than absorbing the overhead of shared services.

Negative

• Complexity of Egress: Cross-region data transfer is billed at the subscription level and is difficult to attribute. We accept the tradeoff of maintaining a custom Kusto query to calculate this specific cost.
• Maintenance: The logic for splitting shared costs (Python functions/Kusto queries) is custom code that must be maintained and verified monthly against Azure invoices.

Context

The AI Services Hub infrastructure was originally deployed as a single Terraform root module with one monolithic state file containing ~174 resources. This created several operational problems:

• Blast radius: Any Terraform error or state corruption could affect all 174 resources simultaneously.
• Lock contention: Only one Terraform operation could run at a time across the entire infrastructure, even for independent resources.
• Serial execution: Foundry projects required parallelism=1 due to Azure ETag conflicts, which forced the entire apply to run serially.
• Apply duration: A full apply took 5m 44s because independent stacks (APIM, foundry, tenant-user-mgmt) had to wait for each other.
• Phased targeting: The deployment script used -target flags to orchestrate a multi-phase apply (Phase 1: everything except foundry, Phase 2: foundry only, Phase 3: validation). This was fragile and hid dependency issues.

Decision

We split the monolithic root module into 5 isolated Terraform stacks, each with its own state file, backend configuration, and dependency management via terraform_remote_state data sources:

A stack engine (deploy-scaled.sh) orchestrates execution in dependency order:

• Phase 1: shared runs first (all other stacks depend on it).
• Phase 2: tenant runs per-tenant in parallel (each tenant gets isolated TF_DATA_DIR and state).
• Phase 3: foundry, apim, and tenant-user-mgmt run concurrently (no cross-dependencies between them).

For destroy, the order is reversed: Phase 3 stacks first (parallel), then tenants (parallel), then shared last.

Rationale

• Reduced blast radius: A state corruption or failed apply in apim cannot affect shared or tenant resources. Each stack can be independently recovered.
• Parallel execution: Phase 3 stacks have no dependencies on each other and can run concurrently, reducing total apply time by ~60 seconds.
• Isolated parallelism: The foundry stack runs with parallelism=1 without forcing the entire infrastructure to be serial. APIM and tenant-user-mgmt run with full parallelism simultaneously.
• Independent state locking: Multiple operators or CI pipelines can work on different stacks without lock contention (e.g., updating APIM policies while a tenant deployment is in progress).
• Cleaner dependencies: Using terraform_remote_state data sources makes cross-stack dependencies explicit and typed, replacing implicit module-to-module references.

Consequences

Positive

• 19% faster applies: Total apply time reduced from 5m 44s to 4m 39s (measured on test environment with 2 tenants).
• Eliminated -target phasing: The old Phase 1/2/3 with -target flags is replaced by natural stack ordering. No more fragile target expressions.
• Auto-recovery: The stack engine includes deposed object cleanup, import-on-conflict, and transient error retry — previously only available at the monolith level.
• Live output streaming: Each stack streams Terraform output in real time via tee, improving debuggability in CI/CD logs.
• Per-tenant state isolation: Each tenant has its own state file, making tenant onboarding/offboarding a state-level operation rather than a resource-level one.

Negative

• More files: 5 stacks × 5 standard files (main.tf, variables.tf, outputs.tf, providers.tf, backend.tf) = 25 files vs. the original 6. Some variable declarations are duplicated across stacks.
• State migration required: One-time migration from the monolith state to 5 stack states using terraform state mv. This was performed manually with verification scripts.
• Cross-stack refactoring: Moving a resource between stacks requires a state move operation, not just a code move. This adds operational complexity for future refactors.
• Remote state coupling: Stacks are coupled via terraform_remote_state data sources. Adding a new output in shared that apim needs requires deploying shared first.

References

• Terraform Remote State Data Source
• infra-ai-hub/stacks/ — Stack root modules
• infra-ai-hub/scripts/deploy-scaled.sh — Stack engine
• infra-ai-hub/scripts/deploy-terraform.sh — Public entrypoint

Context

APIM subscription keys authenticate tenant API calls to the AI Services Hub gateway. Without rotation, these long-lived secrets present a growing risk:

• Credential staleness: Keys provisioned at tenant onboarding remain valid indefinitely unless manually changed. A compromised key grants persistent access.
• No expiry enforcement: Azure APIM subscription keys have no built-in TTL or auto-expiry. The platform must implement rotation externally.
• BC Gov compliance: Government security policy expects secrets to be rotated periodically. The rotation interval and mechanism require STRA sign-off before production use.
• Multi-tenant blast radius: Each tenant has isolated subscription keys, but without rotation a single leaked key provides indefinite access to that tenant’s API surface.

Decision

We implement an alternating primary/secondary key rotation pattern with zero downtime, driven by a Container App Job (scheduled) deployed as a custom container:

1. Alternating slots: APIM subscriptions have two key slots (primary and secondary). Each rotation cycle regenerates one slot while the other remains valid and untouched.
2. Centralized hub Key Vault: After regeneration, both keys are written to a single hub Key Vault ({app_name}-{env}-hkv) with 90-day expiry and rotation metadata. No per-tenant Key Vault is required.
3. Self-service retrieval: Tenants fetch current keys via GET /{tenant}/internal/apim-keys — an APIM policy endpoint that reads from the hub Key Vault using APIM’s managed identity. No Azure SDK required.
4. Configurable schedule: Rotation is controlled by two flags in params/{env}/shared.tfvars: rotation_enabled (master on/off) and rotation_interval_days (7 for dev/test, 30 for prod).
5. Managed identity authentication: The Container App Job uses a system-assigned managed identity for APIM and Key Vault access. No stored secrets required.

Rationale

• Zero downtime: The alternating slot pattern guarantees one key is always valid. Tenants are never locked out during rotation.
• Container App Job over GHA workflow: A scheduled Container App Job runs reliably within Azure (no 60-day inactivity disable risk). The previous Bash script + GHA approach required periodic repo activity to avoid GitHub silently disabling scheduled workflows.
• Centralized over distributed: A single hub Key Vault with tenant-prefixed secrets scales better than per-tenant Key Vaults. One RBAC assignment for APIM’s managed identity covers all tenants.
• Self-service key retrieval: The /internal/apim-keys endpoint eliminates the need for tenants to have Azure portal access or Key Vault Reader roles. Any valid subscription key authenticates the request.
• Idempotent and safe: The function checks elapsed time since last rotation before acting. Multiple invocations within the interval are no-ops. A --dry-run mode allows validation without changes.

Consequences

Positive

• Automated secret hygiene: Keys rotate on a known schedule with full audit trail in Key Vault versioning and GitHub Actions logs.
• Minimal tenant burden: Tenants can use the APIM internal endpoint, daily cron polling, or simply contact the platform team. No Azure SDK or Key Vault access needed.
• Emergency rotation: Both slots can be regenerated immediately via the documented runbook, invalidating all compromised keys.
• Per-environment control: Rotation can be enabled independently per environment — currently active in dev/test, disabled in prod pending STRA approval.

Negative

• Container infrastructure: The Container App Job requires a Container App Environment and Container Registry, adding infrastructure components compared to the previous GHA-only approach. These are managed via the key-rotation-function Terraform module.
• STRA gate: Production rotation cannot be enabled until the STRA process completes. Until then, prod keys are static (same risk as baseline).
• Tenant coordination: Tenants using hard-coded keys (rather than the self-service endpoint) must update their configuration within the rotation interval or face 401 errors.

References

• APIM Key Rotation Guide — full operational documentation
• jobs/apim-key-rotation/ — Container App Job source code
• infra-ai-hub/modules/key-rotation-function/ — Terraform module
• .github/workflows/.builds.yml — container build workflow (matrix entry)
• params/{env}/shared.tfvars — per-environment rotation config

Context

The APIM gateway proxies requests to multiple backend services: Azure OpenAI, Document Intelligence, AI Search, Speech Services, and Storage. When a backend experiences failures (overload, outages, throttling), continued request forwarding wastes resources and degrades the client experience with slow timeouts instead of fast failures.

Azure OpenAI specifically returns 429 Too Many Requests with large Retry-After values (up to 86,400 seconds / 1 day) when quota is exhausted. Without circuit breaking, APIM would continue sending requests to a backend that cannot serve them.

Decision

Implement the circuit breaker pattern on all APIM backend entities using the native circuit_breaker_rule in azurerm_api_management_backend.

Configuration per backend

Backends covered

• ai_foundry — AI Foundry Hub endpoint
• openai — Azure OpenAI endpoint
• docint — Document Intelligence
• ai_search — Azure AI Search
• speech_services_stt — Speech-to-Text
• speech_services_tts — Text-to-Speech
• storage — Blob Storage (higher threshold: 5 failures)

What happens when the circuit trips

1. Backend accumulates failures matching the configured status codes within the failure window.
2. When the failure count exceeds the threshold, the circuit opens (trips).
3. APIM immediately returns HTTP 503 Service Unavailable to all subsequent requests targeting that backend — requests are not forwarded.
4. The global policy <on-error> handler intercepts the 503 and returns a structured JSON error with a Retry-After header (default: 60 seconds).
5. After the trip duration (or the backend’s Retry-After value if accept_retry_after_enabled = true), the circuit resets and traffic resumes.

Client-facing error response (503)

{
  "error": {
    "code": "503",
    "message": "Service temporarily unavailable — backend circuit breaker is open. The service is recovering from excessive failures. Retry after the indicated period.",
    "retryAfter": "60",
    "requestId": "abc-123-def-456"
  }
}

Rationale

• Fast failure: Clients get an immediate 503 instead of waiting for backend timeouts (up to 300 seconds).
• Backend protection: Prevents overwhelming a struggling backend with additional requests.
• Native support: Uses azurerm_api_management_backend.circuit_breaker_rule — no custom policy logic required.
• Retry-After propagation: Passes backend throttle signals directly to clients for proper backoff.
• Event Grid integration: APIM emits Event Grid events on circuit trip/reset for monitoring and alerting.

Consequences

Positive

• Reduced latency during backend outages (instant 503 vs. timeout).
• Backend services get breathing room to recover.
• Consistent error format with Retry-After enables client-side retry logic.
• Zero policy-level code — purely infrastructure configuration.

Negative

• Approximate tripping: APIM gateway instances do not synchronize circuit breaker state. Each instance tracks failures independently, so tripping is approximate in multi-instance deployments.
• Single rule per backend: Only one circuit breaker rule per backend is currently supported by the Azure API.
• 503 during recovery: Legitimate requests during the trip window will be rejected even if the backend has recovered before the trip duration expires.

References

• Azure APIM Backends — Circuit Breaker
• Circuit Breaker Pattern (Azure Architecture)
• APIM Event Grid Events — circuit breaker trip/reset events
• Issue #98: AI Gateway Gap Analysis

Context

The AI Hub needed a tenant onboarding experience that does more than collect a request. The onboarding flow must support structured tenant metadata, governed admin review, future automation from submission through approval, and environment-aware downstream actions such as generating Terraform inputs and preparing non-prod and prod deployment workflows.

We considered using an existing platform instead of building a portal inside the Hub workspace. The main alternatives were the BC Government Platform Product Registry, CHEFS, and the Azure API Management Developer Portal. All three reduce initial build effort, but none matches the lifecycle and control-plane requirements of Hub onboarding.

Options Considered

Decision

We will use a custom tenant onboarding portal inside the AI Hub repository and deployment boundary instead of the BC Government Platform Product Registry, CHEFS, or the APIM Developer Portal.

Rationale

• Governed approval workflow: Hub onboarding requires an admin review and approval process, not just a one-time request capture. The custom portal can model request states, reviewer actions, and follow-up steps directly.
• Structured Hub-specific configuration: The workflow needs to collect and validate data that maps cleanly into tenant-specific Terraform inputs and other structured configuration artifacts. A generic registry or hosted form would require extra translation layers and still would not own the lifecycle.
• Tight integration with Hub logic and storage: The onboarding flow must stay close to Hub-specific validation rules, tenant state, and downstream provisioning behavior. Keeping the portal in the same codebase reduces impedance between intake, approval, generated config, and deployment automation.
• Authenticated post-submission lifecycle: The process does not end at submission. The platform needs room for authenticated follow-up actions, operator review, state transitions, and future self-service interactions that go beyond a write-once form.
• CHEFS is too immutable for this lifecycle: CHEFS is well suited to hosted intake, but its submission model is intentionally form-centric and immutable. It supports status and notes, but it is not designed to mutate form data, drive secure post-approval reveal flows, or act as the system of record for ongoing onboarding operations.
• The Platform Product Registry solves a different problem: The registry is built for teams that need to create or manage a product on Private Cloud OpenShift or the Public Cloud Landing Zone, which may not be a need for AI Hub. AI Hub tenant onboarding can be irrespective of the platform the tenants use, captures Hub-specific configuration, and needs a purpose-built approval and provisioning workflow rather than generic platform registry product change process, similar to what Keycloak and Kong App Gateway have their own portals.
• Secure post-approval actions: The team needs room for actions after approval, including controlled credential-related workflows and other sensitive follow-up behavior. Those concerns should live in a dedicated application boundary rather than in generic form metadata or a public-facing registry experience.
• Future automation path: The chosen design supports evolving from request intake to approval and then to automated promotion and deployment workflows across non-prod and prod environments without replacing the front door later.
• APIM Developer Portal is the wrong abstraction: The APIM Developer Portal is built around API products and subscriptions, not tenant provisioning workflows. Customising it far enough to support multi-step approval, structured config generation, and environment-aware provisioning actions would require extensive delegation and external backend work — effectively rebuilding the same application on a more constrained foundation.

Consequences

Positive

• Single ownership boundary: Intake, review, generated config, and deployment hooks can evolve together in the Hub codebase
• Better security posture for follow-up actions: Sensitive post-approval behavior stays in a purpose-built application instead of being forced into form notes or registry constructs
• Clearer evolution path: The portal can add richer workflow, audit, and automation capabilities without re-platforming
• Operational consistency: The same repo, CI/CD patterns, and Azure deployment model can be reused for the portal and Hub-adjacent automation

Negative

• Custom application to build and maintain: We own the frontend, backend, tests, deployment, and documentation
• Higher initial delivery cost: Building a tailored workflow is slower than standing up a generic form or pointing users at an existing portal
• More platform decisions to maintain: Auth, storage, review workflow, and automation semantics become our responsibility

Mitigations

• Keep the portal thin and focused: Implement only onboarding workflow concerns that are specific to AI Hub
• Automate validation and delivery: Maintain build, unit, E2E, and deployment workflows so sustainment cost stays controlled
• Document the why: This ADR exists so future teams do not revisit CHEFS or the Platform Product Registry without understanding the lifecycle mismatch

References

• BC Government Platform Product Registry
• CHEFS OpenAPI
• APIM Developer Portal Overview
• Customising the APIM Developer Portal
• GitHub Actions Workflows

Context

The AI Hub uses Azure Language Service PII recognition to redact personally identifiable information from chat completion payloads before they reach upstream LLM backends. APIM delegates all PII processing to a dedicated external service rather than calling the Language API inline, because APIM policies have no loop construct and the Language API imposes strict per-call limits.

The Azure Language Service /language/:analyze-text endpoint accepts a maximum of 5 documents per call, and each document is limited to 5 120 characters. Real-world chat completion requests can contain many messages or very long messages that chunk into more than 5 documents. Covering the full payload requires batched calls with deadline enforcement and transient retry handling — logic that belongs in a real programming language, not APIM XML.

Options Considered

Decision

We will deploy a dedicated PII redaction microservice as a Container App on the shared internal Container App Environment, reachable only from APIM via VNet-internal ingress. APIM routes all PII-enabled requests to this service.

Rationale

• Language API batching limit: The /language/:analyze-text endpoint accepts at most 5 documents per call (each up to 5 120 chars). Payloads with many or long messages exceed this and need orchestrated batch calls that APIM policies cannot express.
• APIM cannot loop: Policy XML has no iteration construct. Hard-coding multiple send-request blocks is brittle, hard to test, and caps out quickly. A real programming language (Python + asyncio) handles batching, timeouts, retry/backoff, and error recovery naturally.
• Single routing path: A single route from APIM to the external service keeps the policy fragment simple. The service handles all payloads with the same batching logic regardless of size.
• Reuse existing infrastructure: The shared Container App Environment, Managed Identity RBAC, GHCR image builds, and Terraform module patterns are already in place. Adding one more Container App is incremental.
• Timeout budget alignment: The service enforces an 85 s total processing deadline that fits within the 90 s APIM send-request timeout, with per-attempt timeouts (10 s) and bounded transient retries for 429 and 5xx responses.
• Fail-open / fail-closed flexibility: The service returns a structured response with coverage status. APIM decides whether to block (503) or pass through based on the tenant's fail_closed configuration.
• Testability: Python unit tests cover chunking, batching, reassembly, and API error handling — far easier to maintain than equivalent logic embedded in APIM XML.

Architecture

┌──────────────────────────────────────────────────────────────────────┐
│                 ALL-EXTERNAL PII REDACTION                           │
├──────────────────────────────────────────────────────────────────────┤
│                                                                      │
│   Client → App Gateway → APIM                                        │
│                            │                                         │
│                  ┌─────────┴────────────────────────┐                │
│                  │  POST /redact                     │                │
│                  │  (PII Redaction Service)          │                │
│                  └─────────┬────────────────────────┘                │
│                            │                                         │
│                  ┌─────────┴────────────────────────┐                │
│                  │  Bounded concurrent batches       │                │
│                  │  (max 5 docs × 15 batches)       │                │
│                  │  Word-boundary chunking            │                │
│                  │  Retry-after / backoff handling    │                │
│                  │  Deadline enforcement (85s)        │                │
│                  │  Full-coverage check               │                │
│                  └─────────┬────────────────────────┘                │
│                            │                                         │
│                  Redacted body → APIM → Backend                      │
│                                                                      │
└──────────────────────────────────────────────────────────────────────┘

Key Design Constraints

Consequences

Positive

• Transparent payload handling: Tenants do not need to know about Language API limits; APIM routes all PII-enabled requests to the external service automatically
• Single code path: All PII redaction flows through the Container App, keeping behaviour consistent across all payload sizes
• Testable orchestration logic: Chunking, batching, reassembly, and error handling are covered by Python unit tests rather than being embedded in untestable APIM XML
• Incremental infrastructure cost: Runs on the existing shared CAE with Managed Identity RBAC — no new networking or identity infrastructure
• Structured observability: JSON-formatted logs with correlation IDs, batch counts, and elapsed-time diagnostics

Negative

• Additional component to deploy and maintain: One more Container App, Dockerfile, Terraform module, and deployment phase
• Extra network hop for all PII requests: Every PII-enabled request pays the cost of APIM → Container App → Language API instead of APIM → Language API directly

Mitigations

• Reuse proven patterns: The Container App module, GHCR build workflow, and deploy ordering follow the same conventions as the key-rotation job
• Integration tests: The APIM integration test suite covers PII redaction end-to-end through the external service
• Conservative thresholds: The 75-document cap and 85 s deadline ensure the service stays within APIM timeout bounds even under adverse conditions, including transient retry/backoff

References

• Language Service PII Redaction — operational documentation for PII redaction via the external service
• Services Overview — Container App deployment topology
• Azure Language Service — Analyze Text API
• Azure AI Language — PII Detection Overview