Gateway Subscription Key Rotation

Alternating Primary/Secondary Pattern

APIM subscriptions have two key slots: primary and secondary. Both are valid for API calls at all times. Each rotation cycle regenerates one slot while the other remains untouched:

1. Week 1: Regenerate SECONDARY — tenants continue using PRIMARY (untouched)
2. Week 2: Regenerate PRIMARY — tenants continue using SECONDARY (regenerated last week, still valid)
3. Week 3: Regenerate SECONDARY — alternates indefinitely

Both keys for every subscription-key tenant are stored in the centralized hub Key Vault on first deploy. After rotation, the rotation job updates them with the new values. Tenants can fetch new keys at their convenience via the APIM internal endpoint or by requesting access from the platform team.

Key Lifecycle Timeline

The key NOT being rotated is always safe. You have the full rotation interval to switch to the new key.

Compromised Keys: Rotate Both Slots Immediately

Use this runbook only for emergency compromise scenarios where both keys must be replaced right away. This is a manual operational procedure (documentation only).

1. Identify the APIM subscription ID and tenant name.
2. Regenerate secondary and then primary in APIM.
3. Read latest key values from APIM listSecrets.
4. Write both keys to hub Key Vault with 90-day expiry.
5. Update {tenant}-apim-rotation-metadata with safe_slot and timestamps.
6. Notify tenant team to pull new keys from /internal/apim-keys.

Example CLI Sequence

# Inputs
SUBSCRIPTION_ID="<azure-subscription-id>"
RESOURCE_GROUP="ai-services-hub-dev"
APIM_NAME="ai-services-hub-dev-apim"
TENANT="wlrs-water-form-assistant"
APIM_SUB_ID="<apim-subscription-guid>"
HUB_KV="ai-services-hub-dev-hkv"

# 1) Regenerate both slots (secondary first, then primary)
az rest --method POST \
    --url "https://management.azure.com/subscriptions/${SUBSCRIPTION_ID}/resourceGroups/${RESOURCE_GROUP}/providers/Microsoft.ApiManagement/service/${APIM_NAME}/subscriptions/${APIM_SUB_ID}/regenerateSecondaryKey?api-version=2024-05-01"

az rest --method POST \
    --url "https://management.azure.com/subscriptions/${SUBSCRIPTION_ID}/resourceGroups/${RESOURCE_GROUP}/providers/Microsoft.ApiManagement/service/${APIM_NAME}/subscriptions/${APIM_SUB_ID}/regeneratePrimaryKey?api-version=2024-05-01"

# 2) Get current keys from APIM
SECRETS_JSON=$(az rest --method POST \
    --url "https://management.azure.com/subscriptions/${SUBSCRIPTION_ID}/resourceGroups/${RESOURCE_GROUP}/providers/Microsoft.ApiManagement/service/${APIM_NAME}/subscriptions/${APIM_SUB_ID}/listSecrets?api-version=2024-05-01")

PRIMARY=$(echo "$SECRETS_JSON" | jq -r '.primaryKey')
SECONDARY=$(echo "$SECRETS_JSON" | jq -r '.secondaryKey')
EXPIRES_ON=$(date -u -d "+90 days" +"%Y-%m-%dT%H:%M:%SZ")
NOW=$(date -u +"%Y-%m-%dT%H:%M:%SZ")

# 3) Store keys in hub Key Vault
az keyvault secret set --vault-name "$HUB_KV" --name "${TENANT}-apim-primary-key" --value "$PRIMARY" --expires "$EXPIRES_ON"
az keyvault secret set --vault-name "$HUB_KV" --name "${TENANT}-apim-secondary-key" --value "$SECONDARY" --expires "$EXPIRES_ON"

# 4) Update metadata
METADATA=$(jq -n --arg now "$NOW" '{
    last_rotated_slot: "primary",
    last_rotation_at: $now,
    next_rotation_at: "manual-emergency",
    rotation_number: 999,
    safe_slot: "secondary"
}')

az keyvault secret set \
    --vault-name "$HUB_KV" \
    --name "${TENANT}-apim-rotation-metadata" \
    --value "$METADATA" \
    --content-type "application/json" \
    --expires "$EXPIRES_ON"

Tenant Retrieval After Emergency Rotation

After platform team rotates both keys, tenant teams should fetch current keys from APIM and switch to the safe_slot key.

curl -s -H "api-key: YOUR_CURRENT_VALID_KEY" \
    "https://your-apim-gateway.azure-api.net/YOUR_TENANT/internal/apim-keys" | jq .

# Read the recommended key:
# .rotation.safe_slot
# Then use .primary_key or .secondary_key accordingly.

If the tenant has no valid key left, platform team must provide one-time bootstrap access/key out-of-band.

Shared Config (params/{env}/shared.tfvars)

Key rotation has a global master toggle that acts as the central on/off switch for the entire environment:

apim = {
  # ... existing APIM config ...

  key_rotation = {
    rotation_enabled       = true   # Master flag (on/off for entire environment)
    rotation_interval_days = 7      # Rotate every N days (same for all tenants)
  }
}

Per-Tenant Toggle (params/{env}/tenants/{tenant}/tenant.tfvars)

Each tenant can individually opt in to key rotation via the key_rotation_enabled flag in their apim_auth block. This is an opt-in model — tenants must explicitly set true to participate:

apim_auth = {
  mode                 = "subscription_key"
  key_rotation_enabled = true   # Opt-in to automatic key rotation
}

Tenant Prerequisites

For a tenant to participate in key rotation, it needs:

• apim_auth.mode = "subscription_key" (default)
• apim_auth.key_rotation_enabled = true (opt-in per tenant)

Option 1: APIM Internal Endpoint (Simplest)

Call GET /{your-tenant}/internal/apim-keys with ANY valid subscription key. Returns both keys plus rotation metadata as JSON.

Bash

#!/bin/bash
# Fetch both APIM keys via the internal endpoint
APIM_GATEWAY="https://your-apim-gateway.azure-api.net"
TENANT="your-tenant-name"
CURRENT_KEY="your-current-valid-key"

RESPONSE=$(curl -s \
    -H "api-key: ${CURRENT_KEY}" \
    "${APIM_GATEWAY}/${TENANT}/internal/apim-keys")

echo "${RESPONSE}" | jq .

# Example response:
# {
#   "tenant": "your-tenant-name",
#   "primary_key": "abc123...",
#   "secondary_key": "def456...",
#   "rotation": {
#     "last_rotated_slot": "primary",
#     "last_rotation_at": "2026-02-11T02:00:00Z",
#     "next_rotation_at": "2026-02-18T02:00:00Z",
#     "rotation_number": 5,
#     "safe_slot": "secondary"
#   },
#   "keyvault": {
#     "uri": "https://hub-kv.vault.azure.net/",
#     "primary_key_secret": "your-tenant-name-apim-primary-key",
#     "secondary_key_secret": "your-tenant-name-apim-secondary-key"
#   }
# }

# Use the safe_slot key (the one NOT just rotated)
SAFE_SLOT=$(echo "${RESPONSE}" | jq -r '.rotation.safe_slot')
if [[ "${SAFE_SLOT}" == "primary" ]]; then
    NEW_KEY=$(echo "${RESPONSE}" | jq -r '.primary_key')
else
    NEW_KEY=$(echo "${RESPONSE}" | jq -r '.secondary_key')
fi
echo "Safe key to use: ${NEW_KEY:0:8}..."

Python

import requests

apim_gateway = "https://your-apim-gateway.azure-api.net"
tenant = "your-tenant-name"
current_key = "your-current-valid-key"

response = requests.get(
    f"{apim_gateway}/{tenant}/internal/apim-keys",
    headers={"api-key": current_key}
)
data = response.json()

safe_slot = data["rotation"]["safe_slot"]  # "primary" or "secondary"
safe_key = data[f"{safe_slot}_key"]
print(f"Safe slot: {safe_slot}, key: {safe_key[:8]}...")

Option 2: Contact Platform Team

If you've missed a rotation schedule or need a fresh set of keys, contact the platform team directly. The platform team manages the centralized hub Key Vault and can provide your current keys or trigger an ad-hoc rotation.

Option 3: Automatic Key Refresh (Production Pattern)

For production apps, implement a periodic key check. This pattern uses the APIM endpoint so no Azure SDK is needed.

#!/bin/bash
# rotate-my-key.sh - Run via cron: 0 3 * * * /path/to/rotate-my-key.sh
set -euo pipefail

APIM_GATEWAY="https://your-apim-gateway.azure-api.net"
TENANT="your-tenant-name"
CONFIG_FILE="/etc/myapp/apim-key"

# Read the current key your app is using
CURRENT_KEY=$(cat "${CONFIG_FILE}" 2>/dev/null || echo "")

if [[ -z "${CURRENT_KEY}" ]]; then
    echo "ERROR: No current key found in ${CONFIG_FILE}"
    exit 1
fi

# Fetch both keys via APIM (authenticates with current key)
RESPONSE=$(curl -s -H "api-key: ${CURRENT_KEY}" \
    "${APIM_GATEWAY}/${TENANT}/internal/apim-keys")

SAFE_SLOT=$(echo "${RESPONSE}" | jq -r '.rotation.safe_slot')
if [[ "${SAFE_SLOT}" == "primary" ]]; then
    LATEST_KEY=$(echo "${RESPONSE}" | jq -r '.primary_key')
else
    LATEST_KEY=$(echo "${RESPONSE}" | jq -r '.secondary_key')
fi

if [[ "${CURRENT_KEY}" != "${LATEST_KEY}" ]]; then
    echo "$(date -u +'%Y-%m-%dT%H:%M:%SZ') Key changed - updating config"
    echo "${LATEST_KEY}" > "${CONFIG_FILE}"
    # Optionally: systemctl reload myapp
else
    echo "$(date -u +'%Y-%m-%dT%H:%M:%SZ') Key unchanged"
fi

All subscription-key tenants have their keys stored in the centralized hub Key Vault ({app_name}-{env}-hkv) on first deploy, with tenant-prefixed names:

The Container App Job runs on a cron schedule (configurable NCRONTAB), but the rotation logic checks whether the rotation interval has actually elapsed before regenerating keys. A 30-day interval means most invocations will be no-ops.

GET /{tenant-name}/internal/apim-keys

When key rotation is enabled, each tenant API exposes an internal endpoint for fetching both subscription keys.

Authentication

Requires a valid APIM subscription key (either primary or secondary) in the api-key header. APIM validates the key before serving the response.

How It Works

1. APIM validates the incoming subscription key (standard APIM behavior)
2. APIM policy uses its managed identity to read secrets from the centralized hub Key Vault
3. Returns JSON with primary key, secondary key, rotation metadata, and Key Vault info

Response Example

{
  "tenant": "your-tenant-name",
  "primary_key": "abc123def456...",
  "secondary_key": "ghi789jkl012...",
  "rotation": {
    "last_rotated_slot": "primary",
    "last_rotation_at": "2026-02-11T02:00:00Z",
    "next_rotation_at": "2026-02-18T02:00:00Z",
    "rotation_number": 5,
    "safe_slot": "secondary"
  },
  "keyvault": {
    "uri": "https://hub-kv.vault.azure.net/",
    "primary_key_secret": "your-tenant-name-apim-primary-key",
    "secondary_key_secret": "your-tenant-name-apim-secondary-key"
  }
}

Infrastructure

The endpoint is implemented purely in APIM policy (no backend service). APIM's system-assigned managed identity is granted Key Vault Secrets User on the centralized hub Key Vault via a single Terraform RBAC assignment (scales to 1000+ tenants).

See also: APIM Internal Endpoints reference — full API reference for all internal endpoints including the new /internal/tenant-info endpoint.

My API calls return 401 after rotation

Your app is using the key that was just rotated. The other key is still valid. Fix:

1. Fetch both keys: curl -H "api-key: YOUR_OTHER_KEY" https://apim/{tenant}/internal/apim-keys
2. Use the safe_slot key from the response
3. Both keys are also in the hub Key Vault: {tenant}-apim-primary-key and {tenant}-apim-secondary-key

How do I check rotation metadata?

az keyvault secret show \
    --vault-name HUB_KV_NAME \
    --name "YOUR_TENANT-apim-rotation-metadata" \
    --query "value" -o tsv | jq .

Output:

{
  "last_rotated_slot": "primary",
  "last_rotation_at": "2026-02-11T02:00:00Z",
  "next_rotation_at": "2026-02-18T02:00:00Z",
  "rotation_number": 5,
  "safe_slot": "secondary"
}

The /internal/apim-keys endpoint returns empty keys

APIM's managed identity may not have access to the hub Key Vault. Verify the RBAC assignment:

terraform output apim_key_rotation_summary

Ensure the azurerm_role_assignment.apim_keyvault_secrets_user resource exists on the hub Key Vault. Check that the hub Key Vault ({app_name}-{env}-hkv) is deployed and accessible.

Container App Job not rotating keys

Check the Container App Job logs (Log Stream in Azure Portal or Log Analytics) for errors. Common causes:

1. Managed identity missing Key Vault Secrets Officer role on hub Key Vault
2. Managed identity missing Contributor role on APIM for key regeneration
3. Container App Job is suspended or the Container App revision is unhealthy
4. Contact the platform team if rotation was missed — they can issue new keys or trigger an ad-hoc rotation

I want to temporarily disable rotation for my environment

Set rotation_enabled = false in params/{env}/shared.tfvars and deploy. The rotation workflow will skip all tenants.

I want to disable rotation for a single tenant

Set key_rotation_enabled = false in the tenant's apim_auth block in params/{env}/tenants/{tenant}/tenant.tfvars and deploy. That tenant will be excluded from rotation while others continue normally.

My tenant is not eligible for rotation

Check:

terraform output apim_key_rotation_summary

Common reasons:

• apim_auth.key_rotation_enabled != true — tenant has not opted in to rotation
• apim_auth.mode != "subscription_key" — rotation only applies to subscription key auth
• Rotation is globally disabled (rotation_enabled = false in shared.tfvars)

Component Overview

Security Considerations

• Managed Identity Authentication: The Container App Job uses a system-assigned managed identity for APIM and Key Vault access. No stored secrets.
• Centralized Hub Key Vault: All subscription-key tenant keys are stored in a single hub KV on first deploy, regardless of rotation opt-in. APIM reads keys via its system-assigned MI with one RBAC assignment. Scales to 1000+ tenants.
• No Per-Tenant KV Required: Tenants do not need their own Key Vault. The hub KV handles all tenants centrally.
• Audit Trail: Every key change is versioned in Key Vault with rotation metadata tags.
• No Key in Logs: The rotation function never prints key values. Only metadata appears in logs.
• Terraform Lifecycle: Key Vault secrets use ignore_changes = [value, tags] so Terraform won't overwrite rotated keys.
• Internal Endpoint Security: The /internal/apim-keys endpoint requires a valid subscription key. Only authenticated tenants can read their own keys.

Tenant administrators can view their APIM subscription keys directly in the Tenant Onboarding Portal without contacting the platform team.

Who Can Access

Only users listed in a tenant's admin_users field (set during onboarding) can view credentials. The tenant must also have approved status. Portal logins are gated by Azure AD authentication and the tenant-admin check is enforced server-side.

What Is Shown

• Primary and Secondary APIM subscription keys — displayed masked (••••••••••••••••) with a copy button. Values are never rendered in the page DOM.
• Rotation metadata — last rotation timestamp, next scheduled rotation, current safe slot, and rotation number (when rotation is enabled for the tenant).
• Tenant info — base URL, enabled services, and deployed model list (collapsible panel, lazy-loaded from APIM).
• Three environment tabs — dev, test, and prod credentials are shown in separate tabs and loaded on demand.

How It Works

The portal's system-assigned Managed Identity is granted Key Vault Secrets User on each hub Key Vault (one RBAC assignment per environment, provisioned by tenant-onboarding-portal/infra/main.tf). The backend reads secrets directly from the hub KV using DefaultAzureCredential — no connection strings or stored credentials required.

The secrets read are: {tenant}-apim-primary-key, {tenant}-apim-secondary-key, and {tenant}-apim-rotation-metadata.

Security Properties

• Key values are returned from the backend only to authenticated, authorised sessions — the response is never cached client-side.
• The frontend copies keys to the clipboard via navigator.clipboard.writeText directly from the API response value; the key is never placed in React state or rendered as text in the DOM.
• If the portal Managed Identity is not granted access to a given environment's Key Vault, the endpoint returns HTTP 503 and the UI displays a “not configured” message rather than an error.