TL;DR

When operating Azure API Management in an internal virtual network, already integrated with Azure Application Gateway, an upcoming feature AllowApplicationGatewayPrivateLink allows you to connect this configuration to another virtual network using Private Link and Private Endpoint.

Motivation

In a post I made beginning of February 2022 I linked a virtual network with limited address space - like in a corporate / ExpressRoute / SD-WAN connected scenario - to a Container Apps environment:

That setup works as long as ingress only goes from resources in Hub network to Spoke network. But if Container Apps needs to call (as in my target scenario) Azure API Management - calling from Spoke network into Hub network - that solution is not sufficient.

Looking for options I had this post which Marcel.L pointed me to back then, where he forwards calls to API Management in another virtual network using a Private Endpoint/Link and Virtual Machine Scale Set combination.

However, as pointed out in my earlier post, my endgame should be to replace Azure Service Fabric IaaS container hosting with a higher level abstraction, PaaS like Container Apps, hence I did not necessarily want to add just another IaaS in the process - even one with a lower complexity.

Searching for alternatives I checked on private linking capabilities of Azure API Management itself. However this cannot be used and mixed when it is already operated in external or internal virtual network mode - see preview limitations. Hence no option for me.

"As with container compute I need my API Management with 100% ingress and egress within a virtual network"

Researching on private IP address options on Azure Application Gateway I stumbled over a private link feature in Azure CLI which - lacking tangible documentation - I exploited here and converted to Bicep

Solution Elements

To achieve this configuration

following solution elements additionally to my earlier post are required:

a Private Link Configuration on the Application Gateway; still to subnet within Hub virtual network
a Private Endpoint in Spoke virtual network
a Private DNS Zone linked to Spoke virtual network so that Container Apps resolve to Private Endpoint IP address

complete configuration for all snippets shown in this post can be found in this repo / tag; I'll state the filenames so that snippets can be found more easily

Prerequisites

Azure CLI
Bicep
Linux shell / bash / ...

Stage 1 - get preview feature working (as of Feb'2022)

When first trying to deploy Application Gateway with specifying privateLinkConfiguration, I was rewarded with a nice error code SubscriptionNotRegisteredForFeature and a message like Subscription /subscriptions/... is not registered for feature PrivateLinkConfigurations required to carry out the requested operation. Finding no suitable documentation I tried to find the feature switch myself with

az feature list --namespace Microsoft.Network -o table | grep Priv

I then activated what seemed to make sense for my case:

az feature register --name AllowApplicationGatewayPrivateLink --namespace Microsoft.Network
az feature register --name AllowAppGwPublicAndPrivateIpOnSamePort --namespace Microsoft.Network
az provider register -n Microsoft.Network

DISCLAIMER: The above steps I figured out being unaware that this is (as of Feb'22) a private preview.
The official process: For inquiries about private preview features, please send your subscription id, region, and customer name to appgwpreview@microsoft.com for enrollment.
This will entitle you officially to the preview and you will get proper documentation and sample code.

Stage 2 - configuration Application Gateway

To add private link, an entry in privateLinkConfigurations section is required. It needs to be put in the same virtual network but a different subnet as the gateway itself (once Application Gateway is deployed to a subnet, it only allows for other Application Gateways to share this subnet, but no other resources).

That privateLinkConfiguration is then to be referenced on the frontendIPConfiguration. In my sample I share it with the public IP, but I guess, this also can be separated to allow various forwarding rules depending from where ingress is coming from.

appgw-priv.bicep

...
var fipName = 'appgw-frontend'
...
frontendIPConfigurations: [
  {
    name: fipName
    properties: {
      publicIPAddress: {
        id: pip.id
      }
      privateIPAllocationMethod:'Dynamic'
      privateLinkConfiguration:{
        id: resourceId('Microsoft.Network/applicationGateways/privateLinkConfigurations', appGwName, 'private')
      }
    }
  }
]
privateLinkConfigurations: [
  {
    name: 'private'
    properties: {
      ipConfigurations: [
        {
          name: 'private-ip'
          properties: {
            primary: true
            privateIPAllocationMethod:'Dynamic'
            subnet: {
              id: subnetJumpHubId
            }
          }
        }
      ]
    }
  }
]
...

for simplification in my sample I am using HTTP on port 8080; in production this will be replaced by a proper HTTPS, FQDNs and certificates

Based on private link configuration in Hub network now a private endpoint can be added in Spoke network:

appgw-priv.bicep

resource pep 'Microsoft.Network/privateEndpoints@2021-05-01' = {
  name: 'pep-priv-gateway'
  location: location
  properties: {
    subnet: {
      id: subnetJumpSpokeId
    }
    privateLinkServiceConnections: [
      {
        properties: {
          privateLinkServiceId: appgw.id
          groupIds: [
            fipName
          ]
        }
        name: 'pep-priv-gateway'
      }
    ]
  }
}

fipName

Important to highlight: the name of the groupId added to the Private Endpoint has to match the name of the frontendIPConfiguration of the Application Gateway.

Stage 3 - private DNS zone

Spoke network needs a private DNS zone so that traffic towards the Private Endpoint is resolved correctly:

appgw-priv-dns.bicep

param vnetSpokeId string
param apiName string
param pepIp string

var privateDNSZoneName = 'internal-api.net'

resource privateDnsZone 'Microsoft.Network/privateDnsZones@2020-06-01' = {
  name: privateDNSZoneName
  location: 'global'
}

resource privateDnsZoneLink 'Microsoft.Network/privateDnsZones/virtualNetworkLinks@2020-06-01' = {
  parent: privateDnsZone
  name: '${privateDnsZone.name}-link'
  location: 'global'
  properties: {
    registrationEnabled: false
    virtualNetwork: {
      id: vnetSpokeId
    }
  }
}

resource privateDnsZoneEntry 'Microsoft.Network/privateDnsZones/A@2020-06-01' = {
  name: apiName
  parent: privateDnsZone
  properties: {
    aRecords: [
      {
        ipv4Address: pepIp
      }
    ]
    ttl: 3600
  }
}

I use my own DNS zone internal-api.net here as - unlike other Azure private link capable resources - Application Gateway does (and I guess will) not have its own private endpoint DNS zones.

Stage 4 - deploy API Management, Application Gateway and DNS zone

I had to split deployment of API Management & Application Gateway, return and extract private IP address information from private endpoint and then continue with private DNS deployment. I was not able (or just too lazy) to wire this up within one Bicep file.

deploy-stage-2.sh

...
az deployment group create --resource-group $RESOURCE_GROUP \
    --template-file apim.bicep \
    --parameters apimName=$APIMNAME \
    appInsightsName=$APPINSIGHTNAME \
    logAnalyticsWorkspaceName=$LOGANALYTICSNAME \
    fapp1Fqdn=$fapp1Fqdn \
    fapp2Fqdn=$fapp2Fqdn

VNET_SPOKE_ID=`az network vnet list --resource-group ${RESOURCE_GROUP} --query "[?contains(name,'spoke')].id" -o tsv`
PEP_NIC_ID=`az network private-endpoint list -g $RESOURCE_GROUP --query "[?name=='pep-priv-gateway'].networkInterfaces[0].id" -o tsv`
PEP_IP=`az network nic show --ids $PEP_NIC_ID --query ipConfigurations[0].privateIpAddress -o tsv`

az deployment group create --resource-group $RESOURCE_GROUP \
    --template-file appgw-priv-dns.bicep \
    --parameters "{\"pepIp\": {\"value\": \"$PEP_IP\"},\"vnetSpokeId\": {\"value\": \"$VNET_SPOKE_ID\"},\"apiName\": {\"value\": \"$APIMNAME\"}}"

along with API Management instance I deploy an API to test calls to Function Apps hosted in Container Apps environment; appgw-priv.bicep shown above is a module of apim.bicep, hence deployed in the first block

Stage 5 - testing

To check, that I have not introduced regressions I first test access to Function Apps over public IP of Application Gateway:

test-api.sh

...
APIMID=`az apim show -n $APIMNAME -g $RESOURCE_GROUP --query id -o tsv`
APIMURL=`az apim show -n $APIMNAME -g $RESOURCE_GROUP --query gatewayUrl -o tsv`
GWURL=`az network public-ip show -n ${APIMNAME}-priv-gateway-pip -g $RESOURCE_GROUP --query dnsSettings.fqdn -o tsv`
GWPORT=`az network application-gateway show -n ${APIMNAME}-gateway -g $RESOURCE_GROUP --query frontendPorts[0].port -o tsv`
SUBKEY=`az rest --method post  --uri ${APIMID}/subscriptions/test-subscription/listSecrets?api-version=2021-08-01 --query primaryKey -o tsv`

curl -s ${GWURL}:${GWPORT}/test/fapp1?subscription-key=$SUBKEY
printf '\n'
curl -s ${GWURL}:${GWPORT}/test/fapp2?subscription-key=$SUBKEY
printf '\n'

To make more thorough tests of the Functions Apps in combination with calls to API Management I have to hop on a jump VM in Spoke network:

test-fapps.sh

...
IP=$(az vm list-ip-addresses -g $RESOURCE_GROUP --query "[?contains(virtualMachine.name, 'hub')].virtualMachine.network.publicIpAddresses[0].ipAddress" -o tsv)

declare -a apps=("fapp1" "fapp2")

for app in "${apps[@]}"
do
    echo "$app"
    fqdn=$(az containerapp show -n $app -g $RESOURCE_GROUP --query configuration.ingress.fqdn -o tsv --only-show-errors)
    ssh ca@$IP curl -s https://$fqdn/api/health
    echo " <<-- check APIM internal status"
    ssh ca@$IP curl -s https://$fqdn/api/apim-status
    echo " <<-- check $app APIM health"
    ssh ca@$IP curl -s https://$fqdn/api/apim-internal-status
    echo " <<-- check $app APIM internal status"
done

where

curl -s https://$fqdn/api/health checks Function App only with a call to a HTTP trigger
curl -s https://$fqdn/api/apim-status and curl -s https://$fqdn/api/apim-internal-status call HTTP triggers which themselves forward calls to API Management instance (using the internal-api.net domain)

[FunctionName("Apim-Status")]
public static async Task<IActionResult> ApimStatus([HttpTrigger(AuthorizationLevel.Anonymous, "get", Route = "apim-status")] HttpRequest req)
  => new OkObjectResult(await httpClient.GetAsync("http://ca-kw.internal-api.net:8080/status-0123456789abcdef"));

[FunctionName("Apim-Internal-Status")]
public static async Task<IActionResult> ApimInternalStatus([HttpTrigger(AuthorizationLevel.Anonymous, "get", Route = "apim-internal-status")] HttpRequest req)
  => new OkObjectResult(await httpClient.GetAsync("http://ca-kw.internal-api.net:8080/internal-status-0123456789abcdef"));

Public IP vs SKU vs Private Link Configuration

For my scenario a public IP would not be required. Removing it is not supported for Standard_v2 SKU:

Application Gateway /subscriptions/.../ca-kw-priv-gateway does not support Application Gateway without Public IP for the selected SKU tier Standard_v2. Supported SKU tiers are Standard,WAF.

On the other hand PrivateLinkConfigurations is only supported for Standard_v2:

Application Gateway /subscriptions/.../ca-kw-priv-gateway does not support PrivateLinkConfigurations for the selected SKU tier Standard. Supported SKU tiers are Standard_v2,WAF_v2.

So for the moment I'd keep the public IP and will check again at a later stage of the preview.

Conclusion

With this solution a major roadblock for me is out of the way and I do not need to operate my own compute resources for network traffic forwarding. It still requires a some polishing and hardening before I can use it even close to our production resources.

A positive effect for me is that with reconfiguration of the existing Application Gateway and a very low footprint of IP addresses consumed in my corporate virtual network, I can even operate the existing environment with Service Fabric in coexistence with the new Container Apps environment, which reduces migration risks tremendously.

Use Azure Application Gateway private link configuration for an internal API Management