Potential Asymmetric Routing Issue with Stateful Firewalls in Multi-Hub Mesh Topology

### Check for previous/existing GitHub issues - [x] I have checked for previous/existing GitHub issues ### Issue Type? Bug ### (Optional) Module Version _No response_ ### (Optional) Correlation Id _No response_ ### Description We are experiencing connectivity issues in a multi-hub mesh topology where traffic appears to follow asymmetric paths through different Azure Firewalls, potentially causing stateful firewall inspection to drop return packets. We would like to understand if this is an expected behavior or if there might be a routing design consideration we're missing. ## Environment - Module Version: linked from `Azure/avm-ptn-alz-connectivity-hub-and-spoke-vnet/azurerm` v0.11.2 - Topology: Multi-hub with mesh_peering_enabled = true - Firewalls: Azure Firewall (stateful) deployed in each hub - Issue: ICMP ping fails between subnets in different hubs, Azure Network Watcher shows "GuestFirewall" error ## Problem Description In our multi-hub setup, we're seeing what appears to be asymmetric routing behavior: Forward Path: Traffic from Hub A to Hub B goes through Hub A's firewall initially, then gets routed to Hub B's firewall Return Path: Traffic from Hub B back to Hub A goes through Hub B's firewall initially, then gets routed to Hub A's firewall This creates a scenario where: 1. Hub A's firewall creates a stateful connection entry for the outbound flow 2. Return traffic comes back through Hub A's firewall, but the firewall may not have the proper state information since the return path was processed by Hub B's firewall first Looking at https://github.com/Azure/terraform-azurerm-avm-ptn-hubnetworking/blob/v0.12.3/locals.routing.tf#L77-L93, we notice the mesh routing logic uses destination-based next hop selection: ```terraform mesh_route_map_user_subnets = { for route in flatten([ for k_src, v_src in var.hub_virtual_networks : [ for k_dst, v_dst in var.hub_virtual_networks : [ for index, cidr in v_dst.routing_address_space : { virtual_network_key = k_src key = "${k_src}-${k_dst}-${index}" name = "${k_src}-${k_dst}-${replace(cidr, "/", "-")}" address_prefix = cidr next_hop_type = "VirtualAppliance" next_hop_in_ip_address = try(local.firewall_private_ip[k_dst], v_dst.hub_router_ip_address) resource_group_name = try(v_src.resource_group_name, azurerm_resource_group.rg[k_src].name) } if v_dst.mesh_peering_enabled && can(v_dst.routing_address_space[0]) && local.create_route_tables_user_subnets[k_dst] ] ] if v_src.mesh_peering_enabled && local.create_route_tables_user_subnets[k_src] ]) : route.key => route } ``` ## Expected vs Observed Routing **What we observe:** - Route from Hub A → Hub B: next_hop = firewall_private_ip[hub_b] (destination hub's firewall) - Route from Hub B → Hub A: next_hop = firewall_private_ip[hub_a] (destination hub's firewall) This creates asymmetric paths where: 1. Outbound traffic: Hub A subnet → Hub B firewall → Hub B subnet 2. Return traffic: Hub B subnet → Hub A firewall → Hub A subnet Since Azure Firewalls are stateful, the return traffic might be dropped because: - Hub B firewall sees the outbound flow and creates a connection state - Hub A firewall sees the return traffic but has no state information about the original connection - Hub A firewall drops the return packet as unsolicited traffic We may be misunderstanding the intended architecture or missing some configuration. Any guidance on the proper way to handle multi-hub stateful firewall routing would be appreciated.