Skip to main content
Calico Cloud documentation

L2 reachability for pods and services without BGP

Big picture​

Make pods and LoadBalancer service IPs reachable from the local layer 2 (L2) network without configuring BGP, by having Calico Cloud answer ARP and NDP requests for those IPs directly.

Value​

When a pod or LoadBalancer VIP is assigned an IP from the same subnet as a node's physical interface, external hosts on that L2 segment try to reach it with ARP (IPv4) or Neighbor Solicitation (IPv6). By default nobody answers: the pod IP lives in a network namespace behind a veth, and a LoadBalancer VIP does not exist on any interface.

Traditionally the only way to make these IPs reachable is to advertise them over BGP, which requires peering with your network infrastructure. In flat L2 environments — on-prem environment, edge deployments, kind clusters, or networks where you cannot configure the upstream router — BGP may not be an option.

With localSubnetL2Reachability enabled, Calico Cloud runs a userspace ARP/NDP responder on the relevant host interfaces and answers for local pod IPs and selected LoadBalancer VIPs that fall within the host subnet. No BGP, no overlay encapsulation, and no changes to your network infrastructure.

Concepts​

Userspace ARP/NDP responder​

Calico Cloud opens a raw socket on each host physical interface that has at least one pod IP or LoadBalancer VIP within its subnet, and replies to ARP/NDP requests for those IPs. The reply carries the node's MAC address, so the external host sends the workload's traffic to that node. From there Calico Cloud's normal data plane forwards it the rest of the way — directly to a local pod for a pod IP, or load-balanced to a service backend for a LoadBalancer VIP. The responder only answers ARP/NDP; it does not change how traffic is forwarded once it reaches the node.

One node answers per LoadBalancer VIP​

For a LoadBalancer VIP, exactly one node answers ARP. Calico Cloud elects that node consistently across the cluster using a consistent-hash ring, so every node independently agrees on the owner without coordination. If the owning node's Node object is deleted, the VIP is reassigned to another node, which proactively updates external caches with a gratuitous ARP / unsolicited Neighbor Advertisement.

Only no-encapsulation IP pools are eligible​

Calico Cloud answers only for IPs that belong to an IP pool with no encapsulation (ipipMode and vxlanMode both Never). Encapsulated pools are treated as private pod networks and are never advertised on the host L2 segment.

Before you begin...​

Limitations

  • One node per LoadBalancer VIP. There is no active-active high availability. If the owning node goes down, the VIP is unreachable until its Node object is removed and a new owner is elected.
  • Cloud provider ARP filtering. Some cloud networks (for example, AWS VPC) filter ARP at the hypervisor. The responder will not work unless the VIP/pod IPs are assigned to the node's cloud interface. This is an infrastructure constraint outside Calico Cloud's control.
  • Heterogeneous subnets. Node selection for LoadBalancer VIPs hashes over all nodes without checking whether a node has an interface on the VIP's subnet. This is only a concern when nodes do not share the same subnets.
  • Multiple host interfaces on the same L2 segment. Calico Cloud answers on every host interface whose subnet contains a proxied IP. If a node has two or more interfaces on the same L2 segment (the same broadcast domain) and a proxied IP falls in that shared subnet, each of those interfaces replies to ARP/NDP for the IP with its own MAC address, so external hosts see the IP resolve to several MACs and their neighbor caches flip between them.
  • Only externalTrafficPolicy: Cluster is supported for LoadBalancer services. Calico Cloud elects the node that answers for a VIP with a consistent hash over all nodes, independent of where the service's backends run. Under externalTrafficPolicy: Local the elected node may have no local backend, so traffic that reaches it is dropped. Leave externalTrafficPolicy at its default of Cluster.
  • kube-proxy in IPVS mode requires strictARP. In IPVS mode, kube-proxy binds every LoadBalancer VIP to the kube-ipvs0 dummy interface on all nodes. Because the Linux kernel answers ARP for any IP configured on the host by default, every node replies for the VIP — not just the one Calico Cloud elected — so external hosts receive conflicting replies from several MAC addresses. Either run kube-proxy in iptables or nftables mode, or keep IPVS mode and set strictARP: true in kube-proxy's configuration (the ipvs.strictARP field of the KubeProxyConfiguration, which is stored as YAML inside the kube-proxy ConfigMap) so the kernel stops answering ARP for the dummy-interface VIPs and leaves it to Calico Cloud's responder.
note

The ARP/NDP responder runs inside Felix (calico-node). While a node's Felix is restarting it briefly does not answer ARP or NDP. Established connections keep working because the upstream router and switches still have the node's MAC cached for those IPs, but a new connection to an IP that is not already cached upstream may fail until Felix finishes starting and rebuilds its responder state (typically under a minute).

To reduce the impact, increase the ARP/neighbor cache timeout (aging time) on your upstream router and switches so cached entries survive a Felix restart.

How to​

Enable local subnet L2 reachability​

Set localSubnetL2Reachability to PodsAndLoadBalancers on the default FelixConfiguration:

kubectl patch felixconfiguration default --type='merge' \
   -p '{"spec": {"localSubnetL2Reachability": "PodsAndLoadBalancers"}}'

The feature is Disabled by default. This setting is read when the data plane starts, so changing it makes Felix (calico-node) restart automatically to pick up the new value — no manual restart is required. The feature takes effect once Felix finishes restarting. For the full field reference, see Felix configuration resource.

note

Enabling the feature alone does nothing. Calico Cloud only answers for IPs in a no-encapsulation IP pool that overlaps a host subnet, so you must also create that pool.

Optional: tune periodic re-announcement. While the feature is enabled, Felix periodically re-announces every IP it proxies with a gratuitous ARP / unsolicited Neighbor Advertisement, keeping upstream neighbor caches and switch forwarding tables warm even when the set of proxied IPs has not changed. localSubnetL2ReachabilityRefreshInterval controls how often this happens (default 120s). Lower it if your upstream devices age out ARP/NDP entries quickly, or set it to 0 to disable periodic re-announcement and keep only the one-shot announcement sent when an IP is first proxied:

kubectl patch felixconfiguration default --type='merge' \
   -p '{"spec": {"localSubnetL2ReachabilityRefreshInterval": "30s"}}'

Create a dedicated IP pool in the host subnet​

Create an IP pool whose CIDR lies inside your host interface's subnet, with no encapsulation.

caution

Reserve a sub-range of the host subnet for this pool that does not overlap DHCP scopes, static host assignments, or infrastructure addresses (routers, switches, management interfaces). For example, for a host subnet of 10.0.0.0/23, use 10.0.0.2–10.0.0.255 for hosts and a separate 10.0.1.0/24 for the pool.

For pods:

apiVersion: projectcalico.org/v3
kind: IPPool
metadata:
  name: host-subnet-pod-pool
spec:
  cidr: 10.0.1.0/24
  ipipMode: Never
  vxlanMode: Never
  natOutgoing: false
  disabled: false
  assignmentMode: Manual

assignmentMode: Manual keeps the pool opt-in: Calico Cloud never assigns from it automatically, so only the workloads you explicitly steer into it get host-subnet IPs. Without this, Calico Cloud could allocate IPs from the pool to arbitrary pods.

For LoadBalancer services, add allowedUses: [LoadBalancer]:

apiVersion: projectcalico.org/v3
kind: IPPool
metadata:
  name: host-subnet-lb-pool
spec:
  cidr: 10.0.2.0/24
  ipipMode: Never
  vxlanMode: Never
  natOutgoing: false
  disabled: false
  assignmentMode: Manual
  allowedUses:
    - LoadBalancer

With assignmentMode: Manual, ordinary LoadBalancer services keep drawing VIPs from your existing automatic pools — you do not need to change the LoadBalancer kube-controller's assignIPs mode. Only services you annotate take a VIP from this host-subnet pool.

Your cluster's default workload pool is unchanged, and workloads allocated from it behave exactly as before. For more on LoadBalancer IP pools, see Configure LoadBalancer IP address management.

Steer pods or services into the pool​

Pods and services do not use the new pool automatically — direct the specific workloads you want reachable on the host L2 segment into it.

Pods: annotate the pod or its namespace with the pool name:

cni.projectcalico.org/ipv4pools: '["host-subnet-pod-pool"]'

LoadBalancer services: annotate the Service with the pool name:

projectcalico.org/ipv4pools: '["host-subnet-lb-pool"]'

For IPv6, use projectcalico.org/ipv6pools. Services without this annotation continue to get VIPs from your automatic LoadBalancer pools as before. See Configure LoadBalancer IP address management for more options.

Verify reachability​

On the node, confirm ARP requests arrive and replies leave with the node's MAC:

tcpdump -i <iface> arp

For IPv6 NDP solicitations and advertisements:

tcpdump -i <iface> 'icmp6 and (icmp6.type == 135 or icmp6.type == 136)'

From an external host on the same L2 segment, confirm end-to-end resolution:

arping -I <client-iface> <pod-or-vip>

If there is no reply, check that the IP falls within both a host interface subnet and a no-encapsulation IP pool, that the feature is enabled, and that your network does not filter ARP (see Limitations).

Additional resources​