Troubleshooting EVPN with Arista EOS (Control Plane Edition)
November 18, 2022 Leave a comment
“It’s all fun and games until you can’t ping your default gateway.”
While EVPN/VXLAN brings a number of benefits when compared to a more traditional Core/Aggregation/Access layer style network with only VLANs and SVIs, it is different enough that you’ll need to learn some new troubleshooting techniques. It’s not all that different than what you’ve probably done before, but it is different enough to warrant a blog post.
This article is on how to troubleshoot EVPN/VXLAN on Arista EOS switches, and the command line commands will reflect that. However, as EVPN/VXLAN are a collection of IETF standards, the overall technique will translate to any EVPN/VXLAN platform.
The scenario this article is going to explore is endpoint to endpoint connectivity, though it can also be easily modified for endpoint to network connectivity. It doesn’t matter if the host is on the same VXLAN segment or a different one.
The primary strategy will be to verify the control plane. EVPN/VXLAN has a control plane, a data plane, an overlay and an underlay. Generally, I’ve found that most issues occur on the control plane. The control plane process looks like this:
- MAC address is learned on the ingress leaf VLAN
- MAC learning triggers creating of Type 2 routes (a MAC route and a MAC-IP route)
- Type 2 routes are propagated to all other switches in the fabric (including the egress leaf)
- Type 2 routes are installed into the appropriate forwarding tables (Layer 2, Layer 3) on the egress leaf
While this method only verifies the control plane, going through this list one by one should expose any issues with the underlay or overlay. If the issue still isn’t identified, verification can move to the data plane which will be covered in a different article. We’re staring with the control planes as if the control plane hasn’t told the data plane how to forward, the data plane can’t forward.
Here’s a couple of things to keep in mind with EVPN/VXLAN
- You need the MAC address of the endpoint in question to start. This all flows from the MAC address. Other information from the endpoint, such as ARP table (do we have the MAC of the default gateway?), default gateway and any static routes, and even link status (is it even plugged in/connected to a virtual port on a virtual switch?)
- All leafs and spines (and superspines) will have every EVPN route. They don’t all need to install the routes into a forwarding table, but they will have every EVPN (type 2, type 3, etc.) route. If they don’t, that means something is likely wrong.
- Every VXLAN segment requires a local VLAN to be connected do it.
The Hypothetical Environment
The physical connectivity is listed here below. There’s two hosts, on two different segments.
In this hypothetical environment, there are two hosts that can’t communicate. Host1 is on L2VNI 10010, and Host2 is on L2VNI 10020. There is symmetric IRB, with a L3VNI for the VRF of 10000.
It’s important to know the logical layout (L2VNIs, IP-VRF, etc.) before starting.
- Endpoint MAC/IP (source)
- Local VLAN ID
- VXLAN ID (L2VNI)
- IP-VRF
- L3VNI for IP-VRF
- Egress VXLAN ID (may or may not be different)
- Egress VLAN ID (may or may not be different)
- Endpoint MAC/IP (destination)
For this exercise, here is that information:
- 0000.0000.0001 and 10.1.10.11/24
- VLAN 10
- VNI 10010
- VRF: Red, L3VNI 10000
- VNI 10020
- VLAN 20
- 0000.0000.0002 and 10.1.20.12/24
A Note On The Non-Network Team Environment
As is often the case, the packets from the host may traverse configuration domains outside of the network team’s purview. Most commonly these are virtual switches and blade switches. Ideally the network team would at least have read-only access to these environments, but that’s not always possible.
Hopefully the teams responsible for these environments can check to see the MAC address of the host ended up in the appropriate forwarding tables. For example, the port group on the hypervisor in VMware has a way to check MAC addresses. Cisco UCS, a popular blade system, also has a way to see the MAC addresses learned on the switch (the command is the same as on EOS, in fact).
The Strategy
The diagram below shows the strategy, starting from the ingress leaf to the egress leaf. Normally the hosts would be connected to multiple leafs in either MLAG or EVPN multihoming, but to simplify things this scenario involves two single-homed hosts. Everything will be working correctly, so no issues will be found in the verification as to show you what it should look like.
In addition, the EOS configuration relevant to the verification will be shown.
MAC Address Learned on the Ingress Leaf
This is fairly straightforward, and the verification is the same as you’d be used to in a more traditional network. You’ll want to know the VLAN and VXLAN segment it’s assigned to. You should see the host’s MAC in the local leaf’s MAC table. If you don’t, that needs to be resolved before you can continue. Nothing will work unless the ingress leaf sees the MAC address.
You’ll have to generate some type of traffic of course for the MAC address to be learned. Running a constant ping from the host will do it, or you can also setup (or pre-stage in a tightly change controlled environment) a diagnostic loopback in the Tenant VRF to ping from the leaf to the host.
The command is show mac address-table vlan 10 in this case.
leaf1#show mac address-table vlan 10
Mac Address Table
------------------------------------------------------------------
Vlan Mac Address Type Ports Moves Last Move
---- ----------- ---- ----- ----- ---------
10 0000.0000.0001 DYNAMIC Po67 1 0:00:56 ago
Total Mac Addresses for this criterion: 1
The configuration that enables this is below:
interface Ethernet6
channel-group 67 mode active
interface Ethernet7
channel-group 67 mode active
interface Port-Channel67
switchport access vlan 10
Type 2 Route Generation
So the MAC address was learned on the correct VLAN. We can move onto the next step, which is to see if learning the MAC address generated Type 2 routes. Because we’re routing from one VXLAN segment to another VXLAN segment, there will two EVPN Type 2 routes created: One for the MAC address and one for the MAC and IP combination.
The command to see these routes is show bgp evpn route-type mac-ip. You can optionally specify the MAC address, which can be useful is there’s a lot of endpoints in the routing table.
#show bgp evpn route-type mac-ip 0000.0000.0001
BGP routing table information for VRF default
Router identifier 192.168.101.11, local AS number 65101
Route status codes: s - suppressed, * - valid, > - active, E - ECMP head, e - ECMP
S - Stale, c - Contributing to ECMP, b - backup
% - Pending BGP convergence
Origin codes: i - IGP, e - EGP, ? - incomplete
AS Path Attributes: Or-ID - Originator ID, C-LST - Cluster List, LL Nexthop - Link Local Nexthop
Network Next Hop Metric LocPref Weight Path
* > RD: 192.168.101.11:10010 mac-ip 0000.0000.0001
- - - 0 i
* > RD: 192.168.101.11:10010 mac-ip 0000.0000.0001 10.10.10.11
- - - 0 iThis switch has generated two type 2 routes: One for the MAC address by itself, and one for the combination of MAC and IP. For symmetric IRB (the most common EVPN configuration) this will be the case. Normally a VLAN just learns the MAC address, but with EVPN the VLAN also learns the IP address if the initial frame contains that information (it would in an ARP, for instance).
MP-BGP with the EVPN address family is responsible for generating these routes based on MAC and IPs learned in a VLAN. There are two common ways to configure a leaf to generate Type 2 routes for a VLAN/VNI: Individually or in bundles. They’re called VLAN services or VLAN-aware bundle services.
Below is what the configuration would look like for an individual VLAN service for two VLANs:
router bgp 65101
...
vlan 10
rd 192.168.101.1:10010
route-target both 10010:1000
redistributed learned
vlan 20
rd 192.168.101.1:10020
route-target both 10020:10020
redistribute learned
vrf Red
rd 192.168.101.1:10000
route-target import evpn 10000:10000
route-target export evpn 10000:10000
redistribute connectedNote that each VLAN has its own route distinguisher and its own route target.
For VLAN aware bundles, all the VLANs in a Tenant can be placed into a single instance:
router bgp 65101
...
vlan-aware-bundle Red
rd 192.168.101.1:1000
route-target both 1000:1000
redistributed learned
vlan 10,20Route Propagation to Spines
The route generated on the ingress leaf should propagate to the connected spines, and from there all other leafs and perhaps superspines. They won’t necessarily be installed into any forwarding tables (for example, they won’t be installed on the spines), but they should be on all fabric switches, regardless of role.
The next step is to verify that the route made it to the spine. Log into one of the spines and run the same show command, show bgp evpn summary.
* > RD: 192.168.101.11:10010 mac-ip 10010 0000.0000.0001
192.168.102.11 - 100 0 65101 i
* > RD: 192.168.101.11:10010 mac-ip 10010 0000.0000.0001 10.1.10.11
192.168.102.11 - 100 0 65101 iIf MLAG is being used, you’ll see the same two routes twice: One from each leaf in the MLAG group. They will be differentiated by their route distinguishers.
* > RD: 192.168.101.11:10010 mac-ip 10010 001c.73cd.693d
192.168.102.11 - 100 0 65101 i
* > RD: 192.168.101.12:10010 mac-ip 10010 001c.73cd.693d
192.168.102.11 - 100 0 65101 i
* > RD: 192.168.101.11:10010 mac-ip 10010 001c.73cd.693d 10.1.10.11
192.168.102.11 - 100 0 65101 i
* > RD: 192.168.101.12:10010 mac-ip 10010 001c.73cd.693d 10.1.10.11
192.168.102.11 - 100 0 65101 iIf they’re not on the spines, resolve that before continuing. This is entirely configured in the overlay configuration and underlay. I would check the overlay first, then the underlay to resolve any reasons that the routes aren’t being propagated. If the routes aren’t making it to the spines connected to the ingress leaf, then none of the other leafs will get those routes and won’t know what to do with packets destined for that MAC or IP.
Egress Leaf
The routes should make it to the egress leaf. To verify, it’s the same show bgp evpn route-type mac-ip command. You will see the same routes multiple times. For instance, if you have one ingress leaf and three spines, you’ll see the MAC and MAC-IP routes both three times each. One MAC route each from three spines, one MAC-IP route each from three spines.
* >Ec RD: 192.168.101.11:10010 mac-ip 10010 0000.0000.0001
192.168.102.11 - 100 0 65001 65101 i
* ec RD: 192.168.101.11:10010 mac-ip 10010 0000.0000.0001
192.168.102.11 - 100 0 65001 65101 i
* ec RD: 192.168.101.11:10010 mac-ip 10010 0000.0000.0001
192.168.102.11 - 100 0 65001 65101 i
* >Ec RD: 192.168.101.11:10010 mac-ip 10010 0000.0000.0001 10.1.10.11
192.168.102.11 - 100 0 65001 65101 i
* ec RD: 192.168.101.11:10010 mac-ip 10010 0000.0000.0001 10.1.10.11
192.168.102.11 - 100 0 65001 65101 i
* ec RD: 192.168.101.11:10010 mac-ip 10010 0000.0000.0001 10.1.10.11
192.168.102.11 - 100 0 65001 65101 i
If the egress leafs are in an MLAG pair, you’ll see even more. Two leafs, two route types, three spines: You’ll see 12 routes for the endpoint. That’s normal. Note the routes all point the same VTEP in this example (192.168.102.11).
* >Ec RD: 192.168.101.11:10010 mac-ip 10010 0000.0000.0001
192.168.102.11 - 100 0 65001 65101 i
* ec RD: 192.168.101.11:10010 mac-ip 10010 0000.0000.0001
192.168.102.11 - 100 0 65001 65101 i
* ec RD: 192.168.101.11:10010 mac-ip 10010 0000.0000.0001
192.168.102.11 - 100 0 65001 65101 i
* >Ec RD: 192.168.101.12:10010 mac-ip 10010 0000.0000.0001
192.168.102.11 - 100 0 65001 65101 i
* ec RD: 192.168.101.12:10010 mac-ip 10010 0000.0000.0001
192.168.102.11 - 100 0 65001 65101 i
* ec RD: 192.168.101.12:10010 mac-ip 10010 0000.0000.0001
192.168.102.11 - 100 0 65001 65101 i
* >Ec RD: 192.168.101.11:10010 mac-ip 10010 0000.0000.0001 10.1.10.11
192.168.102.11 - 100 0 65001 65101 i
* ec RD: 192.168.101.11:10010 mac-ip 10010 0000.0000.0001 10.1.10.11
192.168.102.11 - 100 0 65001 65101 i
* ec RD: 192.168.101.11:10010 mac-ip 10010 0000.0000.0001 10.1.10.11
192.168.102.11 - 100 0 65001 65101 i
* >Ec RD: 192.168.101.12:10010 mac-ip 10010 0000.0000.0001 10.1.10.11
192.168.102.11 - 100 0 65001 65101 i
* ec RD: 192.168.101.12:10010 mac-ip 10010 0000.0000.0001 10.1.10.11
192.168.102.11 - 100 0 65001 65101 i
* ec RD: 192.168.101.12:10010 mac-ip 10010 0000.0000.0001 10.1.10.11
192.168.102.11 - 100 0 65001 65101 i
Verify Forwarding Tables
Now that the egress leaf has the routes, it can program them into the forwarding tables. For EVPN, that can mean either the IP VRF or the VLAN forwarding table (or both).
If the ingress VXLAN segment isn’t configured to a VLAN on the egress leaf, then the MAC address won’t go into a VLAN. That’s normal. However, if the VXLAN segment is configured for a local VLAN, you can verify the route was programmed with a simple show mac address-table vlan. Also you can run show vxlan address-table.
#show mac address-table vlan 10
Mac Address Table
------------------------------------------------------------------
Vlan Mac Address Type Ports Moves Last Move
---- ----------- ---- ----- ----- ---------
10 0000.0000.0001 DYNAMIC Vx1 1 0:01:13 ago
Total Mac Addresses for this criterion: 1
#show vxlan address-table vlan 10
Vxlan Mac Address Table
----------------------------------------------------------------------
VLAN Mac Address Type Prt VTEP Moves Last Move
---- ----------- ---- --- ---- ----- ---------
10 001c.73cd.693d EVPN Vx1 192.0.254.3 1 0:05:18 ago
Total Remote Mac Addresses for this criterion: 1
The egress host in the example is on another VXLAN segment/VLAN, however. For 10.1.20.12 to get to 10.1.10.11, there would need to be a /32 host route in the IP VRF. You can see this with a show ip route vrf [IP-VRF]. In this case, the IP VRF is “Red”.
#show ip route vrf Red
VRF: Red
...
Gateway of last resort is not set
B E 10.1.10.11/32 [20/0] via VTEP 192.168.102.11 VNI 10 router-mac 02:1c:73:32:07:cb local-interface Vxlan1
C 10.1.10.0/24 is directly connected, Vlan10
C 10.1.20.0/24 is directly connected, Vlan20
...
If the hosts in question are on the same VXLAN segment, then the address needs to be in the VLAN forwarding table associated with that VXLAN segment (on both sides). If they’re on different segments, the /32 host route must be in the IP VRF forwarding table.
The same configuration on the ingress leaf in the BGP section will take the routes that have been propagated and install them into the appropriate forwarding tables. For non-VLAN aware bundles it would look like this (leaf3):
router bgp 65103
...
vlan 10
rd 192.168.101.13:10010
route-target both 10010:1000
redistributed learned
vlan 20
rd 192.168.101.13:10020
route-target both 10020:10020
redistribute learned
vrf Red
rd 192.168.101.13:10000
route-target import evpn 10000:10000
route-target export evpn 10000:10000
redistribute connected
For VLAN aware bundles, it would look something like the following:
router bgp 65103
...
vlan-aware-bundle Red
rd 192.168.101.13:1000
route-target both 1000:1000
redistributed learned
vlan 10,20Note that any route distinguishers must be unique in every leaf, segment, and bundle. The Route targets need to match across segments.
The Data Center Overlords 