In Defense of OSPF In The Underlay (In Some Situations)

When choosing an underlay for an EVPN/VXAN network, the prevailing wisdom is that BGP is the best choice for the underlay routing protocol. And overall, I think that’s true. But OSPF can make a compelling underlay too, as it has a few logistical advantages over BGP in certain cases.

When building out EVPN/VXLAN networks, I like to break the build into four components. They are layers that are built one-by-one on top of each other.

• Topology (typically leaf/spine)
• Underlay (provides IP connecitivity for loopbacks)
• Overlay (exchanges EVPN routes)
• EVPN services (these are the Layer 2 and Layer 3 networks internal hosts and external networks connect to)

This article is exclusively about the underlay portion. It’s a very simple routed network that has one job, and job only:

Provide routes to enable IP connectivity from any loopbacks on a device to any loopback on any other device.

That’s it.

In normal operation the routing table will be incredibly static. The only time the routing table would change is when a switch is added or removed, or a link goes down, or a switch is upgraded, etc. In regular operation it won’t change.

The underlay is important, but the underlay isn’t complicated.

You have a number of routing protocols to choose from to enable this. Technically any routing protocol (or even static routes!) would work, but the most common and sensible routing protocols are as follows:

• BGP (eBGP or iBGP, depends on your overlay)
• OSPF
• ISIS (to a lesser extent)

I even built a working underlay with RIP, though its lack of ECMP would preclude it from being used. I just did it to see if it could be done. Don’t use RIP, kids.

BGP and OSPF are the most common underlays. ISIS can be used, but ISIS is not a skillset that’s terribly common in the DC. So let’s talk about the two remaining choices: OSPF and BGP.

For any kind of EVPN/VXLAN fabric, BGP has no scaling issues, so you can scale up to as many nodes as you dare have in a single failure domain.

For smaller scales and labbing, I generally don’t like BGP as the underlay. Here’s a few reasons I prefer OSPF for some smaller-scale environments and labs:

• IP unnumbered (not needing IP addresses on the point-to-point links)
• Stupidly simple routing protocol configuration (just set everything to area 0 and set a router-id)
• Separation of function: BGP handles the overlay (EVPN routes) and OSPF handles the underlay

Separation of Function

When I was learning EVPN/VXLAN I made sure to put OSPF as an underlay. I knew it well enough to do the job it was intended for, and I was less confident with BGP. It really helped to have BGP only be involved with the overlay. As I was playing with settings, figuring out what was required and not required, how everything behaved… it really helped to know that anything under “router bgp” was EVPN.

In certain production environments this may also be advantageous, especially for smaller organizations that are just getting up to speed with EVPN/VXLAN. It can make troubleshooting a bit easier.

IP Unnumbered

I love IP unnumbered. As an instructor, I do a lot of EVPN configuration demonstrations. In front of the students I will build, by hand, a soup-to-nuts EVPN/VXLAN network. Of course, EVPN/VXLAN in production should almost always be configured through automation (specifically configuration generation, assuming the switches use config file), but for labbing you’re going to do it manually. You have to understand how to do something manually before automation/troubleshooting it.

BGP, being TCP-based, requires an IP address in order to peer with another device. This typically means you’ll have to come up with an IP scheme of point-to-point IP addresses between every leaf and spine. With three spines and four leafs, that’s 24 unique IP addresses that need to be assigned total, two per each each leaf-to-spine link. With three spines and 20 leafs, that’s 60 unique IPs. This isn’t horrible by any means, but it’s a pain.

Here’s an example of the configuration for point-to-point interface configurations between an Arista EOS leaf and spine:

LEAF1
interface Ethernet3
  ip address 192.168.103.0/31
  no switchport
  mtu 9214

SPINE1
interface Ethernet1
  ip address 192.168.103.1/31
  no switchport
  mtu 9214

And those IPs need to be plugged into the BGP configuration:

router bgp 65101
...
   neighbor 192.168.103.1 peer group Underlay
   neighbor 192.168.103.3 peer group Underlay
   neighbor 192.168.103.5 peer group Underlay

So with BGP as an underlay, a significant portion of the underlay configuration will be unique to each leaf/spine. If you’re doing this by hand, it’s easy to make a mistake with all of those /31s. This alone is one of the reasons one should only configure EVPN/VXLAN in production using automation.

You’ll also want to do something to keep the /31s out of the routing tables, such as route-maps, otherwise the /31s get everywhere and they’re not needed. They’ll eat up forwarding table space (not a super big deal, but still) and crowd a “show ip route”.

With OSPF, the situation is much more simplified. The interface configurations are the same for all p2p links by using ip unnumbered:

interface Ethernet2
  no switchport
  ip address unnumbered loopback0
  ip ospf area 0
  ip ospf network point-to-point
  mtu 9214
  

With OSPF, a neighbor adjacency can be brought up without an individual IP address. As for the OSPF protocol configuration, usually all you need to do is setup the router-id (always set up the router-id). The following is all that’s required:

router ospf 10
  router-id 1.1.1.1 <--- this is usually the loopback0 address

Very simple, and very repeatable (when compared to BGP as the underlay).

Of course most of the automation platforms will do this for you. For example, Arista AVD and Arista CloudVision Studios will generate the p2p IPs from a block of addresses you specify. But if you’re labbing with manual configuration or rolling your own automation (such as Ansible/Jinja templates) this can complicate matters.

But “IPv6 Unnumbered”?

The ability to use “IPv6 unnumbered” does simplify things a little bit, as using rfc 5549 doesn’t require assigning IP addresses to the p2p interfaces. Technically it’s not unnumbered, but instead makes use of the link local address and neighbor discovery aspects of IPv6.

#show ip bgp summary <--- showing the underlay routing protocol
BGP summary information for VRF default
Router identifier 192.0.200.3, local AS number 65101
Neighbor Status Codes: m - Under maintenance
Description   Neighbor                        V AS      MsgRcvd   MsgSent  InQ OutQ  Up/Down State   PfxRcd PfxAcc
  spine1      fe80::20c:73ff:fe10:335d%Et3    4 65001         8        11    0    0 00:01:55 Estab   4      4
  spine2      fe80::20c:73ff:fe76:1b6b%Et4    4 65001         8        11    0    0 00:01:56 Estab   4      4
  spine3      fe80::20c:73ff:fefd:a9d8%Et5    4 65001         9         9    0    0 00:01:56 Estab   4      4

On the p2p interace you must enable ipv6, and the switch will auto-assign a link-local IPv6 address.

interface Ethernet3
   description P2P_LINK_TO_SPINE1_Ethernet2
   no switchport
   ipv6 enable

And in the BGP configuration rather than specifying the neighbor IP address, you just specify the interface and the NOS will auto-find the peer IP.

router bgp 65100
...   
   neighbor interface Et3 peer-group IPv4-UNDERLAY-PEERS remote-as 65001

You can see the link-local IPv6 address that the interface chose:

1#show ipv6 interface brief 
   Interface       Status        MTU       IPv6 Address                     Addr State    Addr Source
--------------- ------------ ---------- -------------------------------- ---------------- -----------
   Et3             up           1500       fe80::21c:73ff:fe32:7cb/64       up            link local 

So that does get rid of the complexity of assigning IPs to p2p links, though there’s still more simplicity in OSPF configurations.

Stupid Simple

OSPF has another advantage as an underlay: It’s incredibly simple. It has everything you want in an underlay with the only knob you need to touch is setting the router id. ECMP, clutter-free routing tables, loopback-to-loopback connectivity is there with the defaults. Just throw everything in area 0 and set the router-id, set the interfaces to unnumbered and point-to-point, and that’s it.

When you’re starting out on your learning EVPN journey, or for smaller organizations, this can be huge benefit.

BPG is more customizable of course, but for an underlay there’s usually no need for customization. We want the simplest, most symmetrical network possible.

The Cases Against OSPF

There are a few cases against OSPF as an underlay. In all cases with EVPN, the overlay (exchanging endpoint information, external networks, etc.) is done with BPG. Specifically, MP-BGP (either iBGP or eBGP can be used). But as we’ve shown, the underlay is up for grabs. So why not use OSPF?

The biggest issue I think is scale. Or rather, concern about scale. There’s been enough large implementations of BGP that we don’t really worry about its scale. But there has historically been concern and assumptions about OSPF scaling. Here’s an article talking about large scale Clos networks with OSPF as the routing protocol.

Given the stability, simplicity, and relatively small size of the routing table of most underlay networks, I think that OSPF is fine for a small to medium sized environment. For larger environments, you could probabably do multi-areas for various leaf/spine groups connected to super spines on area 0, etc. But at that point you might as well do BPG on the underlay, as at that size you’ll have a pretty sophisticated automation system.

Automation systems level the playing field with underlay routing protocols for the most part. If you’re making your own automation system using Jinja and Ansible, for instance, OSPF makes some of that easier. But if you’re using a pre-made system like I mentioned before, such as Arista AVD, the difference between OSPF and BPG in the underlay is as simple as a key value pair in a data model.

TL;DR

So here’s the too-long, didn’t read: For labbing, use OSPF. If you’re making your own templates in Jinja and Ansible, OSPF probably is easier until you hit a certain scale. If you’re using a pre-made automation system, just use what’s easiest (probably BGP). Automation can erase and difference between complexity of one versus the other.