OpenFlow’s Awkward Teen Years

April 12, 2012 2 Comments

During the Networking Field Day 3 (The Nerdening) event, I attended a presentation by NEC (yeah, I know, turns out they make switches and have a shipping OpenFlow product, who knew?).

This was actually the second time I’ve seen this presentation. The first was at Networking Field Day 2 (The Electric Boogaloo), and my initial impressions can be found on my post here.

So what is OpenFlow? Well, it could be a lot of things. More generically it could be considered Software Defined Networking (SDN). (All OpenFlow is SDN, but not all SDN is OpenFlow?) It’s adding a layer of intelligence to a network that we have previously lacked. On a more technical level, OpenFlow is a way to program the forwarding tables of L2/L3 switches using a standard API.

Rather than each switch building their own forwarding tables through MAC learning, routing protocols, and policy-based routing, the switches (which could be from multiple vendors) are brainless zombies, accepting forwarding table updates from a maniacal controller that can perceive the entire network.

This could be used for large data centers, large global networks (WAN), but the NEC demonstration was a data center fabric representation of OpenFlow. In some respects, it’s similar to Juniper’s QFabric and Brocade’s VCS. They all have a boundary, and the connections to networks outside of that boundary is done through traditional routing and switching mechanisms, such as OSPF or MLAG (Multi-chassis Link Aggregation). Juniper’s QFabric is based on Juniper’s own proprietary machinations, Brocade’s VCS is based on TRILL (although not [yet?] interopable with other TRILL implementations), and NEC’s OpenFlow is based on, well, OpenFlow.

While it was the same presentation, the delegates (a few of us had seen the presentation before, most had not) were much tougher on NEC than we were last time. Or at least I’m sure it seemed that way. The fault wasn’t NEC, it was the fact that understanding of Openflow and its capabilities have increased, and that’s the awkward teen years of any technology. We were poking and prodding, thinking of new uses and new implications, limitations and caveats.

As we start to understand a technology better, we start to see it’s potential, as well as probe it’s potential drawbacks. Everything works great in PowerPoint, after all.  So while our questions seemed tougher, don’t take that as a sign that we’re souring on Openflow. We’re only tough because we care.

One thing that OpenFlow will likely not be is a way to manipulate individual flows, or at least all individual flows. The “flow” in Openflow does not necessary (and in all likelihood would rarely, if ever) represent an individual 6 tuple TCP connection or UDP flow (a source and destination MAC, IP, and TCP/UDP port). One of the common issues that Openflow haters/doubters/skeptics have brought up that an Openflow controller can only program about 700 flows per second into a switch. That’s certainly not enough to handle a site that may see hundreds of thousands of new TCP connections/UDP flows per second.

But that’s not what Openflow is meant to do, nor is that a huge issue. Think about routing protocols and Ethernet forwarding mechanisms. Neither handle deal with specific flows, only general flows (all traffic to a particular MAC address goes to this port, this /8 network goes out this interface, etc.). OpenFlow isn’t any different. So a limit of 700 flows per second per switch? Not a big deal.

OpenFlow is another way the build an Ethernet fabric, which means offering services and configurations beyond just a Layer 2/Layer 3 network.

Think of firewalls as how we deal with them now. Scaling is an issue, so they’re often choke points. You have to direct traffic in and out of them (NAT, transparent mode, routed mode), and they’re often deployed in pairs. N+1 firewalls is not that common (and often a huge pain in the ass to configure, although it’s been a while).  With Openflow (or SDN in general) it’s possible to define an endpoint (VM or physical) and say that endpoint requires flows pass through a firewall. Since we can steer flows (again, not on a per-individual flow basis, but general catch-most rules) scaling a firewall isn’t an issue. Need more capacity? Throw another firewall/IPS/IDS on the barbie. Openflow could put in forwarding rules on the switches and steer flows between active/active/active firewalls. These services could also be tied to the VM, and not the individual switch port (which is a characteristic of a fabric).

Myself, I’m all for any technology that abolishes spanning-tree/traditional Layer 2 forwarding. It’s an assbackwards way of doing things, and if that’s how we ran Layer 3 networks, networking administrators would have revolted by now. (I’m surprised more network administrators aren’t outraged by spanning-tree, but that’s another blog post).

NEC’s OpenFlow (and OpenFlow in general) is an interesting direction in the effort to make network more intelligent (aka “fabrics”). Some vendors are a bit dismissive of OpenFlow, and of fabrics in general. My take is that fabrics are a good thing, especially with virtualization. But more on that later. NEC’s OpenFlow product is interesting, but like any new technology it’s got a lot to prove (and the fact that it’s NEC means they have even more to prove).

Disclaimer: I was invited graciously by Stephen Foskett and Greg Ferro to be a delegate at Networking Field Day 3. My time was unpaid. The vendors who presented indirectly paid for my meals and lodging, but otherwise the vendors did not give me any compensation for my time, and there was no expectation (specific or implied) that I write about them, or write positively about them. I’ve even called a presenter’s product shit, because it was. Also, I wrote this blog post mostly in Aruba while drunk.

Filed under Ethernet, NetworkingFieldDay

I, For One, Welcome Our New OpenFlow Overlords

November 7, 2011 4 Comments

When I first signed up for Networking Field Day 2 (The Electric Boogaloo), I really had no idea what OpenFlow was. I’d read a few articles, listened to a few podcasts, but still only had a vague idea of what it was. People I respect highly like Greg Ferro of Packet Pushers were into it, so it had my attention. But still, not much of a clue what it was. I attended the OpenFlow Symposium, which preceeded the activites of Networking Field Day 2, and had even less of an idea of what it was.

Then I saw NEC (really? NEC?) do a demonstration. And my mind was blown.

Side note: Let this be a lesson to all vendors. Everything works great in a PowerPoint presentation. It also conveys very little about what a product actually does. Live demonstrations are what get grumpy network admins (and we’re all grumpy) giddy like schoolgirls at  Justin Bieber concert. You should have seen Ivan Pepelnjak. 

I’m not sure if I got all my assumptions right about OpenFlow, so feel free to point out if I got something completely bone-headedly wrong. But from what I could gather, OpenFlow could potentially do a lot of things:

  • Replace traditional Layer 2 MAC learning and propagation mechanisms
  • Replace traditional Layer 3 protocols
  • Make policy-based routing (routing based on TCP/UDP port) something useful instead of a one-off, pain in the ass, ok just-this-one time creature it is now
  • Create “traceroute on steroids”

Switching (Layer 2)

Switching is, well, rather stupid. At least learning MAC addresses and their locations are. To forward frames, switches need to learn which ports to find the various MAC addresses. Right now the only way they learn about it is listening to the cacophony of hosts broadcasting and spewing frames. And when one switch learns a MAC address, it’s not like it tells the others. No, in switching, every switch is on its own for learning. In a single Layer 2 domain, every switch needs to learn where to find every MAC address on its own.

Probably the three biggest consequences of this method are as follows

  • No loop avoidance. The only way to prevent loops is to prevent redundant paths (i.e. spanning-tree protocol)
  • Every switch in a Layer 2 domain needs to know every frickin’ MAC address. The larger the Layer 2 domain, the more MAC addresses need to be learned. Suddenly, a CAM table size of 8,000 MAC addresses doesn’t seem quite enough.
  • Broadcasts like woah. What happens when a switch gets a frame that it doesn’t have a CAM entry for? BROADCAST IT OUT ALL PORTS BUT THE RECEIVING PORT. It’s the all-caps typing of the network world.
For a while in the early 2000’s we could get away with all this. Multi-layer switches (switches that did Layer 3 routing as well) got fast enough to route as fast as they could switch, so we could easily keep our Layer 2 domains small and just route everything.

That is, until VMware came and screwed it all up. Now we had to have Layer 2 domains much larger than we’d planned for. 4,000 entry CAM tables quickly became cramped.

MAC learning would be more centralized with OpenFlow. ARP would still be there at the edge, so a server would still think it was communicating with a regular switch network. But OpenFlow could determine which switches need to know what MAC addresses are where, so every switch doesn’t need to learn everything.

And no spanning-tree. Loop avoidance is prevented by the OpenFlow controller. No spanning-tree (although you can certainly do spanning-tree at the edge to communicate with legacy segments).

Routing (Layer 3)

Routing isn’t quite as stupid as switching. There are a number of good protocols out there that will scale pretty well, but it does require configuration on each device. It’s dynamic in that it can do multi-pathing (where traditional Layer 2 can’t), as well as recover from dead links without taking down the network for several (dozens of) seconds.  but it doesn’t quite allow for centralized control, and it has limited dynamic ability. For instance, there’s not mechanism to do “oh, hey, for right now why don’t we just move all these packets from this source to that source” in an efficient way. Sure, you can inject some host routes to do that, but it’s got to come from some sort of centralized controller.

Flow Routing (Layer 4)

So why stop at Layer 3? Why not route based on TCP/UDP header information? It can be done with policy-based routing (PBR) today, but it’s not something that can be communicated from router to router (OSPF cares not how you want to direct a TCP port 80 flow versus a TCP port 443 flow).  There is also WCCP, the Web Cache Communication Protocol, which today is not used for web caches, but WAN Optimization Controllers, like Cisco’s WAAS, or Cisco’s sworn enemy, Riverbed (seriously, just say the word ‘Riverbed’ at a Cisco office).

Sure it’s watery and tastes like piss, but at least it’s not policy-based routing

A switch with modern silicon can look at Layer 3 and Layer 4 headers as easily as they can look at Layer 2 headers. It’s all just bits in the flow, man. OpenFlow takes advantage of this, and creates, for lack of a cooler term, a Layer 2/3/4 overlord.

I, for one, welcome our new OpenFlow overlords

TCAMs or shared memory, or whatever you want to call the forwarding tables in your multi-layer switches can be programmed at will by an OpenFlow overlord, instead of being populated by the lame-ass Layer 2, Layer 3, and sometimes Layer 4 mechanisms on a switch-by-switch basis.

Since we can direct traffic based on flows throughout a multi-switch network, there’s lots of interesting things we can do with respect to load balancers, firewalls, IPS, caches, etc. Pretty interesting stuff.

Flow View (or Traceroute on Steroids)

I think one of the coolest demonstrations from NEC was when they showed the flow maps. They could punch up any source and destination address (IP or MAC) and there would be a graphical representation of the flow (and which devices they went through) on the screen. The benefits for that would be obvious. Server admin complain about slowness? Trace the flow, and check the interfaces on all the transit devices. That’s something that might take quite a while in a regular route/switch network, but can be done in a few seconds with an OpenFlow controller.

An OpenFlow Controller Tracks a Flow

To some extent, there are other technologies that can take care of some of these issues. For instance, TRILL and SPB take a good wack at the Layer 2 bullshit. Juniper’s QFabric does a lot of the ain’t-nothin-but-a-tuple thang and switches based on Layer2/3 information. But in terms of potential, I think OpenFlow has them all beat.

Don’t get too excited right now though, as NEC is the only vendor that has working implementation of OpenFlow controller, and other vendors are working on theirs. Standford apparently has OpenFlow up and running in their environment, but its all still in the early stages.

Will OpenFlow become the future? Possibly, quite possibly. But even if what we now call OpenFlow isn’t victorious, something like it will be. There’s no denying that this approach, or something similar, is a much better way to handle traffic engineering in the future than our current approach. I’ve only scratched the surface of what can be done with this type of network design. There’s also a lot that can be gained in terms of virtualization (an OpenFlow vSwitch?) as well as applications telling the network what to do. Cool stuff.

Note: As a delegate/blogger, my travel and accommodations were covered by Gestalt IT, who vendors paid to have spots during the Networking Field Day. Vendors pay Gestalt IT to present, so while my travel (hotel, airfare, meals) were covered indirectly by the vendors, no other remuneration (save for the occasional tchotchke) from any of the vendors, directly or indirectly, or by Gestalt IT was recieved. Vendors were not promised, nor did they ask for any of us to write about them, or write about them positively. In fact, we sometimes say their products are shit (when, to be honest, sometimes they are, although this one wasn’t). My time was unpaid. 

Filed under Ethernet, NetworkingFieldDay, OpenFlow, Virtualization

The Problem

November 3, 2011 13 Comments

One recurring theme from virtually every one of the Network Field Day 2 vendor presentations last week (as well as the OpenFlow symposium) was affectionately referred to as “The Problem”.

It was a theme because, as vendor after vendor gave a presentation, they essentially said the same thing when describing the problem they were going to solve. For us the delegates/bloggers, it quickly went from the problem to “The Problem”. We’d heard it over and over again so often that during the (5th?) iteration of the same problem we all started laughing like a group of Beavis and Butt-Heads during a vendor’s presentation, and we had to apologize profusely (it wasn’t their fault, after all).

Huh huhuhuhuhuh… he said “scalability issues”

In fact, I created a simple diagram with some crayons brought by another delegate to save everyone some time.

Hello my name is Simon, and I like to do draw-wrings

But with The Problem on repeat it became very clear that the majority of networking companies are all tackling the very same Problem. And imagine the VC funding that’s chasing the solution as well.

So what is “The Problem”? It’s multi-faceted and interrelated set of issues:

Virtualization Has Messed Things Up, Big Time

The biggest problem of them all was caused by the rise of virtualization. Virtualization has disrupted much of the server world, but the impact that it’s had on the network is arguably orders of magnitude greater. Virtualization wants big, flat networks, just when we got to the point where we could route Layer 3 as fast as we could switch Layer 2. We’d just gotten to the point where we could get our networks small.

And it’s not just virtualization in general, much of its impact is the very simple act of vMotion. VMs want to keep their IPs the same when they move, so now we have to bend over backwards to get it done. Add to the the vSwitch sitting inside the hypervisor, and the limited functionality of that switch (and who the hell manages it anyway? Server team? Network team?)

4000 VLANs Ain’t Enough

If you’re a single enterprise running your own network, chances are 4000+ VLANs are sufficient (or perhaps not). In multi-tenant environments with thousands of customers, 4000+ VLANs quickly becomes a problem. There is a need for some type of VLAN multiplier, something like QinQ or VXLAN, which gives us 4096 times 4096 VLANs (16 million or so).

Spanning Tree Sucks

One of my first introductions to networking was accidentally causing a bridging loop on a 10 megabit Ethernet switch (with a 100 Mbit uplink) as a green Solaris admin. I’d accidentally double-connected a hub, and I noticed the utilization LED on the switch went from 0% to 100% when I plugged a certain cable in. I entertained myself with plugging in and unplugging the port to watch the utilization LED flucutate (that is, until the network admin stormed in and asked what the hell was going on with his network).

And thus began my love affair with bridging loops. After the Brocade presentation where we built a TRILL-based Fabric very quickly, with active-active uplinks and nary a port in blocking mode, Ethan Banks became a convert to my anti-spanning tree cause.

OpenFlow offers an even more comprehensive (and potentially more impressive) solution as well. More on that later.

Layer 2 Switching Isn’t Scaling

The current method by which MAC addresses are learned in modern switches causes two problems: Only one viable path can be allowed at a time (only way to prevent loops is to prevent multiple paths by blocking ports), and large Layer 2 networks involve so many MAC addresses that it doesn’t scale.

From QFabric, to TRILL, to OpenFlow (to half a dozen other solutions), Layer 2 transforms into something Layer 3-like. MAC addresses are routed just like IP addresses, and the MAC address becomes just another tuple (another recurring word) for a frame/packet/segment traveling from one end of your datacenter to another. In the simplest solution (probably TRILL?) MAC learning is done at the edge.

There’s A Lot of Shit To Configure

Automation is coming, and in a big way. Whether it’s a centralized controller environment, or magical software powered by unicorn tears, vendors are chomping at the bit to provide some sort of automation for all the shit we need to do in the network and server world. While certainly welcomed, it’s a tough nut to crack (as I’ve mentioned before in Automation Conundrum).

Data center automation is a little bit like the Gom Jabbar. They tried and failed you ask? They tried and died.

“What’s in the box?”

“Pain. And an EULA that you must agree to. Also, man-years of customization. So yeah, pain.”

Ethernet Rules Everything Around Me

It’s quite clear that Ethernet has won the networking wars. Not that this is any news to anyone who’s worked in a data center for the past ten years, but it has struck me that no other technology has been so much as even mentioned as one for the future. Bob Metcalfe had the prophetic quote that Stephen Foskett likes to use: “I don’t know what will come after Ethernet, but it will be called Ethernet.”

But there are limitations (Layer 2 MAC learning, virtualization, VLANs, storage) that need to be addressed for it to become what comes after Ethernet. Fibre Channel is holding ground, but isn’t exactly expanding, and some crazy bastards are trying to merge the two.

Oof. Storage.

Most people agree that storage is going to end up on our network (converged networking), but there are as many opinions on how to achieve this network/storage convergence as there are nerd and pop culture reference in my blog posts. Some companies are pro-iSCSI, others pro FC/NFS, and some like Greg Ferro have the purest of all hate: He hates SCSI.

“Yo iSCSI, I’m really happy for you and imma let you finish, but Fibre Channel is the best storage protocol of all time”

So that’s “The Problem”. And for the most part, the articles on Networking Field Day, and the solutions the vendors propose will be framed around The Problem.

Filed under Ethernet, FCoE, Fibre Channel, NetworkingFieldDay, Storage, Virtualization

Brace Yourself: Networking Field Day 2 Posts Are Coming

October 31, 2011 1 Comment

What is Networking Field Day? It’s the brainchild of Stephen “The Gandalf of Storage” Foskett (@sfoskett),  storage guy extraordinaire and all around awesome dude. Also working the event was Matt “No Nerd Left Behind” Simmons (@standalonesa).  He had the idea to put together these two day events, bringing vendors and bloggers together. Vendors pitch their wares in a way that is probably a bit scary: They are in no way guaranteed that the bloggers will write about them, and if we do, whether or not it will be positive or negative. In fact, it’s happened where bloggers have called a vendor’s product shit, to their face and in writing. What’s more, all the presentations (including a oftentimes tough blogger Q&A) is available online at Vimeo.

Stephen runs Tech Field Day, which has evolved into more focused field days such as Networking Field Day, Wireless Field Day, and the last Tech Field Day (Field Day 8) which was an unofficial Storage Field Day for the most part.

The Field Days are great for a couple of reasons. It allows vendors to spread their ideas to people that aren’t just going to parrot a press release. We’re the technical folk most qualified to call bullshit on a vendor claim.  The bloggers get to learn about technologies they were previously unaware of, or only peripherally aware of. And boy did I get my learn on. It’s hard not to with this fantastic group of people who comprised the other delegates/bloggers:

There were two overall themes to this Networking Field Day. One is “The Problem”, the common state of networking and the common set of challenges we face, and the second is OpenFlow and what it means. There was some pretty exciting stuff discussed, and brace yourselves because many of the next posts are going to involve my experiences at tech field day. Fear not, it’s not vendor ass-kissing. We’re tough but fair, and if a product is shit, I’m not afraid to say so (like I did with Symantec).

Note: As a delegate/blogger, my travel and accommodations were covered by Gestalt IT, who vendors paid to have spots during the Networking Field Day. Vendors pay Gestalt IT to present, so while my travel (hotel, airfare, meals) were covered indirectly by the vendors, no other remuneration (save for the occasional tchotchke) from any of the vendors, directly or indirectly, or by Gestalt IT. My time was unpaid. 

Filed under NetworkingFieldDay