Where servers, storage and networking combine to form Voltron.
September 7, 2011 3 Comments
Want to know what I do when I’m not writing snark-filled posts about various data center technologies? No? Well too bad. I sometimes fly airplanes for fun. Here’s a video of some aerobatics training I did earlier this week.
August 16, 2011 21 Comments
Preamble: Chances are I’m going to get something wrong in this article. Please feel free to point anything out so long as you state the correction. You can’t just say “that’s wrong” and not say why. One of the great mysteries of the data center right now is FCoE.
Ah, Fibre Channel over Ethernet. It promises to do away with separate data and storage networks, and run everything on a single unified fabric. The problem though is that FCoE is a bit of a mystery. It involves two very different protocols (Ethernet and Fibre Channel), it involves the interaction between the protocols, and vendors can bicker over requirements, make polar opposite statements, and both can be technically correct.
So that makes it kind of a mess. I’ve been teaching basics of FCoE (mostly single-hop) for a bit now, and I think I’ve come across a way to simplify perception of FCoE: Realize FCoE is implemented in three different ways.
When we talk about FCoE in general, we should be talking about which specific method that’s being referenced. That came to me when I read Ivan Pepelnjak’s article on the two ways to implement multi-hop FCoE , although I’m also adding single-hop as a separate way to implement FCoE.
While all three ways are technically “FCoE”, they are implemented in very different manners, have very different hardware and topology requirements, and different vendors support different methods. They’re almost three completely different beasts. So let’s talk about them separately, and be specific when we talk about it.
So let’s talk about FCoE.
This is the simplest way to implement FCoE, as it doesn’t really require any of the new data center standards on the rest of your network devices. Typically, a pair of switches is enabled for FCoE, as well as some server network/storage adapters known as CNAs (Converged Network Adapter).
In the Cisco realm, this is either a Nexus 5000 series or Fabric Interconnects which are part of the Cisco UCS server system. In HP, this might be part of Virtual Connect. A CNA is a Ethernet/Fibre Channel combo networking card. The server’s operating system is presented with separate native Ethernet and native Fibre Channel devices, so the OS doesn’t even know that FCoE is going on. It just thinks there’s native Ethernet and native Fibre Channel.
Oh hey, look! An actual diagram. Not just proof you were alive in the 80’s.
Ethernet frames containing FC frames are isolated onto their own FCoE VLANs. When the Ethernet frames reach the FCoE switch they are de-encapsulated and forwarded via regular Fibre Channel methods to their final destination as native Fibre Channel.
This method has been in place for a few years now, and it works (and works well). It’s pretty well understood, and there’s plenty of stick time for it. You also don’t need to do anything special on your Ethernet networks, and most of the time nothing special needs to be done on your Fibre Channel SAN (although NPV/NPIV may be needed to get the FCoE switch connected to the Fibre Channel switch). You don’t have to worry about any of the new DCB standards, such as DCBX, PFC, ETS, etc., because they only need to be on the FCoE single-hop switch, and are already there. No tweaking of those standards is typically necessary.
There are two types of multi-hop FCoE, where the FCoE goes beyond just the initial switch. J Metz from Cisco elaborated on the various definitions (and types) of multi-hop in this great blog article here, but I think we can even make it more simple by saying that multi-hop means more than one FCoE switch.
With DMFCoE, a FCoE frame is received at the DMFCoE switch and de-encapsulated into a regular FC frame. The FCF (Fibre Channel Forwarder) portion of the DMFCoE switch makes the forwarding decision and sends it to the next port. At that port, the FC frame is re-encapsulated into an FCoE Ethernet frame and send out an Ethernet port to the next hop.
With DMFCoE, each of your Ethernet switches is also a full-stack Fibre Channel switch. You’re running essentially a Fibre Channel SAN overlay on top of your Ethernet switches. Zoning, name services, FSPF, etc., are all the same as on your regular Fibre Channel network. Also, FCoE frames are routed along not by Ethernet, but by Fibre Channel routing (FSPF) which is multi-path (so no bridging loops).
The drawback is that it requires a pretty advanced switch to do it. In fact, it wasn’t until July of 2011 that Cisco had more than one switch that could even do DMFCoE (the MDS and Nexus 7000 needed 5.2 to do DMFCoE, which wasn’t released until July).
Alternative names for dense-mode FCoE:
Sparse Mode FCoE (SMFCoE) is when an Ethernet network forwards FCoE frames via regular Ethernet forwarding mechanisms. Unlike DMFCoE, the Fibre Channel frame is not de-encapsulated (although but it might be snooped with FIP snooping if the switch supports it). For the most part, the Ethernet switches have little to no awareness of the Fibre Channel layers.
The benefit of SMFCoE is that it doesn’t require quite the beefiness that DMFCoE needs, as you don’t need silicon that can understand and forward FCP (Fibre Channel Protocol) traffic. You still need priority flow control and other DCB standards, and probably DCBx (to set up the FCoE lossless CoS and so forth).
The drawback is that you’ll usually need some sort of multi-path Ethernet protocol, such as TRILL/SPB/Fabric Path as spanning-tree would likely be a disaster for a storage protocol. Since none of the potential multi-path Ethernet protocols are in wide use with the various vendors, that makes SMFCoE somewhat dead right now.
Alternative names for SMFCoE might be:
Because it gets damn confusing otherwise. Recently Juniper and Cisco had a dustup about the requirement of TRILL for FCoE. Juniper posted the article on why TRILL won’t scale for data centers, and mentioned that TRILL is required for FCoE. J Metz from Cisco counter-reponded with essentially “no, FCoE doesn’t need TRILL“. Who’s right? Well they both are.
Cisco has gone the DMFCoE route, so no you don’t need TRILL (or other multi-path Ethernet). Since Juniper is going SMFCoE, it will need some sort of multi-path (and his article is calling for QFabric to be that solution).
Whither FCoE?
So can you do FCoE multi-hop right now, either DMFCoE or SMFCoE? It probably would be wise to wait. In the Cisco realm, the code that supports DMFCoE was just released in July for their Nexus 7K and MDS lines, and the 5Ks could have done DMFCoE since December I think (although I don’t know any one that did).
Right now, I don’t know of any customers actually doing mutli-hop FCoE (and I don’t know anyone who’s all that interested). SMFCoE is a moot point right now until more switches can get multi-path Ethernet, whether that be QFabric, TRILL, SPB or another method.
June 24, 2011 1 Comment
The CCIE certification from Cisco is widely considered to be one of the best, toughest certifications to get.
Generally obtaining this certification requires months, if not years of preparation, abanonding free time (and in some cases, hope). You hear of CCIE widows/widowers. It has a high failure rate the first attempt, and some (really smart people too) take several attempts.
I haven’t seriously considered getting a CCIE, despite working a lot in the Cisco realm (I’m a Cisco Certified Systems Instructor). And it’s not because of the insane prep and soul crushing defeats. I mean, something difficult and insane? Sign me up. (I enjoy insane goals, like running marathons and training to be an aerobatic pilot.)
The problem is relevance. Right now there are six different CCIE tracks: CCIE Route & Switch, CCIE Storage, CCIE Service Provider, CCIE Security, CCIE Wireless, and CCIE Voice. The vast majority are CCIE R&S. CCIE Wireless has less than 50 at last count.
Not one of them would dramatically increase my skills in areas that I typically work in. I deal with switching, a bit of spanning-tree, virtualization, and storage (some FC, more FCoE, and iSCSI). Things I never deal with, ever: ATM, voice, metro Ethernet, routing protocols (although IS-IS maybe a new skill I need to pick up).
This will require intense study. Right after I check Twitter.
For a year or so now however, there’s been a rumor that a CCIE Data Center is coming. It would likely involve MDS/storage, FCoE, Nexus switching, UCS, even some load balancing and WAAS.
So I’m hoping it gets released soon. I would be all over that shit.
June 24, 2011 1 Comment
“Do said skills pay the bills?” -Professor Hubert Farnsworth
That data center landscape is changing rapidly. If you’re a network admin, you’re dealing with server stuff you never thought you’d have to put up with. If you’re a server admin, there’s all this networking stuff that you can’t ignore anymore. If you’re storage-oriented, Fibre Channel is about to jump onto an Ethernet network near you, or you’ll find your storage connecting via iSCSI.
We all need additional skills.
So where do we start? I have a few suggestions.
Networking Admins
If you’re a networking admin, I’d start looking at virtualization as soon as possible. If you can get your employer to pay for it, I recommend getting the VCP certification from VMware (which requires taking a VMware class). While there are other virtualization technologies out there, VMware has about 90% of the server virtualization market and it’s a good foundation for virtualization technology in general. The VMware training is generally excellent, and the VCP (currently VCP4) is a good certification to have in the industry.
Also, look into setting up your own home lab running the free version of ESXi, or some other virtualization technology such as Xen or Hyper-V, both which can be obtained free (I think that’s the case for Hyper-V).
Also, Linux. Learn Linux. From Juniper to Cisco’s Nexus to Arista to most of the routers and switches coming out are based on Linux (or one of the BSDs, but the skills are very transferable). Plus, most of the virtualization technologies are based on Linux. So yeah, Linux.
Server Admins
If you’re a server admin, you really, really need to learn some networking. Specifically, Ethernet switching. You’ll also need to learn IPv4 and IPv6, TCP behavior (like sliding windows) and the HTTP protocol. Fortunately routing protocols isn’t something you’ll typically need to deal with, even today.
Cisco’s CCNA exam is a good start. It’s tough for a server admin (I failed my first attempt so bad I thought John Chambers was going to get a phone call), but it fills in a lot of blanks about networking.
You can play with routers using GNS3, a graphical front end for a Cisco IOS router emulator called DynaMIPs/Dynagen (you need to provide your own copy of an IOS router image).
A cohort at Firefly Communications Chris Welsh has put together an Ubuntu-based virtual machine that has GNS3 already pre-installed and ready to go and can be found on his site rednetctar.net.
Always Be Learning
Podcasts and webinars are a great way to brush up and expand skills. A couple of months ago I found the fantastic Packet Pushers Podcast, and I’ve listened to just about every episode (and some, like the episode on Shortest Path Bridging, a multi-path Layer 2 protocol to replace Spanning Tree Protocol, several times).
Ivan Pepelnjak at ioshints.info has a great blog, very technical, and also a series of webinars you can purchase (I bought the year subscription for $199, and it’s already paid for itself in brain filling goodness).
And I’m amazed how fantastic Twitter is for keeping up with technical stuff. Starting off with myself and @etherealmind and moving from there is a great way to branch out.
If you’re comfortable in the traditional silo’d environment, prepare to be uncomfortable very soon. There’s no turning back. Time to get more skillz.
I’d love to hear any other tips, resources, sites, etc., that you would suggest for the overlord conversions.
The Data Center Overlords 
