FCoE: I’m not Dead! Arista: You’ll Be Stone Dead in a Moment!

I was at Arista on Friday for Tech Field Day 8, and when FCoE was brought up (always a good way to get a lively discussion going), Andre Pech from Arista (who did a fantastic job as a presenter) brought up an article written by Douglas Gourlay, another Arista employee, entitled “Why FCoE is Dead, But Not Buried Yet“.

FCoE: “I feel happy!”

It’s an interesting article, because much of the player-hating seems to directed at TRILL, not FCoE, and as J Metz has said time and time again, you don’t need TRILL to do FCoE if you do FCoE the way Cisco does (by using Fibre Channel Forwarders in each FCoE switch). Arista, not having any Fibre Channel skills, can’t do it this way. If they were to do FCoE, Arista (like Juniper) would need to do it the sparse-mode/FIP-snooping FCoE way, which would need a non-STP way of handling multi-pathing such as TRILL or SPB.

Jayshree Ullal, The CEO of Arista, hated on TRILL and spoke highly of VXLAN and NVGRE (Arista is on the standards body for both). I think part of that is that like Cisco, not all of their switches will be able to support TRILL, since TRILL requires new Ethernet silicon.

Even the CEO of Arista acknowledged that FCoE worked great at the edge, where you plug a server with a FCoE CNA into an FCoE switch, and the traffic is sent along to native Ethernet and native Fibre Channel networks from there (what I call single-hop or no-hop FCoE). This doesn’t require any additional FCoE infrastructure in your environment, just the edge switch. The Cisco UCS Fabric Interconnects are a great example of this no-hop architecture.

I don’t think FCoE is quite dead, but I have to imagine that it’s not going as well as vendors like Cisco have hoped. At least, it’s not been the success that some vendors have imagined. And I think there are two major contributors to FCoE’s failure to launch, and both of those reasons are more Layer 8 than Layer 2.

Old Man of the Data Center

Reason number one is also the reason why we won’t see TRILL/Fabric Path deployed widely: It’s this guy:

Don’t let him trap you into hearing him tell stories about being a FDDI bridge, whatever FDDI is

The Catalyst 6500 series switch. This is “The Old Man of the Data Center”. And he’s everywhere. The switch is a bit long in the tooth, and although capacity is much higher on the Nexus 7000s (and even the 5000s in some cases), the Catalyst 6500 still has a huge install base.

And it won’t ever do FCoE.

And it (probably) won’t ever do TRILL/Fabric Path (spanning-tree fo-evah!)

The 6500s aren’t getting replaced in significant numbers from what I can see. Especially with the release of the Sup 2T supervisor for the 6500es, the 6500s aren’t going anywhere anytime soon. I can only speculate as to why Cisco is pursuing the 6500 so much, but I think it comes down to two reasons:

Another reason why customers haven’t replaced the 6500s are that the Nexus 7000 isn’t a full-on replacement. With no service modules, limited routing capability (it just recently got the ability to do MPLS), and a form factor that’s much larger than the 6500 (although the 7009 just hit the streets with a very similar 6500 form factor, which begs the question: Why didn’t Cisco release the 7009 first?).

Premature FCoE

So reason number two? I think Cisco jumped the gun. They’ve been pushing FCoE for a while, but they weren’t quite ready. It wasn’t until July 2011 that Cisco released NX-OS 5.2, which is what’s required to do multi-hop FCoE in the Nexus 7000s and MDS 9000. They’ve had the ability to do multi-hop FCoE in the Nexus 5000s for a bit longer, but not much. Yet they’ve been talking about multi-hop for longer than it was possible to actually implement. Cisco has had a multi-hop FCoE reference architecture posted since March 2011 on their website, showing a beautifully designed multi-hop FCoE network with 5000s, 7000s, and MDS 9000s, that for months wasn’t possible to implement. Even today, if you wanted to implement multi-hop FCoE with Cisco gear (or anyone else), you’d be a very, very early adopter.

So no, I don’t think FCoE is dead. No-hop FCoE is certainly successful (even Arista’s CEO acknowedged as such), and I don’t think even multi-hop FCoE is dead, but it certainly hasn’t caught on (yet). Will multi-hop FCoE catch on? I’m not sure. We’ll have to see.

7 Responses to FCoE: I’m not Dead! Arista: You’ll Be Stone Dead in a Moment!

  1. Pingback: Tech Field Day 8: The Links

  2. The point I was making is somewhat about TRILL, but mostly about technologies such as OTV, VXLAN, LISP, etc. They do not work with FCoE.

    I believe that the priority order for decision making would put the ability to move a virtual machine anywhere in your DC and across DCs at a higher value than consolidating FC and Ethernet onto one link for one hop. I also think operating a scalable and stable network with broadcast containment and unknown unicasts contained to a reasonable number of hosts is probably also a higher priority than limited storage consolidation.

    If there was a way to make storage globally addressable, work with VXLAN, OTV, and/or LISP, consolidate storage to a single wire coming from the host, and build a scalable and stable network architecture that path would be the dominant one. Oh wait – it does exist: NAS, iSCSI, HDFS, and other global file systems such as Nexenta, etc all support this model…


  3. Stuart Miniman says:

    Cisco has a long history of inflated expectations as to how fast users will adopt new technology.
    The latest data that I have heard is that FCoE product is < 10% of the overall FC revenue. Seeing that the standard was only ratified 2 years ago, that's pretty good – much faster adoption than iSCSI which is also doing just fine. The big driver for FCoE was to enable the next generation of blade servers (such as Cisco's entrance into the market) which could not afford the real estate or power of FC and Ethernet. Cisco, HP and IBM are all taking advantage of converged networking, so mission 1 is accomplished.
    Stu Miniman
    Twitter: @stu

    • tonybourke says:

      Hi Stu,

      I have to imagine virtually 100% of that 10% is no-hop FCoE, which is from the blade/server to the FCoE switch. That’s been quite successful, but it requires a native Fibre Channel infrastructure.

      From what I’ve been able to see, multi-hop FCoE has not displaced any native Fibre Channel implementations, and companies haven’t (yet?) moved to build out FCoE fabrics to replace their FC fabrics.


  4. Stu,

    As I have been ranting on Twitter, “10%” of the FC SAN market (as measured by ports or revenue) does not equal adoption. A very large percentage of the FCoE market is based on FCoE capable ports on Cisco UCS/Nexus and HP switches. Licensing and being FCoE-capable does not equal adoption and deployment. The port counts will also be skewed over time by FCoE LOM, the vast majority of which may never be used for FCoE. At best, it’s being deployed ToR single-hop. Multi-hope FCoE is still a pipe dream that’s waiting for a financial and technical justification.

    So my breakdown would be that 30% of the FCoE market is single-hop or ToR FC/LAN split, 65% is FCoE capable, and maybe 5% is in some form of multi-hope FCoE. If those are reasonable assumptions, than it means that your 10% figure is really closer to 3-5%. Again, my opinions…

  5. Tony
    Thanks for your write up!
    I do agree with you on FCOE multi hop implementations, still nascent
    However, clarifying Arista/ CEO view on TRILL and what I said, I am big believer in standards based TRILL,( not a hater) which is supported in 2012/2013 and in some 2011 silicon.
    I do hate most of the vendor marketing which has been around “Fabric-happy” architectures propping up proprietary precursors to TRILL. Cloud Customers rightly expect multivendor interoperability and debugging capability based o standards and this is the Arista way with n-way cloud scale and EOS support for present MLAG,ECMP and future TRILL standards based solutions.

    Orthogonally, VXLANS indeed promising also for tunneling and enabling VM workload mobility across L2/L3 as IETF standards firm up.

    Jayshree Ullal

  6. Pingback: Internets of Interest:23 Sep 2011 — My Etherealmind

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: