The Case for FCoE Terminology
August 25, 2011 9 Comments
A previous post of mine (Jinkies! It’s an FCoE Mystery) talked about the need for some additional terminology in the FCoE world, specifically three different types of FCoE deployments. It’s generated a lot of comments, some which seem even longer than the actual post. I wanted to do a follow up, specifically regarding my reasoning for having the topology definitions.
FCoE, as a term, is very broad: It means that you’re taking a Fibre Channel frame and encapsulating it into an Ethernet frame. That’s it. There’s only one “FCoE” method in terms of this encapsulation. However, my point is that there are a number of very different ways you can go about moving those FCoE frames onto your Ethernet network.
Take this scenario: You’re presented with a switch. It has a nice sticker on it that says “FCoE switch”. Now what does that tell you about how you can fit it in your network?
Attention Cisco: You’re welcome
Almost nothing.
If you said it was a data center bridge (DCB) switch, you would then know that it’s a transit switch. No FCoE frames will be encap’d/decaped on that switch, but it supports at least PFC (priority flow control) so that FCoE frames can be guaranteed to be lossless.
Now, if you were told the FCoE switch is a FCF switch (has a full Fibre Channel stack), what does that tell you about how you can deploy it?
Still, almost nothing.
Take the example of a Cisco 6X00 Fabric Interconnect, the brains behind Cisco’s UCS server system. They are FCoE devices, and they are Fibre Channel Forwarders (FCFs). However, you can’t do what I would consider multi-hop FCoE. You can connect to a native Fibre Channel fabric, but not an FCoE fabric. That is, you can’t set up an FCoE ISL (Inter-Switch Link, but not the old Cisco pre-802.1Q VLAN tagging, it means something different in Fibre Channel) to another FCoE capable switch. This is why I added a third method to Ivan Pepelnjak’s sparse-mode (SMFCoE) and dense-mode (DMFCoE) definitions. (Note: That’s an embarassing number of acronyms).
So by having those three different distinctions (dense-mode/FCF, sparse-mode/DCB, one-hop/zero-hop) you can then tell immediately how you can deploy a FCoE switch in your network. Some switches will likely support multiple ways, but most right now are limited to one in how they’re deployed on your network.
I understand the concerns that both J Metz from Cisco and Erik Smith from EMC about adding complexity, but I think having these three different topology definitions can go a long way to help simplify discussions on FCoE topology, and in fact removes a lot of complexity (and mystery).
This morning I attended a webinar held by the Ethernet Alliance (based near me in Beaverton, Oregon) and I was happy to hear they also make a distinction between FCF FCoE switches and non-FCF FCoE switches. It really helps simplify things in terms of deployment.
The Data Center Overlords 
Hey Tony,
As in any debate you should have multiple points of view, correct? I’ll go ahead and outline mine here. 🙂
First, though, I must disagree with you on the Nexus 6000 series switch. From an FCoE perspective, it does precisely what is designed to do. Your argument that the switch does not do all of what is in the FC-BB-5 standard, and therefore “means something different to another FCoE-capable switch,” is simply misleading.
No FCoE switch on the market today covers all of FC-BB-5. No FCoE switch covers all the element of FC-BB_IP, FC-BB_GFPT and FC-BB_PW at the same time as FC-BB_E (the E part is the part the concerns itself with FC over lossless Ethernet). You use a standard to solve a particular problem, and if you do not need a standard way of solving a problem then you do not have to implement it.
The N6k is exactly the same FCoE switch as the N5k was before December, 2010. That is, it provided the ability to encapsulate/decapsulate FCoE frames and forward them to FC switches precisely the way the standard defined. To follow your argument to its logical conclusion, the N5k was not a FCoE switch until December because it didn’t have a particular feature either. Obviously, this is a very difficult assertion to sustain. Retrofitting the definition – despite functioning exactly the way the FC-BB-5 standard is defined – is problematic.
Now, let’s take a quick look at your implementation methods. Erik’s got the best rationale listed in his comment on your “Jinkies” piece, but again I believe that it’s making things waaaaay more complicated than it needs to be.
Why?
Because technologies evolve, new features and capabilities are added, as are deployments.
Where, for instance, does FCoE-NPV – just released on the Nexus 5k – fit into your “Sparse Mode”/”Dense Mode”/”Single Hop”/”Zero Hop” model? The technology is the same encap/decap process that Erik outlined, but the FLOGI process is slightly different. It’s not “dense mode” because it doesn’t have an FCF at every stage, but it keeps visibility into the fabric along the way. It’s not “sparse mode” because you don’t lose visibility. Are you going to come up with another, fourth, label? After all, it’s available right now.
What about, as Erik points out, VN2VN? Completely standards-compliant but useful particularly for mainframe environments because it’s possible to do switchless implementations, but the encap/decap of FCoE remains intact. What’s your fifth new term?
What about FDFs and the ability to do Adjacency ports? Are you going to have a sixth new term, despite the fact that – again – the process remains the same?
Now, having said all that, I agree that people are getting confused by something as vague as “FCoE-capable.” Having had some intimate experience with how Cisco is promoting FCoE I do not recall us using that terminology, but I’m just one guy. 🙂
Part of the issue comes from people not really understanding the right questions to ask, which is why I started my series of blogs long before I joined Cisco and continued doing them in my current role. It’s also why I’m very grateful to you for expanding the conversation.
To me, it’s putting the cart before the horse to look at a data center in terms of where FCoE implementations fit by calling it a particular “mode.”
Look at it this way. When you drive your car, the way you drive your car remains the same. You follow the same rules, maintain the same operational model. If you’re just going back and forth to work every day, you still use your car in the same way to get from one location to another.
If you’re making a cross-country road trip, you do not suddenly use a different “mode” for your car, despite using different highways, paths, detours, or infrastructure. Even if you take a ferry across a lake you still use your car in exactly the same way as you did before, just that the ferry has to work its own navigation during that interim. But it does not change the way that the car operates.
There is no different “mode” for operating the car, and you do not need to worry about how to use the car depending on whether or not you are driving to the convenience store or to the in-laws across the country. If we add a new road, or a new bridge, or a new highway infrastructure, it remains the same. It makes no sense to say you need a “work mode” car and a “highway mode” car and a “vacation mode” car or a “ferry mode” car.
That’s FCoE. It really is that simple and people should be encouraged to think of it as that simple.
By the way, you make the illustration on a number of occasions on how Juniper and Cisco can both be “right” in terms of notions like TRILL and FCoE.
I agree that the confusion arises from asking improper questions, but I disagree that Juniper and Cisco are both right on the “need” for TRILL. You either “need” it or you don’t. There is no “both ways are right” here. Juniper’s case was that you need TRILL whether or not you have multihop FCoE (FCFs with VE_Ports) in place.
Their argument centers around erroneous implication that you can no longer use FC forwarding in an Ethernet environment, something that I showed is simply not true. Moreover, if that weren’t enough, if you are running a SAN and wish to run over other media types (whether it be IP, GPFT, PW or E) you still need to maintain your best practices. So no, Virginia, not only do you not “need” TRILL to run multihop FCoE but in current iteration of available technology it would break visibility and best practices of the SANs in the first place.
I think I’ve contributed more than was expected (or probably even recommended! 🙂 ) so I’ll pause (see what I did there?) to let someone else get a word in edgeways…
J
Hi Tony, let’s build off your example where the switch vendor claims “FCoE awesomeness!” I always approach this question by putting the FCF in the middle of the diagram and asking “how can I connect to the FCF from end devices (lets call this southbound connectivity) and how can I connect the FCF to the rest of the Fabric (let’s call this northbound connectivity). BTW, I don’t know where I stole the northbound / southbound terminology from but suffice it to say I am not coining new terminology here…
From a southbound (end device) perspective, you really only need to have the vendor answer a single question:
1) “Do you allow a FIP Snooping Bridge (a.k.a. transit switch) between the FCF and the end device?”
Today, Juniper and Cisco will both say yes and Brocade will say no (at least on the 8000/FCoE 10-24 blade).
From a northbound (FCF to rest of fabric) perspective, things get a bit more complicated but I think you can figure out what they support by asking three additional questions:
2) Do you support FCoE VE_Ports? The answer will be yes for Cisco and for Brocade VDX, and no for Juniper and the Brocade 8000/FCoE 10-24 blade. BTW, a yes answer means they support what you are calling DMFCoE but a no answer doesn’t necessarily mean they support SMFCoE. This is part of the reason why I don’t like the DM/SMFCoE terms.
3) Do you support FC ISLs? The answer will be yes for all products except Nexus 7k and Juniper.
4) Do you support an N_Port Virtualizer mode? Right now all of the FCF vendors will say yes.
If the switch vendor answers “Yes” to all four questions, then you have a fully functional FCoE switch. If the answer is no to any of the four questions, then you still have an FCoE switch that may fully meet your requirements, you just need to know how you intend to deploy the switch and figure out how the “No” answer will impact your use case.
This will probably seem like a very simple question but I hope you’ll excuse my ignorance this once and help me to better understand the FCoE hardware available today.
I have read plenty of information that attempts to explain what FCoE is and how it preserves FC standards etc but I haven’t seen anything that tells me if it possible using current CNAs to make a direct connection from a server to a FC array.
What I would like to know is, do the CNA options that are currently available only use 10Gb Ethernet or are they capable of sending information as Fibre Channel traffic like a standard FC HBA?
I should probably say that my understanding is that the FCoE standard as it stands always uses a 10Gb Ethernet network between the CNA and the switch and that is fine and makes perfect sense. However, the question above isn’t about FCoE it is about the ability of a CNA to work with standard FC traffic.
Assuming a CNA will only ever deal with 10Gb Ethernet traffic, is there a way to take the FCoE traffic sent from the CNA and forward it on to a native FC appliance using something other than a switch?
Hi Chris,
To answer the first part of your question, a CNA is an Ethernet connection. The standard doesn’t require 10 Gbit, but no one supports less than 10 Gbit for CNAs.
With a CNA, the server sees two interfaces: A native fibre channel interface, and a native Ethernet interface. However, when both frames leave the CNA, they’re Ethernet. The native Ethernet traffic runs unaltered, while the native FC frames are put onto special VLANs and encapsulated into FCoE Ethernet frames. All the while, the server has no idea the FC frames are being turned into FCoE frames and vice versa.
I’m not quite sure on your second part: Can you plug a CNA directly into an FCoE storage array? Theoretically, it’s possible. In Fibre Channel that would be a direct N_port to N_port connection. In FCoE, that would be VN_port to VN_port. The question is, would any storage vendor support it? I’m not sure.
The last part of your question, is there a way to make a CNA talk directly to a FC appliance without a switch? No. You’d have to plug a CNA into an FCoE switch with native FC interfaces (such as a Nexus 5000 or Cisco UCS Fabric Interconnect), and the FC interface would plug into the native FC storage array. Or the array would have to have FCoE interfaces (some do now).
Check out this link from Cisco, it’s a multi-hop FCoE reference architecture:
http://www.cisco.com/en/US/docs/solutions/Enterprise/Data_Center/VMDC/tech_eval/mFCoEwp.html
Hi Chris, there are some CNAs that support running in an FC mode. Take a look at the Brocade 1860 Fabric Adapter, it can support 16/8/4 Gb/s FC as well as 10GbE / FCoE.
That having been said, the 1860 does not support FC-AL. This is relevant to your question because there are storage arrays on the market that run either in an FC-SW mode (they expect to use FLOGI and login to a switch) or a direct connect mode (actually uses FC-AL for link initialization, etc). Because of this you will need to verify that your array supports FC Point-to-Point and validate that this has been tested and is supported with the 1860.
Also, in regards to the question about directly connecting a CNA to a Storage port via FCoE, this is not possible today. See http://brasstacksblog.typepad.com/brass-tacks/2011/03/pt2pt-and-vn2vn-directly-connecting-an-fcoe-initiator-to-an-fcoe-target-part-1-overview.html for more information.
Thank you both for taking the time to respond and answer my questions. 🙂
Pingback: Internets of Interest:14 Sep 2011 — My Etherealmind
Pingback: A Tale of Two FCoEs | The Data Center Overlords