Always Be Learning | The Data Center Overlords

Ethernet Congestion: Drop It or Pause It

February 2, 2013 5 Comments

Congestion happens. You try to put a 10 pound (soy-based vegan) ham in a 5 pound bag, it just ain’t gonna work. And in the topsy-turvey world of data center switches, what do we do to mitigate congestion? Most of the time, the answer can be found in the wisdom of Snoop Dogg/Lion.

Of course, when things are fine, the world of Ethernet is live and let live.

We’re fine. We’re all fine here now, thank you. How are you?

But when push comes to shove, frames get dropped. Either the buffer fills up and tail drop occurs, or QoS is configured and something like WRED (Weight Random Early Detection) kicks in to proactively drop frames before taildrop can occur (mostly to keep TCP’s behavior from causing spiky behavior).

The Bit Grim Reaper is way better than leaky buckets

Most congestion remediation methods involve one or more types of dropping frames. The various protocols running on top of Ethernet such as IP, TCP/UDP, as well as higher level protocols, were written with this lossfull nature in mind. Protocols like TCP have retransmission and flow control, and higher level protocols that employ UDP (such as voice) have other ways of dealing with the plumbing gets stopped-up. But dropping it like it’s hot isn’t the only way to handle congestion in Ethernet:

Please Hammer, Don’t PAUSE ‘Em

Ethernet has the ability to employ flow control on physical interfaces, so that when congestion is about to occur, the receiving port can signal to the sending port to stop sending for a period of time. This is referred to simply as 802.3x Ethernet flow control, or as I like to call it, old-timey flow control, as it’s been in Ethernet since about 1997. When a receive buffer is close to being full, the receiving side will send a PAUSE frame to the sending side.

Too legit to drop

A wide variety of Ethernet devices support old-timey flow control, everything from data center switches to the USB dongle for my MacBook Air.

One of the drawbacks of old-timey flow control is that it pauses all traffic, regardless of any QoS considerations. This creates a condition referred to as HoL (Head of Line) blocking, and can cause higher priority (and latency sensitive) traffic to get delayed on account of lower priority traffic. To address this, a new type of flow control was created called 802.1Qbb PFC (Priority Flow Control).

PFC allows a receiving port send PAUSE frames that only affect specific CoS lanes (0 through 7). Part of the 802.1Q standard is a 3-bit field that represents the Class of Service, giving us a total of 8 classes of service, though two are traditionally reserved for control plane traffic so we have six to play with (which, by the way, is a lot simpler than the 6-bit DSCP field in IP). Utilizing PFC, some CoS values can be made lossless, while others are lossfull.

Why would you want to pause traffic instead of drop traffic when congestion occurs?

Much of the IP traffic that traverses our data centers is OK with a bit of loss. It’s expected. Any protocol will have its performance degraded if packet loss is severe, but most traffic can take a bit of loss. And it’s not like pausing traffic will magically make congestion go away.

But there is some traffic that can benefit from losslessness, and and that just flat out requires it. FCoE (Fibre Channel of Ethernet), a favorite topic of mine, requires losslessness to operate. Fibre Channel is inherently a lossless protocol (by use of B2B or Buffer to Buffer credits), since the primary payload for a FC frame is SCSI. SCSI does not handle loss very well, so FC was engineered to be lossless. As such, priority flow control is one of the (several) requirements for a switch to be able to forward FCoE frames.

iSCSI is also a protocol that can benefit from pause congestion handling rather than dropping. Instead of encapsulating SCSI into FC frames, iSCSI encapsulates SCSI into TCP segments. This means that if a TCP segment is lost, it will be retransmitted. So at first glance it would seem that iSCSI can handle loss fine.

From a performance perspective, TCP suffers mightily when a segment is lost because of TCP congestion management techniques. When a segment is lost, TCP backs off on its transmission rate (specifically the number of segments in flight without acknowledgement), and then ramps back up again. By making the iSCSI traffic lossless, packets will be slowed down during congestions but the TCP congestion algorithm wouldn’t be used. As a result, many iSCSI vendors recommend turning on old-timey flow control to keep packet loss to a minimum.

However, many switches today can’t actually do full losslessness. Take the venerable Catalyst 6500. It’s a switch that would be very common in data centers, and it is a frame murdering machine.

The problem is that while the Catalyst 6500 supports old-timey flow control (it doesn’t support PFC) on physical ports, there’s no mechanism that I’m aware of to prevent buffer overruns from one port to another inside the switch. Take the example of two ingress Gigabit Ethernet ports sending traffic to a single egress Gigabit Ethernet port. Both ingress ports are running at line rate. There’s no signaling (at least that I’m aware of, could be wrong) that would prevent the egress ports from overwhelming the transmit buffer of the ingress port.

Many frames enter, not all leave

This is like flying to Hawaii and not reserving a hotel room before you get on the plane. You could land and have no place to stay. Because there’s no way to ensure losslessness on a Catalyst 6500 (or many other types of switches from various vendors), the Catalyst 6500 is like Thunderdome. Many frames enter, not all leave.

Catalyst 6500 shown with a Sup2T

The new generation of DCB (Data Center Bridging) switches, however, use a concept known as VoQ (Virtual Output Queues). With VoQs, the ingress port will not send a frame to the egress port unless there’s room. If there isn’t room, the frame will stay in the ingress buffer until there’s room.If the ingress buffer is full, it can have signaled the sending port it’s connected to to PAUSE (either old-timey pause or PFC).

This is a technique that’s been in used in Fibre Channel switches from both Brocade and Cisco (as well as others) for a while now, and is now making its way into DCB Ethernet switches from various vendors. Cisco’s Nexus line, for example, make use of VoQs, and so do Brocade’s VCS switches. Some type of lossless ability between internal ports is required in order to be a DCB switch, since FCoE requires losslessness.

DCB switches require lossless backplanes/internal fabrics, support for PFC, ETS (Enhanced Transmission Selection, a way to reserve bandwidth on various CoS lanes), and DCBx (a way to communicate these capabilities to adjacent switches). This makes them capable of a lot of cool stuff that non-DCB switches can’t do, such as losslessness.

One thing to keep in mind, however, is when Layer 3 comes into play. My guess is that even in a DCB switch that can do Layer 3, losslessness can’t be extended beyond a Layer 2 boundary. That’s not an issue with FCoE, since it’s only Layer 2, but iSCSI can be routed.

Filed under Always Be Learning, data center, Ethernet, FCoE, Storage

Goals for 2013

January 1, 2013 1 Comment

As the year closes, and it turns out the world didn’t end, it’s time to start planning for 2013 (especially since I don’t know when the next doomsday is supposed to be).

My 2012 in review:

Obtained CCNA Data Center (possibly the first outside of Cisco, literally days after it was available)
Obtained CCNP Data Center (probably not the first, I know I tied with one guy at least)
Didn’t pass the CCIE Data Center written (beta or actual)
Ran a marathon in Australia (continent number 4 for marathons, shooting for all 7)
Saw a total solar eclipse (part of the previous trip)
Australia is the 30th country that I’ve visited (and I’m not counting airport layovers, such as Egypt and Japan)
Did more aerobatic pilot training

Fruity drinks with Kurt Bales in Australia in 2012

May career goals for 2013:

Pass CCIE Data Center written in Janurary
Obtain CCIE Data Center in 2013
Obtain VCAP-DCA
ABL (Always Be Learning)

Flying a plane upside down in 2012

I think career wise, getting CCIE DC and VCAP-DCA are plenty enough for a 12-month span, as both are very tall orders. And though ambitious, with the current support system I have and resources publicly (such as vBrownbag) and that I have through Firefly, they’re both doable for 2013. I’ve got some thoughts on that particular combination of certifications which I’ll go into in another post.

There are a couple of technologies that look exciting for 2013 that I’d like to take a (closer) look at. Openstack for one, and how it relates to data center as I have only a vague conceptual understanding of it. VXLAN, STT in VMware, NVGRE in Windows 2012 Server, and the overlay technologies in general. Checking out the other hypervisor vendors, especially (and the condescending Unix administrator in me is going to throw up a bit in my mouth when I say this) Hyper-V 3.

So those are my goals for 2013. Yours?

Filed under Always Be Learning

CCIE Data Center Beta Written Results Are In! (351-080)

September 20, 2012 1 Comment

And Cisco probably couldn’t be happier that the results are finally in. It’s been more than 3 months since the beta closed, and after a few promises of “soon”, we finally got our results today. Over at the Cisco learning community message boards for CCIE DC, there was a virtual riot going on.

Guys? I think we’d better get those results posted…

Once I got word they were live on PearsonVUE, I logged in and…. I failed.

Smug Cisco Guy: Way to go, dumbass.

At least we got our results.

To find out your status, go to PearsonVUE, log into your account, and check your history. It’ll show the pass or fail. Beyond pass/fail, we have to await the score report to find our what our weak areas were. My guess I was really weak on the 7K/5K stuff. I know I got all the ACE-related questions right, and most of the storage and UCS seemed pretty evident to me. I’ll have to wait and see, of course. I’ve scheduled a re-take for October 5th, so I’ve got some books to hit. Queue the montage…

Filed under Always Be Learning, CCIE Data Center Prep, Certifications

A Different Kind of Loop

September 8, 2012 3 Comments

I’m not all nerd and memes, sometimes I do stuff like this. Starting not long after I got my pilot’s license, I started doing occasional aerobatics lessons. In this clip, I’m the one flying under the watchful eye of the instructor behind me.

Filed under Always Be Learning

Po-tay-to, Po-ta-to: Analogies and NPIV/NPV

May 21, 2012 2 Comments

In a recent post, I took a look at the Fibre Channel subjects of NPIV and NPV, both topics covered in the CCIE Data Center written exam (currently in beta, take yours now, $50!). The post generated a lot of comments. I mean, a lot. Over 50 so far (and still going). An epic battle (although very unInternet-like in that it was very civil and respectful) brewed over how Fibre Channel compares to Ethernet/IP. The comments look like the aftermath of the battle of Wolf 359.

Captain, the analogy regarding squirrels and time travel didn’t survive

One camp, lead by Erik Smith from EMC (who co-wrote the best Fibre Channel book I’ve seen so far, and it’s free), compares the WWPNs to IP addresses, and FCIDs to MAC addresses. Some others, such as Ivan Pepelnjak and myself, compare WWPNs to MAC addresses, and FCIDs to IP addresses. There were many points and counter-points. Valid arguments were made supporting each position. Eventually, people agreed to disagree. So which one is right? They both are.

Wait, what? Two sides can’t be right, not on the Internet!

When comparing Fibre Channel to Ethernet/IP, it’s important to remember that they are different. In fact, significantly different. The only purpose for relating Fibre Channel to Ethernet/IP is for the purpose of relating those who are familiar with Ethernet/IP to the world of Fibre Channel. Many (most? all?) people learn by building associations with known subjects (in our case Ethernet/IP) to lesser known (in this case Fibre Channel) subjects.

Of course, any association includes includes its inherent inaccuracies. We purposefully sacrifice some accuracy in order to attain relatability. Specific details and inaccuracies are glossed over. To some, introducing any inaccuracy is sacrilege. To me, it’s being overly pedantic. Pedantic details are for the expert level. Using pedantic facts as an admonishment of an analogy misses the point entirely. With any analogy, there will always be inaccuracies, and there will always be many analogies to be made.

Personally, I still prefer the WWPN ~= MAC/FC_ID ~= IP approach, and will continue to use it when I teach. But the other approach I believe is completely valid as well. At that point, it’s just a matter of preference. Both roads lead to the same destination, and that is what’s really important.

Learning always happens in layers. Coat after coat is applied, increasing in accuracy and pedantic details as you go along. Analogies is a very useful and effective tool to learn any subject.

Filed under Always Be Learning, CCIE Data Center Prep, Certifications, data center, Ethernet, Fibre Channel, Learning, Storage

The VDI Delusion Book Review

April 14, 2012 1 Comment

Sitting on a beach in Aruba (sorry, I had to rub that one in), I finished Madden & Company’s take on VDI: The VDI Delusion. The book is from the folks at brianmadden.com, a great resource for all things application and desktop delivery-related.

The book title suggests a bit of animosity towards VDI, but that’s not actually how they feel about VDI. Rather, the delusion isn’t regarding the actual technology of VDI, but the hype surrounding it (and the assumption many have that it’s a solve-all solution).

So the book isn’t necessarily anti-VDI, just anti-hype. They like VDI (and state so several times) in certain situations, but in most situations VDI isn’t warranted nor is it beneficial. And they lay out why, as well as the alternative solutions that are similar to VDI (app streaming, OS streaming, etc.).

It’s not a deep-dive technical book, but it really doesn’t need to be. It talks frankly about the general infrastructure issues that come with VDI, as well as delivering other types of desktop services to users across a multitude of organizations.

It’s good for the technical person (such as myself) who deal in an ancillary way with VDI (I’ve dealt with the network and storage aspects, but have never configured a VDI solution), as well as the sales persons and SE that deal with VDI. In that regard, it has a wide audience.

Brian Drew over at Dell I think summed it up the best:

For anyone dealing with VDI (who isn’t totally immersed in the realities of it and similar technologies) this is a must-read. It’s quick and easy, and really gets down to the details.

Filed under Always Be Learning, Learning, VDI, Virtualization

Initial Thoughts on Apple’s New Initiative

January 25, 2012 Leave a comment

When I heard about Apple’s new education initiative, I got excited. For one, it’s Apple. And yes, I’m a fanboy. So, like… Squeeeeeeee.

Tony, you have a problem

But it’s not algebra or geography books geared towards primary education that excites me (although that’s pretty cool), it’s how it could revolutionize IT ebooks.

Right now the primary market for technical books is print books. There are technical eBooks available on a variety of eBook platforms, but for the most part, technical books are a print business, with eBooks as an afterthought.

This approach has worked since the tech industry begain, but it does have its limiations.

For one, tech books usually have a percentage of its content that’s out of date by the time it reaches the shelves. Technical books can take over a year to get from outline to ending up on the shelves, and naturally the fast-paced moves from under the book. And going an update or corrections to a book is a major effort. If it’s C programming, it’s probably not too much of an issue. But a book on FCoE or VXLAN? There’s bound to be lots of changes and corrections within the span of a year.

What do you mean my book on cell phones isn’t current?

Also, eBooks right now are mostly just electronic versions of the paper books (ed: duh). The electronic format could do a whole lot more than just words on page, as shown by Apple in their presentation. With a fully interactive eBook, there could be animations (really awesome for networking flows), interactive quizzes (and huge test banks, not just 10 questions per chapter).

And right now eBooks seem to be an afterthought. Not all physical titles are available in eBook format (hint, several important and influential Fibre Channel books), and the ones that are can seem like a rush job. In my preparation for the CCIE Storage written test, I picked up this ebook on the Kindle platform: CCIE Network Storage. The ebook version was riddled with formatting errors which made it sometimes difficult to follow. Also, it looks like they’ve seem to have even taken it off Kindle.

Right now my favorite eBook format is the Kindle. Despite being an Apple fanboy, Kindle has the largest library of technical books, by far. And Kindle’s reader and cloud storage make managing your library stupid easy. Apple also makes it easier, although the platform is limited to Apple devices, and the tech library doesn’t seem to be as comprehensive. All of this this is in stark contrast to Adobe’s shitty eBook platform, which seems to want to destroy eBooks.

The Controversy

So the controversy is in Apple’s EULA. If you create an iBook with the iBook Author, that “Work” must be distributed through the Apple iBook store if you charge a fee for it. The tricky part is how Apple defines the term “Work”. Right now it’s a bit ambiguous. Some claim that the term “Work” defines the totality of the book. Others (like the Ars article) say “Work” only defines the output of the iBook Author program (PDF of Apple’s proprietary eBook format).

So if I write a book, and create an eBook version of it with Apple’s iBook Author (which looks like it create amazingly interactive ebooks), can I take the material from the book and make a (probably less interactive) Kindle version of the book?

Tony’s Take

Whether you like Apple or not, you have to admit this certainly ups the game. It’s high time eBooks took center stage for technical eBooks, instead of being an afterthought.

Right now the networking and data center landscape is changing fast, and we need new and better ways to cram new knowledge into our brainbags. A good interactive ebook, riddled with animations, audio, and large test banks would certainly go a long way to help. I don’t really care if it’s Apple or Amazon that provide that format. But right now, it looks like Apple is the only one saddling up.

Filed under Always Be Learning, Certifications, Learning

Adobe’s eBook Platform Is A Piece of Shit

December 1, 2011 10 Comments

Adobe’s eBook platform is utter shit. To those of you that have dealt with ACSM files, that statement is as controversial as saying “the sky is blue”. To those of you that haven’t, and are wondering what makes it such shit, read on.

It all started with a deal that Cisco Press had on cybermonday this year, offering 50% off if you buy three books. As a certified Cisco course instructor (I do not work for Cisco, I just teach Cisco courses) who is also working on my CCIE Storage, I can always do with a few more books, especially if they’re on the recommended reading list for CCIE Storage.

Also, since I travel quite a bit (150,000 miles this year), eBooks are the preferred knowledge delivery vector, since books are, well, frickin’ heavy. I took a nearly 800 page CCNP route book with me all over Europe last year, and it almost killed me. eBooks it is. I’ve got an iPad, and I absolutely love the Kindle reader app. If I’ve got a long flight ahead of me (such as to say, India) then I make sure I’ve got plenty of books loaded up into my first generation iPad and iPhone 4 (which is also a surprisingly good e-reader). I also have a half decent PDF viewer for non-eBook format documents to read on the road.

I found three eBooks from Cisco Press that fit the bill, loaded them up in my shopping cart, and pulled the trigger. $150 worth of books for $75, not too bad. Two of the books were in an unprotected PDF format (watermarked with my name to discourage rampant sharing, which is fine), the other book downloaded as a tiny little file, with an .acsm extension.

I’d never heard of a .acsm file, but I would soon come to loath those four letters with the burning hatred of a thousand suns. My Canadian friend Jaymie Koroluk (@jaymiek) had this to say about it:

FFUUUUUU indeed. And thus began my Zeldian quest to get a friggin’ eBook on a friggin’ eBook reader. How hard could it be?

Well, of course my Mac didn’t recognize the .acsm file type. I tried loading it into a couple of readers, such as Kindle (it laughed at it) and a PDF viewer that I use. It turns out that .acsm didn’t actually contain the eBook, just a reference to it (and I believe the DRM rights to open the book). I had no idea what to do with it. The Cisco Press site didn’t have any specific instructions that I could find, so I Googled .acsm and eBook.

What I found was link after link that all said essentially “How the fuck do I get an .acsm book onto my reader???” Searching for acsm on Google reveals a world of woe, frustration, and hopelessness.

Google searches for “.acsm” should just show this

After sifting through a few links, I found out that I needed to download something called Adobe Digital Editions. So I go to Adobe’s site, and I get this is the message I get when I try to download it:

What? I’ve got a new MacBook Air with MacOS Lion. There’s no “here’s what you need to do”, just that obnoxious error. With a bit of digging, I’m able to download it anyway.

I install Adobe Digital Editions, which is not intuitive and bizarrely laid out, and I’m finally able to load up the acsm file, and download a copy of the eBook. And the eBook is… a protected PDF. All that shit for a protected PDF.

But hey, at least I got it, right? Horray! But wait, I can only read it on my laptop, however. I need to get it on my iPad for this book to be of any use.

Yes, I’ve just experienced the eBook version of “The Princess is in another castle”.

But I told her to meet me here like five… fine. You know what? Tell here she’s on her own. I’m gonna go find a girl who can manage to stay un-kidnapped for say, 30 minutes at a time.

Laptops are generally not great eBook readers, because among other issues, the batteries don’t last as long. The iPad’s battery lasts 10 hours of active use, and the various Kindle readers have their active battery life measured in days. If I can’t find a way to get this onto my iPad, then there’s not much point in me having spent the money for this book.

I try to find some iPad app at the App Store that reads that format, that would allow me to open the protected PDF, but I came up blank. Or at least, none of them would obviously work. And most of them cost money, so I wasn’t about to do trial and error on which ones might work.

Jaymie mentioned she found an app called txtr, which I downloaded an installed. Txtr apparently was a failed ebook reader, and moved to a purely software play. They also had the ability to read Adobe eBooks (and as far as I can tell, the only iPad app that can). So Finally, I’m able to read the eBook on my iPad.

All told, it takes me over an hour and lots of tinkering, installing, and Googling to get an Adobe eBook onto my iPad.

So how does the Adobe eBook platform compare to other eBook platforms when you finally get the fucking book loaded up on your fucking eBook reader (which again, should not be nearly as difficult as it was)? Let’s compare.

First, ease of getting a book. How long does it take me to get an eBook on the Kindle, iBook, or Nook platforms? About 10 fucking seconds with a decent Internet connection. On Adobe’s platform? About an hour. By my math, Adobe’s platform is 360 times worse than the competition.

So how about usability? The book is a PDF, and PDFs are not ideal as a book format, even the non-DRMd ones that can be opened up on any reader. They’re just not optimized for eReaders and it shows. When you turn a page, the page is blurry for a split second before coming into focus. You can’t zoom in on individual photos like you can with the other readers. And there are about a dozen other nit-picky yet important UI niceties that Kindle and the others have that a PDF eBook lacks. Adobe’s platform seems like they took their existing PDF format, and slapped an eBook layer onto it in a half-assed manner.

In studying for my CCIE Storage, I came across a fantastic free Fibre Channel eBook from EMC (the storage vendor). It’s in an unprotected PDF format, but I’d happily pay $10 to get it in the Kindle format, which is much more conducive to eBook formats.

Final Thoughts

I have a simple plea to anyone thinking of publishing an eBook: For the love of all that is sacred and good in the world, do not use the Adobe book format. It will annoy your readers, and severely limit your eBook sales.

Adobe either has no clue about the eBook market, or they’re trying to sabotage it with a platform so shitty, so mind-bogglingly difficult for even tech-savvy consumers, that no one will ever want to read an eBook ever again.

That’s right, sometimes you have a product so bad, that it doesn’t just leave a bad taste in your mouth, it actually does harm to the industry. And that’s what we have with Adobe.

So Adobe, what did eBooks ever do to you?

Filed under Always Be Learning, Dick Move

Dare To Be Stupid

October 3, 2011 2 Comments

I was fortunate to be a guest again on the Packet Pusher’s Podcast recently, and one of the topics was an audience question regarding how to keep up with all that’s going on in the networking world. The group as a whole came up with some great insights, but I thought this would also make a great blog post.

Depending on your point of view, it can either be an exciting time or a terrifying time to be in data center networking. Here’s a small list of all the new stuff that you’re likely going to have to be familiar with: LISP, OTV, SPB, Fabric Path, TRILL, FCoE, FCoTR, BSP, IPv6, IS-IS, VXLAN, NVGRE, NPV, NPIV, EVB, as well as technologies that have been around for a little while but are much more prominent in a networker’s life such as iSCSI and Fibre Channel. And that’s just the data center. With campus and enterprise networking, you’ve got VOIP, unified communications, MPLS, VPLS, metro Ethernet, and more.

*BSP: The Bullshit Protocol. Used to see if you’re paying attention.

So how do you keep up with all this? I’ll admit, it can be a bit overwhelming. But the answer comes from the timeless wisdom of Weird Al Yankovic: Dare to be stupid.

One of the greatest mistakes I see people making in IT is that they stop learning. This is a common folly, and it never ends well. I know this because this is a mistake I’ve made big time. Let’s take the wayback machine to the late ’90s, early 2000’s.

This was a period in my career where I thought I was hot shit. In the late 90s and early 2000s, I was an expert in load balancing, and everyone who wanted to know information about load balancing came to me. I was Mr. Load Balancer.

We all have an inner one of these

But there were huge, huge gaps in my knowledge. Gaps in networking, gaps in system administration, and gaps in my HTTP knowledge. During the heyday of the First Great Internet Bubble, technical talent was a scarce and precious resource, and anyone with experience and skills did very, very well. It made for a great living, but the downside was that it made it very easy to ignore skills gaps, and ignore those gaps I did. I thought that because I was hot shit, that I didn’t need to spend too much time learning. I didn’t dare be stupid.

But it caught up with me. I did a telephone interview with a load balancing vendor, and I got ripped to shreds. They found the gaping holes in my knowledge easily, and it was quite a humbling experience. Initially I was angry, and I thought they were being overly pedantic (something I still dislike). But it wasn’t the IP header overhead of an unlaiden swallow that I didn’t know, it was core concepts that I didn’t know.

It took a while, but my ego healed enough to realize I had a problem: I had to get my shit together. They were right to rip me to shreds (they were nice about it, but having large areas of ignorance in an area you thought you knew well is fairly unpleasant).

Moral of the story? Don’t rest on your laurels, and dare to be stupid. Otherwise, it will be your undoing. And if you’ve been too chicken to be stupid, it’s not too late. I eventually got my shit together. When I started my tract to become a Cisco Certified Systems Instructor (CCSI), I confronted those huge gaps head on, and it was humbling. On my first attempt at the CCNA, I failed so badly that I thought Johns Chambers was going to get a phone call. I thought I was good at networking, but I couldn’t even do proper subnetting. (Like most sysadmins, if it wasn’t a class C subnet, 255.255.255.0, I was completely lost.)

What the fuck does 255.255.255.224 mean?

Eventually I learned subnetting, networking, and filled in the gaps. And I know what 255.255.255.224 means. So always be learning. And a trick I’ve used to continually learn is to learn something not related to computers. You’d be amazed at the insights you can get from learning a completely unrelated skill. For instance, in the past 5 years I’ve learned how to scuba dive, fly a plane, and ball room dance. Each one of those gave me incredible insights into how I learn. Keep at it.

The Magic Words

The three magic words in IT are also among the most painful to say: “I don’t know”. That’s especially true for me, an IT instructor. I’m supposed to know the answer, but I don’t always do. So saying “I don’t know” is quite painful.

In IT, knowledge is our currency, and ignorance is poverty. So it’s really tough to admit ignorance. But it’s important to fight that urge, and say the words “I don’t’ know”.

Even with that motto, part of me still cringes when those words escape my lips. I have a confession to make: During the most recent podcast I was on, Ethan asked me if I knew about vPC with the Nexus 2000 FEX. My response was “It’s been so long since I taught Nexus 7000”. That was basically me being too much of a chicken shit to say “I have no frakkin’ clue.”

Don’t Be An Asshole

Have you ever worked with someone who made you feel small? Where they seem to take delight in showing you how you fucked up? Do they take delight in highlighting your ignorance? Someone who enjoys a good gotcha?

Fuck those people.

Also, stay away from them. Avoid them like the plague. They create environments that are not conducive to learning. Learning is filling in the gaps of knowledge, and it’s tougher to do that when you don’t feel safe to admit you don’t know the answer.

I used to work with a guy like that back in 1998. I was a green Unix administrator whippersnapper, and there was a senior admin who used his powers for evil. He would lord his knowledge over us lesser experienced people. It was a hostile environment for growing. It backfires on them, however, since they stop growing too. They’ll be stuck at their skill level, because they’ll avoid areas where they aren’t the smartest person in the room. They don’t dare to be stupid.

And for Kirk’s sake, don’t be one of those people. Don’t be an asshole, be a teacher. If someone has a lesser level of knowledge on a subject, don’t berate them, don’t lord it over them, help them understand. Want to know how well you know a subject? Explain it to someone who ins’t familiar. You’ll figure out a topic much more comprehensively to that. That’s one of the secrets of blogging, you learn more about a subject simply by writing about it and organizing your thoughts on it (and coming up with clever pictures and captions).

Pull A Superman 2

I’m fortunate enough to have been invited to be a delegate for Network Field Day 2. If you’re not familiar with Network Field Day, it’s a networking-oriented offshoot of Tech Field Day, the brain child of Stephen Foskett, storage expert extrordinarre (check out his great talk on iSCSI and FCoE). If you want to keep up with the future of IT developments, whether it’s storage, networking, or virtualization, pay attention to Tech Field Day and its offshoots. The companies that present (for the most part) aren’t pitching old ideas, they’re pitching what’s next. (For instance, Fsck It! We’ll Do It All in SSDs!)

When I take a look at the other delegates for the upcoming Network Field Day 2, I can only come to one conclusion: I’m not worthy.

Ivan Pepelnjak, Greg Ferro, Ethan Banks, Tom Hollingsworth, Brandon Carrol, (along with my fellow former condescending Unix administrator Mrs Y.) these some of the smartest, most experienced people in networking. And they love to share. I’m not at their level, and I’m likely going to embarrass myself. But I’m going anyway, because it’s a great opportunity to soak up as much knowledge from them as I can. I’m even preparing my own Superman 2 chamber, where I can steal their powers and abilities. And I’m doing it by daring to be stupid.

Surround yourself with people who know more than you, and like sharing that knowledge. You’ll naturally soak up their power.

So if you want to increase your kung fu, learn all the things, and bring out your inner “fuck yea”, then dare to be stupid.

Filed under Always Be Learning, Certifications

SSL: Who Do You Trust?

September 25, 2011 1 Comment

Note: This is a post that appeared on the site lbdigest.com about a year or so ago, but given that SSL is back in the news lately, I figured it’s worth updating and re-posting. Also, it features the greatest SSL diagram ever created. Seriously, if you fire up Visio/Omnigraffle, know the best you can hope for is second place.

One of the most important technologies used on Internet is the TLS/SSL protocol (typically called just SSL, but that’s a whole different article). The two benefits that TLS/SSL gives us are privacy and trust.

Privacy comes through the use of digital encryption (RSA, AES, etc.) to keep your various Internet interactions, such as credit card numbers, emails, passwords, saucy instant messages, confidential documents, etc., safe from prying eyes. That part is pretty well understood by most in the IT industry.

But having private communications with another party is all for naught if you’re talking to the wrong party. You also need trust. Trust that someone is who they say they are. For Internet commerce to work on a practical level, you need to able to trust that when you’re typing your username and password into your bank’s website, that you’re actually connecting to a bank, and not someone pretending to be your bank.

Trust is accomplished through the use of SSL certificates, CAs (certificate authorities), intermediate certificates, and certificate chains which combined is known as PKI (Public Key Infrastructure). To elaborate on the use of these technologies to provide trust, I’m going to forgo the traditional Bob and Alice encryption examples, and go for something a little closer to your heart. I’m going to drop some Star Trek on you.

Let’s say you’re in the market for a starship. You’re looking for a sporty model with warp drive, heated seats, and most importantly, a holodeck. You go to your local Starfleet dealer, and you find this guy.

Ensign Tony.

Seriously Tony, have you even talked to a girl?

The problem is, you don’t trust this guy. It’s nothing personal, but you just don’t know him. He says he’s Ensign Tony, but you have no idea if it’s really him or not. But there is one Starfleet officer you do know and trust implicitly, even though you never met him. You trust Captain Jean-Luc Picard.

Picard’s Law: Set up a peace conference first, ask questions later

Captain Picard is the kind of guy you start out automatically trusting. His reputation precedes him. Your browser is the same way, in that right out of the gate there are several sources (such as Verisign) that your browser trusts implicitly.

If you want to check out who your browser trusts, you can typically find it somewhere in the preferences section. For example, in Google Chrome, go into Preferences, and then Under the Hood. On a Mac this opens up the system-wide keystore for all the trusted certificates and keys. In other operating systems and/or browsers, you may have different certificate stores for different browsers, or like with a Mac all the programs may share a single centralized certificate store.

Back to the Star Trek analogy.

But you’re not dealing with Picard directly. Instead, you’re dealing with Ensign Tony. So Picard vouches for Ensign Tony, and thus a trust chain is built. You trust Picard, and Picard trusts Ensign Tony, so by the transitive property, you can now trust Ensign Tony.

That is the essence of trust in SSL.

Intermediate Certificates

One of the lesser understood concepts in the us of SSL certificates is the intermediate certificates. These are certificates that sit between the CA (Picard) and the site certificate (Ensign Tony).

You see, Picard is an important man. The Enterprise has over a thousand crew members and he can’t possibly personally know and trust all of them. (In Ensign Tony’s case, there’s also the little matter of a restraining order.) So he farms the trust out to his subordinates. And one crew member he does implicitly trust is Chief Engineer Geordi La Forge.

Ensign Tony works for Geordi, and Geordi trusts Ensign Tony. Thus Geordi becomes the intermediate certificate in this chain of trust. You can’t trust Ensign Tony directly through Picard because Picard can’t vouch for Tony, but Geordi can vouch fro Tony, and Picard can vouch for Geordi, so we have built a chain of trust. This is why load balancers and web servers often require you to install an intermediate certificate.

This is the greatest SSL diagram ever made.

Here’s what happens when you don’t install an intermediate certificate onto your load balancer/ADC/web server:

You’re 34 years old Tony, you’d think you would have made Lieutenant by now

One of the practical issues that comes up with intermediate certificates is which one do you use? The various SSL certificate vendors such as Thawte, Digicert, and Verisign have several intermediate certificates depending on the type of certificate you purchase. Sometimes it’s not always obvious. If you have any doubts, use one of the SSL certificate validation tools from the various vendors , including this one by Digicert. It will tell you if the certificate chain works or not. Do not let a test from your browser determine whether your certificate works. Browsers handle certs differently, and a validation tool will tell you if it will work with all browsers.

Filed under Always Be Learning, HTTP, Security

← Older posts

Newer posts →