Last month (with little fanfare) Cisco released 5(1.0) for the ACE 4710 appliance and ACE30 Service Modules, bringing IPv6 support for the first time.
Wait, what?
IPv6 was around when we were partying like it was… 1999
Yes, September of 2011, and Cisco’s load balancing platform finally gets IPv6. It’s a dual-stack implementation for free, and with an extra license fee, you can get the protocol translation (IPv6 VIP with an IPv4 server as the most common example) as well. Honestly, I’m not sure why Cisco decided to charge extra for the NAT64, since IPv6 is pretty much useless on load balancers without that ability. F5, A10, and several other load balancing vendors don’t charge for the IPv6/4 translation component. Also, the ACE10 and ACE20 service modules (the later which has a pretty large install base) will never have IPv6 support. (Cisco has an aggressive pricing plan for ACE10/20 to ACE30 upgrades).
So why are IPv6 load balancers worthless without 6/4 conversion? It’s very likely that web applications servers will be among the laggards in the transition from IPv4 to IPv6. You’ll pry IPv4 out of their cold, dead, unpatched hands. The 6/4 conversion allows you to setup an IPv6 VIP to communicate with the future Internet, while the servers run their familiar IPv4.
Honestly, I’m very underwhelmed by the Cisco ACE product line lately. They’re pretty far behind the competition (F5, A10, Citrix NetScaler, Radware) in terms of features, and Cisco doesn’t seem to be doing much about it. Don’t get me wrong, it’s fine for what it does. But other companies are innovating, and Cisco seems to be content with letting the ACE lineup stagnate, just like they did with the LocalDirector and the CSS. I’d like to see Cisco up their game with true content logic (like F5’s iRules). But considering Cisco discontinued their line of XML Gateways/Web Application Firewalls, it seems pretty unlikely they will.
Traffic control languages like iRules are double edged swords: They can solve a lot of problems, but they can also create a lot of problems when trying to solve problems. I’ve seen them save the day, and I’ve seen them consume an entire network department in a DevOps nightmare worthy of DevOps Borat. Still, I’d rather have it, than not.
Aright implementors of services that utilize TLS/SSL, shit just got real. TLS 1.0/SSL 3.0? Old and busted. TLS 1.2? New hotness.
We config together, we die together. Bad admins for life.
There’s an exploit for SSL and TLS, and it’s called BEAST. It takes advantage of a previously known (but though to be too impractical to exploit) weakness in CBC. Only BEAST was able to exploit that weakness in a previously unconsidered way, making it much more than a theoretical problem. (If you’re keeping track, that’s preciously the moment that shit got real).
The cure is an update to the TLS/SSL standard called TLS 1.2, and it’s been around since 2008 (TLS 1.1 also fixes it, and has been available since 2006, but we’re talking about new hotness here).
So no problem, right? Just TLS 1.2 all the things.
Well, virtually no one uses it. It’s a chicken and egg problem. Clients haven’t supported it, so servers haven’t. Servers didn’t support it, so why would clients put the effort in? Plus, there wasn’t any reason to. The CBC weakness had been known, but it was thought to be too impractical to exploit.
But now we’re in a state of shit-is-real, so it’s time to TLS up.
So every browser and server platform running SSL is going to need to be updated to support TLS 1.2. On the client side, Google Chrome, Apple Safari, Firefox, IE (although IE 9 supports TLS 1.1, but previous version will need to be back ported) will need to be updated.
On the server side, it might be a bit simpler than we think. Most of the time when we connect to a website that utilizes SSL (HTTPS), the client isn’t actually talking SSL to the server, instead they’re talking to a load balancer that terminates the SSL connection.
Since most of the world’s websites have a load balancer terminate the SSL, we can update the load balancers with TLS 1.2 and take care of a major portion of the servers on the Internet.
Right now, most of the load balancing vendors don’t support TLS 1.2. If asked, they’ll likely say that there’s been no demand for it since clients don’t support it, which was fine until now. Now is the time for the various vendors to upgrade to 1.2, and if you’re a vendor and you’re not sure if it’s worth the effort, listen to Yoda:
Right now the only vendor I know of that supports TLS 1.2 is the market leader F5 Networks with their version 11 of their LTM, for which they should be commended. However, that’s not good enough, they need to backport version 10 (which has a huge install base). Vendors like Cisco, A10 Networks, Radware, KEMP Technologies, etc., need to also update their software to TLS 1.2. We can no longer use the excuse “because browsers don’t support it”. Because of BEAST, they will soon, and so do they.
In the meantime, if you’re running a load balancer that terminates SSL, you may want to change your cipher settings to prefer RC4-SHA instead of AES (which uses CBC). It’s cryptographically weaker, but is immune to the CBC issue. In the next few days, I’ll be putting together a page on how to prefer RC4 for the various vendors.
Rembmer, TLS 1.0/SSL 3.0: Old and busted. TLS 1.2? New hotness.
Between OCSP and CRLs, browser updates and rogue certificates, it’s called into question the very fabric of trust that we’ve taken for granted. Some even claim that PKI is totally broken (and there’s a reasonable argument for this).
This is what happens when there’s no trust. Also, this is how someone loses an eye. Do you want to lose an eye? Because this will totally do it.
Then someone found a way to straight up decrypt an SSL connection without any of the keys.
Wait, what?
It’s getting’ all “hack the planet” up in here.
The exploit is called BEAST, and it’s one that can decrypt SSL communications without having the secret. Thai Duong, one of the authors (the other is Juliano Rizzo) of the tool saw my post on BEAST, and invited me to watch the live demonstration from the hacker conference. Sure enough, they could decrypt SSL. Here’s video from the presentation:
Let me say that again. They could straight up decrypt that shit.
Granted, there were some caveats, and the exploit can only be used in a in a somewhat limited fashion. It was a man-in-the-middle attack, but one that didn’t terminate the SSL connection anywhere but at the server. They found a new way to attack a known (but thought to be too impractical to exploit) vulnerability in the CBC part of some encryption algorithms.
The security community has known this was a potential problem for years, and it’s been fixed in TLS 1.1 and 1.2.
Wait, TLS? I thought this was about SSL?
Quick sidebar on SSL/TLS. SSL is the old term (the one everyone still uses, however). TLS replaced SSL, and we’re currently on TLS 1.2, although most people use TLS 1.0.
And that’s the problem. Everyone, and I do mean the entire planet, uses SSL 3.0 and TLS 1.0. TLS 1.1 has been around since 2006, and TLS 1.2 has been around since 2008. But most web servers and browsers, as well as all manner of other types of SSL-enabled devices, don’t use anything beyond TLS 1.0.
And, here’s something that’s going to shock you:
Microsoft IIS and Windows 7 support TLS 1.1. OpenSSL, the project responsible for the vast majority of SSL infrastructure used by open source products (and much of the infrastructure for closed-source projects), doesn’t. As of writing, TLS 1.1 or above hasn’t made it yet into OpenSSL libraries, which means Apache, OpenSSH, and other tools that make use of the OpenSSL libraries can’t use anything above TLS 1.0. Look at how much of a pain in the ass it is right now to enable TLS 1.2 in Apache.
We’re into double-facepalm territory now
No good, very bad month indeed.
So now we’re going to have to update all of the web servers out there, as well as all the clients. That’s going to take a bit of doing. OpenSSL runs the SSL portion of a lot of sites, and they’ve yet to bake TLS 1.1/1.2 into the versions that everyone uses (0.9.8 and 1.0.x). Load balancers are going to play a central role in all of this. so we’ll have to wait for F5, Cisco, A10 Networks, Radware, and others to support TLS 1.2. As far as I can tell, only F5’s LTM version 11 supports anything above TLS 1.0.
The tougher part will be all of the browsers out there. There are a lot of systems that run non-supported and abandoned browsers. At least SSL/TLS is smart enough to be able to downgrade to the highest common denominator, but that would mean potentially vulnerable clients.
This highlights the importance of keeping your software, both clients and servers, up to date. Out of convenience we got stale with SSL, and even security the security obsessed OpenSSL project got caught with their pants down. This is really going to shake a lot of systems down, and hopefully be a kick in the pants to those that think it’s OK to not run current software.
I worked at an organization once where the developers forked off Apache (1.3.9 I believe) to interact with an application they developed. This meant that we couldn’t update to the latest version of Apache, as they refused to put the work in to port their application module to the updated versions of Apache. I bet you can guess how this ended. Give up? The Apache servers got p0wned.
So between BEAST and PKI problems, SSL/TLS has had a rough go at it lately. It’s not time to resort to smoke signals just yet, but it’s going to take some time before we get back to the level of confidence we once had. It’s a scary-ass world out there. Stay secure, my friends.
It looks like Riverbed is making a move into F5’s territory by acquiring Zeus. F5 has WOC (WAN optimization controllers) technology, but the last time I took a look at them (admittedly several years ago) they were pretty terrible. However F5 has the market leader for load balancing. Riverbed is known as the leader in WOCs, but until now had no load balancing capabilities.
Strong WOCs, new load balancing. F5 has strong load balancing, iffish WOC. Is Riverbed looking to be F5 with a goatee?
Meet Riverbed
Everyone I’ve ever talked to loves the Riverbed WOCs. Their primary competitors on this space have been Cisco (with WAAS), Bluecoat, F5 (used to have WanJets, now they’re modules in the LTMs) and a handfull of others. But they haven’t had a load balancer, until now.
It makes sense, the WOC and load balancer spaces both live in that tricky realm of Layer 7 network devices. It is a realm where many network admins dare not tread. It’s tough enough keeping STP, OSPF, ISIS, let alone all the new stuff like FCoE, TRILL, DCBX, and so forth without it all bursting one’s cranial cavity like an over-ripened fruit, these products throw all those Layer 7 protocols (HTTP, IMAP, POP3, Exchange, etc.) into the mix.
To be honest, I haven’t seen a lot of Zeus in the market (while I always see F5). My impression is that, like my favorite band Pop Will Eat Itself, they’re bigger in Europe.
The issue is that it’s easy to make the mistake of not uploading to both. Because the rest of the configuration is sync’d automatically, it’s easy to make the assumption the keys and certs are sync’d too. When you import the certs and keys there’s no warning that tells you that you should upload to both the active and standby.
And if you did make that mistake, there’s no red light that says “hey, I’m in STANDBY_COLD!”. If you don’t know to look for it, you could be in it and not know it. Also, STANDBY_COLD isn’t exactly a descriptive error state.
So it’s not a bug per say, but it is a bit tricky. It’s more of an operational bug. There should be more of an indication that the system is in STANDBY_COLD. That’s why many of my students are surprised to find that they are, in fact, in STANDBY_COLD and didn’t know it.
It may already be too late! OK, it’s not too late, but there’s a common scenario I run into with Cisco ACE load balancers. Around 25% of the ACE load balancers (4710 appliance and Service Module) have this condition called STANDBY_COLD.
The horror of STANDBY_COLD
So here’s a command you should run when logged into the Admin context of your redundant ACE deployment:
show ft group detail
You’re looking for “Peer State” to stay STANDBY_HOT. STANDBY_HOT is good, and you don’t need to do anything else. However, it’s very common to see something else:
FT Group : 1
Configured Status : in-service
Maintenance mode : MAINT_MODE_OFF
My State : FSM_FT_STATE_ACTIVE
Peer State : FSM_FT_STATE_STANDBY_COLD
Peer Id : 1
No. of Contexts : 1
STANDBY_COLD is a peer state where the standby ACE context is not receiving automatic configuration syncs from the active ACE. If you had a failover right now with the status of STANDBY_COLD, you would be running on an older version of the configuration, potentially months old.
How Did We Get Here?
When you make a configuration change on the primary ACE, it DOES get automatically copied automatically to the standby ACE.
When you upload a certificate and key to the primary ACE, it DOESNOT get automatically copied to the standby.
The problem is typically that the configuration on the standby ACE references a key and certificate file that don’t exist on the standby, only the active. The standby ACE looks for the files, can’t find them, then stops accepting configuration updates.
How Do We Fix It?
The fix is to upload manually all of the certificates and keys to the standby ACE that were referenced in the configuration. You can import them into the ACE with the crypto import command through either terminal (cut and paste in the SSH/Telnet window), SFTP, TFTP, or FTP.
Then, reboot the standby. To fix STANDBY_COLD you need to reboot. It will do a fresh configuration sync (it might take a few minutes), but then it should be in STANDBY_HOT again. You’ll need to do this on a context by context basis, as you can have soms contexts in STANDBY_HOT and others in STANDBY_COLD. If it doesn’t fix it, make sure that you’ve got the file names matched exactly.
How Do We Avoid It In The Future?
Keep in mind that when you add SSL certificates and keys, you must add them manually to both the active and standby ACE contexts. So far, no version of the ACE code (that I’m aware of) does certificate and key automatic sync. And make sure to add the files beforeyou put them in the configuration file.
IPv6 was around when we were partying like it was… 1999
The Data Center Overlords 