Cisco ACE: Insert Client IP Address

Source-NAT (also referred to as one-armed mode) is a common way of implementing load balancers into a network. It has several advantages over routed-mode (where the load balancer is the default gateway of the servers), most importantly that the load balancer doesn’t need to be Layer 2 adjacent/on the same subnet as the servers.  As long as the SNAT IP address of the load balancer has bi-directional communication with the IP address of the servers, the load balancer can be anywhere. A different subnet, a different data center, even a different continent.

However, one drawback is that with Source NAT the client’s IP address is obscured. The server’s logs will show only the IP address of the SNAT address(s).

There is a way to remedy that if the traffic is HTTP/HTTPS, and that’s by having the load balancer insert the true source IP address into the HTTP request header from the client. You can do it with the ACE by putting it into the load balance policy-map.

policy-map type loadbalance http first-match VIP1_L7_POLICY
  class class-default
     serverfarm FARM1
     insert-http x-forwarded-for header-value "%is"

But alone is not enough. There are two extra steps you need to take.

The first step is you need to tell the web server to log the x-forwarded-for. For Apache, it’s a configuration file change. For IIS, you need to run an ISAPI filter in IIS.

The other thing you need to do is fix the ACE’s attention span. You see, by default the ACE has a short attention span. The HTTP protocol allows you to make multiple HTTP requests on a single TCP connection. By default, the ACE will only evaluate/manipulate the first HTTP request in a TCP connection.

So your log files will look like this:

1.1.1.1 "GET /lb/archive/10-2002/index.htm"
- "GET /lb/archive/10-2003/index.html"
- "GET /lb/archive/05-2004/0100.html HTTP/1.1"
2.2.2.2 "GET /lb/archive/10-2007/0010.html"
- "GET /lb/archive/index.php"
- "GET /lb/archive/09-2002/0001.html"

The “-” indicates Apache couldn’t find the header, because the ACE didn’t insert it. The ACE did add the first source IP address, but every request after it in the same TCP connection was ignored.

Why does the ACE do this? It’s less work for one, only evaluating/manipulating the first request in a connection. Since browsers will make dozens or even hundreds of requests over a single connection, this would be  a significant saving of resources. After all, most of the time when L7 configurations are used, it’s for cookie-based persistence. If that’s the case, all the requests in the same TCP connection are going to contain the same cookies anyway.

How do you fix it? By using a very ill-named feature called persistence-rebalance. This gives the ACE a longer attention span, telling the ACE to look at every HTTP request in the TCP connection.

First, create an HTTP parameter-map.

parameter-map type http HTTP_LONG_ATTENTION_SPAN
  persistence-rebalance

Then apply the parameter-map to the VIP in the multi-match policy map.

policy-map multi-match VIPsOnInterface
  class VIP1
    loadbalance vip inservice
    loadbalance policy VIP1_L7_POLICY
    appl-parameter http advanced-options HTTP_LONG_ATTENTION_SPAN

When that happens, the IP address will show up in all of the log entries.

1.1.1.1 "GET /lb/archive/10-2002/index.htm"
2.2.2.2 "GET /lb/archive/10-2003/index.html"
1.1.1.1 "GET /lb/archive/05-2004/0100.html HTTP/1.1"
2.2.2.2 "GET /lb/archive/10-2007/0010.html"
1.1.1.1 "GET /lb/archive/index.php"
2.2.2.2 "GET /lb/archive/09-2002/0001.html"

But remember, configuring the ACE (or load balancer in general) isn’t the only step you need to perform. You also need to tell the web service (Apache, Nginx, IIS) to use the header as well. None of them automatically use the X-Forwarded-for header.

I don’t know if they’ll try to trick you with this in the CCIE Lab, but it’s something to keep in mind for the CCIE and for implementations.

NPV and NPIV

The CCIE Data Center blueprint makes mention of NPV and NPIV, and Cisco UCS also makes heavy use of both topics, topics that many may be unfamiliar with most. This post (part of my CCIE Data Center prep series) will explain what they do, and how they’re different.

(Another great post on NPV/NPIV is from Scott Lowe, and can be found here. This is a slightly different approach to the same information.)

NPIV and NPV are among the two most ill-named of acronyms I’ve come across in IT, especially since they sound very similar, yet do two fairly different things. NPIV is an industry-wide term and is short for N_Port ID Virtualization, and NPV is a Cisco-specific term, and is short for N_Port Virtualization. Huh? Yeah, not only do they sound similar, but the names give very little indication as to what they do.

NPIV

First, let’s talk about NPIV. To understand NPIV, we need to look at what happens in a traditional Fibre Channel environment.

When a host plugs into a Fibre Channel switch, the host end is called an N_Port (Node Port), and the FC switch’s interface is called an F_Port (Fabric Port). The host has what’s known as a WWPN (World Wide Port Name, or pWWN), which is a 64-bit globally unique label very similar to a MAC address.

However, when a Fibre Channel host sends a Fibre Channel frame, that WWPN is no where in the header. Instead, the host does a Fabric Login, and obtains an FCID (somewhat analagous to an IP addres). The FCID is a 24-bit number, and when FC frames are sent in Fibre Channel, the FCID is what goes into the source and destination fields of the header.

Note that the first byte (08) of the FCID is the same domain ID as the FC switch that serviced the host’s FLOGI.

In regular Fibre Channel operations, only one FCID is given per physical port. That’s it. It’s a 1 to 1 relationship.

But what if you have an ESXi host, for example, with virtual fibre channel interfaces. For those virtual fibre channel interfaces to complete a fabric login (FLOGI), they’ll need their own FCIDs. Or, what if you don’t want to have a Fibre Channel switch (such as an edge or blade FC switch) go full Fibre Channel switch?

NPIV lets a FC switch give out multiple FCIDs on a single port. Simple as that.

The magic of NPIV: One F_Port gives out multiple FCIDs on (0x070100 to the ESXi host and 0x070200 and 0x070300 to the virtual machines)

NPV: Engage Cloak!

NPV is typically used on edge devices, such as a ToR Fibre Channel switch or a FC switch installed in a blade chassis/infrastruture. What does it do? I’m gonna lay some Star Trek on you.

NPV is a clocking device for a Fibre Chanel switch.

Wait, did you just compare Fibre Channel to a Sci-Fi technology? 

How is NPV like a cloaking device? Essentially, an NPV enabled FC switch is hidden from the Fibre Channel fabric.

When a Fibre Channel switch joins a fabric, it’s assigned a Domain_ID, and participates in a number of fabric services. With this comes a bit of baggage, however. See, Fibre Channel isn’t just like Ethernet. A more accurate analogue to Fibre Channel would be Ethernet plus TCP/IP, plus DHCP, distributed 802.1X, etc. Lots of stuff is going on.

And partly because of that, switches from various vendors tend not to get a long, at least without enabling some sort of Interopability Mode. Without interopability mode, you can’t plug a Cisco MDS FC switch into say, a Brocade FC switch. And if you do use interopability mode and two different vendors in the same fabric, there are usually limitations imposed on both switches. Because of that, not many people build multi-vendor fabrics.

Easy enough. But what if you have a Cisco UCS deployment, or some other blade system, and your Fibre Channel switches from Brocade? As much as your Cisco rep would love to sell you a brand new MDS-based fabric, there’s a much easier way.

Engage the cloaking device.

(Note: NPV is a Cisco-specific term, while other vendors have NPV functionality but call it something else, like Brocade’s Access Gateway.) A switch in NPV mode is invisible to the Fibre Channel fabric. It doesn’t participate in fabric services, doesn’t get a domain ID, doesn’t do fabric logins or assign FCIDs. For all intents and purposes, it’s invisible to the fabric, i.e. cloaked. This simplifies deployments, saves on domain IDs, and lets you plug switches from one vendor into a switch of another vendor.  Plug a Cisco UCS Fabric Interconnect into a Brocade FC switch? No problem. NPV. Got a Qlogic blade FC switch plugging into a Cisco MDS? No problem, run NPV on the Qlogic blade FC switch (and NPIV on the MDS).

The hosts perform fabric logins just like they normally would, but the NPV/cloaked switch passes FLOGIs up to the NPIV enabled port on the upstream switch. The FCID of the devices bears the  the Domain ID of the upstream switch (and appears directly attached to the upstream switch).

The NPV enabled switch just proxies the FLOGIs up to the upstream switch. No fuss, no muss. Different vendors can interoperate, we save domain IDs, and it’s typically easier to administer.

TL;DR: NPIV allows multiple FLOGIs (and multiple FCIDs issued) from a single port. NPV hides the FC switch from the fabric.

Common ACE Gotchas

OK, so the Cisco ACE is not my favorite load balancer. It’s certainly not my go-to load balancer when clients come a-callin. It lacks many features that its competitors have, and market share-wise it’s getting its ass handed to it by F5. (And honestly? Deservedly so.)  Also, that blue light will seer your retinas like a fine tofu steak.

This is what the ACE 4710 Appliance’s blue light can do if you’re not careful

But with the Cisco ACE being part of the CCIE Data Center track, it’s kind of like a distant relative who won the lottery: I suddenly find it more interesting.

So my thoughts turn to the CCIE Data Center lab test, and trying to figure out what they could test me on, and I started to think of some of the common gotchas I see in the field in terms of ACE configuration, so I’ve listed a few of them here.

HTTP Health Probes Failed (With Healthy Server)

With the ACE software (including the latest 5.x code) there’s a requirement for an HTTP/HTTPS-based health probe that often trips people up: HTTP status code (min/max). HTTP responses have a status code associated with them, ranging from 200 (OK) to 404 (Not Found) or even 418 “I’m a teapot”. (No seriously, it’s a valid code.) Most load balancers will accept 200 by default, and fail on something 400 or above. But not the ACE. You must explicitly configure what the ACE will accept, or it will accept nothing.

When you configure a health probe in the ACE, it has defaults for URL (/), timeout (10 seconds), and interval (15 seconds). But there’s no default for “expect status”.  If you don’t set it, all health probes will fail, even though the server is perfectly healthy.

No. Probes skipped : 0 Last status code : 200
No. Out of Sockets : 0 No. Internal error: 0
Last disconnect err : Received invalid status code

As you can see in the show probe detail above, the last HTTP response was a 200 (which is good), but the ACE still considered that an invalid status code. The solution is to set the status.

probe http HTTP_PROBE1
 expect status 200 200

Once you do that, it will take the passdetect interval and timeout to mark the server up (default is 1m30s, since the interval and timeout values when a server is down can be different than when a server is considered up). You can have it do it faster by disabling and re-enabling the probe.

Cold Standby

When you upload a certificate and key to the ACE, you typically do so on the active ACE in a HA pair. What a lot of people don’t realize is that you need to upload the certificate and key on both ACEs, and you need to do so before you reference the key and certificate in the configuration file.

If you put the key and certificate in the configuration file, and it doesn’t exist on both the active and standby ACE, the standby goes into a mode called COLD_STANDBY. With COLD_STANDBY, the standby ACE will take over in the even of the active ACE going down, however it no longer accepts configuration updates from the active ACE. You can still failover, but the config could be months old.

  1. Upload certificate and key to active ACE
  2. Upload certificate and key to standby ACE
  3. Go into config mode, setup the ssl-proxy referencing the key and cert.

What happens a lot of the time is people do this:

  1. Upload certifiate and key to active ACE
  2. Go into config mode, setup the ssl-proxy referencing the key and cert
If you don’t do step 2 before step 3, then the ACE will go into STANDBY_COLD mode.

If you’re in STANDBY_COLD (many are and don’t even realize it), make sure all referenced certificates and keys are uploaded to both ACEs, then reboot the standby box or run the command no ft auto-sync run followed by ft auto-sync run. When it comes up, it’ll sync again, and you should be in STANDBY_HOT, which is what you want.

You Forgot the Intermediate Certificate

This isn’t an ACE thing, this is a load balancing thing. As I outlined in my article on SSL and trust, many CAs (including Verisign) require an intermediate certificate in addition to the server certificate you obtain. Both the server and intermediate certificate need to be installed on the ACE (or other load balancer) for the certificate chain to be complete.

This is an often missed step, as it’s not always obvious from the CA if you need one or not, and even when it is explicitly stated, they don’t often tell you which is the right intermediate (some CAs have several to choose from).

Cert Expired: Health Checks Fail

If you’re doing health check against and HTTPS device, and the certificate (whether self signed or certificate authority-signed) has expired, the health checks will fail. Not matter what. So make sure your back end servers don’t have an expired cert.

That’s all I can think of for now. Feel free to post questions or other possible gotchas in the comments section.