Run a Cisco ACE? Then Do This Command Right Now!
July 2, 2011 5 Comments
It may already be too late! OK, it’s not too late, but there’s a common scenario I run into with Cisco ACE load balancers. Around 25% of the ACE load balancers (4710 appliance and Service Module) have this condition called STANDBY_COLD.
So here’s a command you should run when logged into the Admin context of your redundant ACE deployment:
show ft group detail
You’re looking for “Peer State” to stay STANDBY_HOT. STANDBY_HOT is good, and you don’t need to do anything else. However, it’s very common to see something else:
FT Group : 1 Configured Status : in-service Maintenance mode : MAINT_MODE_OFF My State : FSM_FT_STATE_ACTIVE Peer State : FSM_FT_STATE_STANDBY_COLD Peer Id : 1 No. of Contexts : 1
STANDBY_COLD is a peer state where the standby ACE context is not receiving automatic configuration syncs from the active ACE. If you had a failover right now with the status of STANDBY_COLD, you would be running on an older version of the configuration, potentially months old.
How Did We Get Here?
When you make a configuration change on the primary ACE, it DOES get automatically copied automatically to the standby ACE.
When you upload a certificate and key to the primary ACE, it DOES NOT get automatically copied to the standby.
The problem is typically that the configuration on the standby ACE references a key and certificate file that don’t exist on the standby, only the active. The standby ACE looks for the files, can’t find them, then stops accepting configuration updates.
How Do We Fix It?
The fix is to upload manually all of the certificates and keys to the standby ACE that were referenced in the configuration. You can import them into the ACE with the
crypto import command through either terminal (cut and paste in the SSH/Telnet window), SFTP, TFTP, or FTP.
Then, reboot the standby. To fix STANDBY_COLD you need to reboot. It will do a fresh configuration sync (it might take a few minutes), but then it should be in STANDBY_HOT again. You’ll need to do this on a context by context basis, as you can have soms contexts in STANDBY_HOT and others in STANDBY_COLD. If it doesn’t fix it, make sure that you’ve got the file names matched exactly.
How Do We Avoid It In The Future?
Keep in mind that when you add SSL certificates and keys, you must add them manually to both the active and standby ACE contexts. So far, no version of the ACE code (that I’m aware of) does certificate and key automatic sync. And make sure to add the files before you put them in the configuration file.