Creating Your Own SSL Certificate Authority (and Dumping Self Signed Certs)

Jan 11th, 2016: New Year! Also, there was a comment below about adding -sha256 to the signing (both self-signed and CSR signing) since browsers are starting to reject SHA1. Added (I ran through a test, it worked out for me at least).

November 18th, 2015: Oops! A few have mentioned additional errors that I missed. Fixed.

July 11th, 2015: There were a few bugs in this article that went unfixed for a while. They’ve been fixed.

SSL (or TLS if you want to be super totally correct) gives us many things (despite many of the recent shortcomings).

  • Privacy (stop looking at my password)
  • Integrity (data has not been altered in flight)
  • Trust (you are who you say you are)

All three of those are needed when you’re buying stuff from say, Amazon (damn you, Amazon Prime!). But we also use SSL for web user interfaces and other GUIs when  administering devices in our control. When a website gets an SSL certificate, they typically purchase one from a major certificate authority such as DigiCert, Symantec (they bought Verisign’s registrar business), or if you like the murder of elephants and freedom, GoDaddy.  They range from around $12 USD a year to several hundred, depending on the company and level of trust. The benefit that these certificate authorities provide is a chain of trust. Your browser trusts them, they trust a website, therefore your browser trusts the website (check my article on SSL trust, which contains the best SSL diagram ever conceived).

Your devices, on the other hand, the ones you configure and only your organization accesses, don’t need that trust chain built upon the public infrastrucuture. For one, it could get really expensive buying an SSL certificate for each device you control. And secondly, you set the devices up, so you don’t really need that level of trust. So web user interfaces (and other SSL-based interfaces) are almost always protected with self-signed certificates. They’re easy to create, and they’re free. They also provide you with the privacy that comes with encryption, although they don’t do anything about trust. Which is why when you connect to a device with a self-signed certificate, you get one of these: So you have the choice, buy an overpriced SSL certificate from a CA (certificate authority), or get those errors. Well, there’s a third option, one where you can create a private certificate authority, and setting it up is absolutely free.


OpenSSL is a free utility that comes with most installations of MacOS X, Linux, the *BSDs, and Unixes. You can also download a binary copy to run on your Windows installation. And OpenSSL is all you need to create your own private certificate authority. The process for creating your own certificate authority is pretty straight forward:

  1. Create a private key
  2. Self-sign
  3. Install root CA on your various workstations
Once you do that, every device that you manage via HTTPS just needs to have its own certificate created with the following steps:
  1. Create CSR for device
  2. Sign CSR with root CA key
You can have your own private CA setup in less than an hour. And here’s how to do it.

Create the Root Certificate (Done Once)

Creating the root certificate is easy and can be done quickly. Once you do these steps, you’ll end up with a root SSL certificate that you’ll install on all of your desktops, and a private key you’ll use to sign the certificates that get installed on your various devices.

Create the Root Key

The first step is to create the private root key which only takes one step. In the example below, I’m creating a 2048 bit key:

openssl genrsa -out rootCA.key 2048

The standard key sizes today are 1024, 2048, and to a much lesser extent, 4096. I go with 2048, which is what most people use now. 4096 is usually overkill (and 4096 key length is 5 times more computationally intensive than 2048), and people are transitioning away from 1024. Important note: Keep this private key very private. This is the basis of all trust for your certificates, and if someone gets a hold of it, they can generate certificates that your browser will accept. You can also create a key that is password protected by adding -des3:

openssl genrsa -des3 -out rootCA.key 2048

You’ll be prompted to give a password, and from then on you’ll be challenged password every time you use the key. Of course, if you forget the password, you’ll have to do all of this all over again.

The next step is to self-sign this certificate.

openssl req -x509 -new -nodes -key rootCA.key -sha256 -days 1024 -out rootCA.pem

This will start an interactive script which will ask you for various bits of information. Fill it out as you see fit.

You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
Country Name (2 letter code) [AU]:US
State or Province Name (full name) [Some-State]:Oregon
Locality Name (eg, city) []:Portland
Organization Name (eg, company) [Internet Widgits Pty Ltd]:Overlords
Organizational Unit Name (eg, section) []:IT
Common Name (eg, YOUR name) []:Data Center Overlords
Email Address []

Once done, this will create an SSL certificate called rootCA.pem, signed by itself, valid for 1024 days, and it will act as our root certificate. The interesting thing about traditional certificate authorities is that root certificate is also self-signed. But before you can start your own certificate authority, remember the trick is getting those certs in  every browser in the entire world.

Install Root Certificate Into Workstations

For you laptops/desktops/workstations, you’ll need to install the root certificate into your trusted certificate repositories. This can get a little tricky. Some browsers use the default operating system repository. For instance, in Windows both IE and Chrome use the default certificate management.  Go to IE, Internet Options, go to the Content tab, then hit the Certificates button. In Chrome going to Options and Under The Hood, and Manage certificates. They both take you to the same place, the Windows certificate repository. You’ll want to install the root CA certificate (not the key) under the Trusted Root Certificate Authorities tab. However, in Windows Firefox has its own certificate repository, so if you use IE or Chrome as well as Firefox, you’ll have to install the root certificate into both the Windows repository and the Firefox repository. In a Mac, Safari, Firefox, and Chrome all use the Mac OS X certificate management system, so you just have to install it once on a Mac. With Linux, I believe it’s on a browser-per-browser basis.

Create A Certificate (Done Once Per Device)

Every device that you wish to install a trusted certificate will need to go through this process. First, just like with the root CA step, you’ll need to create a private key (different from the root CA).

openssl genrsa -out device.key 2048

Once the key is created, you’ll generate the certificate signing request.

openssl req -new -key device.key -out device.csr

You’ll be asked various questions (Country, State/Province, etc.). Answer them how you see fit. The important question to answer though is common-name.

Common Name (eg, YOUR name) []:

Whatever you see in the address field in your browser when you go to your device must be what you put under common name, even if it’s an IP address.  Yes, even an IP (IPv4 or IPv6) address works under common name. If it doesn’t match, even a properly signed certificate will not validate correctly and you’ll get the “cannot verify authenticity” error. Once that’s done, you’ll sign the CSR, which requires the CA root key.

openssl x509 -req -in device.csr -CA rootCA.pem -CAkey rootCA.key -CAcreateserial -out device.crt -days 500 -sha256

This creates a signed certificate called device.crt which is valid for 500 days (you can adjust the number of days of course, although it doesn’t make sense to have a certificate that lasts longer than the root certificate). The next step is to take the key and the certificate and install them in your device. Most network devices that are controlled via HTTPS have some mechanism for you to install. For example, I’m running F5’s LTM VE (virtual edition) as a VM on my ESXi 4 host. Log into F5’s web GUI (and should be the last time you’re greeted by the warning), and go to System, Device Certificates, and Device Certificate. In the drop down select Certificate and Key, and either past the contents of the key and certificate file, or you can upload them from your workstation.

After that, all you need to do is close your browser and hit the GUI site again. If you did it right, you’ll see no warning and a nice greenness in your address bar.

And speaking of VMware, you know that annoying message you always get when connecting to an ESXi host?

You can get rid of that by creating a key and certificate for your ESXi server and installing them as /etc/vmware/ssl/rui.crt and /etc/vmware/ssl/rui.key.

Fsck It, We’ll Do It All With SSDs!

At Tech Field Day 8, we saw presentations from two vendors that had an all-flash SAN offering, taking on a storage problem that’s been brewing in data centers for a while now, and the skewed performance/capacity scale.

While storage capacity has been increasing exponentially, storage performance hasn’t caught up nearly that fast. In fact, performance has been mostly stagnant, especially in the area where it counts: Latency and IOPS (I/O Operations Per Second).

It’s all about the IOPS, baby

In modern data centers, capacity isn’t so much of an issue with storage. Neither is the traditional throughput metric, such as megabytes per second. What really counts is IOPS and latency/seek time. Don’t get me wrong, some data center applications certainly have capacity requirements, as well as potential throughput requirements, but for the most part these are easily met by today’s technology.

IOPS and latency are super critical for virtual desktops (and desktops in general) and databases. If you computer is sluggish, it’s probably not a lack of RAM or CPU, by and large it’s a factor of IOPS (or lack thereof).

There are a few tricks that storage administrators and vendors have up their sleeve to increase IOPS and drop latency.

In a RAID array, you can scale IOPS linerally by just throwing more disks at the array. If you have a drive that does 100 IOPS per second, add a second drive for a RAID 0 (mirror) and you’ve got double the IOPS. Add a third and you’ve got 300 IOPS (and of course add more for redundancy).

Another trick that storage administrators have up their sleeve is the technique known as “short stroking“, where only a portion of the drive is used. In a spinning platter, the outside is spinning the fastest, giving the best performance. If you only format that out portion, the physical drive head doesn’t have to travel as far. This can reduce seek time substantially.

Tiered storage can help with both latency and IOPS, were a combination of NVRAM, SSDs, and hard drives are combined.”Hot” data is accessed from high-speed RAM cache, “warm” data is on a bank of SSDs, and “cold” data would be stored on cheaper SAS or (increasingly) consumer SATA drives.

And still our demand for IOPS is insatiable, and the tricks in some cases aren’t catching up. Short stroking only goes so far, and cache misses can really impact performance for tiered storage. While IOPS scale linearly, the IOPS we need can sometimes end up with racks full of spinning rust, while only using a tenth of the actual capacity. That’s a lot of wasted space and wasted power.

And want to hear a depressing fact?

A high-end enterprise SAS 15,000 RPM drive (which spins faster than most jet engines) gives you about 150 IOPS in performance (depending on the workload of course). A good consumer grade SSD from Newegg gives you around 85,000 IOPS. That means you would need almost 600 drives to equal the performance of one consumer grade SSD.

That’s enough to cause anyone to have a Bill O’Reilly moment.

600 drives? Fuck it, we’ll do it with all flash!

No one is going to put their entire database or virtual desktop infrastructure on a single flash drive of course. And that’s where vendors like Pure Storage and SolidFire come into play. (You can see Pure Storage’s presentation at Tech Field Day 8 here. SolidFire’s can be seen here.)

The overall premise with the we’ll-do-it-all-in-flash play is that you can take a lot of consumer grade flash drives, use the shitload of IOPS that they bring, and combine it with a lot of storage controller CPU power for deduplication and compression. With that combination, they can offer an all-flash based array at the same price per gig as traditional arrays comprise of spinning rust (disk drives).

How many IOPS are we talking about? SolidFire’s SF3010 claims 50,000 IOPS per 1 RU node. That would replace over 300 drives of traditional drives, which I don’t think you can put in 1RU. Pure Storage claims 300,000 IOPS in 8U of space. With a traditional array, you’d need over 2000 drives, also unlikely to fit in 8 RU. Also, imagine the power savings, with only 250 watts needed for SoldFire’s node, and 1300 Watts for the PureStorage cluster.  And Both allow you to scale up by adding more nodes.

You wire them into your SAN the traditional ways, as well. The Pure Storage solution has options for 10 Gbit iSCSI and 8 Gbit Fibre Channel, while the SolidFire solution is iSCSI only. (Sadly, neither support FCoE or FCoTR.)

For organizations that are doing virtual desktops or databases, an all-flash storage array with the power savings and monster IOPS must look more tantalizing than a starship full of green-skinned girls does to Captain Kirk.

There is a bit of a controversy in that many of the all-flash vendors will tell you capacity numbers with deduplication and compression taken into account. At the same time, if the performance is better than spinning rust even with the compression/dedupe, then who cares?

So SSD it is. And as anyone who has an SSD in their laptop or desktop will tell you, that shit is choice. Seriously, I get all Charlton Heston about my SSD.

You’ll pry this SSD from my cold, dead hands

It’s not all roses and unicorn-powered SSDs. There are two issues with the all-flash solution thus far. One is that they don’t have a name like NetApp, EMC, or Fujitsu, so there is a bit of a trust issue there. The other issue is that many have some negative preconceptions about flash, such as they have a high failure rate (due to a series of bad firmwares from vendors) and the limited write cycle of memory cells (true, but mitigaitable). Pure Storage claims to have never had a drive fail on them (Amy called them flash driver whispers).

Still though, check them (and any other all-SSD vendor) out. This is clearly the future in terms of high performance storage where IOPS is needed. Spinning rust will probably rule the capacity play for a while, but you have to imagine its days are numbered.

Inexpensive VMware ESXi (vSphere Hypervisor) Host

So you want an ESXi (vSphere Hypervisor) server, but you don’t want to spend several grand on a blade chassis or enterprise-grade server. Perhaps you want an inexpensive server for home use, and something that’s going to be quieter than the jet-engines that cool the big stuff. So what do you do?

Like a lot of people, you’ll hit up a message board or two. And invariably, someone will link you to the site’s whitebox HCL. It’s a good list if you have a pile of hardware and you want to see what works, but if you’re looking for which system to buy or what components to obtain to build your own system, it’s mostly useless.

One route I’m particularly fond of is desktop hardware. It’s the least expensive way to get a virtualization host, and they’re a heck of a lot quieter than most servers, making them appropriate for putting them in places that aren’t a data center.

So I’ve written this guide on how to build/spec/buy your own ESXi system.

First off, ESXi is available for free. With ESXi 4.x, you were limited to 6 cores and 256 GB of RAM. With ESXi 5, there’s no core limit although you’re limited to 32 GB of RAM (that shouldn’t be a problem). The free license lets you run as many VMs as you can stuff in there, although you can’t do any of the fancy features like integrate with vCenter or vMotion and other fun stuff. But for a home lab or basic virtualization environment, that’s no big deal.

The issue with ESXi is that it’s somewhat picky about the hardware it will run on (although it’s improved with ESXi 4 and 5). Most server-grade hardware will run fine, but if you’re looking to use something a little more pedestrian, some of the out-of-the-box components may not work, or may not have the features you need.

System Itself

Whether you’re shopping for a motherboard or a pre-built system, I’ve yet to find a fairly recent mid to high-end system that doesn’t load ESXi. Usually the only thing I’ve had to add is a server-grade NIC (recommendations in the NIC section).

RAM Rules Everything Around Me

For virtualization, RAM is paramount. Get as much RAM as you can. When researching a system, it’s a good idea to make sure it has at least 4 memory DIMM slots. Some of the i7 boards have 6, which is even better. Most 4 DIMM slot motherboards these days can have up to 16 GB of RAM, 6 DIMM slots can do up to 24 GB. Given RAM prices these days, it doesn’t make sense not to fill them to the brim with RAM. I can get 16 GB of good desktop memory for less than $100 USD now.

Haters gonna hate

Also, make sure to get good RAM, and don’t necessarily go for fast RAM, and it’s definitely not a good idea to overclock RAM in a virtualized environment. Remember, this is a virtualization host, not a gaming rig. I tried once using off-brand RAM, and all I got was a face full of purple-screens of death.


For a long time, it seems that processors got all the glory. With virtualization and most virtualization workloads, processors are secondary. Cores are good, speed is OK. RAM is usually the most important aspect. That said, there are a couple of things with processors you want to keep in mind. On the Intel side, new processors such as i5’s or i7’s make good virtualization processor.


The more cores the better. These days, you can easily afford a quad-core system. I wouldn’t worry too much about the core speed itself, especially on a virtualization system. I would recommend putting more emphasis on the number of cores.

Then there’s hyper threading. A processor with hyper threading support will make 4 cores look like 8 to the the operating system. Each core would have two separate execution contexts. VMware makes pretty good use of hyper threading by being able to put a vCPU (what the virtual machine sees as a core) on each context, so get it if you can.

Proof that I’m, like, technical and stuff.

But again, for most VM workloads don’t go for a monster processor if it means you can only afford 4 GBs of RAM. RAM first, then processor.

Here’s where it gets a bit tricky. There are a number of new features that processors may or may not have that will affect your ability to have a functioning ESXi host.

Virtualization Support

Virtually (get it?) every processor has some sort of virtualization support these days.  For Intel, this is typically called VT-x. For AMD processors, this is called AMD-V. You’ll want to double-check, but any processor made in the past five years likely supports these technologies.  While there are ways to do virtualization (paravirtualization) on processors without these features, it’s not nearly as full featured and most operating systems won’t run.

Direct Connection

Some hypervisors allow a virtual machine to control a peripheral directly. VMware calls this DirectPath I/O, and other vendors have other names for it. If you had a SATA drive in your virtualization host, you could present it directly to a particular VM.  You can also do it for USB devices, network interfaces, crypto accelerators, etc.

Keep in mind if you do this, no other VM will be able to access that particular device. Doing DirectPath I/O (or similar) usually prevents vMotion form occuring.

It’s a somewhat popular feature if you’re building yourself a NAS server inside a virtual system, but otherwise it’s not that popular in the enterprise.

Intel calls this technology VT-d, and AMD calls it IOMMU. Not all processors have these features, so be sure to check it out. You may also need to enable this in your system’s BIOS (sometimes it’s not on by default). For instance, my new Dell XPS 8300 has an i5-2300 processor. It does not support VT-d, although the i5-2400 processor does.

For most setups it’s not a big deal, but if you’ve got a choice, get the processor with the VT-d/IOMMU support.


This is a lesser known feature, AES-NI is an Intel-only feature right now (although AMD is supposed to support something like it in the upcoming Bulldozer processor family).

AES-NI first appeared in laptop and server processors, but is now making it’s way into all of Intel’s chips. Essentially, it’s an extra set of processor instructions specifically geared towards AES encryption. If you have software that’s written to specifically take advantage of these instruction sets, you can significantly speed up encyrption operations. Mac OS X Lion for instance uses AES-NI for File Vault 2, as does BitLocker for Windows 7. Best of all, it’s passed down to the guest VMs so they can take advantage of it as well.


VMWare has been notoriously picky about its networking hardware. The built-in Ethernet ports on most desktop-class motherboards won’t cut it. What you’ll likely need is to buy a server-grade network adapter, and with ESXi 4.0 and on, it will have to be Gigabit or better (there’s no more Fast Ethernet support). ESXi won’t even install unless it detects an appropriate NIC.

This has changed somewhat with ESXi 5.0. The ESXi 5.0 installer comes pre-loaded with more Ethernet drivers than its predacessors, and accept more mundane NICs. On my Dell XPS 8300, the built-in BCM57788 Broadcom-based NIC was recognized with the default ESXi 5.0 installer. It was not with 4.1.

If your NIC is not recognized, my favorite go-to NIC for VMware systems is an Intel Pro 1000 NIC. I can pick a single or dual port NIC off eBay for less than $50 usually. Make sure to get the right kind (most modern systems use PCI-e). Make sure it’s the Pro NICs, and not the desktop NICs. I have a couple of Intel Pro 1000 PTs for my ESXi boxen that I got for $40 a piece, and they work great.

We’ll have to see how many more drivers ESXi 5.0 support, but chances are you’ll need to pick up an Intel Pro 1000 NIC if you’re using 4.x with desktop hardware.


Standard SATA drives work fine in ESXi 4 and 5. For a desktop system, it doesn’t make much sense usually to try to put a SAS drive, but it’s your money.

If you’re going to use desktop or basic server hardware, there’s something to keep in mind, something that may surprise you: The RAID you think your motherboard has isn’t hardware RAID — it’s software RAID. Even if you set it up in the BIOS, it’s still software RAID. VMware doesn’t support it. If you want hardware RAID, you’ll need to buy a separate RAID controller (SATA or SAS). Typically these RAID controllers are a couple hundred dollars.

But lets say you’re going to host a NAS device inside your virtualized environment, with something like FreeNAS or OpenFiler. Here’s an option: Put three SATA drives into your host. Each drive would then become a data store. Create a virtual machine, and give it three drives, each one contained on a different data store. You can then create a RAID 5 array from the operating system. This would all be software RAID, but performance should be fine.

SSD drives are dropping in price. Unfortunately, none of the hypervisor vendors I know of support the TRIM feature (important for keeping performance good with an SSD). I wouldn’t recommend using an SSD as a datastore just yet.

You can also use a NAS device (such as a Drobo or Synology) as either an iSCSI or NFS server and store your virtual machines there. That would get you RAID protection as well as some flexibility and the ability to run a cluster.

Pre-built or Build Your Own

The allure of building your own system is strong, especially for the nerd core set. But you have to be a bit careful. The last time I tried to build my own ESXi host, this was the result.

Things went horribly, horribly wrong

The picture above is from the actual build I did. There is a samurai sword, a bottle of absinthe, and three server carcasses. I won’t say how each was used, as you can see things did not go as planned. I eventually got the system up and running, but I wasted about two days of effort. If I’d just bought a pre-built system and added some RAM, I could have saved a lot of time.

Also, I really didn’t save any money by building it myself, even if you don’t account for the wasted time. I recently purchased a Dell XPS 8300 with 6 GByte of RAM, 1.5 TB HD, 4 core i5-2300 processor as a virtualzation system. It cost me about $650 USD at Best Buy. Another $90 will get that system to 16 GB of RAM. Building a similar system on Newegg probably would cost a bit more.

But your time is yours, and perhaps your server builds don’t include ancient weapons and high-proof spirits. If so, more power to you.

Some may scoff at the hardware choices here, and that’s fine. I wouldn’t run a Fortune 500 company off the components discussed here, but that’s not what this equipment is for. This is for the smaller side of the SMB, home labs, and dev/test labs. And for that, it works beautifully.

You Changed, VMware. You Used To Be Cool.

So by now we’ve all heard about VMware’s licensing debacle. The two biggest changes where that VMware would be charging for RAM utilization (vRAM) as well as per CPU, and the free version of ESXi would go from a 256 GB limit to an 8 GB limit.

There was a bit of pushback from customers. A lot, actually. A new term has entered the data center lexicon: vTax, a take on the new vRAM term. VMware sales reps were inundated with angry customers. VMware’s own message boards were swarmed with angry posts. Dogs and cats living together, mass hysteria. And I think my stance on the issue has been, well, unambiguous.

It’s tough to imagine that VMware anticipated that bad of a response. I’m sure they thought there would be some grumblings, but we haven’t seen a product launch go this badly since New Coke.

VMware is in good company

VMware introduced the concept of vRAM, essentially making customers pay for something that had been free. The only bone they threw at customers was the removal of the core limit on processors (previously 6 or 12 cores per CPU depending on the license). However, it’s not more cores most customers need, it’s more RAM.  Physical RAM is usually the limiting factor for increasing VM density on a single host, not lack of cores. Higher VM density means lower power, lower cooling, and less hosts.

VMware knew this, so they hit their customers where it counts. The only way to describe this, no matter how VMware tries to spin it, is a price increase.

VMware argued that the vast majority of customers would be unaffected by the new licensing scheme, that they would pay the same with vSphere 5 as they did with vSphere 4. While that might have been true, IT doesn’t think about right now, they think about the next hardware they’re going to order, and that next hardware is dripping with RAM. Blade systems from Cisco and HP have to ability to cram 512 GB of RAM in them. Monster stand-alone servers can take even more, like Oracle’s Sun Fire X4800 can hold up to 2 TB of RAM. And it’s not like servers are going to get less RAM over time. Customers saw their VMware licensing costs increasing with Moore’s Law. We’re supposed to pay less with Moore’s Law, and VMware figured out a way to make us pay more.

So people were mad. VMware decided to back track a bit, and announced adjusted vRAM allotments and rules. So what’s changed?

You still pay for vRAM allotments, but the allotments have been increased across the board. Even the free version of the hypervisor got ups to 32 GB from 8 GB.

Also, the vRAM usage would be based on a yearly average, so spikes in vRAM utilization wouldn’t increase costs so long as they were relatively short lived. vRAM utilization is now capped at 96 GB, so no matter how large a single VM is, it will only count as 96 GB of vRAM used.

Even with the new vRAM allotments, it’s still a price increase

The adjustments to vRAM allotments have helped quell the revolt, and the discussion is starting to turn from vTax to the new features in vSphere 5. The 32 GB limit for the free version of ESXi also made a lot more sense. 8 GB was almost useless (my own ESXi host has 18 GB of RAM), and given what it costs to power and cool even a basement lab, not even worth powering up. 32 GB means a nice beefy lab system for the next 2 years or thereabouts.

What’s Still Wrong

While the updated vRAM licensing has alleviated the immediate crisis, there is some damage done, some of it permanent.

The updated vRAM allotments make more sense for today, and give some room for growth in the future, but it still has a limited shelf life. As servers get more and more RAM over the next several years, vTax will automatically increase. VMware is still tying their liscensing to a deprecating asset in RAM.

That was part of what got people so peeved about the vRAM model. Even if they ran the numbers and it turned out they didn’t need additional licensing from vSphere 4 right now, most organizations had an eye on their hardware refresh cycle, because servers get more and more RAM with each refresh cycle.

VMware is going to have to continue to up vRAM allotments on a regular basis. I find it uncomfortable to know that my licensing costs could increase as exponentially as RAM becomes exponentially more plentiful as time goes on. I don’t doubt that they will increase allotments, but we have no idea when (and honestly, even if) they will.

You Used to be Cool, VMware

The relationship between VMware and its customers base has also been damaged. VMware had incredible goodwill from customers as a vendor, a relationship that was the envy of the IT industry. We had their back, and we felt they had our back. No longer. Customers and the VMware community will now look at VMware with a somewhat wary eye. Not as wary with the vRAM adjustments, but wary still.

I have to imagine that Citrix and Microsoft have sent VMware’s executives flowers with messages like “thank you so much” and “I’m going to name my next child after you”. I’m hearing anecdotal evidence that interest in HyperV and Citrix Xen has skyrocketed, even with the vRAM adjustments. In the Cisco UCS classes I teach, virtually everyone has taken more than just a casual look at Citrix and HyperV.

Ironically Citrix and Microsoft have struggled to break the stranglehold that VMware had on server virtualization (over 80% market share). They’ve tried several approaches, and haven’t been successful. It’s somewhat amusing that it’s a move that VMware made that seems to be loosening the grip. And remember, part of that grip was the loyal user community.

Thank about how significant that is. Before vSphere 5, there was no reason for most organizations to get “wandering eye”. VMware has the most features and a huge install base, plus plenty of resources and expertise around to have made VMware the obvious choice for most . The pricing was reasonable, so there wasn’t a real need to look elsewhere.

Certainly the landscape for both VMware and the other vendors has changed substantially. It will be interesting to see how it all plays out. My impression is that VMware will still make money, but they will lose market share, even if they stay #1 in terms of revenue (since they’re charging more, after all).

Free ESXi 5 (vSphere 5 Hypervisor) Now With 32 GB

Update: Check out my buyers/builder’s guide for an inexpensive ESXi host.

Yeah, we all know about the licensing debacle with VMware, and how it dominated the conversation of the vSphere 5 release. A side issue got a bit less attention, however. I’d written previously about the new licensing model screwing home/lab users of the free ESXi hypervisor with the new licensing model by allocating a paltry 8 GB of vRAM. The free version is limited in functionality (which I’m fine with), but the RAM limit used to be 256 GB. When they announced vSphere 5, the free version was then restricted to only 8 GB of vRAM. Dick move.

However, it seems the huge backlash has changed their minds, and they’ve increased the vRAM allocation for the free version of ESXi 5.0 to 32 GB of RAM.

RAM everything around me (get the DIMMs, y’all)

The FAQ still shows the free version with only 8 GB of vRAM, but the updated pricing PDF confirms that it’s actually 32 GB of vRAM allocation, as well as the announcement on VMware’s blog. From the pricing PDF:

Users can remotely manage individual vSphere Hypervisor hosts using the vSphere Client. vSphere Hypervisor is entitled to 32GB of vRAM per server (regardless of the number of processors) and can be utilized on servers with up to 32GB of physical RAM.

I think right now 32 GB for the free version is reasonable, although I think it’s only going to be reasonable for the next 18-24 months.  After that, they’ll need to increase it, and increase it on a regular basis if they continue with vRAM licensing. RAM is a depreciating asset, after all.

This should go a ways to repair a fairly damaged relationship with the VMware advocates such as myself and the VMware user community in general who relied on the free version of ESXi for our home and lab setups. The relationship isn’t quite where it was, to be sure, but it’s improving.

Also, yes that’s me in the picture. That I just took. For this entry. Because that’s how I roll.

VMware? Surprised. Oracle? Not Surprised.

There’s a rumor that VMware is backing away a bit from their vRAM (vtax) licensing. Nothing confirmed yet, but according to, they’re not getting rid of vRAM licensing entirely, but are making some adjustments. They’re upping the vRAM allotments, and 96 GB is the maximum a VM can count towards vRAM allocation, If you had a VM with 256 GB of RAM, only 96 of it would count towards vRAM usage. I’d still like to see vRAM go away entirely, but it seems the new limits are less catastrophic in terms of future growth.

On a related note, it seems Oracle is one-upping VMware in the “dick move” department. A blog post on points out Oracle’s new Java 7 licensing: In virtualized environments, Java 7 only officially supported by an Oracle hypervisor. Knowing Larry Ellison, it’s not all the surprising. He’s a lock’em in kinda guy. I avoid Oracle at all costs.

VMware’s dick move was a surprise. That didn’t seem like the VMware we’d been working with for years. Oracle on the other hand, you’re only surprised they didn’t do it sooner.


Yes, VMware’s License Change Is That Bad: It’s A Trap!

So VMware announced last week the new vSphere 5. There were lots of new features and improvements announced, but the conversation that has resulted isn’t quite what VMware had in mind. Virtually (get it? virtually?) all discussion has centered around the new licensing scheme.

And some of us feel that the change is, well… a trap.

For pete’s sake Tony, they’re going to take your VCP away for this

There have been many, many blog posts with people weighing in. Most of the feedback on blogs and messages boards has been fairly negative.  A few have been more positive, although the positive blogs (retweeted rapid fire by a somewhat desperate-sounding @vSphere5 twitter account) have enthusiastic sounding topics like 7 Reasons Why VMware vSphere 5 vRAM Licensing Is Not As Bad As It Looks At First Glance.

My thoughts on this are it’s actually much worse than at first glance. To quote a certain raspy-voiced naval officer: “It’s a trap”.

What Changed

At the heart of the change is the new concept of vRAM. vRAM is an allotment of RAM to use for powered-up VMs. If you have 10 4 GB VMs, you’d need 40 GB of vRAM allotments to power them up.

With the old licensing, you could use as much RAM as you could cram into a box. You only paid per CPU (with limits on the number of cores per CPU depending on the license).  As RAM prices dropped, it made it very attractive to cram lots and lots of VMs into fewer hosts, requiring less power and cooling. Servers with 128-512 GB of RAM from Cisco, HP, and Dell are fairly common now.

With the new licensing, you pay for CPU like before, and they’ve removed any core limits. However you now get a limited vRAM allotment with each license. No more unlimited RAM. And that’s the part that’s a trap.

It’s About The Future

The @vsphere5 twitter account is busy retweeting posts from users who checked their licensing and found that to upgrade to vSphere 5, the current vRAM allotments are within the range of their physical RAM.  So yeah, right now your licensing may work out and you don’t need to purchase any extra licenses, especially if you’re running on hardware more than a year old.

But what about the hardware you were going to refresh too? Chances are, it’s a nice beefy blade dripping with RAM like jewels on an heiress. And that’s when you’ll need to buy more licenses to use all that RAM.

RAM Rules Every Thing Around Me (Get the DIMMS, Y’All)

We’ve gotten used to the idea that RAM was essentially free in terms of licensing. In physical terms, RAM had a cost associated to it, but it could easily offset those costs in terms of savings with regard increased density.

The biggest issue with vRAM allotments are that they’re too small today, and they’ll definitely be too small tomorrow. RAM is a depreciating asset.  RAM price continuously goes down, and servers are getting more and more of it. The issue isn’t if the new licensing model will screw you, it’s when.

VMware’s blog post on why they made the new licensing, and they included some graphs. Well here’s a graph I came up with.

Tony, you have a gift for visual aids.

The vRAM allotments are pretty small today, especially if you recently did a hardware refresh.  Chances are if you bought a brand new 2-way server recently, it probably came dripping with RAM like jewels on an heiress.

And RAM keeps getting less and less expensive.  A year or so from now, 1 TB systems (probably with no more than 2 or 4 10 core CPUs) won’t be that uncommon. And then the licensing for VMware will be really bad compared to vSphere 4.

Under current licensing, licensing a full 1TB system with 4 CPUs would cost $76,890 USD. That’s 22 licenses for a 4 CPU system.

“$76,890 a server? Our IT budgets can’t repel license increases of that magnitude!”

VMware’s Other Dick Move

I’ve already written about how VMware has dramatically changed the licensing for the free version of the vSphere 5 hypervisor (ESXi 5.0). The limit used to be 256 GB of RAM, now it’s 8 GB.  Was 256 GB too much? Sure. But 8 GB is way too low.

To those of us that have home labs, that means we’ll be looking elsewhere. I think this is going to have a dramatic impact on the number of virtual appliances released for VMware. The focus will then move to XenServer (since most virtual appliances are based on Linux).

I’ve Got A Bad Feeling About This

Let’s face it, this new licensing scheme (and the 8 GB limit on the free version of ESXi 5) are dick moves. Up until now, VMware has been very friendly to its customer base. Users have supported them, championed them, and advocated on their behalf. I’ve never felt the need to even look at another hypervisor, because VMware rocked it.

Now, I look at VMware like I look at vendors like Oracle and EMC. For IT managers and techies alike, we know there are some vendors where the relationship is symbiotic. We’ll sit at the same side of the table. Networking vendors like Juniper, F5, and even Cisco (with a few exceptions) take a cordial and symbiotic view of the customer/vendor relationship.  Some vendors *cough Oracle cough Checkpoint couch* take a more adversarial and parasitical view.

I’m not saying VMware is quite at the level of parasitic right now, but they’ve suddenly taken a step in that direction. And that means the honeymoon is over. There is a general feeling that the loyal customer base is being taken for granted, as if somehow we owe VMware for the awesomeness they provide us. Well it’s the IT managers and nerdarati that helped make VMware the company it is today. And I can’t help but feel they’ve turned on us. One might say even say, turned to the dark side.

Wandering Eye

Look VMware, I’ve got to be honest. I’ve been… I’ve been seeing other people. I downloaded XenServer the other day. It asked me to download it, and I was being polite, but considering you might limit us to 8 GB, I went ahead and installed it. And I’m liking what I see.

Don’t Uninstall Angry

A wise sysadmin once said: “Never uninstall angry”.  So I’m not making any moves yet. I’m not quite ready to break up with VMware. It’s still a dynamic ecosystem, and I hope its not too damaged by all this. It’s been almost a week since the announcements so I’ve been able to get a little perspective. I’m still disappointed, to be sure, but I’m not quite as aggro.

It’s clear VMware felt it was getting the short end of the stick with the trend towards high server density. And perhaps VMware licensing needed to be changed to reflect this. I’m not all together against vRAM pools, but I do think they need to be dramatically increased. Today they’re too small, and a year or two from now they’ll be laughable.