Did VMware vSphere 6.0 Remove the Layer 2 Adjacency Requirement For vMotion? No.
May 24, 2016 2 Comments
I’ve seen this misconception a few times on message boards, reddit, and even comments on this blog: That Layer 2 adjacency is no longer required with vSphere 6.0, as VMware now supports Layer 3 vMotion. The (mis)perception is that you no longer need to stretch a Layer 2 domain between ESXi hosts.
That is incorrect. VMware did remove a Layer 2 adjacency requirement for the vMotion Network, but not for the VMs. Lemme explain.
It used to be (before vSphere 6.0) that you were required to have the VMkernel interfaces that performed vMotion on the same subnet. You weren’t supposed to go through a default gateway (though I think you could, it just wasn’t supported). So not only did your VM networks need to be stretched between hosts, but so did your VMkernel interfaces that performed the vMotion sending/receiving.
What vSphere added was a separate TCP/IP stack for vMotion networks, so you could have a specific default gateway for vMotion, allowing your vMotion VMkernel interfaces to be on different subnets.
This does not remove the requirement that the same Layer 2 network exist on the sending and receiving ESXi host. The IP of the VM needs to be the same, so the VM network you vMotion to needs to have the same default gateway (for outbound packets) and inbound routing (for inbound packets).
Inside of a data center this adjacency is typically done by simply making the same VLAN available (natively or now through VXLAN) on all the ESXi hosts in the cluster.
If it’s between datacenter, things tend to get a more complicated. As in dumpster fire. Here’s a presentation I recently did on the topic, and Ivan Pepelnjak has far more high-brow explanations of why it’s a bad idea.
You’ll need solutions like LISP (for inbound), FHRP filtering (for outbound), OTV (for stretching the VLAN), and a whole host of other solutions to handle all the other problems long distance vMotion can introduce.
Where is your God now?!?!?
So when you hear that vSphere 6 no longer requires Layer 2 adjacency between ESXi hosts, that’s only for the vmkernel interfaces, not the VM networks. So yes, Virginia, you still need Layer 2 adjacency for vMotion. Even in vSphere 6.0.
The Data Center Overlords 
You can use NSX to stretch L2 between datacenters.
LISP actually supports “Across Subnet Mode” mobility of an IP address. This means that you do not need any layer-2 adjacency between different data centres. For example, you could have 10.1.0.0/24 as a subnet in DC-A, and 10.2.0.0/24 in DC-B. For simplicity, let’s say that the vNIC port-group in both DC’s that maps to those subnets is called “MY-PG”. A virtual machine in DC-A might have IP address 10.1.0.11. You can migrate that VM to DC-B using a layer-3 vMotion and keep it in the MY-PG port-group. This means that in DC-B, the VM with IP address 10.1.0.11 sits within the 10.2.0.0/24 subnet. That is obviously not natural to a lot of people, BUT LISP plays some clever tricks and will ensure that communication to/from the migrated VM routes efficiently across the network along optimised paths. User traffic to the VM will go straight to DC-B for example, rather than take a detour to DC-A. If the migrated VM needs to communicate with other VM’s within the 10.1.0.0/24 subnet that are still sat in DC-A…well LISP takes care of all that too. All of this where the network connections between DC-A and DC-B do NOT need to be layer-2; a routed network is just fine.
Across Subnet Mode is quite a good fit for maintenance or recovery scenarios where if some infrastructure in a DC goes off-line (either planned or due to some disaster), then you want to move your workloads into another DC without having to modify their IP addresses (and sort out DNS etc. etc.). For a lot of people, this might be their only use case to require IP address mobilty between DC’s. If that’s the case, then extending a subnet between DC’s (over a layer-2 interconnect) could be an undesirable solution. Hence, LISP Across Subnet Mode provides a good fit.