Taking advantage of multiple NIC vMotion requires having multiple vMotion vmkernel interfaces on each host. You need to assign unique IP address for each of those vmkernel interface and you should bind each vmkernel interface to a different uplink NIC.

Configuring standard vSwitch for multi-NIC vMotion

To enable multiple NIC vMotion with standard vSwitch you could either use single vSwitch with multiple uplinks, or you could use multiple vSwitches with single uplink.

a. Single vSwitch with multiple uplinks

b. Multiple vSwitches with single uplink

vMotion vmkernel interface

Add new vmkernel interface and enable vMotion on it. vSphere 5 can load balance vMotion transfer over up 16 interfaces with 1 Gigabit NICs or up to 4 interfaces with 10 Gigabit NICs.

You can assign IP addresses from same subnet to all vMotion vmkernel interfaces, but IP address has to be unique on each interface.

Override switch failover order

If you have multiple vmkernel interfaces on same vSwitch you need to assign dedicated uplink for each vMotion portgroup. Go to the “NIC Teaming” tab on portgroup settings, enable “Override switch failover order” and configure only single vmnic as “Active Adapters”, any other adapters should be set as “Unused Adapters”.

Repeat this step for each vMotion vmkernel interface, assign unique vmnic adapter for each vMotion portgroup, eg. “vMotion 1” portgroup has “vmnic2” as active adapter and “vMotion 2” portgroup has “vmnic3” as active adapter.

Verifying multi-NIC vMotion functionality

ESXi 5 hosts will take advantage of multiple vMotion interfaces automatically once they have been properly set. Any single vMotion will be load balanced on all vMotion interfaces you have configured. To verify that vMotion load balancing takes place check ESXi host vmkernel.log, you should see multiple connections for single vMotion session.

Outgoing vMotion session

cpu1:10496)MigrateNet: 1155: 1328264432389309 S: Successfully bound connection to vmknic ’10.0.0.4′
cpu1:10496)VMotionUtil: 3118: 1328264432389309 S: Stream connection 1 added.
cpu1:10496)MigrateNet: 1155: 1328264432389309 S: Successfully bound connection to vmknic ’10.0.0.1′
cpu1:10496)VMotionUtil: 3118: 1328264432389309 S: Stream connection 2 added.

Incoming vMotion session

cpu1:10321)MigrateNet: 1155: 1328264187746848 D: Successfully bound connection to vmknic ’10.0.0.4′
cpu0:2666)MigrateNet: vm 2666: 1979: Accepted connection from <10.0.0.5>
cpu0:2666)MigrateNet: vm 2666: 2049: dataSocket 0x4100154e60c0 receive buffer size is 563272
cpu1:2666)VMotionUtil: 3118: 1328264187746848 D: Stream connection 1 added.
cpu1:2666)MigrateNet: vm 2666: 1979: Accepted connection from <10.0.0.2>
cpu1:2666)MigrateNet: vm 2666: 2049: dataSocket 0x4100154e6760 receive buffer size is 563272
cpu1:2666)VMotionUtil: 3118: 1328264187746848 D: Stream connection 2 added.

Open Advanced Settings of your ESXi host

In ESXi host Advanced Settings browse to Config –> HostAgent and you will see "Config.HostAgent.plugins.hostsvc.esxAdminsGroup” setting which you can change to match your preference. Change takes place within a minute or so, ESXi host reboot is not necessary.

Please note that each ESXi host will store esxAdminsGroup setting locally once any member in that group logs successfully to a ESXi host, setting is stored in ESXi host local permissions configuration (see screenshot below). If you change esxAdminGroup setting you have to make sure that you check local permissions on each ESXi host and remove any references to old AD group you might have previously used.

NetFlow will export only information about source, destination, protocol and volume of network traffic passing the distributed switch, it will not have any visibility to actual payload.

NetFlow Analyzer

Ability to export NetFlow data doesn’t do much as-is, you need NetFlow analyzer application in which you import NetFlow data from vSphere distributed switch. While searching for application to do this I stumped on ManageEngine NetFlow Analyzer which is available as free download for 30 day evaluation, http://www.manageengine.com/products/netflow/download.html.

ManageEngine NetFlow analyzer is available as Windows and Linux versions, I downloaded Windows version and had it running in no time as installation and configuration was very easy. Once installed I glanced though NetFlow Analyzer configuration for TCP port to which I should send NetFlow data from vDS. I found this quickly under “Admin”, “Server Settings”, “NetFlow Listener Port”.

As I knew IP address of my NetFlow Analyzer server and NetFlow listener port of application I was ready to proceed with vDS configuration.

vSphere Distributed Switch Configuration

NetFlow export is configured at distributed switch level, go to the networking view in vSphere Client click “edit settings” on switch you wish to export NetFlow data from. In switch Settings you can see “NetFlow” tab and on that page you will have following settings.

First setting you should configure is “Collector IP address”, that is the IP of your NetFlow analyzer or collector application. You also need to configure correct port as every NetFlow analyzer and collector may have different default settings.

By default each ESXi host will export NetFlow data by using their own management address, because of this you will see multiple sources of traffic in your NetFlow analyzer. If you wish to have only single source for all your distributed switch traffic, you can configure specific source IP address for your vDS. This IP should be entered in “VDS IP address” field. Please note that you need to allow access from your ESXi host management IP addresses to your NetFlow analyzer/collector no matter what IP address you would type in vDS IP address.

By default distributed switch will send information about active network flow while 60 seconds have passed since flow was initiated, if you wish to have data about new flows quicker you could lower the value on “Active flow export timeout” but I do not recommend it unless you fully understand all implications it might have. I do not, so I don’t try to explain it. Data of idle network flows will be exported after 15 seconds of initial packet has passed.

Exporting NetFlow data from a vDS which is passing a LOT of traffic might have performance hit on ESXi host CPU, if you do not mind having lower resolution on your NetFlow data you could increase “Sampling rate” value. By default it is set to 0 which means that every packet is inspected and included in NetFlow export. If you set “Sampling rate” to 2 it means that every other network packet is inspected and included in NetFlow export, and so on. Increasing sampling rate will improve performance if you are having CPU utilization issues but it will also decrease NetFlow data resolution.

As last setting there is “Process internal flows only” check box. By default distributed switch will export data about all traffic passing internally on vDS or passing to or from physical network. If you have NetFlow enabled on your physical network devices it could be unnecessary to send information about flows which are passing on physical network layer also, as those are already seen and reported by other network devices. In this case you could check “Process internal flows only” setting and have distributed switch to export data only about flows which is switched directly by vDS and not passed to physical network layer, this would decrease any overhead NetFlow data exporting might have on your ESXi hosts.

Inspecting NetFlow data

Once you have your NetFlow analyzer and distributed switch properly set you should eventually have some data about network traffic taking place in your virtual infrastructure. Below is a screenshot of ManageEngine NetFlow Analyzer output about my vSphere lab traffic, you can see larger version of image by clicking of it.

Data Migration Tool can be used to back up vCenter Server settings such as:

I hope that my new position also allow me to work more on my blog which I feel I have neglected a bit lately, I feel that I can contribute more to the excellent community that we have around VMware.

With these changes I really feel energized and full of new ideas, even though Capgemini was a great employer I expect that year 2012 will be most exciting time I have had for a while!