NexentaStor released version 3.1 of their storage appliance in the end of July, this new version of NexentaStor is very interesting for VMware vSphere users as it adds VAAI support to this already very feature rich ZFS storage appliance. VAAI in NexentaStor 3.1 release is a second generation VAAI implementation which is based on T10 committee SCSI standards, because of this NexentaStor VAAI is compatible only with vSphere 5 or higher and not with vSphere 4.1.
NexentaStor is available in two flavors, Community Edition (later CE) is free for non-production use and supports up to 18 TBs of data. For production use there is Enterprise Edition which is licensed by required capacity. VAAI is supported on both editions which makes NexentaStor CE very good candidate for vSphere home lab storage appliance.
I have been running NexentaStor on my home vLab for some time now so implementing VAAI was just a simple NexentaStor appliance upgrade task. Once appliance was upgraded to latest 3.1.1 release VAAI hardware acceleration was immediately available to my ESXi 5.0 host:
Below is a copy & paste from NexentaStor 3.1.1 Release Notes explaining available VAAI primitives:
- SCSI Write Same: When creating a new virtual disk VMware must write‟s zeros to every block location on the disk. This is done to ensure no residual data exists on the disk which could be read by the new VM. Without Write Same support the server‟s CPU must write each individual block which consumes a lot of CPU cycles. With the Write
Same command VMware can direct the storage array to perform this function, offloading it from the CPU and thereby saving CPU cycles for other operations. This is supported in ESX/ESXi 5.0 and later.
- SCSI ATS: Without ATS support, when VMware clones a VM it must lock the entire LUN to prevent changes to the VM while it is being cloned. However, locking the entire LUN affects all other VM‟s that are using the same LUN. With ATS support VMware is able to instruct the array to lock only the specific region on the LUN being cloned. This
allows other operations affecting other parts of the LUN to continue unaffected. This is supported in ESX/ESXi 5.0 and later.
- SCSI Block Copy: Without Block Copy support, cloning a VM requires the server CPU to read and write each block of the VM consuming a lot of server CPU cycles. With Block Copy support VMware is able to instruct the array to perform a block copy of a region on the LUN corresponding to a VM. This offloads the task from the server‟s CPU
thereby saving CPU cycles for other operations. This is supported in ESX/ESXi 5.0 and later.
- SCSI Unmap: Provides the ability to return freed blocks in a zvol back to a pool. Previously the pool would only grow. This enables ESX to destroy a VM and return the freed storage back to the zvol. ESX 5.0 and later support this functionality
VAAI performance impact
To see what impact VAAI has on my vSphere lab I ran some test by deploying a VM from template. Test template was a Windows 2008 R2 VM having 40 GB thin disk with 10,4 GB of data and 4 GB VM swap file, I ran VM deploy three times for both tests.
Shortest time to deploy VM without VAAI was 488 seconds, with VAAI enabled it took only 248 seconds at worst to deploy a VM. NexentaStor 3.1.1 with VAAI is almost twice as fast to deploy VMs than without!
For full disclosure here is test setup used. Please note that some parameters shown here are not safe for production use!
ESXi host had two dedicated iSCSI connections to NexentaStor appliance over quad port Intel PRO/1000 adapter. vSphere PSP was set to Round Robin and IOOperation Limit was set to 10 on all devices. MTU on iSCSI connections was set to 9000 on both ends. NexentaStor was configured with two iSCSI target portal groups which both were configured for single iSCSI target.
VM template was stored on a VMFS datastore cluster with four VMFS 5 datastores, new VM was deployed to same datastore cluster using automatic VMDK placement by Storage DRS.
LUN configuration on NexentaStor for VMFS 5 datastores was compression=on, checksum=off, deduplication=off, log bias=throughput and sync=disabled.
NexentaStor appliance preference changes from default was sys_zfs_nocacheflush=yes, sys_zfs_vdev_max_pending=1, Net_tcp_naglim_def=1, net_tcp_recv_hiwat=524288 and net_tcp_xmit_hiwat=524288.
ESXi host software and hardware
- VMware ESXi 5.0 build 469512
- Intel Core i5 2500K CPU
- 16 GB DDR3-1600 RAM
- Intel PRO/1000 Quad Port Ethernet adapter
NexentaStor appliance software and hardware
- NexentaStor 3.1.1
- Intel Core 2 Duo E8400 CPU
- 8 GB DDR2-1066 RAM
- Intel PRO/1000 Quad Port Ethernet adapter
- 6 x Seagate 7200.11 500 GB SATA drives in RAIDZ-0