I double checked VMware HCL and adapter I had, Qlogic 8100 (QLE8152 to be exact), is listed as supported on ESX 4.1 with ethernet driver qlge 1.0.0.39 and FCoE driver qla2xxx 831.k1.23vmw, I had correct drivers so I filed support case to VMware.

While waiting for a reponse from a VMware support I started digging internals of driver zip. Inside of any VMware patch file is a metadata.zip which hosts, like name states, metadata for that particular patch in XML format. I focused my investigation on vmware.xml and in that I noticed something odd, bundle softwarePlatform version was tagged as 4.* but actual driver was tagged to be compatible with 4.0.0. Below is a snippet of XML blocks in question.

<platforms>
 <softwarePlatform locale="" productLineID="esx" version="4.*" />
 <softwarePlatform locale="" productLineID="embeddedEsx" version="4.*" />
</platforms>

<systemReqs>
 <swPlatform locale="" productLineID="esx" version="4.0.0" />
 <swPlatform locale="" productLineID="embeddedEsx" version="4.0.0" />
 <maintenanceMode>true</maintenanceMode>
</systemReqs>

Feeling adventurous I replaced version=”4.0.0″ with version=”4.*” in <systemReqs> block, zipped all files back together and transferred driver zip files to ESX service console.

With a bit excitement I ran esxupdate for FCoE driver, just to get message that bundle is obsolete as newer package is already installed. Quick “rpm -qa | grep qla” revealed that indeed there is a 831.k1.28.1-1vmw included in ESX 4.1. So ESX 4.1 actually supports QLogic FCoE out of the box, while working on service console only and relying on VMware HCL that additional drivers are required I had not noticed that FC adapters were already visible.

FCoE solved I ran esxupdate for remaining QLogic 10 GbE driver which this time passed successfully, after a reboot I also had two QLogic 10 GbE adapters visible!

I reported my findings to VMware support  and they came back with confirmation that driver should have been working with ESX 4.1 and that they have reported this error back to ESX engineers for driver fix. So, until there is fixed QLogic 10 GbE CNA driver available from VMware you can fix the driver with instructions above, but please do that with your own responsibility!

I firmly believe that this is error in documentation, so SQL Server 2008 Standard support is not actually dropped on VUM 4.1, it just makes no sense at all to me that SQL Server Express 2005 is marked as supported but Standard is not, Express is so much more limited in its capabilities than Standard. Also SQL Server 2008 Standard is supported with previous releases so it has already been certified, why remove support now?

I am still waiting confirmation from VMware on this, but until there is public clarification on this issue check your upgrade plans with VMware support to be sure!

Update: I received an answer from VMware today, this is indeed an error in Compatibility Matrix, SQL Server 2008 Standard IS supported with Update Manager 4.1 and error in Compatibility Matrix will be corrected.

One important thing you should note is that as vCenter Server 4.1 is now full-fledged 64-bit application it requires 64-bit DSN, but Update Manager 4.1 is still 32-bit application so it requires 32-bit DSN! See discussion of this at Yellow Bricks

Download pre-upgrade-from-ESX4.0-to-4.1.0-0.0.260247-release.zip and upgrade-from-ESX4.0-to-4.1.0-0.0.260247-release.zip files from vmware.com, you need active support subscription for that.

1. Copy both files ESX host Service Console
2. Put host in maintenance mode
3. Run esxupdate with “update –bundle=/path/to/pre-upgrade-zip-file”
4. Run esxupdate with “update –bundle=/path/to/upgrade-zip-file”
5. Reboot
6. Exit maintenance mode

Tip how to put host in maintenance mode using CLI

# vimsh -n -e /hostsvc/maintenance_mode_enter

Execute pre-upgrade patching

# esxupdate update --bundle=pre-upgrade-from-ESX4.0-to-4.1.0-0.0.260247-release.zip

pre-upgrade-from-ESX4.0-to-4.1.0-0.0.26024.. #################################### [100%]
Unpacking vmware-esx-esxupdate-4.1.0-0.0.2.. #################################### [100%]
Unpacking vmware-esx-uwlibs-4.1.0-0.0.2602.. #################################### [100%]
Unpacking glibc-common-2.5-34.2926.vmw.x86.. #################################### [100%]
Unpacking glibc-2.5-34.2926.vmw.x86_64.vib   #################################### [100%]
Unpacking glibc-2.5-34.2926.vmw.i686.vib     #################################### [100%]
Installing glibc-common                      #################################### [100%]
Installing glibc                             #################################### [100%]
Installing glibc                             #################################### [100%]
Installing vmware-esx-uwlibs                 #################################### [100%]
Installing vmware-esx-esxupdate              #################################### [100%]
Cleaning up vmware-esx-esxupdate             #################################### [100%]
Cleaning up vmware-esx-uwlibs                #################################### [100%]
Cleaning up glibc-common                     #################################### [100%]
Cleaning up glibc                            #################################### [100%]
Cleaning up glibc                            #################################### [100%]

Execute upgrade patching

# esxupdate update --bundle=upgrade-from-ESX4.0-to-4.1.0-0.0.260247-release.zip
Host was not updated, no changes required.
Skipping bulletin ESX410-GA-esxupdate; it is installed or obsoleted.
Unpacking rpm_vmware-esx-likewise-ad.. ########################################## [100%]
... lots of updated RPM packages ...
Cleaning up nss_ldap                   ########################################## [100%]

Running [/usr/sbin/cim-install.sh]...
ok.
Running [/usr/sbin/vmkmod-install.sh]...
ok.
Running [esxcfg-boot -b]...
ok.
The update completed successfully, but the system needs to be rebooted for the
changes to be effective.

And after succesful patching reboot ESX host

# reboot

To exit from maintenance mode using CLI run

# vimsh -n -e /hostsvc/maintenance_mode_exit

Features shown in video are

1. New CLARiiON and Celerra management GUI called Unisphere, based on Adobe Flex technology
2. Sub-LUN tiering – “VMware DRS” equivalency for storage medium, will migrate data based on usage between Flash, FC and SATA
3. Flash cache – AKA FAST cache, up to 2 terabytes(!) of 2nd level read/write cache
4. Block level compression – told to save 30% to 50% on average on your storage consumption

It is not told directly in this video but it looks like thin pools will become an standard feature on CX4, at the moment is is sold as an add-on option.

These and new vStorage API integration features make me feel that this is a good time to be EMC customer.

NexentaStor is available as commercially supported enterprise version and as community supported free to use Community Edition. Community Edition is limited to 12 terabytes of used storage, it is also lacking some optional modules available on enterprise version.

I downloaded NexentaStor Community Edition 3.0.3 and installed it on PC I built of some left over components I had at hand. My finished storage appliance consist of

• AMD Athlon 4850e dual core CPU
• Asus M3A78 Pro motherboard
• 4 GB DDR2 RAM
• 1 x 80GB Seagate 7200.10 ATA disk as boot drive
• 5 x 500GB Seagate 7200.11 SATA disks as ZFS pool
• Intel X25-M G2 as flash cache
• Intel  PRO/1000 PT Quad Port Ethernet adapter

Above system fits to NexentaStor hardware recommendations like a glove, 64-bit CPU, 4GB RAM, dedicated spindles for ZFS pool and dedicated boot drive.

Although Community Edition is free, it requires registration key which is unique for each hardware appliance is installed on. You get registration key in email by submitting machine signature to Nexenta, machine signature is generated and presented during appliance installation. I got mine registration key in email within a minute of submitting signature, which was nice.

NexentaStor recognized all of my storage system components correctly and installation was finished 15 minutes later. Once registration was active I was queried of management IP and WebGUI port settings, those set I launched web browser and pointed it to my new storage appliance.

Configuration wizard

First step at WebGUI is to set attributes such as host and domain name, NTP server and localization.

Second step is network configuration,

including iSCSI target settings.

In third and fourth step you create your ZFS volume, you may also configure any available SSD drives as secondary read cache (L2ARC), or as ZFS intent log device (ZIL) which is used for writes

SSD as secondary cache

Employing fast SSDs as secondary disk cache is a really neat feature. SSDs are much larger in sizes and cheaper per gigabyte than RAM and they still provide incredible amount of IOPS, for example even consumer grade SSD Intel X25-M is rated up to 35,000 IOPS on 4k reads! You need approximately 175 15kRPM spindles to deliver that amount of IOPS!

Folders

Now that you have your ZFS volume created, you are asked to create folders used as network shares, I created single folder named “nfs”. You can also set ZFS attributes such as deduplication and compression

NFS share settings

Once you save and review your configuration you are finished with configuration wizard. Now select “Data Management” from top menu, and in drop down list select “Shares”

If you did not yet enable NFS sharing you may do it now, then click “Configure” in “NFS Server” box on left side of screen

Set Server and Client Version to 3, it’s the version vSphere is using and it is recommended by Nexenta to limit NFS version to 3 when using with vSphere, click Save

Go back to Shares view and click folder link

In this view you see mount point in which folder is available to NFS client, make note of it

If you scroll down the view there is options like deduplication, checksum, compression and access time. Checksum is ZFS feature for detecting data corruption, it works only if you are using RAID-Z or ZFS mirroring, if you use hardware RAID you should disable this since it has small CPU overhead. You might disable access time as well, it is not needed with vSphere datastores and it has small disk IO overhead. You can enable or disable deduplication and compression any time you like, new setting will affect only new data written to ZFS.

Mounting NFS share to vSphere

I assume that you have necessary VMkernel port set at ESX so I won’t go into details with that. Open vSphere Client and mount NAS datastore, folder path is mountpoint you saw at folder settings view

And you have NexentaStor NFS share mounted to vSphere

NexentaStor and ESX 4.0 both support jumbo frames so you can improve NFS performance by configuring MTU up to 9000k or what is maximum supported by your NICs and/or switches.

Performance

I configured my ZFS volume as striped RAID-0 (since this is purely experimental, no valuable data stored in here), I have ZIL enabled and stored at ZFS pool, Intel X25-M SSD as L2ARC (flash cache) device. Compression and deduplication are off.

Raw throughput of NexentaStor NAS is quite good, below is an results of large file write with 8k blocks within CentOS Linux VM

# dd if=/dev/zero of=/sdb/test bs=8k count=838860
838860+0 records in
838860+0 records out
6871941120 bytes (6.9 GB) copied, 58.3875 seconds, 118 MB/s

and same file read in 8k blocks

# dd if=/sdb/test of=/dev/null bs=8k
838860+0 records in
838860+0 records out
6871941120 bytes (6.9 GB) copied, 79.2254 seconds, 86.7 MB/s

By looking read performance there might be something wrong with my L2ARC setup as reads are slower than writes, running iostat in storage appliance also reveals that L2ARC device is not read that much. Write performance could be improved a quite lot (to point of GigE saturation at least) by assigning second SSD drive as ZIL device, or I could just disable ZIL altogether but lose on data integrity. Well, I think I’ll investigate that on some another day.

Things to consider with Linux volumes

If your intention is to use all of your new disk for single file system I recommend that you do not use partitioning, use a plain disk as LVM physical volume instead, it simplifies managing volumes a lot. You could also chose not to use LVM and format new disk as is with ext3 but I do not recommend that, using LVM has many benefits which I might cover on some later articles.

If you need to use partitioning you should be aware of following caveats

1. To avoid performance penalty from file system misalignment you need to match beginning of new partition with underlying storage device using fdisk expert commands
2. Using partitions makes extending file systems much more complicated compared to using plain disks instead

If you fail to align file system properly with your storage, you can lose up to 15% of your available IOPS.

Attach new disks without reboot

After new virtual disk have been added to virtual machine you need to rescan new disks in Linux. Scan new disks with command below, replace “host0″ with SCSI controller ID you added disks to

# echo "- - -" > /sys/class/scsi_host/host0/scan

use dmesg command to check what new devices have been added, watch for “Attached scsi disk” message

# dmesg | tail -n 10 | grep Attached
sd 0:0:1:0: Attached scsi disk sdb

Now create LVM Physical Volume, Volume Group, Logical Volume and file system on your new disk

# pvcreate /dev/sdb
  Physical volume "/dev/sdb" successfully created
# vgcreate VolGroup01 /dev/sdb
  Volume group "VolGroup01" successfully created
# lvcreate -n LogVol00 -l+100%VG VolGroup01
  Logical volume "LogVol00" created
# mkfs.ext3 /dev/VolGroup01/LogVol00

Or if you know that you don’t need LVM you can simply format plain disk with

# mkfs.ext3 /dev/sdb

Your new volume is ready for mounting.

Resizing existing disks

After virtual disk have been resized you need to rescan changes in Linux. You must know SCSI ID of disk changed, replace 0:0:1:0 in example below with correct SCSI ID

# echo 1 > /sys/bus/scsi/devices/0\:0\:1\:0/rescan

use dmesg to check that resize is successful, watch for “capacity change” message

# dmesg | tail -n 10 | grep change
sdb: detected capacity change from 8589934592 to 17179869184

if LVM is not used resizing file system can be done with command

# resize2fs /dev/sdb

Other ways proceed to extend LVM volume

Extending LVM volume and file system, easy way with no partitions

Rescan LVM physical volume size change with pvresize, replace sdb with correct device

# pvresize /dev/sdb

Extend LVM logical volume to new full size of volume group

# lvextend -l+100%VG /dev/VolGroup01/LogVol00

Now you can resize mounted ext3 file system

# resize2fs /dev/VolGroup01/LogVol00

And you are done

Extending LVM volume and file system, hard way with partitions in use

You cannot resize LVM physical device on partition without reboot, when using single disk your only option is to create new partition on space created by extending disk and use partprobe command to reload partition table without need to reboot. You could also create new disk and extend Volume Group to that.

Create a new partition to existing disk

# fdisk /dev/sdb
Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 2
First cylinder (3134-3916, default 3134):
Using default value 3134
Last cylinder or +size or +sizeM or +sizeK (3134-3916, default 3916):
Using default value 3916

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.

WARNING: Re-reading the partition table failed with error 16: Device or resource busy.
The kernel still uses the old table.
The new table will be used at the next reboot.
Syncing disks.

Run partprobe to reload partition table

# partprobe /dev/sdb

Create new LVM physical volume to new partition

# pvcreate /dev/sdb2

Add new physical volume to existing Volume Group

# vgextend VolGroup01 /dev/sdb2

Extend Logical Volume up to maximum size of Volume Group

# lvextend -l+100%VG /dev/VolGroup01/LogVol00

Finally you can resize mounted ext3 file system

# resize2fs /dev/VolGroup01/LogVol00

Done. See how much work there is when using partitions? There is absolutely no point at all to create partition table for standard ext3 file system when intention is to use whole disk for single file system.

http://vizioncore.com/products/vConverter/vConverterSC.php

To use FLR functionality you need command line FLR client on Windows VM protected by VDR appliance. FLR client is just a single executable vdrFileRestore.exe which is available at same download page you got VDR appliance from.

If you are not familiar with VDR appliance let me explain it shortly. After that I’ll show how FLR works.

VMware Data Recovery is a pre-installed Linux appliance which is imported into your vSphere cluster. VDR has GUI which will integrate with help of plugin onto vSphere client. VDR backup works by creating restore points for each of your VM virtual disks you wish to backup. Restore point is kind of full copy of virtual machine state at that current point in time when backup was created. VDR utilizes de-duplication on backup storage and it is very efficient to save backup storage space. VDR can use CIFS share or virtual disk as backup medium, that makes restoring data very quick. There is no support for tape backups.

FLR client works by mounting restore point into Windows VM as read only disk, from that disk you can restore invidual files or whole directories by copying them with standard Windows explorer.

To start using FRL download and copy vdrFileRestore.exe into your Windows VM, place it into some directory and make sure it is on PATH environment.

Make sure you have restore points available for you VM and execute vdrFileRestore.exe -a “IP or DNS name of your VDR” on Windows command line. FLR client uses unique BIOS id to identify which restore points it should make available for mounting for each VM.

FLR client will query which restore point in time you wish to mount

After you selected restore point FLR client will do some of its magic and backup of your Windows drives will show up in your VM as additional drives, FLR client will tell you which drive backup is available at which drive letter.

After you copied files you need, type “unmount” and press enter. This will close restore point mounts and FLR client will exit.