Processor Intel Core2Duo
Memory 4 GB
SCSI Controller : LSI 8888ELP with 512 MB cache with BBU
HDD : WD5000AAKS (500 GB) x 4 (2 HW RAID0 (2 x 2) ) (Total two data volumes)
HDD : WD 80 GB for boot
NIC : Intel MT 1000 PCI NIC

I am using iSCSI amd created two VOLs on 1st RAID0.. If I am starting / shudown more than 3 VMs at a given time, the performance is not so good. I configured MTU 9000 on vSwitch, VMK and in Nexenta.

If I add a SSD as cache would it help me in performance improvement ?

iSCSI or NFS which is best option ?

In the case of the L2ARC, the “warm-up” period is not that rapid ~ 18MB/sec. MFU and MRU items in the ARC (main memory cache) are committed to the L2ARC as ARC demands dictate. Since this is a block-based cache, block frequency and recency are the deciding factors.

With the ZIL – which ALWAYS exists unless you explicitly (and dangerously) disable it – all writes that are synchronous get committed there immediately. In the absence of a synchronous logging device (i.e. SLOG or ZIL accelerator), these are first committed to the pool to conform with non-volatile requirements (hard commit) and then re-committed to the pool along with the non-synchronous components of the associated transaction groups.

It is a common misunderstanding, but the ZIL never gets “read” (from disk or cache) but always reads from the ARC. In case of a checksum failure from ARC data (oops!), the ZIL data would be re-read from disk and reconstructed (if possible, as necessary) to then commit with the TX group. If that is happening, your architecture is flawed

A note on disabling ZFS checksum: don’t. For that matter, don’t use hardware RAID unless you have a LUN aggregation issue that compels it. The ZFS checksum is a key factor in data integrity and protects you from silent data corruption. Yes there is overhead, but your data is usually worth it. Plus it’s better that having to fall-back to application based data integrity…

One thing I wanted to mention is your closing comments about performance. My understanding is that it takes a while for ZIL and L2ARC to populate, so it makes sense that you would not see an immediate performance benefit. After all they are performing a cache function, and a cache is only useful the second time you try to transfer data (or on other subsequent tries). I have seen several mentions of letting the cache “warm up”, but am not sure how quickly that occurs.

With that in mind, have you had a chance to experiment with more performance testing scenarios that mimic real world usage? e.g booting multiple VMs that are nearly identical, once using the SSD based L2ARC/ZIL and once without? Same thing goes for file reads/writes like what you did in your example, but over a longer time frame and with/without the SSD based disks as cache?

One thing I have done in similar circumstances is to use an Ubuntu based VM and then read/write data between the /dev/shm mountpoint and your virtual hard disk. This should allow you to better isolate what the Nexentia based storage system bottlenecks are.

FYI /dev/shm is a on-demand allocated ramdisk equal to half the size of the RAM you allocate to the virtual machine. Similarly you can use tmpfs to create a ramdisk (allocated manually, not on demand) that has even better performance than /dev/shm. Check out this link I found for a good writeup: http://kevin.vanzonneveld.net/techblog/article/create_turbocharged_storage_using_tmpfs/

I’ve tried the Nexentia Store community edition and I think it needs a little polishing for use-cases like this. I’m a big fan of FreeNAS and I have my fingers crossed that the upcoming v0.8 of FreeNAS is going to support the newer versions of ZFS that allow use of L2ARC, ZIL, dedup, etc.

http://www.lessfs.com/wordpress/