Jump to content
RESET Forums (homeservershow.com)

Low I/O performance with ESXi 6.0


Altecice
 Share

Recommended Posts

Hello together!

 

I have a HP Microserver Gen 8 with 16GB RAM, a Xeon 1230 v2 CPU and a Dell Perc H200 HBA (flashed to IT Mode) for passthrough the HBA to a VM.

 

My findings are these:

 

- The latest ESXi 5.5 with Express Patch 6 (Build 2456374) runs fine and fast with passthrough of the HBA! Even the Onboard-USB3 is running fine and fast with passthrough to the same VM.

 

- An update or a new installation of ESXi 6 have the same results. The installation is done with the newest ISO from HP (VMware-ESXi-6.0.0-2494585-HP-600.9.2.38-Mar2015.iso).

 

-- After the upgrade to ESXi 6 everything runs nice till I enable the passthrough of the Dell HBA again. As soon as I try to power on the VM with the passtroughed HBA the whole system freezes!

 

-- if I take a look with iLO at the direct console and press F12 to see the messages I can see a lot AHCI port resets and timeouts. If I remove the Dell HBA as passthrough device the AHCI port resets and timeouts are gone and all is fast and stable again.

 

I went back to the latest ESXi 5.5 with Express Patch 6 (Build 2456374) till the issue is solved. It seems for me that with ESXi 6 something that is solved with the latest 5.5 doesn't make the way to ESXi 6.

 

There you can see a lot of AHCI port resets and timeouts if the passthrough of the HBA is enabled:

AHCI_resets.jpg

  • Like 1
Link to comment
Share on other sites

Ok, on vmware forum got one solution (https://communities.vmware.com/message/2496155#2496155)

Problems are with hpsva drivers' versions 090 and 092. I installed 088 and Ubuntu speed improves dramatically. But problem is still when I copy from one VM to another on same SSD.

Ahh. Thank you. That is helpful.

 

scsi-hpvsa-5.5.0-84OEM.550.0.0.1198611.x86_64.vib

scsi-hpvsa-5.5.0-88OEM.550.0.0.1331820.x86_64.vib

scsi-hpvsa-5.5.0-92OEM.550.0.0.1331820.x86_64.vib

Edited by GotNoTime
  • Like 1
Link to comment
Share on other sites

Thank you very much for the solution! :)

 

I will test it but also I will later test Hyper-V too. For a little Homelab it is good to have it for play around and I also have the possibility to make a nested esxi-playground with vmware workstation (like the one on my notebook). :D

  • Like 1
Link to comment
Share on other sites

Now that the performance is where it should be, I've enabled vFlash Read Cache on the SSD. It isn't a supported SSD from the HCL but it works fine. I enabled vFRC, formatted the SSD as VFFS, setup a cache for the VMDKs in the VM and now I'm getting 500MB/s on data that is present in the cache. Requesting anything not in the cache will fallback to the HD. It is all transparent to the OS inside the VM as it is handled by ESXi.

 

You'll need Enterprise Plus or the trial license to use vFRC. The C# client isn't supported so you're supposed to use the vSphere Web Client but all of the required operations for vFlash are exposed via the VI SDK so you can manually do it through that if you don't want to also run vCenter on this little microserver.

 

Enabling vFlash Host Cache is also possible via the VI SDK.

Edited by GotNoTime
  • Like 2
Link to comment
Share on other sites

Thanks a lot for posting this info!

It really helped improving my system performance but I still think it didn't completely fix my Disk I/O performance.

I am new to Microserver G8 and ESXi so I might be missing an obvious trick!

 

If anyone could help that would be greatly appreciated. Below are details of my system, what I did to fix it and the performance I am getting.

 

I just bought and configured a G1610T Gen8 Microserver for a homelab, I upgraded it with:

16 Gb

Xeon 1240V2

And added 4x disks WD Red 3Tb each

 

Configured a Raid 10 (using the raid B120i)

Installed ESXi (HP custom) vSphere 6.0 (March 15) on a 16Gb SD card

Installed a fresh Ubuntu VM, installed VMware tools

I did provide 4Gb of Ram and 1 Virtual Socket with 2 Core to the VM.

There is no other VM and every settings are default.

 

Before I reverted to the old driver as indicated by this thread I was getting the following performance on my Ubuntu VM:

sudo hdparm -Tt /dev/sda

gives me the following:

Timing cached reads:   19800 MB in  2.00 seconds = 9907.62 MB/sec

Timing buffered disk reads: 150 MB in  3.11 seconds =  48.15 MB/sec

 

On a 3 years old iMac (no SSD, just good old standard HD) running VMware fusion 7 a similar VM gives me the following results for the same command:

Timing cached reads: 10594 MB in 2.00 seconds = 5298.41 MB/sec
Timing buffered disk reads: 1152 MB in 3.01 seconds = 383.12 MB/sec

 

Worse.. trying the following test on The Gen8 and iMac:

time sh -c "dd if=/dev/zero of=tmp1 bs=8k count=1000000 && sync"

on my iMac it completes in 16s

On my Gen8 it completes in over 4 minutes!

 

Now, AFTER reverting back to the old driver (scsi-hpvsa-5.5.0-88OEM.550.0.0.1331820.x86_64.vib)

I am getting better performance but I still think they are not what they should be, because they are still much slower than the VM on my iMac:

/dev/sda:

 Timing cached reads:   20618 MB in  2.00 seconds = 10317.72 MB/sec

 Timing buffered disk reads: 428 MB in  3.00 seconds = 142.59 MB/sec

 

The buffered disk read went from 50 Mb/Sec to 142.59 Mb/sec

 

Creating the 8Gb file went from 4 minutes to 2 minutes.

 

I was expecting the perf from my Gen8 to be at the very least as fast as my 3 years old iMac...

I admit I am new to all this as I am a desktop user, not a system admin. So if anyone could please help me how to improve those poor performance it would be great!

 

Thanks,

Gavin.

 
  • Like 1
Link to comment
Share on other sites

iMac is cheating (if it have HDD not SSD)  :)  Even 4 fastest HDD in RAID0 cannot create 8 GB of 8k blocks in 16 sec. 1 million I/O in 16 s = 62 kIOps, it is not real for HDD.
Try /dev/urandom instead of /dev/zero. And check when really disk finished activity.

I also enabled physical write cache of disks on B120i (I have UPS).

There is also newer HP ESXi image from 30 March (hpsva driver should be reverted after upgrade)

Edited by ArnisR
  • Like 1
Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share

×
×
  • Create New...