Jump to content
RESET Forums (homeservershow.com)

Low I/O performance with ESXi 6.0


Altecice
 Share

Recommended Posts

Performance under ESXi is not an ESXi issue, its a fakeraid card issue. 

If not an ESXi issue, why thiss issue not found under KVM or Hyper-V?

Link to comment
Share on other sites

I have the same problem.

System:
HP Gen8 1610T; 10 GB RAM (8+2); SSD: Samsung 850 Pro (RAID0, in ODD bay, SATA2); HDD 2x Seagate Archive 8 TB (on SATA3) with RAID1 and RAID0 arrays. Latest BIOS & drivers as of beggining of March.
B120i read cache disabled, physical write cache of disks enabled (switching these doesn't change anything noticable IMHO)
ESXi 6.0 (HP image) on SD card.

2 VMs: Ubuntu 14.04 (with Samba) and Windows 10 technical preview. Both assigned 4 GB RAM and unlimited resources. Running both or alone doesn't change anything in terms of disk performance.

I provide both VMs with separated base SSD datastore and additional RAID1 from HDD. I've tried thin and thick (lazy zeroed) provisioning for datastores

I was very frustated with read/write speeds to Windows/Samba shares through network. (VMware vSphere client work even slower with datastores.)

Sometimes there was fully saturated 1Gbps link and then drop to 10-20%. I've tested only with large file (>1-2GB) from local SSD to Gen8 SSD datastore or to RAID1 datastore.
At first I blame the network configuration, but then I decided to test local VMs disk performance.

So. Ubuntu are ok with write - write speed are ~270 MB/s for SSD and >100 MB/s for HDD RAID1. But read speeds are horrible... and identical(!) - 22.5 MB/s - the same for SSD and HDD. It's quite clear that there is as some artifical limit on reading.

However if I tried to read the file without cleaning cache, the speed is fantastic - GB/s. It explains why I have received something very good results when copying from network the "fresh" file.

With Windows it isn't so easy to understand. HD tune shows very interesting diagram - >90 % of disk read was about 5-6 MB/s with 3 spikes about 1500 MB/s giving average ~150 MB/s. But generally, the speed (even copy file within Windows VM SSD datastore) is something from past century. When I try to copy file from Windows VM to Ubuntu VM share (both on same SSD), the speed after initial fast spike drops nearly to zero.

Then I have tried to check ESXi host performance. I have enabled SSH&ESXi shell, logged in. Read and write speeds to datastores was even lower ~10MB/s. It explains slow VMware vSphere client's work with datastores.

Conclusions and usual suspects:
1. ESXi 6.0 somehow limits allocated host's resources (I couldn't change them in ESXi 6.0 contrary to ESXi 5.5). I'll try to take VMware vSphere Web client's trial and change allocated resources to host. 
2. HP wrote bad drivers. I'll try to boot server from some liveUSB and check disk performance outside ESXi. I have tried to use HP Insight diagnostic with sequential read of disks, but it gives also the same result for SSD and HDD - ~65 MB/s. So the bottleneck couldn't be detected.

Any ideas?


 

Link to comment
Share on other sites

You use the last HP ESXi 6.0 Image from Mar2015 ?

 

Ich have same Problems with the frist Image from HP on my ML310e with B120i.

Link to comment
Share on other sites

What I've discovered about the performance issues:

 

1. ESXi 6.0 really doesn't like my Sandisk SDSSDXP-120G in AHCI mode on any of the internal ports. It causes ESXi to hang and looking in the logs, there are loads of errors regarding ATA PASS-THROUGH commands failing then it gets stuck endlessly resetting the SATA port. The cable is fine and I've tried on both a SAS forward cable and on the ODD port with a regular SATA cable.

2. ESXi 6.0 is better with the Sandisk SDSSDXP-120G with the b120i but performance is awful and it doesn't seem entirely reliable. Reads will be fast but writes will linger around 20-30MB/s. I've had mysterious issues occasionally with the SSD connected. Long pauses when scanning for datastores etc...

3. ESXi 6.0 is fine with 2x WD Red 3TB in RAID1 with the b120i but performance is awful. Reads will be fast but writes will linger around 20-30MB/s. The SSD was attached to the ODD port at the time.

4. ESXi 6.0 is fine with 4x WD Red 3TB in RAID5 with a P222 and performance is very good both ways. The Sandisk SDSSDXP-120G on port 1 of the b120i again caused mystery issues like the datastore scanning delays. Removing the SSD or disabling the b120i controller entirely fixed it.

4. Linux (CentOS 7.1) is fine with the WD Red and the Sandisk SDSSDXP-120G with the b120i but again, performance is awful in one direction only.

5. Windows (Server 2012 R2) however is fine with 2x WD Red 3TB in RAID1 and the Sandisk SDSSDXP-120G on the b120i. Reads and writes are both fast.

 

I'm wondering about #3 whether it was the SSD that was causing problems with the mirrored WD Reds. If the SSD was generating excessive interrupts or resets then it might be causing problems with the b120i driver or controller.

 

I've no idea what is going on. I'd assume there is a fault with the controller or the SSD except for the fact it all works well in Windows. My only conclusion is that there is some issue with the Linux + ESXi b120i drivers possibly relating to my SSD.

 

My intended use for this Microserver is to run ESXi with the P222 so I'm just going to remove the SSD and disable the b120i controller.

 

For people with problems and who also have SSDs attached, can you temporarily remove the SSDs to see if it fixes the problem with your HDs?

Edited by GotNoTime
  • Like 1
Link to comment
Share on other sites

Thanks, FrankvanLight

I didn't notice latest HP iso from 30 March. Reinstalled ESXi.

Windows read performance on large files through network now is almost good, but HDtune still shows awfull results. File copying within Windows VM on SSD ir very slow and uneven. 

On Ubuntu write performance improves, but reading not - 22.8 MB/s whatever SSD or HDD.

There are HDtune results for Windows VM SSD

 

Edited by ArnisR
Link to comment
Share on other sites

You don't need to reinstall. You can just do the HP updates manually using the vSphere CLI. Same with the latest ESXi 6.0 patch (ESXi600-201504001) which just got released.

 

esxcli --server=YOUR_SERVER software vib update --depot=http://vibsdepot.hp.com/hpq/latest/index.xml --force

esxcli --server=YOUR_SERVER software vib update --depot=http://vibsdepot.hp.com/hpq/latest/index-drv.xml --force

Edited by GotNoTime
  • Like 2
Link to comment
Share on other sites

Thanks, GotNoTime

I'm novice to vmware and really highly appreciate Your advice.
I theoretically knew that here were several upgrade scenarios, but intentionally made clean install because I experimented with advanced host settings and therefore wanted restart with clean configuration.
I updated HP and ESXi patch with these commands in ESXi host's SSH shell (Without --server option. I suppose that this option is used somewhere else, because esxi shell refuse this option).  Still no improvement in read speed  on Ubuntu.

I wonder where is this "bare metal hypervisor with minimal overhead" :D

Link to comment
Share on other sites

I updated HP and ESXi patch with these commands in ESXi host's SSH shell (Without --server option. I suppose that this option is used somewhere else, because esxi shell refuse this option).

Yes. If you're running it from the ESXi shell on the console or have temporarily enabled SSH then you don't need the --server argument.

 

I was talking about https://developercenter.vmware.com/web/dp/tool/vsphere_cli/6.0which allows you to run the esxcli commands remotely.

 

There are alternative methods to update the system like vSphere Update Manager which automates it all for you but that costs money for vCenter and has significant resource requirements. For a small single host installation at home or small business then you'll need to use vSphere CLI or the ESXi shell.

 

Still no improvement in read speed  on Ubuntu.

The problems with the SATA controller in the Gen8 Microserver are very odd. It works great in Windows but Linux and ESXi both have throughput issues. I've worked around it using the P222 card but it should be fine with the built-in b120i.
Link to comment
Share on other sites

I solved my issue originally by dropping ESXi, its clear it limits your transfers. I put 2012 R2 straight onto my SSD and used Hyper-V (and loving it). All my issues are solved (Raid controller or ACHI)

 

I suggest you look into Hyper-V for home use its really great and its Microkernalized nature means amazing support for any OS and due to the nature of the parent VM (2012 in my case) you have no limits. Its truly like a Type 2 Hypervisor but is in fact a Type 1. Microsoft seem to make a habit of getting into the game late but when they do they knock the competition out.

 

Its worth a shot/Test if you are having the issues.

Edited by Altecice
Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share

×
×
  • Create New...