How to best test disk IO performance on ESXi

[zxdavb]

Hi,

 

I have an HP Gen8 running the HP version of ESXi v6.0u1 running off a 32GB sd-card.

 

I have 1x 250GB Samsung 850 EVO (will soon be paired in RAID 1 with a 840 EVO currently used elsewhere) for VMs, and 2x 1GB  WD REDs in RAID 1 for storage (maybe xpenology); they are all connected by a P222/512MB controller - nothing is connected to the B120i controller.

 

I have been unimpressed with the disk performance, and before sorting it out, wanted to ask what was the best way of benchmarking disk I/O?  My preference would be to run some utility inside a Windows 7 VM, or a Windows 10 VM, but which?

 

Any suggestions would be appreciated.

 

I have tried ATTO disk benchmark v3.05 with default settings. read about 500MB/sec, but writes are 50 MB/sec at best (and usually lower than that).

[GotNoTime]

Hmm. That is odd. I don't have that trouble with my P222 under ESXi but I'm using a RAID5 array and my VMs are Linux based. Neither of those should affect it though.

 

How is your VM drive that you're benchmarking configured? Single drive RAID0 of just the SSD?

 

Atto should be fine for benchmarking disk performance. You'll get lower values than if you were running Windows on bare metal but it shouldn't go down to < 50MB/s.

[zxdavb]

How is your VM drive that you're benchmarking configured? Single drive RAID0 of just the SSD?

 

Yes, single drive RAID0 of just the SSD.

 

Do I need to disable the B120i, or such?  AFAIK, I pretty well much used defaults for everything when setting up the arrays, etc...

[GotNoTime]

Do I need to disable the B120i, or such?  AFAIK, I pretty well much used defaults for everything when setting up the arrays, etc...

No. You don't need to disable the B120i controller. I've got a SSD attached to mine for vFRC.

 

Can you run the following from the vSphere CLI?

esxcli -u AdminUsername -s YourESXiHost hpssacli cmd -q "controller all show config detail"

If you're running it on the host itself then you can skip the username and server options.

[zxdavb]

Can you run the following from the vSphere CLI?

 

In the end I used:

/opt/hp/hpssacli/bin/hpssacli controller all show config detail

The result is:

Smart Array P222 in Slot 1
   Bus Interface: PCI
   Slot: 1
   Serial Number: PDSXH0BRH8Q223
   Cache Serial Number: PBKUA0BRH7X0FA
   RAID 6 (ADG) Status: Enabled
   Controller Status: OK
   Hardware Revision: B
   Firmware Version: 6.34
   Rebuild Priority: Low
   Expand Priority: Medium
   Surface Scan Delay: 15 secs
   Surface Scan Mode: Idle
   Parallel Surface Scan Supported: No
   Queue Depth: Automatic
   Monitor and Performance Delay: 60  min
   Elevator Sort: Enabled
   Degraded Performance Optimization: Disabled
   Inconsistency Repair Policy: Disabled
   Wait for Cache Room: Disabled
   Surface Analysis Inconsistency Notification: Disabled
   Post Prompt Timeout: 0 secs
   Cache Board Present: True
   Cache Status: OK
   Cache Ratio: 25% Read / 75% Write
   Drive Write Cache: Disabled
   Total Cache Size: 512 MB
   Total Cache Memory Available: 304 MB
   No-Battery Write Cache: Disabled
   Cache Backup Power Source: Capacitors
   Battery/Capacitor Count: 1
   Battery/Capacitor Status: OK
   SATA NCQ Supported: True
   Spare Activation Mode: Activate on physical drive failure (default)
   Controller Temperature (C): 88
   Cache Module Temperature (C): 36
   Capacitor Temperature  (C): 22
   Number of Ports: 2 (1 Internal / 1 External )
   Driver Name: HP HPSA
   Driver Version: 6.0.0
   PCI Address (Domain:Bus:Device.Function): 0000:07:00.0
   Host Serial Number: CZ1538035V
   Sanitize Erase Supported: False
   Primary Boot Volume: None
   Secondary Boot Volume: None

   Port Name: 2I
         Port ID: 0
         Port Connection Number: 0
         SAS Address: 5001438034C9BDD0
         Port Location: Internal

   Port Name: 1E
         Port ID: 1
         Port Connection Number: 1
         SAS Address: 5001438034C9BDD4
         Port Location: External

   Array: A
      Interface Type: Solid State SATA
      Unused Space: 0  MB (0.0%)
      Used Space: 232.9 GB (100.0%)
      Status: OK
      MultiDomain Status: OK
      Array Type: Data
      HP SSD Smart Path: enable

      Logical Drive: 1
         Size: 232.9 GB
         Fault Tolerance: 0
         Heads: 255
         Sectors Per Track: 32
         Cylinders: 59844
         Strip Size: 256 KB
         Full Stripe Size: 256 KB
         Status: OK
         MultiDomain Status: OK
         Caching:  Disabled
         Unique Identifier: 600508B1001CFAE86895CAA31E5EBDE4
         Logical Drive Label: 0010C235PDSXH0BRH8Q2230414
         Drive Type: Data
         LD Acceleration Method: HP SSD Smart Path

      physicaldrive 2I:0:1
         Port: 2I
         Box: 0
         Bay: 1
         Status: OK
         Drive Type: Data Drive
         Interface Type: Solid State SATA
         Size: 250 GB
         Drive exposed to OS: False
         Native Block Size: 512
         Firmware Revision: EMT01B6Q
         Serial Number: S21NNSAG575652N
         Model: ATA     Samsung SSD 850
         SATA NCQ Capable: True
         SATA NCQ Enabled: True
         Current Temperature (C): 25
         Maximum Temperature (C): 70
         SSD Smart Trip Wearout: Not Supported
         PHY Count: 1
         PHY Transfer Rate: 6.0Gbps
         Drive Authentication Status: Not Applicable
         Sanitize Erase Supported: False

   Array: B
      Interface Type: SATA
      Unused Space: 0  MB (0.0%)
      Used Space: 1.8 TB (100.0%)
      Status: OK
      MultiDomain Status: OK
      Array Type: Data
      HP SSD Smart Path: disable

      Logical Drive: 2
         Size: 931.5 GB
         Fault Tolerance: 1
         Heads: 255
         Sectors Per Track: 32
         Cylinders: 65535
         Strip Size: 256 KB
         Full Stripe Size: 256 KB
         Status: OK
         MultiDomain Status: OK
         Caching:  Enabled
         Unique Identifier: 600508B1001C59A584F9B9A4398B7A43
         Logical Drive Label: 04832AA4PDSXH0BRH8Q223FA77
         Mirror Group 1:
            physicaldrive 2I:0:3 (port 2I:box 0:bay 3, SATA, 1 TB, OK)
         Mirror Group 2:
            physicaldrive 2I:0:4 (port 2I:box 0:bay 4, SATA, 1 TB, OK)
         Drive Type: Data
         LD Acceleration Method: Controller Cache

      physicaldrive 2I:0:3
         Port: 2I
         Box: 0
         Bay: 3
         Status: OK
         Drive Type: Data Drive
         Interface Type: SATA
         Size: 1 TB
         Drive exposed to OS: False
         Native Block Size: 4096
         Rotational Speed: 5400
         Firmware Revision: 82.00A82
         Serial Number: WD-WXV1E949AN87
         Model: ATA     WDC WD10JFCX-68N
         SATA NCQ Capable: True
         SATA NCQ Enabled: True
         Current Temperature (C): 27
         Maximum Temperature (C): 41
         PHY Count: 1
         PHY Transfer Rate: 6.0Gbps
         Drive Authentication Status: Not Applicable
         Sanitize Erase Supported: False

      physicaldrive 2I:0:4
         Port: 2I
         Box: 0
         Bay: 4
         Status: OK
         Drive Type: Data Drive
         Interface Type: SATA
         Size: 1 TB
         Drive exposed to OS: False
         Native Block Size: 4096
         Rotational Speed: 5400
         Firmware Revision: 82.00A82
         Serial Number: WD-WX91E94164RZ
         Model: ATA     WDC WD10JFCX-68N
         SATA NCQ Capable: True
         SATA NCQ Enabled: True
         Current Temperature (C): 27
         Maximum Temperature (C): 41
         PHY Count: 1
         PHY Transfer Rate: 6.0Gbps
         Drive Authentication Status: Not Applicable
         Sanitize Erase Supported: False

   SEP (Vendor ID PMCSIERA, Model SRCv8x6G) 380 
      Device Number: 380
      Firmware Version: RevB
      WWID: 5001438034C9BDDF
      Vendor ID: PMCSIERA
      Model: SRCv8x6G

Edited January 11, 2016 by zxdavb

[rotor]

Two things:

 

- Are you using build 88 of the HP driver? If you are using a newer version, you will get terrible performance

- Is the P222 Write-Cache enabled for that volume? Could be turned off for some reason.

 

Also, just noticed you have this (seems very high, but I don't have a P222 so can't check):

 

   Controller Temperature (C): 88

Edited January 11, 2016 by rotor

[GotNoTime]

- Are you using build 88 of the HP driver? If you are using a newer version, you will get terrible performance

That is the HP DSA driver which is used for the B120i. The P222 uses the HP SA driver which is completely different.

 

- Is the P222 Write-Cache enabled for that volume? Could be turned off for some reason.

No write cache shouldn't affect the performance that badly. It is more of an issue with RAID5 but these are mirrored drives.

 

Also, just noticed you have this (seems very high, but I don't have a P222 so can't check):

 

Controller Temperature (C): 88

That is normal for a P222 inside a Gen8 Microserver. The RAID controller chip gets very hot but only has a tiny heatsink with no fan and the Gen8 Microserver doesn't have very good airflow over the card slot. Taking off the case however completely stops the small amount of air flowing over the heatsink and the P222 will start to overheat.

[zxdavb]

Thanks for everyone's input. 

 

I read: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2092190, which gave me a hint...

 

...and I noticed the following: 

Smart Array P222 in Slot 1
...
   Drive Write Cache: Disabled

and

Logical Drive: 1
...
   Caching:  Disabled

So I set the first (Drive Write Cache) to Enabled and I got my performance back!  I understand that this setting configures the write cache on the drive itself, not the controller.

 

Also, it seems that 'When SSD Smart Path is enabled, I/O for the SSD bypasses the firmware of the controller' and thus doesn't utilize the controller's write cache (which is turned off for this logical drive anyway?).

 

Given that the controller's write cache is backed-up by a capacitor (and the SSD's is not), I may switch Drive Write Cache back to Disabled, and turn off the SSD Smart Path feature, but I will have to do some more reading first! (I think the answer is that SSD Smart Path is good for repetitive high-volume transaction-based IO).

 

Unless anyone has any recommendations in this area?

Edited January 11, 2016 by zxdavb

[zxdavb]

OK, this page is a good reference: https://kallesplayground.wordpress.com/useful-stuff/hp-smart-array-cli-commands-under-esxi/

 

After turning SSD Smart Path off (forcing IO through the controller), I tried two combinations:

 1) left - as my original configuration, but the the physical SSD's own write cache enabled (as I expected, its appears same with/without Smart Path)

 

 2) right - as left, but with the controller caching IO for that logical drive

[root@vm-esxhost:~] /opt/hp/hpssacli/bin/hpssacli controller slot=1 show config detail | grep achi
         Caching:  Disabled
         Caching:  Enabled
[root@vm-esxhost:~] /opt/hp/hpssacli/bin/hpssacli ctrl slot=1 logicaldrive 1 modify arrayaccelerator=enable
[root@vm-esxhost:~] /opt/hp/hpssacli/bin/hpssacli controller slot=1 show config detail | grep achi
         Caching:  Enabled
         Caching:  Enabled

Here is the results: 

 

 

Edited January 11, 2016 by zxdavb

[zxdavb]

And this is with the controller cache on, but the physical SSD's write cache off via:

/opt/hp/hpssacli/bin/hpssacli ctrl slot=1 modify dwc=disable