Jump to content
RESET Forums (homeservershow.com)

ESXi 5.5 U2 samba speeds at 50 MB/s Help needed


wolololo
 Share

Recommended Posts

Hello everyone!

 
I've been reading this forum for about 10 days, since I got my Microserver Gen8 and I need some help with finishing my setup. This is my first whitebox and I don't have experience with servers and virtualisation. Till now, I worked with a Debian 8 server on an old laptop, some basic mysql and samba stuff. 
 
Specs:
HP ProLiant Microserver Gen8
Intel® Pentium® G2020T 2.5GHz Ivy Bridge
2 GB UDIMM stock +  Kingston ValueRAM ECC UDIMM DDR3 4GB 1600MHz CL11 Single Rank x8 1.35v w/TS Intel 
1 x WD Red 4TB SATA-III
1 x Old Western Digital Caviar 500 GB @ 7200 RPM
 
Configuration:
ESXi 5.5 U2, HP customized image, on an 8GB USB stick attached to the internal USB port.
5 x Debian Linux VMs. They are not resource intensive, but I like to compartmentalise. 
 
Problem:
I can't get more than 50 MB/s transfer speeds from my Samba VM (read/write), residing on the WD RED disk, no matter what I do.
 
I did and tried so far, in this order:
  • update to latest BIOS and ROM with HP's service pack (note: now if I try to update the firmware from Intelligent Provisioning, it suggests to "update iLO" from 2.30 to 2.10). ROM date 07/16/2015.
  • tried to install ESXi 5.5 U2 and 6.0 U1 with customized HP images in ACHI mode. After volume and VM creation, the system hangs on VM Power On.
  • switched to B120i without configuring RAID (so I think it falls back to AHCI mode). Installed 5.5U2 custom iso and it worked. Just for good measure, I reinstalled 88 hpvsa (following this thread).
  • created all 5 VMs and installed open-vm-tools. Samba has 3 GB RAM, 2.5 TB disk space, EXT4, Unlimited CPU resources. Having only this one turned on, I had 20-25 MB/s transfer speeds. I modified smb.conf with the following:
    [global]
    max protocol = SMB2
    read raw = no
    write raw = no
    socket options = TCP_NODELAY IPTOS_LOWDELAY SO_RCVBUF=8192 SO_SNDBUF=8192 SO_KEEPALIVE
    #min receivefile size = 16384
    use sendfile = true
    aio read size = 16384
    aio write size = 16384
    

    With this config, the speed increased to 45-55 MB/s (read/write) . I've tried different variations and this one is the fastest. Sometimes it would start at 110 MB/s  and slowly drop to 50 MB/s, other times it would start at 50 MB/s, increase to 80 MB/s shortly and drop back to 50.

  • Ran Ubuntu Live 14.04 bare-metal and copied a 30 GB vmdk to a windows share on the network. It ran at 110 MB/s the entire time. I couldn't test write speeds though, because vmfs-tools only allows read access. It's double the speed in the samba VM.
  • several hdparm test results: cached 4198 MB/s , buffered 127 MB/s

  • Tried a Windows VM and an Ubuntu 14.04 VM on the 4 TB disk and on the 500 GB disk. same speed.

  • installed proxmox on a different stick. It worked at 12 MB/s and noticed that it wasn't using Gigabit connectivity. Tried to change the speed with ethtool, but the system crashed, so I abandoned any other effort on Proxmox (even though I think I'd like it more than ESXi)

  • Upgraded to 6.0U1. Same speed.
  • Switched to ACHI. After remounting the volumes, the system hangs just like it did in the beggining, when trying to power on any VM. Switched back to b120i.
  • Tried a different NIC, TP-link TG-3468 (with the custom driver). Also tried 2 NICs with load balanicng. Same result.
  • Note: when I switched to ACHI the last time, I noticed something strange. During POST, it showed AHCI version 90 and then beneath it, briefly, AHCI version 88. I didn't get a good look so I'll check this out again.  

So, does anyone have any suggestions? It should work considering I got 110 MB/s with Ubuntu Live.

Link to comment
Share on other sites

Are these VM to VM speeds or VM to an external machine?  If external you might want to look at another network cable.

Link to comment
Share on other sites

Are these VM to VM speeds or VM to an external machine?  If external you might want to look at another network cable.

VM to external speeds. I checked the cable with another PC and it worked at full speed. 

 

EDIT: also tried with different cables, but the original worked at 110 MB/s on ubuntu Live. 

Edited by wolololo
Link to comment
Share on other sites

When you switched back to the B120i did you use the older driver? This is the last post in a long thread:

http://homeservershow.com/forums/index.php?/topic/9141-low-io-performance-with-esxi-60/?p=109409

 

Before I upgraded to 6.0U1, I cloned the USB disk with the installed 5.5U2. In 6.0 I had the same speed as in 5.5 so I switched to AHCI just to see if it works (didn't). When I reverted to 5.5U2, I switched to the backup USB clone and from AHCI to b120i at the same time (so the system was exactly as before the upgrade to 6.0).

 

Edit: double checked right now. I'm on  5.5.0-88OEM.550.0.0.1331820

Edited by wolololo
Link to comment
Share on other sites

Is this Wheezy or Jessie?

 

From 14.04 ubuntu uses Samba 4, which performs *much* better if you are connecting a Mac to the storage. Wheezy is back on 3.6. Of course if the client is a windows box, then I doubt the issue is samba. I would remove the socket options and any attempts to tune samba for now - generally it works well out of the box.

 

Are you sure smbd has enough CPU to push gigabit at line rate in a VM?

Link to comment
Share on other sites

Is this Wheezy or Jessie?

 

From 14.04 ubuntu uses Samba 4, which performs *much* better if you are connecting a Mac to the storage. Wheezy is back on 3.6. Of course if the client is a windows box, then I doubt the issue is samba. I would remove the socket options and any attempts to tune samba for now - generally it works well out of the box.

 

Are you sure smbd has enough CPU to push gigabit at line rate in a VM?

Jessie with Samba 4.1.17. I'm not testing from the Mac because it's usually slower with Samba than Windows. I'm only testing with Win machines. Now I've upgraded one to Win10 and it's the same.

As for resources, I think gave it enough. I tried again with 3 GB RAM, unlimited CPU, Paravirtual SCSI controller, 2 CPU, removed the socket options. It started at 80 MB/s and dropped to 40-50 in about 10 seconds.

Link to comment
Share on other sites

Are you using the send file option? I've seen this cause issues with various releases - seems to break.

 

Check your smb logging too.

 

As for increasing memory, I reckon it probably won't help. I too never see great performance with VBOX and SMB, but on bare hardware (Samba4 + OS X 10.10) sees saturated gigabit, but the CPU load on something like a Pentium 2020 is fairly high. Comparatively netatalk server is *much* lighter...

 

 

Having been on the samba mailing lists for a while, they seem to recommend leaving smb tuning alone these days, although there are definitely some improvements to be had.

 

Have you benchmarked the storage? Whats Bonnie++ / dd saying?

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share

×
×
  • Create New...