Jump to content
RESET Forums (homeservershow.com)

Very slow disk speeds in certain scenarios


molnart
 Share

Recommended Posts

I have recently started to play around with Proxmox on my microserver and i noticed huge IO delays and slow disk write speeds when cloning/restoring VMs. After several attempts i have identified my ssd may be the culprit, but a brand new disk does the same. After several tries i have concluded that the following command provides a pretty accurate description of the drives behaviour in proxmox. 

dd if=/dev/zero bs=1M count=64 | pv -s 64M | dd of=/target/drive oflag=direct

I have tried this command on other PC as well, inserting the pv part in the middle gives significanlty slower write speeds than expected. but the difference is *HOW* slow it is.

 

So i put in a bunch of different drives I have at home, fired up systemrescuecd, formatted them all to ext4 and run the above command on different drives.

 

Here is the results:

Drive 1: ancient OCZ Vertex3 128 GB SSD, drive health 78%, Drive bay 1

dd if=/dev/zero bs=1M count=64 | pv -s 64M | dd of=/mnt/Vertex3/tempfile oflag=direct
64+0 records in [11.3MiB/s] [=======================================================>        ] 89% ETA 0:00:00
64+0 records out
67108864 bytes (67 MB, 64 MiB) copied, 5.56159 s, 12.1 MB/s
64.0MiB 0:00:05 [11.5MiB/s] [==============================================================>] 100%            
131072+0 records in
131072+0 records out
67108864 bytes (67 MB, 64 MiB) copied, 5.574 s, 12.0 MB/s

[root@sysresccd ~]# dd if=/dev/zero bs=1M count=512 of=/mnt/Vertex3/tempfile oflag=direct
512+0 records in
512+0 records out
536870912 bytes (537 MB, 512 MiB) copied, 1.20429 s, 446 MB/s

Drive 2: Kingston SM200 mSATA SSD in a mSATA-SATA adapter, drive health 97%, drive bay 2

 

dd if=/dev/zero bs=1M count=64 | pv -s 64M | dd of=/mnt/SM200/tempfile oflag=direct
64+0 records in [11.0MiB/s] [=====================================================>          ] 85% ETA 0:00:00
64+0 records out
67108864 bytes (67 MB, 64 MiB) copied, 5.8027 s, 11.6 MB/s
64.0MiB 0:00:05 [11.0MiB/s] [==============================================================>] 100%            
131072+0 records in
131072+0 records out
67108864 bytes (67 MB, 64 MiB) copied, 5.81387 s, 11.5 MB/s

[root@sysresccd ~]# dd if=/dev/zero bs=1M count=512 of=/mnt/SM200/tempfile oflag=direct
512+0 records in
512+0 records out
536870912 bytes (537 MB, 512 MiB) copied, 1.24386 s, 432 MB/s

Drive3: Seagate Momentus 7200.1 74GB HDD, drive health 100%, barely used (i have decreased the dataset because it took ages), Bay 3

[root@sysresccd ~]# dd if=/dev/zero bs=1M count=16 | pv -s 16M | dd of=/mnt/M72001/tempfile oflag=direct
16+0 records in [60.0KiB/s] [==============================================================> ] 99% ETA 0:00:01
16+0 records out
16777216 bytes (17 MB, 16 MiB) copied, 275.365 s, 60.9 kB/s
16.0MiB 0:04:36 [59.2KiB/s] [==============================================================>] 100%            
32768+0 records in
32768+0 records out
16777216 bytes (17 MB, 16 MiB) copied, 277.541 s, 60.4 kB/s

[root@sysresccd ~]# dd if=/dev/zero bs=1M count=512 of=/mnt/M72001/tempfile oflag=direct
512+0 records in
512+0 records out
536870912 bytes (537 MB, 512 MiB) copied, 17.5119 s, 30.7 MB/s

Drive 4: WD RED 4TB 40EFRX, drive health 100% (this one is actually formatted and contains some data), Bay 4

[root@sysresccd ~]# dd if=/dev/zero bs=1M count=16 | pv -s 16M | dd of=/mnt/WD40EFRX/tempfile oflag=direct
16+0 records in [22.7KiB/s] [==============================================================> ] 99% ETA 0:00:03
16+0 records out
16777216 bytes (17 MB, 16 MiB) copied, 729.189 s, 23.0 kB/s
16.0MiB 0:12:12 [22.4KiB/s] [==============================================================>] 100%            
32768+0 records in
32768+0 records out
16777216 bytes (17 MB, 16 MiB) copied, 734.868 s, 22.8 kB/s

[root@sysresccd ~]# dd if=/dev/zero bs=1M count=512 of=/mnt/WD40EFRX/tempfile oflag=direct
512+0 records in
512+0 records out
536870912 bytes (537 MB, 512 MiB) copied, 11.6139 s, 46.2 MB/s

Drive 5: Samsung 860EVO 512GB, brand new disk, ODD bay via sata adaptor

root@sysresccd ~]# dd if=/dev/zero bs=1M count=16 | pv -s 16M | dd of=/mnt/840evo/tempfile oflag=direct
16+0 records in [ 288KiB/s] [=============================================================>  ] 98% ETA 0:00:00
16+0 records out
16777216 bytes (17 MB, 16 MiB) copied, 54.7701 s, 306 kB/s
16.0MiB 0:00:55 [ 297KiB/s] [==============================================================>] 100%            
32768+0 records in
32768+0 records out
16777216 bytes (17 MB, 16 MiB) copied, 55.2053 s, 304 kB/s

[root@sysresccd ~]# dd if=/dev/zero bs=1M count=512 of=/mnt/840evo/tempfile oflag=direct
512+0 records in
512+0 records out
536870912 bytes (537 MB, 512 MiB) copied, 3.21884 s, 167 MB/s

The question is, why are these other drives so extremely slow? On a different computer I get more "normal" speeds both when writing to SSD (12 MB/s) and HDD (  3,5 MB/s).

I know inserting the pv into the pipe slows down the process and I am also aware bays 3-4 and the ODD bay are SATA II only. However the speeds with pv seem extremely slow to me.

 

I will do further tests switching the bays the drives are located in, just in case.

Edited by molnart
Link to comment
Share on other sites

i did some further tests, exchanging the disks between bays and created a chart with the read speeds:

 

i don't think there's anything wrong with the Samsung 860 EVO, Crucial MX300, WD40EFRX and Seagate 7200.1 disks. I have tested the MX300 on a windows machine extensively, running full write tests and the disk reached consistent full SATA speed on the whole surface.

 

image.png.822a9940b1fa79dabc750bd730e19aa5.png

 

so what's wrong with my machine?

 

My system config is Xeon E3-1265Lv2, 16GB ECC RAM. I am thinking about getting a SmartArray RAID controller, but I don't need RAID and i think it would be wasted money

Link to comment
Share on other sites

so i have just switched the B120i to RAID mode and suddenly all my drives have "normal" speeds.

 

dd if=/dev/zero bs=1M count=64 of=/target/drive oflag=direct

reaches full SATA speeds respective to the speed of the bay the disk is in (440 MB/s or 250 MB/s)

 

and using

dd if=/dev/zero bs=1M count=64 | pv -s 64M | dd of=/target/drive oflag=direct

has 12-14 MB/s across all drives (this time i tested only the SSDs, no HDDs in the server at the moment)

 

what i did was, that at took two of the drives (Vertex3 and Crucial MX300) and created two RAID 0 arrays out of them. interestingly their content remained untouched. also the rest of the drives that are not in an array have now normal read speeds and their content is reachable.

 

so this raises new questions and possibilities:

1) why is AHCI mode in my case so much slower than RAID mode? HSS knowledge says performance should be the same

2) if I keep my system in RAID mode, my experience from this experiment suggests, that I could put my system SSD in to the ODD Slot, create a single RAID 0 array from it, pop in my 4 prefilled HDDs and mount them normally. Even if this post is telling otherwise https://homeservershow.com/forums/topic/8595-time-to-start-b120i-faq/?do=findComment&comment=92355

 

thanks again for your time

Link to comment
Share on other sites

  • 1 year later...

Hello @molnart

 

Last year i faced the same issues... i have 3 gen8 like yours (same ram and cpu) using 3 nodes in proxmox but cannot get it right... always get a lot of delay i/o is painfull... as i boot in odd sata i always use it in raid mode, also i have 10gb pci card, so cannot try with other raid controller... i never find a solution and end up without the ceph cluster...

 

Do you have done any more test?

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share

×
×
  • Create New...