Jump to content
RESET Forums (homeservershow.com)
Sign in to follow this  
JackoUK

Converged 10G Home Storage Network

Recommended Posts

JackoUK

Overview

I intend to design 2 new machines: a workstation and a data server, replacing an old workstation (of 7 years) and 2 data servers (5 years). While HP Microservers have proved to be inexpensive reliable machines, I am beginning to feel the constraints of only 4 disks and the slow speed of GbE networking … and looking towards the future with these builds. The pull of trying something new has nothing to do with it J

Network Segmentation

I view the network in 4 parts.

1.       The WAN connection. My current ISP speeds of 40/20 Mbs are adequate. I could double that but I don’t use cloud backup (all local) or download much other than a few ISO’s and so feel no pressing need here.

2.       The wireless network provided by the ISP’s commodity router is also adequate. The few devices I wish to connect are in the next room - no swimming pool in the NE of England! Again I could obtain a more powerful signal (from another ISP) if I chose.

3.       The workstation and data servers are on a commodity Netgear GbE switch. Under failure conditions - a hard disk fails or on one occasion a server lost its RAID configuration and the whole server was potentially compromised - I find myself relocating TB’s of data. Now that disks are 8TB and still growing the time for recovery can run into days. Increasingly dangerous.

4.       I’d like to relegate the GbE ports in 3 to access alone and create a 10GbE storage network.

Fast Disk Subsystem

Now there is no point in having 10GbE network bandwidth if the disk subsystem only has the bandwidth of a single hard disk (say 125MBs) … and I am attracted to the automatic parallelised rebuilding process across all disks which is supported by Storage Spaces under fault conditions. I am therefore of a mind to equip both the new workstation and the new data server with 8 disks apiece and configure a 4-column mirrored Storage Space on each …

… but I’m not sure Windows will be up to the task (even if I go to Windows Pro for Workstations with ReFS). So I’d like to try INTEL software RAID and then a specialised RAID card if the software route doesn’t perform well.

10Gb Switching

Although the price of 10GbE switches is coming down, I am inclined to treat the storage network in the manner of a 2-node, hyper-converged cluster and direct-wire a separate subnet between only 2 machines, avoiding the need for 10Gb switching altogether (at least in the first instance). This idea solidified when I looked at page 34 of this Microsoft working paper.

I have also found much inspiration and useful information on the Cinevate site and the questions discussed on this site.

Attachment: Network diagram

 

 

Network diagram.JPG

Component Selection

Since both old machines are to be replaced rather than modified, the idea of purchasing motherboards with in-built 10GbE capability surfaced. The ASROCK range looks especially suitable with either Aquantia or Intel Thunderbolt circuitry on board … and sometimes both, providing a link to newer laptops which tend to have TB built in.

Data Server Option 1

Motherboard

Fatal1ty Z370 Pro Gaming 7

 

245

CPU

I3-8100 3.6GHz

4 cores

99

RAM

Corsair dual channel

2 x 8GB

170

Video

Intel 620 on CPU

 

 

PSU

Corsair CX550

550W

55

SATA DISKS

Intel RAID 5 on 6 + 2

8

 

SSD DISKS

Kingston 60GB

 

40

Case

Fractal design R5

8 disk bays

100

LAN

On board

2 x 1GbE

 

Thunderbolt

Header on board

 

 

Storage

Aquantia on board

10Gbe BaseT

 

 

TOTAL £709

Given that a new dual core, 8GB Gen 10 microserver is £306, that’s not bad.

The cost of the OS and disks are omitted for the present since they are often common to different systems.

Performance questions remain:

-          How well will the Intel software RAID work with 6 disks?

-          Is the i3-8100 powerful enough to perform the RAID computations and fill the disks?

-          How good is the Aquantia silicon?

Why Not a NAS?

The Synology 1817+ goes for 983 in the UK with:

Intel Atom quad core CPU at 2.4Ghz

8GB RAM (max 16GB)

8 disk bays

The dual port 10GbE add-in is 307 … and is SFP not 10BaseT

TOTAL £1,290.

At its maximal configuration there seems to be little flexibility and no room for expansion. 

A Real Server?

My favourite HP Ml110 Gen 10 configuration goes for 1355.

XEON Silver 4108 8 core 1.8GHz base, turbo 3 GHz

16GB ECC RAM in one slot

550W PSU

Software RAID S100i for 8 disks

10 SATA ports, 6 memory slots, 5 PCI slots including two at x16

So adding in:

-          Small SSD for the OS 40

-          2nd 4-bay LFF drive cage 138

-          INTEL dual port 10GbE NIC X540 T2 180

TOTAL £1,713

Smashing machine but too rich for my blood. Maybe one of our gurus will take such under his wing?

Workstation

The design uses the same motherboard as the data server … I have one eye on testing the data server and if it does not live up to expectations, convert it to a workstation: I am not a qualified builder or system administrator!

Motherboard

Fatal1ty Z370 Pro Gaming 7

 

245

CPU

I7-8600 3.2GHz

6 cores

280

RAM

Corsair dual channel

4 x 8GB

340

Video

nVidia GTX 1050

2GB

140

PSU

Corsair CX650

650W

83

SATA DISKS

Intel RAID 5 on 6 + 2

8

 

SSD DISKS

Samsung 960 M.2 NVME

256GB

109

Case

Fractal design R5

8 disk bays

100

LAN

On board

2 x 1GbE

 

Thunderbolt

Header on board

2 x 40GB TB

73

Storage

Aquantia on board

10Gbe BaseT

 

 

TOTAL £1,370

Testing Disk Performance

No doubt the general recommendation will be use a hardware RAID card but I would like to see for myself how INTEL software RAID 5 performs on a modern CPU.

I would also like to test the performance of a 4–column Storage Space, without and maybe with ReFS – so Windows 10 Pro for Workstations.

Before testing I need to identify … probably ¾ … of my data for cold storage. Realistically I rarely look at it and indeed may never look at some of it again. The 2 servers taken out of service can be secured with the cold data for the rare occasions the same is required.

Testing Networking

I’d also like to test Windows SMB Multichannel on my remaining Gen 8 microservers, both of which have 2x1GbE ports, and the new machines.

I’ll need to master how Windows handles multiple NIC’s on the same machine.

And of course correcting/tuning the 10GbE link.

Your turn!

Share this post


Link to post
Share on other sites
Poppapete

Very interesting.
Questions & Observations.

Is i3 4 core a bit underpowered for the data server?
What is the TB connection to laptop used for?
Will you need to do any transcoding for your video streaming?
What media server will you use?
Very important to get a case and motherboard that will handle lots of disks as you will always need more.
What JBOD do you intend to use and what card to support them?
Cisco bought into Aquantia early this year!!??

Share this post


Link to post
Share on other sites
JackoUK
Posted (edited)

Answers (and evasions!)

 

Q1.  Is i3 4 core a bit underpowered for the data server?

A1. Probably not ... because much lower specification parts appear to be enough for NAS devices offering 10GbE (e.g. the Synology 1817+). But if it is I just buy an 8600 and recover some of the £99 from the 8100.

 

Q2. What is the TB connection to laptop used for?

A2. Transferring files ... but this is not a major requirement, more of a 'future proofing' measure. The availability of mobos with both 10GbE and TB allows the possibility of adding a high-end laptop later. I can only see USB-C and TB becoming more common. The marginal additional cost is negligible.

(Further ... though not explained ... I have one eye on a modern laptop as the workstation. High-end laptops are now quad-core with 2 TB ports, so I could use one for the data server and the other for a 4K monitor.)

 

Q3. Will you need to do any transcoding for your video streaming?

A3. Dunno. I believe I've nailed audio on Windows 10: files on a Gen8 Microserver; wireless connection to the streamer; remote desktop to the streamer from a tablet on the sofa. The bandwidth for even DSD is way below wireless.

Can I turn this question round? What would  make a good 4K streamer?

If my CHUWI Hi13 tablet will do, done.

If not, surely a modern laptop will be enough (overkill probably): I am either at home watching video using a tablet ... or on  holiday using the laptop.

I've posted in a few places asking if anyone has an end-to-end 4K streaming design.

NETFLIX are saying 15Mbs ... easy for home wireless. What problem do you see?

 

Q4. What media server will you use?

A4. First copy of all data will be on the workstation. 2nd copy on the data server. 3rd (backup) copy in the garage.
 

Q5. Very important to get a case and motherboard that will handle lots of disks as you will always need more.

A5. Covered by scaling out. The mobo's 10GbE port will be connected directly to the 10GbE port on the data server. If I want another 8 disks ... I buy another 10GbE card for the workstation and another data server with 8 disks. So I don't need more disk bays, I need more PCI slots. Note too that since the workstation and data server both have 2xTB (the ASROCK card has 2 ports) ... I could chain TB data servers instead of adding 10GbE PCI NIC's.

 

Q6. What JBOD do you intend to use and what card to support them?

A6. Not clear from my explanation but DATA4 and DATA5 are just my 2 existing Gen8 Microservers. Backup  to them can run at its own speed when I can be bothered or I make a mistake. Indeed I hope only to use them if the hosue burns down: with RAID 5 an workstation and server I would be unlucky to see both devices compromised. 

 

"Cisco bought into Aquantia early this year!!??"

Sounds good for the long term viability of AQ ... and cheap 10GbE networking ... oh.oh, ... need to take a rain check on the cheap. Buy now!

Edited by JackoUK

Share this post


Link to post
Share on other sites
JackoUK
Posted (edited)

One other motherboard interests me - the Gigabyte UD4.

https://www.amazon.co.uk/Gigabyte-Intel-X299-UD4-Chipset-Motherboard/dp/B072VL89BZ/ref=sr_1_1?ie=UTF8&qid=1522770425&sr=8-1&keywords=x299+ud4

Although it has a TB connector there is only 1 GbE port and crucially no 10GbE ... but it has other redeeming features:

- it is X299

- all 8 SATA ports are exposed to Intel software RAID

- low cost than the fancier ASROCK equivalent

Data Server Option 2

Motherboard Gigabyte UD4   185
CPU i5-7640 4.0GHz 4 cores 115
RAM Corsair Dual 2 x 8 GB 170
Video nVidia GTX 1050   140
PSU Corsair CX550 550W 55
SATA Disks RAID 5 on 8 8  
M.2 Disk Samsung 960 M2 NVME 256 GB 109
Case Fractal Design R5   100
LAN on board 1 x  1GbE  
Thunderbolt Dual TB add in card Yes 53
OS W 10 Pro OEI   130
      1057

 

                                                TOTAL £1,057 

In various ways this is a more balanced system ...

... and the X299 chipset affords great extensibility: the CPU could be upgraded to a 7800X or even a 7900X and beyond, opening up all the PCI lanes.

 

 

Edited by JackoUK

Share this post


Link to post
Share on other sites
Poppapete
Posted (edited)

I like that 2nd MB option, much better in most aspects.

 

By Media server I mean "What media software" do you use?  Plex, Serviio!

 

It seems you are only serving media to a few laptops not 3 Sony TV's like I am!

Also I have to worry about VPN's so we in Australia can get BBC iPlayer and Netfix USA.

Edited by Poppapete

Share this post


Link to post
Share on other sites
JackoUK

I don't use any media server software for video or audio ... I only use players accessing remote file systems and pumping data into the target device.

If I want to listen to Vokuro I switch the audio file server, streamer, tablet and hi-fi on.

I remote desktop from the tablet to the streamer and then navigate in File Explorer to ...

/Network/HPMS3/Music/Pop/90s/Bjork/Medulla/Vokuro.flac

... and double-click, which brings up Groove (or Foobar if the file is DSD not PCM).

If I wanted music in another room (I don't) then I'd just add another streamer.

 

I'm expecting to do the same for video when I get a 4K TV:

double-click /Network/HPMS4/Videos/Family/Holidays/Azores/Terra Nostra Gardens.mp4

 

No problem with VPN: the streamer is Windows 10, so I'd just slap IPVanish thereon.

Of course we natives back in Blighty have a BBC account old chap ...

... watch American TV? You cannot be serious! :rolleyes:

Share this post


Link to post
Share on other sites
JackoUK

The final option I've been researching follows the release of the C3000 CPU ...

https://www.servethehome.com/denverton-day-official-sth-intel-atom-c3000-launch-coverage-central/

... which looks like a very good match for a storage server:

- low power consumption

- ECC

- lots of SATA ports

- 10GbE

... but I can't find a motherboard I like ... and no Thunderbolt.

Share this post


Link to post
Share on other sites
JackoUK

Changed my mind about the i3-8100 ... considering that it is only c£50 more expensive, the i5-8400 with 6 cores looks much more cost effective and powerful.

 

Found a server grade motherboard I like now:

Gigabyte MB51-PS0

https://www.servethehome.com/gigabyte-mb51-ps0-review-big-intel-xeon-d-platform/

14xSATA ports and 2x10GbE BaseT ports.

Could be expensive :(

Case? Maybe http://www.lian-li.com/pc-a79/

2 ports for SSD

12 ports for 6 column Storage Space.

Share this post


Link to post
Share on other sites
JackoUK

More ideas after further research and some new product announcements.

 

Just when I became happy with the i5-8400 … I noticed the ‘T’ series processors e.g. i5-8600T. With a lower power consumption of 35W when idling and turbo to 3.7GHz when motoring they seem well suited to a home data server role. However I can’t find anywhere that sells them!

An i9 setup looks great … but the power consumption and cost is huge. I really liked this motherboard too:

https://www.servethehome.com/supermicro-x11sph-nctf-atx-storage-motherboard-review/

but the value of the XEON CPU’s is terrible.

 

8 disks/SATA ports/bays seemed to be pretty much the general limit without expanding into external cabinets …

… until I started reading recent announcements by ICY DOCK. That company is developing a range of compact disk bays to fit in the obsolete 5.25 bays still contained in tower cases. For example it is possible to fit 4 by 15mm disks into a 1 by 5.25 bay using:

https://www.icydock.com/goods.php?id=254

The idea here is to buy an inexpensive JBOD HBA for an additional 8 SATA ports opening up the possibility of a 16 disk Storage Space – an 8 column mirror. 8 of the bays would be 3.5” disks fed from the motherboard and another 8 fed by the HBA.

Now 15mm disks means a limit of 4TB per disk and a little more expense of course …

… but theoretically those 8 columns should be getting somewhere near filling a 10GBE network connection and a 4TB disk failure shared out between 7 columns rebuilding in parallel should be over pretty quick. I should add that I’ve never got anywhere near testing any of this!

 

The ICYDOCK solution is splendidly general, opening up the choice of case. My current preference is the FRACTAL DESIGN R5 for its quietness, 8 x 3.5” bays and 2 x 5.25” bays.

http://www.fractal-design.com/home/product/cases/define-series/define-r5-black

Indeed one can even contemplate the FRACTAL DESIGN XL R2 with 4x 5.25” bays for a 24 disk system.

http://www.fractal-design.com/home/product/cases/define-series/define-xl-r2-black-pearl

ICY DOCK are also heading (but not there yet) in the right direction by employing mini-SAS connectors. Clearly we would prefer to combine SATA cabling in bundles of 4 with a 16 disk system!

Edited by JackoUK

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  



×