Jump to content
RESET Forums (homeservershow.com)

Going Big: Building A 16+ Bay Archival Server


santacruzskim
 Share

Recommended Posts

I think i may have finally lost it! I'm going to build a massive storage array for my home.  After spending way too much time shopping for and collecting external JBOD enclosures, I realized it's just a matter of time before I'm surrounded by piles of half-functional enclosures and have a storage mess on my hands. I'm thinking of doing 1 of the following:


 


1. i3 TS140 running WHS2011 (already own) -> SAS Expander Card -> External 16-24 Drive SAS JBOD enclosure


in this scenario, the TS140 would be my always-on server, serving up ONLINE files, backing up PC's, etc using all-internal drives. The external storage array would be for OFFLINE files, mainly backups and archival data, broken into ~3 pools via drivepool. It wouldn't be on unless I was specifically accessing that data (maybe once a month).


 


2. Build A New Server From the Ground Up


This would do everything listed in option 1, but be an all-inclusive unit (no external components). Something like a Norco 4220 ?


 


---


My Needs


  • Drive Pooling: I'd be using drivepool + scanner or something very similar so the OS needs direct access to each drive.
  • Form-Factor: For either option rack-mount makes the most sense, but I'm not opposed to another tower if it does the job.

  • Bays: [16-24] 16 will do just fine. I'm seeing a lot of 20-24 bay units in roughly the same size and price range.  Even by doubling my foreseeable storage needs, >24 drives is going to be overkill.

  • I/O Speed:  Anything above a consistent rate of 50MB/s will be fine.  Unless there's going to be a painful bottleneck somewhere, speed isn't that big a factor in deciding on components or an overall strategy.

  • $$$: I'd like to do this "on-the-cheap" but understand there are at least 1-2 components where buying a janky knock-off will not be a smart option.  This whole endeavor is to set myself up long-term and I rather bite the bullet now in order to have everything working properly than spend the next few months wrestling with issues.

My Thoughts


  • Option 1 gives me versatility, but hardware costs and futzing with the compatibility of more links in the chain will take more R&D.  I know there are pre-built SAS Expander boxes that are a little pricy, but are plug-in and go.  However, I don't see why I can't just build a lightweight version of the option 2 box and use that for the external storage, still passing direct disk access on to the TS140 for accessing and managing the data.  With this DIY angle, how low can I go? I'd obviously need a decent PS, but are there autonomous SAS Units that can simply pass direct disk access on to the TS140's card w/o the need of a Motherboard + CPU + RAM... This is very much the gaping hole in my research so far so some enlightening would be helpful.
  • Option 2 seems the most straight-forward, but I would like to avoid running something this massive and loud all day long. That's pretty much my only gripe with going this route, but its a big one.

  • Another option would be to do Option 2, but keep the TS140 around as my primary server.  This way I can build the new storage server on the cheap and still only fire it up when needed since its not responsible for those always-on, day-to-day server tasks.  It would still need a (server) os + drivepooling, but it could just serve up files over the network and I wouldn't have to worry about extra cards, compatibility, etc.

Being in somewhat unknown territory, I'm really just hoping to utilize those smarter than me to point me in the right direction and avoid massive follies, but if you want to provide more detailed info / hardware combo suggestions, and additional guidance, i'm all ears!  Is there an obvious Option 3 I'm overlooking?


 


thanks!


Edited by santacruzskim
Link to comment
Share on other sites

I have a SuperMicro 16-bay rackmount case (836BA maybe?  I'll have to check on that) as my primary server and it's worked out pretty well for me so far.  I does generate some noise, but I have it in the basement and can't hear it upstairs other than when I turn it on.  The fans are all variable speed, so unless it gets really busy, they don't spin up a whole lot.  The startup noise is because there is a approx 1 second burst at startup where they do hit full speed, and then they drop back down to normal operation after that.  I added a cage for 2.5" drives in the back below the power supply, so all of my front-facing drives are data drives.  In my case I use various sets of Storage Spaces (have a 2-way mirror and a parity set, with a single drive for some scratch stuff).  I only have 8 drives in it currently though, my needs haven't expanded quite that much, but they were starting to give me trouble with my 8-bay case.  It does host around 3-4TB of data though, I haven't checked in while, but my needs have leveled off a bit for now too.  I had originally tried a norco case, but after a couple backplanes caught fire (it was a bad drive that caused it, but still, they literally were on fire), I decided to try something a little bit higher-end.

 

As for the size, well, I have a 25U rack in my basement that contains that case, 2 2U cases, a switch, 2 1U boxes (router and I haven't decided yet) and some shelves for a few other misc. things (wireless router, modem, etc...)  The entire setup is only a few square feet of floor space, and is still shorter than me, so it hasn't been too bad so far.  I stuck in a room that is partially my server closet (patch panel for the hardline jacks around the house are in it) and partially storage and for the most part have been able to just ignore it.

 

Price is the hardest part, the initial investment on the server was probably $1500-1700, counting case, motherboard, memory, etc...  I watch for sales on WD reds (3TB mostly right now) and pick them up as cheap as I can, so loading it with hard drives hasn't been too bad.  I also had a number of drives around from before, so although it was a bit piecemeal to migrate data, my initial needs were pretty well covered too.

 

Off the top of my head, here are my parts (I was just doing some maintenance a week ago, so I'm pretty sure they're at least close)

SuperMicro 836BA-R920B case & power supply

SuperMicro X10SLH-F motherboard

Xeon E3-1220v3 processor (I know it's v3, might not be 1220)

8GB ECC RAM (whatever was fast and cheap)

SuperMicro 2-port SAS HBA card (not their RAID card, HBA only)

150GB Velociraptor OS drive

Several 1TB/2TB/3TB WD Red drives and growing as needed

 

One other thought for you, if you're going to store that much data, how do you plan to back it all up?  To do a good job, you theoretically need at least as much backup space and primary storage space.  I don't actually back up everything myself, the MyMovies collection is considered to be replaceable and doesn't need to be backed up (yet).  That accounts for about 2TB of my data currently, so my backup needs are still fairly small.

Link to comment
Share on other sites

Well, I have a Norco RPC-4220. I'd rather have gotten the 4224, but it was $55, so I literally couldn't pass it up.

The 4220 has 20 bays, and the 4224 has 24.  And both have "slots" for two 2.5" drives... though the 4224, you have to get a "tray" that screws in, as well. 

Either way, it uses SAS cables, so great cable management.

Rest of the specs are in my signature. Not to dissimilar from Andne's setup, actually. Though, I have 32GB of RAM, and am running HyperV... and 3x IBM ServeRAID m1015 cards crossflashed to "IT mode". Though I suppose I could have gotten an expander card or two instead of three cards...

Though all the disks are pooled together with StableBit DrivePool, and monitored with StableBit Scanner.

And I have 8-9 bays free, still.

 

Speaking of "on the cheap", Ikea Lack tables are great for rack-mount equipment. The "hacker" name for them is the Lack Rack. :)

For the large rackmount case, the Lack coffee table is perfect (well, almost, a half inch narrower would have been perfect).

 

 

 

To be blunt here, I absolutely hate external solutions. SAS stuff may be more reliable but I've long ago been turned off of ANY external solution. It's always more problematic.

Also, using a single case for everything means that you have much more control over the cooling and airflow.

 

As for operating system, it depends on what you want. For just a file server, then Windows 7 or 8 would do. 
But I'm a big fan of Server 2012R2 Essentials. It's WHS2011 on crack.

Link to comment
Share on other sites

It's not 1 of the options you mentioned but, just to throw it in there, if I was looking at mass storage like you are I think I would investigate building a BackBlaze system. I have no idea if the drives are directly accessible to the OS (so you can run Scanner for example), but it looks like an interesting system.

Link to comment
Share on other sites

Thanks everyone for the great feedback!

 

After taking some time to think about this as an actual, physical thing in my house and how that would logistically work out, I've come to the conclusion that its just not practical to run something so large, loud, and power hungry 24/7. Though there are other aspects that make an all-in-one appealing, its just not a good fit for me. Also, once I mapped everything out I realized the difference between an internal and external in terms of data signal flow would be 1 cable (an 8088 connecting the SAS Expander to the HBA instead of an 8087).

 

So, my question becomes much more specific: I can either get a Norco DS-24E [$~1260] or a comparable pre-built external SAS storage system (though I came up fairly empty-handed googling around last weekend) or get a Norco 4224 [$~450] case and build a system from the ground up inside of it (again, just used to house data and feed it to the main server when needed).

 

Here's my breakdown of the 2 Options

  1. The 24E will be much quicker to get up and running, has a smaller footprint (it'll fit a lot nicer in my closet rack), and would be made of higher end components, some of which I can re-purpose or sell down the road.
  2. The 4224 will be cheaper when its all said and done, but is almost guaranteed to take longer, provide more headaches (though i'll also enjoy putting it all together), and the final product will be made of lower quality components (though i'm definitely not dumpster-diving here!). This route definitely provides me with more flexibility for any future storage plans.  To be clear, this configuration would not need a standard motherboard, ram, cpu, etc..  Here's the price breakdown (based on estimates, after taxes and shipping)
    1. 4224 Case -- $450
    2. (6) 8087 Internal SAS Cables -- $120
    3. PSU -- $130
    4. Powerboard -- $60 (To power the SAS Expander, control the power of the rest of the system, etc.)
    5. SAS Expander -- $300
    6. Random Junk -- $40
    7. (all other purchases either apply to both options or are things i already have)

= $~1100 as configured.  I can get this down around $1000 if I want to spend some more time and headaches.

 

this post has the same purpose as the last: what obvious issue / variable have i overlooked? does anyone have tales of great success or horror stories going down a similar path?  I'll probably be getting an LSI 9207-4i4e HBA or something similar (I have no plausible need for hardware RAID at this time). Also, my needs haven't changed from my 1st post (eg. direct access to individual disks).

 

thanks!

Edited by santacruzskim
Link to comment
Share on other sites

  • 4 weeks later...

This is just my 2 cents, but this mobo would be good for up to 18 drives 1 or 2 for OS and the rest for storage. It runs with a Avoton quad-core cpu,and has the ability to use ECC or Non ECC RAM,up to 32GBs.Test show is only uses 25w to 30w idle & load,that's with quad nics and a iKVM port. With the option for a addin card for 8 more drives(if needed). This cuts out the need for the SAS expander and powerboard.Something along the lines of a IBM M1015 would be a cheaper option - 8-ports  for around $100 on ebay. So theres your 24-ports you need while saving $$$.

 

This also can run with minimal air flow, so running it quiet isn't a problem.There are other options in this platform too, like the SuperMicro A1SAi-2550F,2750F( 8-cores) and A1SRi-2758F( 8-cores) they all use laptop RAM moduals and don't have as many HDD ports. And then there is the AsRock C2750D4I( 8-cores) & C2550( 4-cores). I plan on going the Asus -P9A route here shortly,from my i3- 3220 rig. If you go with WS 2012 R2 for a OS look into Flexraid as well as what the others have suggested.

Link to comment
Share on other sites

Nice! Very interesting approach.  Definitely very sensible way to get the job done. Ya, that's a lot to drop on the mobo, but the rest of the components could be on the cheaper side with very few cons.  You could keep it all-in-one and quiet as well!

 

I ended up getting a DS-24E as it will play well with my long-term storage plans.  No regrets so far. everything's been tested and aside from needing to tweak a few things its been absolutely spectacular.

 

When I originally started this project I spec'd everything out for use with FlexRAID.  Strictly on paper, its nothing short of amazing and the developer has recently announced even more offerings in the way of software RAID.  However, after some basic testing and much forum snooping, it just wasn't solid enough to rely on for such important data.  If the developer keeps at it and works out the kinks, I believe it would be hands down the best option for data storage.  For now, I'm ok with taking the storage hit of file-based duplication in exchange for a simple, reliable system using stablebit drivepool.

 

In the future, I could see myself getting a storage-centric case and a tossing a board in it like you've suggested, but for now, I'm pretty happy with where I'm at!

Link to comment
Share on other sites

  • 4 weeks later...

That's a nice sub system,with plenty of room for you to grow. I ran Flexraid for some years without any problems( I understand your fears).But have now switched to storage spaces.  It's doing what I need so... :D Glad to hear that things are going as planned, be sure to get some pics up of the setup.

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share

×
×
  • Create New...