Jump to content
RESET Forums (homeservershow.com)
Cow1200

40 TB of music help

Recommended Posts

GavinCampbell

40tb is a lot of storage but I know a lot of djs that like to only use wav files and they take up a lot of space. My dj collection is about 300gig but it's all compressed and I worked hard to keep it organized to avoid dupes.

 

If you set up a home server (2011 or 2012 essentials) there is a plugin to help identify dupes to help clean up the library. I'm sure thee are regular windows programs to do the same.

 

I would probably start with a home server with a few hard drives to centralize the data. Then copy over a few hard drives of data and start organizing it. Then add those now empty drives to the pool if possible and repeat.

 

It will take a lot of time but worth it in the end.

 

Also plan out your backup strategy. I'm sure when you are done you won't have 40tb of data but after all that work you don't want to lose it. I just invested in crash plan for my collection. It will take a long time to upload but eventually I will have an online backup of my data as well.

Share this post


Link to post
Share on other sites
Joe_Miner

-snip-

Order a Micro server to work with.... What OS and software do I use.

 

Wow, even if you don't have 40TB of data/music you no doubt have a LOT of data to store. And when you start thinking about backup you're looking at very large storage requirements. I'm not going to address backup -- you can go here and start getting a feel about 3-2-1.

 

OS: You're way beyond the limitations of WHS-2011 IMHO and I would suggest looking at Windows Server 2012 Essentials -- with as much data as you have you don't want to mess with WHS-2011 and it's issues like 2TB backup limits etc.

 

HD's: With 4TB drives you could get 5 drives into a MicroServer (put your OS on a smaller 6th drive) and have 20TB in the MicroServer. With USB 3.0 cards you could attach mass storage devices (preferably ones that support 4TB drives) -- so with some work you could get 40TB in/attached to the MicroServer but I wonder if the MicroServer CPU could manage the OH for all of that storage.

 

I suspect that you could start out with the MicroServer -- but eventually you will have to move up to a more powerful server with a case and HW that will support 12 or more 4TB drives plus a duplicate somewhere as at least one of your 3 backups. There are some posts in the forums about some of those giant servers. There was one guest on BYOB podcast about a year ago that had built some huge Server you might want to check back thru the archives.

 

Hope this helps.

Share this post


Link to post
Share on other sites
jcollison

I think we are making this too hard here. I would take a dedicated PC running Windows 7 with a couple large hard drives installed. I would start moving files over until the the largest drive was 75% full. Then install iTunes and point it at the newly established music folder and let iTunes index it. It has a very easy to use dedup option that will allow you to find and remove the dups. When you have that set done, add more files and repeat until the drive is 75% full with no dups, then move that data off and start with a new set of music. Repeat until all music is deduped. Then think about a WHS to store and back it all up.

Share this post


Link to post
Share on other sites
Dave

Wow, this is crazy. I wonder if the systems could be compared via a method different than actual file comparisons? Perhaps export the a list of files of each system and dump them into a db and start some compare routines and reports. If managed properly you could find out what files were needed on each system and then make your server plan based on that.

 

I think we are making this too hard here. I would take a dedicated PC running Windows 7 with a couple large hard drives installed. I would start moving files over until the the largest drive was 75% full. Then install iTunes and point it at the newly established music folder and let iTunes index it. It has a very easy to use dedup option that will allow you to find and remove the dups. When you have that set done, add more files and repeat until the drive is 75% full with no dups, then move that data off and start with a new set of music. Repeat until all music is deduped. Then think about a WHS to store and back it all up.

 

Great suggestion.

Wonder if you could get a couple of 3TB's and stripe 'em up to a 6TB volume. With that much data you need a little room to work.

Share this post


Link to post
Share on other sites
Joe_Miner

I think we are making this too hard here. I would take a dedicated PC running Windows 7 with a couple large hard drives installed. I would start moving files over until the the largest drive was 75% full. Then install iTunes and point it at the newly established music folder and let iTunes index it. It has a very easy to use dedup option that will allow you to find and remove the dups. When you have that set done, add more files and repeat until the drive is 75% full with no dups, then move that data off and start with a new set of music. Repeat until all music is deduped. Then think about a WHS to store and back it all up.

 

Fantastic! Glad someone is thinking outside the box. With your approach this may even fit in a MicroServer with WHS-2011 -- a much cheaper solution!

Share this post


Link to post
Share on other sites
brianmullins

I think we are making this too hard here. I would take a dedicated PC running Windows 7 with a couple large hard drives installed. I would start moving files over until the the largest drive was 75% full. Then install iTunes and point it at the newly established music folder and let iTunes index it. It has a very easy to use dedup option that will allow you to find and remove the dups. When you have that set done, add more files and repeat until the drive is 75% full with no dups, then move that data off and start with a new set of music. Repeat until all music is deduped. Then think about a WHS to store and back it all up.

 

So, this is great, but the best solution is to take this one step further. Just install iTunes on all the machines, connect them to an account that has iMatch enabled and let it do it's thing. Once all machines are synced, you'd have a cloud library with every track, deduplicated. Then you get a clean machine and install iTunes, turn on iMatch and then download all the tracks. Simple, and you have all the tracks in very high quality. The only draw-back is that you would have to authorized/deauthorize as you go along since iTunes can only work with 5 machines at a time, or something like that.

Share this post


Link to post
Share on other sites
Dave

I think i like the 40TB WHS better. ;)

Share this post


Link to post
Share on other sites
jcollison

So, this is great, but the best solution is to take this one step further. Just install iTunes on all the machines, connect them to an account that has iMatch enabled and let it do it's thing. Once all machines are synced, you'd have a cloud library with every track, deduplicated. Then you get a clean machine and install iTunes, turn on iMatch and then download all the tracks. Simple, and you have all the tracks in very high quality. The only draw-back is that you would have to authorized/deauthorize as you go along since iTunes can only work with 5 machines at a time, or something like that.

 

I have to try this Match you speak of...I'm just a bit out of touch with the iTunes ecosystem at the moment.

 

So $25 a year for Match, 10 device limit and 25,000 songs. Is anyone else using this? Like it? Drawbacks?

 

Too bad it leaves out listening on an Android Phone.

Share this post


Link to post
Share on other sites
ikon

Too bad it leaves out listening on an Android Phone.

 

Shocker; Apple doesn't want to support Android! :o

Share this post


Link to post
Share on other sites
Cow1200

Wow so far I love the thinking going on here. If I don't say it later thanks guys. I wanted to hear what others would do. Within 15 min I had 8 response. At 12 hours I'm just enjoying reading everyone posting on my problem. My friend is lending me his Drobo between us we have 5 2tb drives for me to get started. It will be slow at USB speed but I won't loose any thing.

 

P.S. how do you find copy of music with iTunes. Remember this would be on windows not a Mac.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now



×
×
  • Create New...