Jump to content

  •  

  • Photo

    Backup Limitations - A counterpoint


    • Please log in to reply
    9 replies to this topic

    #1 pcdoc

    pcdoc

      HSS Legend

    • Moderators
    • 3,957 posts
    • LocationLos Angeles, California

    Posted 16 April 2011 - 01:06 PM

    As a follow up to a post from the geek-accountant who had run into some limitations of the 2T backup, I did some experimenting. One of the issues with the 2T limitation is that we are all trying to squeeze everything into one drive and some of us are running into problems. When your break down the key elements, you have critical data, the server backup itself, and the client backups (OS and Local data). Since MS is using single file backup, backing up 10 systems (OS only) is not a problem with the 2T limit. Add the pile of local data on each machine to it and may become at issue and start pushing towards or exceed that limit. Add a couple of critical share folders and the server backup itself, and again you may push towards or exceed that limit. As part of your planning here are some possible considerations that will minimize or eliminate the issues with the backup limitation.

    1. If your client backups are huge because of allot of local data, consider pointing (moving) them to separate drive. It can be a standalone, RAID, or a large drive such as a 3T drive. The size of the destination drive is not limited for the client backups (only the server backup is limited), I store my client backups on a 4T RAID 5 Partition which provides redundancy I am looking for with no concerns of size limitations. If you really need to have a backup of a backup, then consider a mirror of two drives such as 2x2T, or even 2x3T drives dedicated for this task using the built in tools in the OS to create a mirror. (see link below). This way you clients are protected beyond what you had in V1 without the manual process of BDBB.

    2. If you do the above and still find that your server backup and critical shares are still not large enough, again consider splitting off your critical data into a mirror of any size which will be in a sense folder duplication and remove it from the backup. For example, if you regularly have 1.5T of critical data, 1T of local backups, plus the server. This will not obviously fit on a 2T drive. One option would be to have 1 2T drive dedicated for client backups, 2x2t mirror for the data, and a 2T drive for server backup and "selective" shares such as documents or pictures (as a third copy).

    I know that many of us wanted to pile everything together into one continuous stream, but rememeber that we did not have that in V1 either. We did not have the ability to backup the server at all and backing up our client backups required an add-in and a manual process that was painfully slow. If we break down each requirement carefully and look at our options, we will find that the limitations are workable and we can end up with a nice server experience. As a matter of fact, V2 is actually more capable and more flexible that V1 ever was if we start looking at what it can do rather at what it cant. As was the main point of the recent HSS podcast, planning is the key and with planning, there really are no limitations. As a reference, here is how I broke mine down which will shows I have much opportunity for growth.

    Critical data - 2x2T mirror. Only two of the ultra critical shares are included in the backup (about 40 gigs) as third copy (more copies and protection that we had with V1 folder duplications)
    Server backup - Dedicated 2T drive. Server OS, 40 gigs from above and misc other things from the RAID 5 partitions. Could have been a much smaller drive as 87% of it is free.
    Client backups - Redirected to a 4T RAID partition (could be a separate drive mirror dedicated for client redundancy)and not included in the backups as it being protected by the RAID (or mirror) redundancy.

    When you break it down, you will find that the data in most cases can be managed easily for 98% of the use cases. The last 2% will end up with different solutions. The point I am trying to make is that with a bit of planning you do not give anything up at all as a matter of fact you gain by going with V2 as all these task are automated and done without a single add-in.


    Creating a mirror. These instructions work perfectly on WHS-2011 as I have tested it
    http://www.howtogeek...p-in-windows-7/


    Posted Image
    • 0

    Main Server - WHS 2011, Core I5-2500, 12T RAID 5 (5x3T) + 2T of Mirror + 2T of backup
    Second Server - 2012, Core I5-2500, 12T RAID 5
    Main Systems - Core I5-4770, 16 Gigs DDR3-1600, 180 Gig Intel 330 SSD. 240 Gig Intel 520, 3T Seagate - Other systems - Core I7-2600, 2 x Core I5-2500, Core I5-4570S, Core I3-2100, and G620

    My Blog - The Doc's World
    Tablet Resource - BYOB Videos - TheBYOBPodcast

    For More Information:

    Computer Systems

    Home Network


    #2 geek-accountant

    geek-accountant

      HSS Advanced

    • Donating Member
    • 871 posts
    • LocationAtlanta Area

    Posted 16 April 2011 - 04:08 PM

    Great points pcdoc. I am following your #2 option, somewhat. Right now I have a single 2tb drive on my desktop machine which is basically just holding my photos. Moving these to a network drive is not something I want to do because of speed issues. Heck, I am considering getting 2 or 3 drives and putting them in RAID 0 and use that for my photos and other programs with a SSD for the OS to speed things up. Therefore, I will most likely NOT backup my photos to WHS and will just set up and sync with my unRAID server. Of course these are also backed up offsite using Crashplan.

    Another problem I would have is my WHS (both 2011 and v1) are virtual machines so I don't think I can make use of a larger RAID array, right?
    • 0
    unRAID server 18.5TB
    WSS-2011 5TB internal + 4x3TB RAID 5 Mediasonic USB 3.0 external storage
    Hyper-V server running 10 VM's(AMD 6 core with 16gig of ram - Raid 5 & Raid 0, plus USB 3 Mediasonic 4 drive enclosure)
    pfSense & Untangle (aka, SUPER ROUTER) running as seperate machines
    Broadcast server - broadcast the jpeg2RAW podcast - AMD 8 core 4Ghz, 8gig DDR3 1600, RAID 0
    [url="""]The jpeg2RAW podcast site[/url]

    #3 pcdoc

    pcdoc

      HSS Legend

    • Moderators
    • 3,957 posts
    • LocationLos Angeles, California

    Posted 16 April 2011 - 07:33 PM

    Great points pcdoc. I am following your #2 option, somewhat. Right now I have a single 2tb drive on my desktop machine which is basically just holding my photos. Moving these to a network drive is not something I want to do because of speed issues. Heck, I am considering getting 2 or 3 drives and putting them in RAID 0 and use that for my photos and other programs with a SSD for the OS to speed things up. Therefore, I will most likely NOT backup my photos to WHS and will just set up and sync with my unRAID server. Of course these are also backed up offsite using Crashplan.

    Another problem I would have is my WHS (both 2011 and v1) are virtual machines so I don't think I can make use of a larger RAID array, right?



    I am sure you have your reasons for using a VM on WHS but if you want speed, you might want to think about a dedicated hardware. I can tell you speed is not a real issue with the right hardware combo. Just a thought.
    • 0

    Main Server - WHS 2011, Core I5-2500, 12T RAID 5 (5x3T) + 2T of Mirror + 2T of backup
    Second Server - 2012, Core I5-2500, 12T RAID 5
    Main Systems - Core I5-4770, 16 Gigs DDR3-1600, 180 Gig Intel 330 SSD. 240 Gig Intel 520, 3T Seagate - Other systems - Core I7-2600, 2 x Core I5-2500, Core I5-4570S, Core I3-2100, and G620

    My Blog - The Doc's World
    Tablet Resource - BYOB Videos - TheBYOBPodcast

    For More Information:

    Computer Systems

    Home Network


    #4 geek-accountant

    geek-accountant

      HSS Advanced

    • Donating Member
    • 871 posts
    • LocationAtlanta Area

    Posted 16 April 2011 - 09:33 PM

    For me, WHS is only really used for backing up the PC's and not for central storage. I am using unRAID for central storage, and while it is slower than other options, it is fast enough for how I am using it. However, even if I had the right setup for speed (limited to a gig network for this discussion), I still want my photos on the local PC. I have a large database of images (well over 200,000) almost all of which are in RAW format and having them on a network drive would not be as fast as having them on the local PC. It could work and for many people this may be the preferred method, I just want as much speed as possible when working in Lightroom and Photoshop.
    • 0
    unRAID server 18.5TB
    WSS-2011 5TB internal + 4x3TB RAID 5 Mediasonic USB 3.0 external storage
    Hyper-V server running 10 VM's(AMD 6 core with 16gig of ram - Raid 5 & Raid 0, plus USB 3 Mediasonic 4 drive enclosure)
    pfSense & Untangle (aka, SUPER ROUTER) running as seperate machines
    Broadcast server - broadcast the jpeg2RAW podcast - AMD 8 core 4Ghz, 8gig DDR3 1600, RAID 0
    [url="""]The jpeg2RAW podcast site[/url]

    #5 ImTheTypeOfGuy

    ImTheTypeOfGuy

      HSS Master

    • Donating Member
    • 2,425 posts
    • LocationhOUston

    Posted 17 April 2011 - 06:40 AM

    I am sure you have your reasons for using a VM on WHS but if you want speed, you might want to think about a dedicated hardware. I can tell you speed is not a real issue with the right hardware combo. Just a thought.



    I didn't realize VM's would slow down the server? Why does it do that?
    • 0
    ITTOG


    - WHS V1: Dell XPS 420; Quad Core @ 2.66 GHz; 4 GB RAM
    - S2008R2: Lian Li PC-A70F, EVGA X58 3X SLI, i7 920 @ 2.67 GHz; 12 GB RAM, 2 x 250 GB WD Black Caviar in IcyDock Enclosure with Raid 1, EVGA GeForce GT 240, 12TB RAID5
    - HTPC: Silverstone Lascala, Gigabyte GA-H55-USB3, i3 530 @ 2.93 GHz, 4 GB Ram, 60 GB OCZ Vertex 2 SSD, 12TB RAID5
    - Personal Desktop: Lian Li PC-9F, ASUS Sabertooth P67, i7 2600k @ 4.1 GHz, 16 GB RAM, 2 x 120 GB OCZ Vertex 2 SSD's in Raid 0, EVGA GTX580
    - Kids Desktop: Dell Dimension 8400 Pentium 4 560, 3.6GHz, 2 GB RAM - Lets not forget this beauty!
    - Other Devices: iPad, Boxee Box, XBox's, PS3, Wii, and HP TouchPad

    #6 geek-accountant

    geek-accountant

      HSS Advanced

    • Donating Member
    • 871 posts
    • LocationAtlanta Area

    Posted 17 April 2011 - 07:11 AM

    That is a good question. At what point does the hard drive speed exceed the LAN speed (assuming a normal gig LAN)? I think a 2 drive RAID 0 setup can exceed gig lan speeds, but a single 7200rpm SATA drive would most likely not exceed gig lan speeds. So unless you VM is sitting on a RAID, it will not be able to max out the gig lan speed, plus I would imagine there is some additional overhead with a VM that shaves off some speed.

    In my case, my WHS is on a Hyper-V server using a RAID 5 setup. However, it is not the only VM so even if this was a fast setup, the speed is spread out among the other VM's and therefore will be slower than a standalone setup. For me, it is fast enough, but none of these setups are fast enough that I would want to store all my images on the network and access them through Lightroom.

    However, for the average user who does not have a multi 100,000 image database or does not regularly access them through a program like Lightroom, then pcdoc's methods sound like the best way to go.

    Hopefully pcdoc will correct any mistakes I made in that first paragraph as I am speaking from memory and not any facts sitting in front of me.
    • 0
    unRAID server 18.5TB
    WSS-2011 5TB internal + 4x3TB RAID 5 Mediasonic USB 3.0 external storage
    Hyper-V server running 10 VM's(AMD 6 core with 16gig of ram - Raid 5 & Raid 0, plus USB 3 Mediasonic 4 drive enclosure)
    pfSense & Untangle (aka, SUPER ROUTER) running as seperate machines
    Broadcast server - broadcast the jpeg2RAW podcast - AMD 8 core 4Ghz, 8gig DDR3 1600, RAID 0
    [url="""]The jpeg2RAW podcast site[/url]

    #7 pcdoc

    pcdoc

      HSS Legend

    • Moderators
    • 3,957 posts
    • LocationLos Angeles, California

    Posted 17 April 2011 - 10:57 AM

    That is a good question. At what point does the hard drive speed exceed the LAN speed (assuming a normal gig LAN)? I think a 2 drive RAID 0 setup can exceed gig lan speeds, but a single 7200rpm SATA drive would most likely not exceed gig lan speeds. So unless you VM is sitting on a RAID, it will not be able to max out the gig lan speed, plus I would imagine there is some additional overhead with a VM that shaves off some speed.

    In my case, my WHS is on a Hyper-V server using a RAID 5 setup. However, it is not the only VM so even if this was a fast setup, the speed is spread out among the other VM's and therefore will be slower than a standalone setup. For me, it is fast enough, but none of these setups are fast enough that I would want to store all my images on the network and access them through Lightroom.

    However, for the average user who does not have a multi 100,000 image database or does not regularly access them through a program like Lightroom, then pcdoc's methods sound like the best way to go.

    Hopefully pcdoc will correct any mistakes I made in that first paragraph as I am speaking from memory and not any facts sitting in front of me.



    You pretty much nailed it. A single drive will average 70-80 Megabytes/sec depending on the drive of course and your Gig lan should be able to handle 100-110. You will need a very fast drive or raid to feed max throughput of your lan. I am still amazed that you have that many photos. Accountant, IT, and now photographer and graphics expert....

    ITTOG,

    The main reason is resources sharing. Of course a well setup VM will run great and you should see the impact it still has to do with how much you are having it do.
    • 0

    Main Server - WHS 2011, Core I5-2500, 12T RAID 5 (5x3T) + 2T of Mirror + 2T of backup
    Second Server - 2012, Core I5-2500, 12T RAID 5
    Main Systems - Core I5-4770, 16 Gigs DDR3-1600, 180 Gig Intel 330 SSD. 240 Gig Intel 520, 3T Seagate - Other systems - Core I7-2600, 2 x Core I5-2500, Core I5-4570S, Core I3-2100, and G620

    My Blog - The Doc's World
    Tablet Resource - BYOB Videos - TheBYOBPodcast

    For More Information:

    Computer Systems

    Home Network


    #8 geek-accountant

    geek-accountant

      HSS Advanced

    • Donating Member
    • 871 posts
    • LocationAtlanta Area

    Posted 17 April 2011 - 11:14 AM

    Well, lets withhold the term "expert" for everything except the accounting piece. That is the only one I am good enough at to make a living doing. The rest are just hobbies. When the kids were smaller and played a whole host of sports, I would photograph the entire team every week. That worked out to be about 300-400 images per game. Then the "pro" for our league asked me to help her photograph the other teams and I made a bit to offset the cost of the equipment. With them I could shot several thousand per day. Eventually the fun wore off (12 hours on Saturday and 8-10 on Sunday) and it seemed more like a job than a hobby and I already have one of those, so I stopped shooting for them.

    Now, I normally only shoot a couple thousand a month, sometimes less. One of the areas I am really getting into is HDR (High Dynamic Range) photography. I have a SMALL post about it over at my photo blog (wwww.mhfoto.com/blog). For this type of photography, a single HDR image could be made up from as many as 9 individual photos. Therefore, it is not uncommon for me to take 100+ images in the process of capturing what I want from the scene.

    O, and for the graphics part, I wish I was an expert, or at least more knowledgeable. I bet it takes me twice as long to do something in Photoshop as some one who knows what they are doing. Even then, my results will not be as good.

    OK, I need to get back out int the yard before the wife gets home and catches me on the computer. Today is our 18th wedding anniversary and we are going out tonight, but first she wants to get some things done in the yard. :)
    • 0
    unRAID server 18.5TB
    WSS-2011 5TB internal + 4x3TB RAID 5 Mediasonic USB 3.0 external storage
    Hyper-V server running 10 VM's(AMD 6 core with 16gig of ram - Raid 5 & Raid 0, plus USB 3 Mediasonic 4 drive enclosure)
    pfSense & Untangle (aka, SUPER ROUTER) running as seperate machines
    Broadcast server - broadcast the jpeg2RAW podcast - AMD 8 core 4Ghz, 8gig DDR3 1600, RAID 0
    [url="""]The jpeg2RAW podcast site[/url]

    #9 pcdoc

    pcdoc

      HSS Legend

    • Moderators
    • 3,957 posts
    • LocationLos Angeles, California

    Posted 17 April 2011 - 02:38 PM

    Well, lets withhold the term "expert" for everything except the accounting piece. That is the only one I am good enough at to make a living doing. The rest are just hobbies. When the kids were smaller and played a whole host of sports, I would photograph the entire team every week. That worked out to be about 300-400 images per game. Then the "pro" for our league asked me to help her photograph the other teams and I made a bit to offset the cost of the equipment. With them I could shot several thousand per day. Eventually the fun wore off (12 hours on Saturday and 8-10 on Sunday) and it seemed more like a job than a hobby and I already have one of those, so I stopped shooting for them.

    Now, I normally only shoot a couple thousand a month, sometimes less. One of the areas I am really getting into is HDR (High Dynamic Range) photography. I have a SMALL post about it over at my photo blog (wwww.mhfoto.com/blog). For this type of photography, a single HDR image could be made up from as many as 9 individual photos. Therefore, it is not uncommon for me to take 100+ images in the process of capturing what I want from the scene.

    O, and for the graphics part, I wish I was an expert, or at least more knowledgeable. I bet it takes me twice as long to do something in Photoshop as some one who knows what they are doing. Even then, my results will not be as good.

    OK, I need to get back out int the yard before the wife gets home and catches me on the computer. Today is our 18th wedding anniversary and we are going out tonight, but first she wants to get some things done in the yard. :)


    Congrats on your anniversary and thanks for sharing the blog. Pretty impressive. Now get back to your yard work....
    • 0

    Main Server - WHS 2011, Core I5-2500, 12T RAID 5 (5x3T) + 2T of Mirror + 2T of backup
    Second Server - 2012, Core I5-2500, 12T RAID 5
    Main Systems - Core I5-4770, 16 Gigs DDR3-1600, 180 Gig Intel 330 SSD. 240 Gig Intel 520, 3T Seagate - Other systems - Core I7-2600, 2 x Core I5-2500, Core I5-4570S, Core I3-2100, and G620

    My Blog - The Doc's World
    Tablet Resource - BYOB Videos - TheBYOBPodcast

    For More Information:

    Computer Systems

    Home Network


    #10 ImTheTypeOfGuy

    ImTheTypeOfGuy

      HSS Master

    • Donating Member
    • 2,425 posts
    • LocationhOUston

    Posted 17 April 2011 - 06:13 PM

    Nice photo's. That reminds me, I have to get the wife a new camera and more importantly, a zoom lense for inside sports. She is our resident photo taker given I coach about four different sports.
    • 0
    ITTOG


    - WHS V1: Dell XPS 420; Quad Core @ 2.66 GHz; 4 GB RAM
    - S2008R2: Lian Li PC-A70F, EVGA X58 3X SLI, i7 920 @ 2.67 GHz; 12 GB RAM, 2 x 250 GB WD Black Caviar in IcyDock Enclosure with Raid 1, EVGA GeForce GT 240, 12TB RAID5
    - HTPC: Silverstone Lascala, Gigabyte GA-H55-USB3, i3 530 @ 2.93 GHz, 4 GB Ram, 60 GB OCZ Vertex 2 SSD, 12TB RAID5
    - Personal Desktop: Lian Li PC-9F, ASUS Sabertooth P67, i7 2600k @ 4.1 GHz, 16 GB RAM, 2 x 120 GB OCZ Vertex 2 SSD's in Raid 0, EVGA GTX580
    - Kids Desktop: Dell Dimension 8400 Pentium 4 560, 3.6GHz, 2 GB RAM - Lets not forget this beauty!
    - Other Devices: iPad, Boxee Box, XBox's, PS3, Wii, and HP TouchPad




    Skins By Invisioneers