Jump to content
RESET Forums (homeservershow.com)

  • Dave
     Share

    by AJ Peck (aka usacomp2k3)

     

    The purpose of this article isn’t to do a backup 101, but rather more of a 201 or 301 perspective. It assumes a general understanding of the user-side of Backups. Basically what we are going to do is take a look at how Microsoft’s Windows Home Server handles backups. This information is specific to Version 1, which was released in 2007. A future article on how WHS V2 codename Vail does backups is planned closer to when the Vail featureset is locked in.

    Settings

    To begin with, lets go real quick through the setup screens on the WHS. First up is the backup settings:

     

    image

     

    The first is the settings window specific to backup (it should be pointed out that the WHS defaults are 3 months, 3 week, and 3 days, and the window is 12am to 6am). While this may seem fairly straight-forward, there are couple of items worth nothing. First, any computer that is on during the opening of the backup window will start to backup at that time (or it will wake up the PC’s if you have “wake up to backup” selected). For example, if you have 3 PC’s on, all 3 will start to backup. The backups will actually happen one PC at a time.

     

    The other item to note is what happens with a 4th PC that isn’t on at the beginning of the window. When you turn the PC on (or restore the network connection), the WHS connector on the PC will connect to the server. This checks various things such as password synching and stuff like that. One other thing it checks is to see if it needs to be backed up. If the PC is turned on during the backup window, it should still start the backup up during the window. In my home environment we have 3 home pc’s (a desktop and 2 laptops) and my work laptop. The main PC’s backup at 6pm during the window and my work PC backups up whenever I turn it on every few days to do some work. This is usually around 8pm, and one of the first things the machine does is backup.

     

    I would recommend that you set a timeframe that encompasses all the times that the various PC’s will be on. For example, if you have an HTPC, I’d probably setup the backup window to start with a time of day when the PC isn’t likely to be doing anything. This will make sure that the backup doesn’t interfere with your viewing. You could set the window from 6am to 11pm if you really wanted to.

     

    The second part of those backups settings is the “Automatic Backup Management” section. Here is where you configure how long you want WHS to keep the backups. This should be fairly self-explanatory.

     

    Here is what the window looks like when you click on a PC in the “Computers” tab of the console and select “view backups”:

     

    image

    Semi-technical

    Now to begin looking at how the backups work. Let me paraphrase how it works first and then we’ll look at the technical details. Consider the following scenario:

     

    The following file: (C:\Program Files\Zune\Zune.exe) is stored on 2 computers. This is for the Zune software version 4.6. They get backed up on December 31st. On January 1st, Computer A updates to version 4.7 of the Zune software. This means that the zune.exe file changes. On January 2nd, Computer B updates. The backup settings are set to keep 7 daily, 0 weekly, and 0 monthly backups. So what is on the server:

    • Dec 30 = 1 copy of zune.exe, which is linked in both Computer A’s and Computer B’s backup.
    • Dec 31 = Same as Dec 30
    • Jan 01 = 2 copies of zune.exe. The first is version 4.6’s which is linked to all of Computer B’s backups and Computer A’s backups prior to Jan 01. The 2nd is version 4.7’s which is linked to Computer A’s backup on Jan 01.
    • Jan 02 = 2 copies of zune.exe. The first is version 4.6’s which is linked to Computer B’s backups prior to Jan 02 and Computer A’s backups prior to Jan 01. The 2nd is version 4.7’s which is linked to Computer A’s backup on and after Jan 01 and Computer B’s backup on and after Jan 02.
    • Jan 03 = same as Jan 02
    • Jan 04 = same as Jan 02
    • Jan 05 = same as Jan 02
    • Jan 06 = same as Jan 02
    • Jan 07 = 2 copies of zune.exe. The first is version 4.6’s which is linked to Computer B’s backups prior to Jan 02. The 2nd is version 4.7’s which is linked to all of Computer A’s backups and Computer B’s backup on and after Jan 02.
    • Jan 08 = 1 copy of zune.exe, which is linked in both Computer A’s  and Computer B’s backups.

    So what happened here? Each week (at the beginning of the Sunday backup window to be exact), there are 2 actions that take place: 1/ delete backups 2/ clean up backup database. This first action is just a simple check that the server does to see if any backups are outside of the windows defined in the settings. In this scenario, if there are any backups more than 7 days old, those backups get deleted. The 2nd action is to clean up the database. What this does is to check every file that exists in the backup catalog and see if there are any orphaned files, that is, files that were originally linked to an old backup and are not linked to any current backups.

     

    In our scenario above, the first action started taking place on Jan 7th, by deleting both computers backups of Dec 30. On Jan 8th, the backups from Dec 31 were deleted. For action 2, on January 7th, there were no orphaned files, so no files were actually deleted. However on Jan 8th, the old zune.exe file was no longer referenced anywhere, and so that file was considered an orphan and deleted.

    Technical

    Now that we’ve looked at the paraphrased version, lets get into the nitty gritty. First off, the backup “database” isn’t a database like you normally think of that is hosted in a registry file or an SQL server. It is a database that stores its data in actual files.

     

    The WHS stores the backups on the server in a non-duplicated area (unless you are using the BDBB to duplicate them). On my WHS, they are in D:\folders\{00008086-058D-4C89-AB57-A7F909A47AB4} When you look in that folder, you will see basically 4 types of files. The first corresponds to the computer-specific backups. Take a look at the following:

     

    image

     

    This is all of the files for one of my PC’s. The explanation I gave above is a simplistic version. What really happens is that instead of storing files as files, it actually stores files in the form of clusters. A little background: the hard drive in your computer is comprised of clusters. When you format the disk, the file system splits the drive up into little chunks called clusters. NTFS, by default, breaks those chunks into clusters that are each 4KB in size. This means that every file in your computer is split up into 4KB chunks. Take a look at this:

     

    image

     

    What you can see is that while the zune.exe file itself is only 201KB, it actually takes up 204 KB (51 clusters worth) of disk space (any file less than 4KB still takes up a full cluster, as you can’t have multiple files in a single cluster (unless it is inside of a .zip file or the like)).

     

    When you do your initial set up of WHS and attach the first PC, the first backup will take a lot of time. What WHS actually does is copy every cluster to the server. The client computer calculates what are called hashes for each cluster. A hash is basically the output of an algorithm that is performed for a set of data. In the WHS arena, each cluster runs through the algorithm and a hash is generated that (for all intents and purposes) uniquely identifies this cluster. This hash is sent to the WHS along with the cluster for storage.

     

    When you do subsequent backups on that PC or on a different PC, the client connector reads every cluster on the hard drive and looks to see when that cluster was last changed (this is a function of the NTFS file system). If that change date is after the time of last backup, the connector generates a hash for that cluster. It sends the hash to the WHS. The WHS looks up that hash in the backup database and if that hash doesn’t return a result, that cluster is then sent to the WHS for safe keeping. What this practically means is that any time a file changes, the client checks with the WHS to see if the WHS already has that file from another computer and if not, sends the file for storage.

     

    The output from each backup is thus twofold. The first is that each cluster on the client computer will have a copy on the WHS (whether due to being recently copied or from prior existence). The second is that each backup will have a list of every cluster that is on the client hard drive and the hash corresponding to where that cluster is stored in the WHS backup database.

     

    Speaking of which, lets look at where those clusters are stored:

     

    image

     

    WHS stores the clusters in a .dat file that grows to a maximum of 4GB in size (that would be 2^20  or roughly 1 million clusters). It then rolls over to the .1.dat and then a .2.dat and so on and so forth as far as is needed to store every cluster (I’m up to .152.dat on my WHS). These can be thought of as non-compressed .zip files that are store the clusters in a flat (no subfolders) format. These .dat folders don’t associate the clusters with any specific computer or backup. One other thing to notice from the screenshot above is that .dat files don’t stay at the 4GB file size (of my 153 files, only 38 are still at 4GB). When the backup cleanup is done, any orphaned clusters are deleted, but the .dat files aren’t consolidated.

    Misc Notes

    A few other things to note: the reason why WHS can’t do simultaneous backups is because the database needs the backups to be serialized in order for the hash database to be accurate. Because the client backup procedures does a lookup of hash values, you wouldn’t get accurate results if it was doing multiple backups at the same time. The second is that if you have systems with different cluster sizes (such as an HTPC with 64KB clusters (like SageTV sometimes recommends)) the cluster hashes will always be different than a system with 4KB clusters and thus there will be duplicate data stored on the WHS. Lastly, WHS will only backup from NTFS shares, which is why it doesn’t support Windows 98 (which uses fat32) or Mac OS (which uses HFS).

     

    Excluded files. I just wanted to point out real quick that there are files that WHS doesn’t back up. These are:

    • User temporary files
    • System page file
    • Hibernation file
    • Client-side cache folders
    • Shadow volume implementation folders
    • Media Center temporary files &  Recorded TV (this can be changed if desired)

    Conclusion

    I hope this helps to give the reader more information as to how WHS backups work. It truly is a marvelous system and, in my opinion, is one of the most flexible backup systems out there. This is especially true when you consider how it performs differential backups, taking multiple systems into account. You can restore from any backup of any PC onto whatever you want. (You can even backup from/to virtual PC’s which is really nice). You can even mount a backup image as a virtual hard drive and extract any file you want. A truly wonderful  system

     

    Resources:

     

    WHS Backup Technical Doc

     

    Special Thanks to Alex Kuretz for helping ensure the technical accuracy.

     Share


    User Feedback

    Recommended Comments

    Fantastic explanation! Thanks! I have an unusual problem: after a hard disk failure, I'm attempting to backup/ restore the backup volume (on WHS 2003) which is "slightly larger" than the client/ destination volume (by a few Kb). Because of this, the backup and restore program cannot "mount the drive"! Any suggestions as to either: a) extract only the individual clusters I need? or b) reduce the size of the backup clusters to fit" onto the client/ destination PC?
    Link to comment
    Share on other sites

    i like the intro given on the topic WHS back ups. the detailed description has helped me a lot to get involved in the material. you have also made up time to get information on the method of installing back ups. thank you for the share . keep sharing. super beta prostate pill
    Link to comment
    Share on other sites

    Can you just go into the hard drive and delete all of them? The backups take up too much room on the hard drive and it is taking forever to run the clean up.
    Link to comment
    Share on other sites



    Create an account or sign in to comment

    You need to be a member in order to leave a comment

    Create an account

    Sign up for a new account in our community. It's easy!

    Register a new account

    Sign in

    Already have an account? Sign in here.

    Sign In Now

  • Our picks

    • I throw this out every once in a while.  Is anyone interested in writing up "semi-formal" reviews here on the forums?  I say semi-formal because they don't have to be pro level, just a good attempt at telling the story about the gear.  Something you have purchased lately.  You don't have to go buy stuff, just incorporate what you have already have purchased.  Hit me up with any questions.  You never know where it will lead!
        • Like
      • 4 replies
    • D-Link has dropped a couple of new products and additions to their lineup of smart home gear.  Hold on because there is a lot of gear!
      Let's start with cameras.  D-Link has new Wi-Fi cameras, the Full HD Pan and Tilt Pro Wi-Fi Camera (DCS-8302LH) and Full HD Indoor and Outdoor Pro Wi-Fi Camera (DCS-8526LH).  These are not just new, and good looking cams, they can now perform people detection and glass break detection.  The new cameras also feature expanded ways for customers to save video, including ONVIF Profile S, which enables custom storage and streaming options to personal NAS devices, built-in storage microSD capacity up to 256 GB, as well as free and paid cloud storage options. 
       

       
      This year's camera models include both an indoor and an outdoor model. The outdoor model features a spotlight and siren that can be triggered when motion is detected, deterring potential intruders. The indoor model pans around to give a full 360-degree view of any room and tracks motion. Both include two-way audio. 
      Full HD Pan and Tilt Pro Wi-Fi Camera is the model (DCS-8302LH), available in Q2, 2020, and retail pricing will be $99.99.
      Full HD Indoor and Outdoor Pro Wi-Fi Camera is the model (DCS-8526LH), available in Q3, 2020, and retail pricing will be $119.99.
      One of my favorite products is Wi-Fi.  D-Link has a whole lot of new products coming including Wi-Fi 6, Mesh, Alexa and Google Assistant integration, IFTTT, Parental Controls, and more!
       

       
      There are so many new SKU's that I can't make heads or tails of them!  I'm going to be speaking with D-Link more this week and will sort all of these out. I do want you to see all they are offering and what the release date and projected retail pricing will be.

      AC1900 Scalable Mesh Wi-Fi Router (COVR-1900-US), Q1 2020, $119.99
      AC1750 Mesh Wi-Fi Router (DIR-1750-US), Q1 2020, $99.99
      AC1900 Mesh Wi-Fi Router (DIR-1950-US), Q1 2020, $119.99
      AC1750 Mesh Wi-Fi Range Extender (DAP-1755-US), Q1 2020, $99.99
      AC1950 Mesh Wi-Fi Range Extender (DAP-1955-US), Q1 2020, $109.99
      Smart AX1500 Mesh Wi-Fi 6 Router (DIR-X1560-US), Q1 2020, $119.99
      Smart AX1800 Mesh Wi-Fi 6 Router (DIR-X1870-US), Q2 2020, $139.99
      Smart AX2400 Mesh Wi-Fi 6 Router (DIR-X2460-US), Q3 2020, $159.99
      Smart AX5400 Mesh Wi-Fi 6 Router (DIR-X5460-US), Q1 2020, $279.99
      AX1800 Mesh Wi-Fi 6 Range Extender (DAP-X1870-US), Q2 2020, $129.99
      AX1800 Whole Home Mesh System (COVR-X1872-US), QX22020, $269.99
        • Like
      • 0 replies
    • RESET Merch Shop
      Get ya Merch here!  I've created a T-Shirt shop with the famous RESET paperclip. I've pasted that clip on just about everything so you can wear it around town!  Cable bags, Coffee Mugs, T-Shirts, and more. 
       
      https://reset.threadless.com/
       
      or you can get it it via
       
      https://reset.fm/store
       
      Here is the main design.
       

       
      Below is a part of the order that I put in!  
       
       

       
      The Heather Grey above is the Extra Soft version and it is awesome.  I highly recommend the softness!    The "Chili Red" is the Tri-Blend and probably my favorite feel and color.

       
      This is the Zip-Up Hoody, Heather Indigo, Sponge Fleece, Men's, Large.  In my podcast I couldn't remember what style it was but it is Sponge Fleece. It's REALLY soft on the inside. Soft outside as well. The only problem is with the zip up hoody the logo is on the back. Regular hoody, it will be on the front.
       
      https://reset.threadless.com/
       
      or you can get it it via
       
      https://reset.fm/store
       
      Thank you for supporting this community. Everything you purchase will help keep the lights on the hard drives spinning.
       
      • 0 replies
    • OneDrive Personal Vault and expandable storage
      Microsoft's OneDrive has a few new features and options worth pointing out.  Personal Vault and Expandable Storage.
       
      Personal Vault is a protected area in OneDrive that can only be accessed with a strong authentication method or a second step of identity verification, such as your fingerprint, face, PIN, or a code sent to you via email or SMS. 
       
      Personal Vault gives you an added layer of protection for your most important files, photos, and videos—for example, copies of documents such as your passport, driver’s license, or insurance information—should someone gain access to your account or device.
      Plus, this added security won’t slow you down. You can quickly access your important documents, photos, and files with confidence wherever you are, on your PC, OneDrive.com, or your mobile device.

       
      Beyond a second layer of identity verification, Personal Vault also includes the following security measures:
       
      Scan and shoot—Using the OneDrive app, you can scan documents or shoot photos directly into your Personal Vault, keeping them off less secure areas of your device, like your camera roll.
      Automatic locking—No need to worry about whether you left your Personal Vault or your files open—both will close and lock automatically after a period of inactivity.
      BitLocker encryption—On Windows 10 PCs, OneDrive automatically syncs your Personal Vault files to a BitLocker-encrypted area of your local hard drive.
      Restricted sharing—To prevent accidental sharing, files in Personal Vault and shared items moved into Personal Vault cannot be shared.
       
      Taken together, these security measures help ensure that Personal Vault files are not stored unprotected on your PC, and your files have additional protection, even if your Windows 10 PC or mobile device is lost, stolen, or someone gains access to it or to your account.
       
      Expandable Storage
       
      If you are and Office 365 Subscriber you get 1 TB of OneDrive storage space with all the other Office goodies like Word, Excel, etc.  I know personally that I have gone over the 1TB limit and have always wanted to be able to add additional storage to my account.  Now you can!

       
      Pick and option and keep on hoarding, errr, I mean saving! Cancel anytime, upgrade at any moment.
      • 1 reply
    • Ubiquiti adds new items to the Unifi Line including UAP Flex HD and the Unifi Dream machine
      Ubiquiti has been busy.  There area ton of new items to recently released and I'm going to share two of my favorites.
       
      The UAP Flex HD and the Unifi Dream machine. The Flex HD is a mouthful of descriptors like most of UI gear is.  It's a 2Ghz 2x2 MIMO, 5GHz 4x4 MU-MIMO, POE, Indoor/Outdoor, multi mount, mesh point that is no bigger than a can of Coke.

       
      You will still need the Unifi controller although you can configure it with basic functionality with the Unifi App.  I've always found it's best to configure with your controller and then use the app as an add-on.  There are several mounting options that include sitting it on a shelf! That is something that Unifi has not had before unless you count the ceiling AP I have awkwardly mounted placed on top of a few books.  It can be found on the Unifi store for $179.
       
      https://store.ui.com/collections/wireless/products/unifi-flexhd
       
      The Dream Machine is an altogether different beast that I hope lives up to its naming.  This is the gateway drug, for lack of a better term, to the Unifi world.  The starter kit.  It is an Access Point, Gigabit Switch, Security Gateway, and the Cloud Key all in one package.  The latter being the most significant as this is something that has deterred new users from getting started with Unifi.  Requiring new users to purchase a $100 item just to run the AP's has been somewhat of a roadblock in the past.  Granted, that is improving every year with the ability to run it in the cloud, on a NAS, a Pi, Docker, MacOS, and of course Windows, it is still a barricade to getting up and running when manufacturers such as Eero offer simplicity in an app.
       

       
      The switch includes 4 LAN Ports and 1 WAN port.  All of which are Gigabit and security services such as IPS are rated at Gigabit speed. It's $299 in the Unifi store but I'm unsure how nicely it will play with other Unifi gear.  This may be a nice "first AP" with its built in Cloud Key if you can add additional units or other Unifi access points.
       
      https://store.ui.com/collections/routing-switching/products/unifi-dream-machine
      • 4 replies
×
×
  • Create New...