Jump to content
RESET Forums (homeservershow.com)

Data Duplication


Trig0r
 Share

Recommended Posts

IS there a plug in, or some sort of manual way, of checking for duplicated data, I've got loads of old drag and drop "backups" of files and folders from many different devices of mine, I could do with something that;ll compare files and list which ones are the same so I can delete them, most will have different names (multiple drag and drops into the same folder with a keep both copies addition of (1) and so on in the file name) but some might be in dated folders so while the file name is the same the path might be different..

 

Its not like I'm actualyl running out fo disk space but I just want to have a bit of a tidy up and this is the obvious way..

Link to comment
Share on other sites

Well, there is the Data Deduplication role in Windows Server 2012 R2 (standard and Datacenter). That handles this automatically, and will dedup sections of files even.

 

Otherwise, there are dozens of duplicate file finders out there.

Link to comment
Share on other sites

I don't think data deduplication is what he's looking for, that just helps to store the duplicates more efficiently... I think he wants to identify where the duplicates are so that he can remove them manually.

 

Some of the tools I use for this are Space Sniffer (visual file display tool, good for rummaging around larger folders) and for pictures, I love Visipics (visipics.info)

 

 

Sent from my iPhone using Tapatalk

Link to comment
Share on other sites

Just out of interest Chris, how does drivepool play with the built in de-dupe, do you set it on each drive that's in the pool individually or on the pool's virtual drive itself?
Or does it not work at all?

Well, there is the Data Deduplication role in Windows Server 2012 R2 (standard and Datacenter). That handles this automatically, and will dedup sections of files even.

 

Otherwise, there are dozens of duplicate file finders out there.

 

Link to comment
Share on other sites

Just out of interest Chris, how does drivepool play with the built in de-dupe, do you set it on each drive that's in the pool individually or on the pool's virtual drive itself?

Or does it not work at all?

Per drive in the pool, and not the pool drive.  Because Deduplication works by looking (and seperating) the raw data blocks, it wont' work on the pool. Also, it's probably using some other advanced file system features. 

 

 

And you need to disable the "bypass file system filter" option in the release version (2.1.1.561), otherwise you will get corrupted (partial) data.  However, the Public Beta build (2.2.0.651) will disable the feature automatically if it detects that the dedup role is installed (it's not by default).  

 

 

Specifically, the pool driver bypasses file system filters for the pooled data by default. This breaks deduplication.  It works by separating the duplicate blocks of data and storing them in the "System Volume Information" folder and leaves the unique data (or just a "marker" file) on the drive. The DeDup filter intercepts file access and reconstructs the data on the fly. 

 

While this won't get you as much savings, it will still get a good amount. 

 

 

However, because of how it works, I'd recommend backing up the data if you use deduplication... just in case.

Link to comment
Share on other sites

I'll leave it alone then, cba to piss about with it, if its not broken, and all that..

The above software that the other guys linked saved me near on 2tb of disk space anyways so I'll probs be fine for now, the only place I do actaully use DeDup as a service is on the SSD that my VM's sit on..

Link to comment
Share on other sites

I'll leave it alone then, cba to piss about with it, if its not broken, and all that..

The above software that the other guys linked saved me near on 2tb of disk space anyways so I'll probs be fine for now, the only place I do actaully use DeDup as a service is on the SSD that my VM's sit on..

Which is where I use deduplication, and it's so very nice!

 

And glad the rest has already saved you on space!

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share

×
×
  • Create New...