i wanted to check my files match after doing large multi TB file transfers. i have MD5checker but, i it’s generating different check sums for both the source and destination files. pretty sure i’m not doing it right.
maybe i need a separate app to compare destination files from the generated md5/source files
EDIT: I THINK THERE HAS BEEN A MISSCOMMUNICATION
Just use rsync to do the copies? AFAIK, it checksums the resulting copied file to make sure it was copied correctly.
Almost. Default rsync behavior is to quick-check, which just checks filesize and modification time between sender and receiver. You can add --checksum or -c to compare a 128bit checksum between sender and reciever. You can also --checksum-choice=md5 or whatever to specify your favorite algorithm.
You should use SHA because MD5 is so broken that people have found ways to easily cause hash collisions.
If you’re checking files that you copied yourself, I doubt you went to the effort to create MD5 hash collisions.
Or did you …
You’re talking about the rsync comparison behavior, where it decides IF it needs to copy a file or can skip it.
But whenever rsync actually copies a file, it always verifies the checksum (in memory) as part of the “full-file checksum verification step”
If your disk is corrupting the data after, then dropping caches and rsync -ci SRC/ DST/ will tell you so.
Make sure the folder structure and file names are the same.
I have never used MD5checker so it might be checksumming the file attributes (date modified) so if you’re not using Robocopy or an alternative to copy the file attributes, that could explain the md5 mismatch.
If the source and destination are accessible from a Windows machine then you could use Duplicate Cleaner from Digital Volcano and compare the number of duplicate groups and the number of files.
Either that or use HashCheck Shell Extension and move the MD5/SHA file to the source/destination and recheck. I used HashCheck while migrating 6TB of files.
well i only have windows VMs right now because SMB is broken on ubuntu. it might be time to spin an up 18.04 LTS vm.
well that’s nice to know.
I apologize i do not speak your language natively and do not recognize these words and terms such as hash collision.
no not at all. i want to make sure the TBs of data have not be corrupted after transfer/copy so when i delete the source i’m not restoring corrupted data.
i can not keep the folder structer intact. it’s too long for windows. i may have to spin up an 18.04 LTS buntu vm
linux doesn’t seem to see the usb passthrough drive but, windows and osx do 
There’s also Total Commander which has an md5 / sha1 packer plugin. ie. This packer plugin lets you pack a directory, but instead of an archive with compressed file contents, you get an archive with filenames and checksums, which is just a text file itself, with filename+file checksum inside. You can generate one on each side, and compare these text files and see if they’re the same or different, and if different you can look inside and see in what way do they differ.
Also, you can run md5sum or sha1sum utilities on mac, same thing.
Edit, now that I think of it, you can compare directories directly in TC
it has neato “Verify” check box when you go to do file transfers. i’m trying this out for my last transfer, boy is it slow. 73 hours to go
EDIT: guess it’s not stable, it shit the bed and froze.
You might have hardware issues. Total Commander has been around for 20+ years - it’s unlikely it’s buggy that way.
i suppose but, you’d think all of my other VMs would have had issues as well. idk how to check ECC ram logs and such on ESXi.
some googeling showed folks complaining about bugs in years long passed about transfer that just stop or get stuck for one reason or another but, no error displayed in TC. one even suggested Double Commander.
Are you on windows? Teracopy copies and does checksum or sha while transferring.
i’ll see if i can’t give that a try. do i need to buy it or are basic features free?
There is a freeware version and a paid version. It lookups checksum or sha while copying and then verifies it afterwards. It doesn’t verify if you dont use it to copy. It can’t verify post transfer only while transferring. I have the paid version, because it lacks delete function in the free one. The nice thing about it is instead of basic windows transfer, it lets you queue files instead of starting a second file transfer.
SSH into the service console and look through the logs.
No, I think it is not a free feature on Teracopy, and also it is not free on Gs RichCopy 360 and GoodSync. you can check the free tools like FreefileSync and this is a link to how to do this task
https://freefilesync.org/forum/viewtopic.php?t=5203