Checksum functions

May 30, 2008
235
2
Is there anyone using the various built-in checksum functions like @crc32, @md5, @sha512 etc. for generating and maintaining checksums of backup files (on e.g. external HDs) for verification purposes?

I have considered doing so, but maybe a specialized tool like checksum for Windows or similar would be better? (haven't tried that one yet)

Anyone has any experience or (even better!) good BTM files to share?
 
May 29, 2008
47
0
Orion Arm, MWG
Yes, I use @CRC32 extensively for such purposes.

I have one .btm, CRC32TAG, which adds a "CRC: 0123CDEF" (example)
description to any file(s) I specify. It's a little smarter than
that, of course. It first checks to see if a description begins
with "CRC: " and if that's followed by a hex string. If it finds
such a match, it skips that file and moves on to the next. If it
doesn't find that, it adds the string to an undescribed file or
prepends it to any existing description.

Its complement is CRC32CHK, which looks for that same specific
format of string in a description. If one is found, the CRC is
computed anew and compared to the stored value. Results are
displayed in green for a match or red for a mismatch, giving a
visual clue that's very easy to spot. I like and use both .btms on
a regular basis. Writing them was time well spent. I almost never
have any need to create .md5, .sfv or .par2 files for myself
anymore. And since I nearly always use TCC to perform copy and move
operations, the descriptions travel around nicely from directory to
directory or drive to drive.

I also have a few PDIR commands that integrate just that portion of
a description into their output.

I realize that @CRC32 is probably the weakest of my available
choices in TCC but it produces the shortest output and is therefore
best suited to my needs.


-- Dan McMullin
 
May 30, 2008
235
2
Thank you for sharing! That was a very interesting approach.

I had considered storing the checksums in some way (planned to use MD5 rather than CRC perhaps) but updating and querying the stored values in a single file seemed a little troublesome. (I'd rather avoid having the overhead (and clutter) of an extra .md5 file for every archived file)

Never thought to have DESCRIBE and @DESCRIPT do the heavy lifting! Much easier and should hopefully also be faster than parsing custom txt or xml files manually with a btm script. I currently don't use file descriptions at all (never saw the need with long file names) so it will not collide with any existing uses either in my case.

I've since also discovered Microsoft's FCIV tool to generate and compare checksums (MD5 or SHA) to an XML file for either single files or whole directory trees, but it seems it does not support incremental updates of the stored checksums, and I would rather avoid regenerating the checksums for several gigabytes just for adding a few small files! This should be no problem at all for a TCC script on the other hand.
 
May 20, 2008
3,515
4
Elkridge, MD, USA
From: nikbackm
| I had considered storing the checksums in some way (planned to use
| MD5 rather than CRC perhaps) but updating and querying the stored
| values in a single file seemed a little troublesome. (I'd rather
| avoid having the overhead (and clutter) of an extra .md5 file for
| every archived file)
On CRC32 vs. MD5 - I have read, but cannot cite a source, that MD5 - despite its more complex algorithm and longer hash code - has more "collisions" (files that are different but have the same hash code) than CRC32. Since the sole purpose is to detect unwanted changes, I'd go for CRC32.
BTW, the PDIR command's output options include field r - the same CRC32 generated by the @CRC32[] function. You may benefit from its use in the program to generate or to verify the hash codes.
--
Steve
 
May 29, 2008
47
0
Orion Arm, MWG
One of my PDIR aliases places a portion of any description just
before the "r" field so that I get this type of output:

... CRC: DEADBEEF DEADBEEF ...
... CRC: F00DFACE 00C0FFEE ...
... <Undescribed> 01020304 ...

This also provides a convenient way to verify that the files have or
haven't changed but has the "drawback" of always generating the
CRC. In addition to having color-coded output to catch my eye, the
.btm approach allows me to skip generating the CRC on any file(s)
which don't already contain it in the description, which would be
wasted effort for my purposes.


-- Dan McMullin
 
Similar threads
Thread starter Title Forum Replies Date
Charles Dye Documentation Help Nits Forever: Checksum functions Support 0
Charles Dye Optional F parm in checksum functions Support 0
A Problem with functions @int @decimal and identifying Powershell as a shell. Support 12
Joe Caverly "Functions" in cmd.exe batch files Support 6
vefatica Library functions and line continuations? Support 4
Joe Caverly SETLOCAL removes Aliases and Functions Support 3
Joe Caverly @NAME supports library functions Support 2
Charles Dye Functions which no-one has ever tried to use.... Support 5
vefatica WHICH and @functions Support 2
vefatica Two @FUNCTIONs/aliases with the same name Support 4
M Keywords for commands, functions, internal variables Support 3
C Documentation near or fully obsolete functions Support 7
C coding functions and subroutines Support 9
rps Functions @year[date] @isowyear[date] help Support 7
mikea How to? TCC and XMLnodes/XMLxpath functions Support 0
Dan Glynhampton Are these two functions the same? Support 2
M How to? Logic in functions... Support 5
A WAD function return values with parentheses prevent evaluation of additional functions Support 31
M "History" command not giving me what I want when executed by @Exec... functions... Support 6
Charles Dye Web help on functions Support 2
samintz Help examples for the @ISxxx functions Support 3
R specific functions for number conversion Support 2
C Unsuppressable errors from XML functions Support 5
J TCC Out of memory with aliases and functions Support 5
Peter Bratton Disappearing functions Support 1
A I dont see anything in the alias/functions/batchvariables tabs in IDE Support 13
dcantor Comments and suggestions on functions for binary buffers Support 4

Similar threads