1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Checksum functions

Discussion in 'Support' started by nikbackm, Oct 26, 2010.

  1. nikbackm

    Joined:
    May 30, 2008
    Messages:
    194
    Likes Received:
    1
    Is there anyone using the various built-in checksum functions like @crc32, @md5, @sha512 etc. for generating and maintaining checksums of backup files (on e.g. external HDs) for verification purposes?

    I have considered doing so, but maybe a specialized tool like checksum for Windows or similar would be better? (haven't tried that one yet)

    Anyone has any experience or (even better!) good BTM files to share?
     
  2. WavSlave

    Joined:
    May 29, 2008
    Messages:
    47
    Likes Received:
    0
    Yes, I use @CRC32 extensively for such purposes.

    I have one .btm, CRC32TAG, which adds a "CRC: 0123CDEF" (example)
    description to any file(s) I specify. It's a little smarter than
    that, of course. It first checks to see if a description begins
    with "CRC: " and if that's followed by a hex string. If it finds
    such a match, it skips that file and moves on to the next. If it
    doesn't find that, it adds the string to an undescribed file or
    prepends it to any existing description.

    Its complement is CRC32CHK, which looks for that same specific
    format of string in a description. If one is found, the CRC is
    computed anew and compared to the stored value. Results are
    displayed in green for a match or red for a mismatch, giving a
    visual clue that's very easy to spot. I like and use both .btms on
    a regular basis. Writing them was time well spent. I almost never
    have any need to create .md5, .sfv or .par2 files for myself
    anymore. And since I nearly always use TCC to perform copy and move
    operations, the descriptions travel around nicely from directory to
    directory or drive to drive.

    I also have a few PDIR commands that integrate just that portion of
    a description into their output.

    I realize that @CRC32 is probably the weakest of my available
    choices in TCC but it produces the shortest output and is therefore
    best suited to my needs.


    -- Dan McMullin
     
  3. nikbackm

    Joined:
    May 30, 2008
    Messages:
    194
    Likes Received:
    1
    Thank you for sharing! That was a very interesting approach.

    I had considered storing the checksums in some way (planned to use MD5 rather than CRC perhaps) but updating and querying the stored values in a single file seemed a little troublesome. (I'd rather avoid having the overhead (and clutter) of an extra .md5 file for every archived file)

    Never thought to have DESCRIBE and @DESCRIPT do the heavy lifting! Much easier and should hopefully also be faster than parsing custom txt or xml files manually with a btm script. I currently don't use file descriptions at all (never saw the need with long file names) so it will not collide with any existing uses either in my case.

    I've since also discovered Microsoft's FCIV tool to generate and compare checksums (MD5 or SHA) to an XML file for either single files or whole directory trees, but it seems it does not support incremental updates of the stored checksums, and I would rather avoid regenerating the checksums for several gigabytes just for adding a few small files! This should be no problem at all for a TCC script on the other hand.
     
  4. Steve Fabian

    Joined:
    May 20, 2008
    Messages:
    3,520
    Likes Received:
    4
    From: nikbackm
    | I had considered storing the checksums in some way (planned to use
    | MD5 rather than CRC perhaps) but updating and querying the stored
    | values in a single file seemed a little troublesome. (I'd rather
    | avoid having the overhead (and clutter) of an extra .md5 file for
    | every archived file)
    On CRC32 vs. MD5 - I have read, but cannot cite a source, that MD5 - despite its more complex algorithm and longer hash code - has more "collisions" (files that are different but have the same hash code) than CRC32. Since the sole purpose is to detect unwanted changes, I'd go for CRC32.
    BTW, the PDIR command's output options include field r - the same CRC32 generated by the @CRC32[] function. You may benefit from its use in the program to generate or to verify the hash codes.
    --
    Steve
     
  5. WavSlave

    Joined:
    May 29, 2008
    Messages:
    47
    Likes Received:
    0
    One of my PDIR aliases places a portion of any description just
    before the "r" field so that I get this type of output:

    ... CRC: DEADBEEF DEADBEEF ...
    ... CRC: F00DFACE 00C0FFEE ...
    ... <Undescribed> 01020304 ...

    This also provides a convenient way to verify that the files have or
    haven't changed but has the "drawback" of always generating the
    CRC. In addition to having color-coded output to catch my eye, the
    .btm approach allows me to skip generating the CRC on any file(s)
    which don't already contain it in the description, which would be
    wasted effort for my purposes.


    -- Dan McMullin
     

Share This Page