1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

"externally altered" error message

Discussion in 'Support' started by Peter Bratton, Oct 4, 2011.

  1. Peter Bratton

    Joined:
    Jul 1, 2008
    Messages:
    81
    Likes Received:
    0
    I'd appreciate some insight on the error message below. It started showing up after I installed v13, in btm files that had been running well, in some cases for years.

    TCC: (Sys) C:\Btm\stat.btm [189] Le fichier ouvert n'est plus valide car le volume qui le contient a été endommagé de manière externe.
    "%@lines[c:\wrk\stat\reg.txt]"
    TCC: (Sys) C:\Btm\stat.btm [192] Le fichier ouvert n'est plus valide car le volume qui le contient a été endommagé de manière externe.
    "%@line[c:\wrk\stat\reg.txt,4]"

    TCC: (Sys) C:\Btm\stat.btm [189] The volume for a file has been externally altered so that the opened file is no longer valid.
    "%@lines[c:\wrk\stat\reg.txt]"
    TCC: (Sys) C:\Btm\stat.btm [192] The volume for a file has been externally altered so that the opened file is no longer valid.
    "%@line[c:\wrk\stat\reg.txt,4]"

    There is a pattern, the error pops up when a batch directs the output from some command to a file, then processes each line in the file. This snippet is representative, the "offending" lines are 189 and 192, corresponding to the messages quoted above:

    Code:
    185 do root in @%wrk\registry.lst
    186   set abbr=HK%@if[%@index[%root,_LOCAL_] ge 0,LM,CU]
    187
    188   reg query "%root" >&> nul > %wrk\reg.txt
    189   set ll=%@lines[%wrk\reg.txt]
    190   iff %ll ge 4 then
    191     do cnt = 4 to %ll
    192       set txt=%@line[%wrk\reg.txt,%cnt]
    193       echo %abbr!%@trim[%@word["%=t",0,%txt]]!%@word["%=t",2-9,%txt] >> %wrk\df.txt
    194     enddo
    195   endiff
    196 enddo
    In passing, the English error message says "altered", the French says "damaged"
    --
    Peter
     
  2. dcantor

    Joined:
    May 29, 2008
    Messages:
    505
    Likes Received:
    3
    I've been getting the same error using @LINE and @LINES. I've been trying to produce the error in isolation from the batch files where it occurs but haven't succeeded in isolating it yet. From an interactive session, the error doesn't seem to occur; it only seems to occur in batch files (both .BAT and .BTM).

    Under TCC v12 (any build) this does not happen, but in TCC v13, it does.
     
  3. vefatica

    Joined:
    May 20, 2008
    Messages:
    7,784
    Likes Received:
    29
    I don't know what's happening but it's easy enough to reproduce.

    Code:
    v:\> echos > empty.txt
    
    v:\> echo %@lines[empty.txt]
    TCC: (Sys) The volume for a file has been externally altered so that the opened
    file is no longer valid.
     "%@lines[empty.txt]"
    
    v:\>
    That seems to happen whenever redirection results in a 0-byte file.

    While Rex should look into this, the OP should check his REG command.
     
  4. vefatica

    Joined:
    May 20, 2008
    Messages:
    7,784
    Likes Received:
    29
    On Tue, 04 Oct 2011 22:02:43 -0400, vefatica <> wrote:

    |That seems to happen whenever redirection results in a 0-byte file.

    Actually, @LINES seems to do that with any 0-byte file. This one was created
    months ago:

    Code:
    v:\> echo %@filesize[cwatch.log]
    0
    
    v:\> echo %@lines[cwatch.log]
    TCC: (Sys) The volume for a file has been externally altered so that the opened
    file is no longer valid.
     "%@lines[cwatch.log]"
    
    v:\>
     
  5. vefatica

    Joined:
    May 20, 2008
    Messages:
    7,784
    Likes Received:
    29
    And (coincidentally?) my plugin LC (linecount) has a problem with 0-byte files.

    Code:
    v:\> lc cwatch.log
    TCC: (Sys) Invalid access to memory location.
     "LC"
    
    v:\>
    My problem (I think) is trying to read the character before EOF to see if it's a
    newline. Rex, I'm curious about the problem with @LINES.

    On Tue, 04 Oct 2011 22:16:10 -0400, vefatica <> wrote:

    ||That seems to happen whenever redirection results in a 0-byte file.
    |
    |Actually, @LINES seems to do that with any 0-byte file. This one was created
    |months ago:
    |
    |
    |Code:
    |---------
    |v:\> echo %@filesize[cwatch.log]
    |0
    |
    |v:\> echo %@lines[cwatch.log]
    |TCC: (Sys) The volume for a file has been externally altered so that the opened
    |file is no longer valid.
    | "%@lines[cwatch.log]"
    |
    |v:\>
    |---------
     
  6. dcantor

    Joined:
    May 29, 2008
    Messages:
    505
    Likes Received:
    3
    Thanks for narrowing that down, Vince. Fortunately for me, I'm testing for zero or more so I can work around this by using @FILESIZE, but will have to test for existence with IF EXIST first.
     
  7. Peter Bratton

    Joined:
    Jul 1, 2008
    Messages:
    81
    Likes Received:
    0
     
  8. Peter Bratton

    Joined:
    Jul 1, 2008
    Messages:
    81
    Likes Received:
    0
    I'll see if a short pause before the line count makes a difference.
    --
    Peter
     
  9. Peter Bratton

    Joined:
    Jul 1, 2008
    Messages:
    81
    Likes Received:
    0
    Adding a pause as short as 1 millisecond right before the @lines[] function call quashes the error, and the function returns an expected value. Presumably that's enough time for the redirection to complete and its output file to become accessible.

    The redirection change and the message are definitely new to TCC v13. I think the function should return -1 for a zero-byte file, as it used to, instead of an error message.

    Thanks guys for your insights
    --
    Peter
     
  10. vefatica

    Joined:
    May 20, 2008
    Messages:
    7,784
    Likes Received:
    29
    On Wed, 05 Oct 2011 20:09:46 -0400, Peter Bratton <> wrote:

    |---Quote (Originally by vefatica)---
    |I don't know what's happening but it's easy enough to reproduce.
    |
    |
    Code:
    v:\> echos > empty.txt
    |
    |... the OP should check his REG command.
    |---End Quote---
    |reg query always produces at least a file header, so there is never an empty output file.  Moreover I can run the batch from the debugger and view the contents of the output file immediately after the first error is reported.  And this doesn't change the fact that the batch has worked well as is under several previous versions of TCC.
    
    Well, the error in question (1006) is a very odd one.  I have no idea how to
    cause it using "C" or with a plugin using the functions in the TCC API.  But
    being reproducible (@LINES[0-byte file]) will let Rex track it down.  As you
    say, it happens in other circumstances too.  I'm not surprised ... just really
    curious about what causes it.
     
  11. vefatica

    Joined:
    May 20, 2008
    Messages:
    7,784
    Likes Received:
    29
    On Wed, 05 Oct 2011 21:17:23 -0400, Peter Bratton <> wrote:

    |Adding a pause as short as 1 millisecond right before the @lines[] function call quashes the error, and the function returns an expected value. Presumably that's enough time for the redirection to complete and its output file to become accessible.

    I am very surprised. In a later post I pointed out that it seemed to have
    nothing to do with the redirection (or being in a batch file). Right at the
    command line, using a previously-existing 0-byte file, I get

    Code:
    v:\> echo %@filesize[cwatch.log]
    0
    
    v:\> echo %@lines[cwatch.log]
    TCC: (Sys) The volume for a file has been externally altered so that the opened
    file is no longer valid.
     "%@lines[cwatch.log]"
     
  12. Steve Fabian

    Joined:
    May 20, 2008
    Messages:
    3,523
    Likes Received:
    4
    From: vefatica
    | Peter Bratton wrote:
    |
    || Adding a pause as short as 1 millisecond right before the @lines[]
    || function call quashes the error, and the function returns an
    || expected value. Presumably that's enough time for the redirection to
    || complete and its output file to become accessible.
    |
    | I am very surprised. In a later post I pointed out that it seemed to have
    | nothing to do with the redirection (or being in a batch file). Right at the
    | command line, using a previously-existing 0-byte file, I get
    |
    |
    | Code:
    | v:\> echo %@filesize[cwatch.log]
    | 0
    |
    | v:\> echo %@lines[cwatch.log]
    | TCC: (Sys) The volume for a file has been externally altered so that
    | the opened file is no longer valid.
    | %@lines[cwatch.log]

    Same here. Tested on years-old empty file. V8 and V12 return -1, as expected, using the same command line.
    --
    Steve
     
  13. rconn

    rconn Administrator
    Staff Member

    Joined:
    May 14, 2008
    Messages:
    9,730
    Likes Received:
    80
    The error is returned by CreateFileMapping, which fails if the file size is
    0. (Probably not unreasonably, but it's undocumented behavior.)

    I've added a minor kludge in build 25 to test for a 0-byte file.
     
  14. rconn

    rconn Administrator
    Staff Member

    Joined:
    May 14, 2008
    Messages:
    9,730
    Likes Received:
    80
    Not so - nonexistent files have always returned an error. -1 is only
    returned for empty files.
     
  15. vefatica

    Joined:
    May 20, 2008
    Messages:
    7,784
    Likes Received:
    29
    On Wed, 05 Oct 2011 23:45:11 -0400, rconn <> wrote:

    |---Quote---
    |> Well, the error in question (1006) is a very odd one.
    |---End Quote---
    |The error is returned by CreateFileMapping, which fails if the file size is
    |0. (Probably not unreasonably, but it's undocumented behavior.)
    |
    |I've added a minor kludge in build 25 to test for a 0-byte file.

    Thanks for the explanation.

    I read a lot of Google hits for "1006" and the text of the message but didn't
    read far enough to find any about CreateFileMapping. Many dealt with particular
    pieces of software, Python, installers, and USB/network drives.

    Given the chance I would have done it too. I'll have to look through my
    plugins.
     
  16. Steve Fabian

    Joined:
    May 20, 2008
    Messages:
    3,523
    Likes Received:
    4
    From: rconn
    | The error is returned by CreateFileMapping, which fails if the file size is
    | 0. (Probably not unreasonably, but it's undocumented behavior.)

    I think it is unreasonable unless the mapping is created in read-only mode. However, I guess @LINE[] and @LINES[] would create a read-only map (and probably not with exclusive access), so in the instance it is reasonable. Documentation? Standard for MS.

    I presume use of this API call to implement @LINE[] and @LINES[] is new for V13, and is the reason for their faster operation.

    | I've added a minor kludge in build 25 to test for a 0-byte file.

    Thanks for the quick fix. Expected from no other vendor but you.

    I presume the new test also applies to @LINES[], which in V12 returns ***EOF***, but in V13 the same error message for an empty file.
    --
    Steve
     
  17. dcantor

    Joined:
    May 29, 2008
    Messages:
    505
    Likes Received:
    3
    Yes, sorry. I realized that a few minutes after I posted it and I edited my post to remove that statement a few minutes after I wrote it.

    -1 for empty files.
     
  18. vefatica

    Joined:
    May 20, 2008
    Messages:
    7,784
    Likes Received:
    29
    On Thu, 06 Oct 2011 11:03:43 -0400, Steve Fabian <> wrote:

    |From: rconn
    || The error is returned by CreateFileMapping, which fails if the file size is
    || 0. (Probably not unreasonably, but it's undocumented behavior.)
    |
    |I think it is unreasonable unless the mapping is created in read-only mode. However, I guess @LINE[] and @LINES[] would create a read-only map (and probably not with exclusive access), so in the instance it is reasonable. Documentation? Standard for MS.

    I wonder about the speed advantage. In a quick test comparing @LINES to my @LC
    (line count, which does not use a MM file) they performed the same. The test
    was flawed, though, because it used the same collection of files repeatedly ...
    the files were probably cached (Win7 is cache-crazy!), and after that, all
    operations will be fast.

    I have in mind to more carefully devise a statistical test. My thoughts are
    (suggestions welcome) ... With DIR /B create a large list of text files (here,
    g:\*.cpp;*.h ought to come up with a few thousand). Then time routines (for
    %LINES and %LC) which randomly select files from that list for line counting;
    they should generally be uncached.

    But there are other considerations. One is simply resolving the name of the
    function (%LINES vs. @LC). I suspect TCC looks through the names of plugin
    functions before looking through the names of built-in functions (right, Rex?).
    This could affect the results. Another thing I have always wanted to ask Rex:
    When trying to resolve the name of something (_variable, @function, or command
    ... internal or plugin, alias, UDF) is the search a linear one, a binary one,
    something else?

    Long ago I found that BTM's run significantly faster if you unset all
    unnecessary variables, unalias all unnecessary aliases, unfunction any
    unnecessary UDFs, unload all unneseccary plugins, and use very short names for
    environment variables. All of these must affect the speed at which TCC resolves
    the names of things.
     
  19. rconn

    rconn Administrator
    Staff Member

    Joined:
    May 14, 2008
    Messages:
    9,730
    Likes Received:
    80
    Yes. And TCC also does a lot of generic processing of the arguments before passing it to the specific @ function. (This slows things down slightly but vastly reduces the amount of code necessary.

    Command, variable, and @function are all binary searches. The alias and UDF searches are sequential (can't really do it any other way without the overhead making it slower anyway).

    Plugin searches are sequential based on the name passed by the plugin. (Again, it can't be done any other way without requiring the plugin authors to order everything.)

    True, but that's becoming less relevant over time with the increases in system speeds.
     
  20. Steve Fabian

    Joined:
    May 20, 2008
    Messages:
    3,523
    Likes Received:
    4
    From: vefatica
    | I wonder about the speed advantage. In a quick test comparing @LINES to my @LC
    | (line count, which does not use a MM file) they performed the same. The test
    | was flawed, though, because it used the same collection of files repeatedly ...
    | the files were probably cached (Win7 is cache-crazy!), and after that, all
    | operations will be fast.
    |
    | I have in mind to more carefully devise a statistical test. My thoughts are
    | (suggestions welcome) ... With DIR /B create a large list of text files (here,
    | g:\*.cpp;*.h ought to come up with a few thousand). Then time routines (for
    | %LINES and %LC) which randomly select files from that list for line counting;
    | they should generally be uncached.

    Caching and release from cache are often not controllable, esp. that done by the disk drives themselves. One alternative is to use files so big they will not fit into all your internal memory (and onboard disk cache) together, and accessing locations too far from one another in the file to be read in one access. This would mean multigigabyte size! The other alternative is to use files already cached because of prior access. This would mean the execution time of the first call to each function (@LINE and @LC) is not counted, only the subsequent several hundred execution times. E.g.:

    set dummy=%@lines[xxx]
    timer on /1
    do 1000 (set dummy=%@lines[xxx])
    set lines=%@timer[1,s]
    timer off /1
    set dummy=%@lc[xxx]
    timer on /1
    do 1000 (set dummy=%@lc[xxx])
    set lc=%@timer[1,s]
    timer off /1
    echo LINES: %lines s
    echo LC: %lc s

    | But there are other considerations. One is simply resolving the name of the
    | function (%LINES vs. @LC). I suspect TCC looks through the names of plugin
    | functions before looking through the names of built-in functions (right, Rex?).
    | This could affect the results. Another thing I have always wanted to ask Rex:
    | When trying to resolve the name of something (_variable, @function, or command
    | ... internal or plugin, alias, UDF) is the search a linear one, a binary one,
    | something else?

    The fastest search is hash search, esp. when the list is constant, e.g., internals, when a perfect hash is possible, thus totally avoiding collisions. In such case the hash algorithm is unique to the table, but the computed hash code is the exact index into the table for the string (though you must still verify the input string matching the table entry to eliminate invalid strings): 1 match is all. Of course, whenever the list changes (e.g., a command is added), the perfect hash algorithm itself changes, and thus the order of entires in table itself. OTOH a single table for all internals (commands, functions and environment variables) would suffice. Programs for developing a perfect hash algorithm automatically exist. For a random set of strings with possible collisions the average is 2.5 matches if the table is not more than half full. See Donald Knuth: The Art of Computer Programming, vol. 3, Sorting and Searching.

    I suspect that if Rex uses the Windows API version of the Standard-C library function getenv() for resolving environmental variables, it ends up with a linear search - MS is not known for speedy algorithms.

    | Long ago I found that BTM's run significantly faster if you unset all
    | unnecessary variables, unalias all unnecessary aliases, unfunction any
    | unnecessary UDFs, unload all unneseccary plugins, and use very short
    | names for environment variables. All of these must affect the speed at
    | which TCC resolves the names of things.

    I suspect the presence or absence of a category of non-built-ins is more important than the number of members in the category.

    BTW, just as the asterisk (*) prefix for a command name eliminates alias search, it would be useful to have a syntax that selectively restricts function, variable and command searches to built-ins only (ignore plugins and UD entries), or to built-ins and plugins only (ignore aliases and UDFs). OTOH it would be extremely useful to allow UDFs which currently require executing a batch file (typically via @EXECSTR[]) to be able to load the batch file into the UDF space, as is done in some *nix shells.

    NOTE: This post was almost complete when an area-wide power loss killed the original version a while ago. Some of its contents may no longer be current.
    --
    Steve
     
  21. rconn

    rconn Administrator
    Staff Member

    Joined:
    May 14, 2008
    Messages:
    9,730
    Likes Received:
    80
    If you want a fairer test, in addition to all of the preprocessing the parser does for the @ functions, you'll also need to add support to @LC for Unicode files, CON:, and CLIP: (all of which @LINE / @LINES support).
     
  22. Peter Bratton

    Joined:
    Jul 1, 2008
    Messages:
    81
    Likes Received:
    0
    Correct. The @lines[] function behaves differently with no redirection in the mix, to wit
    Code:
    touch /c file.txt
    echo %@lines[file.txt]
    triggers the error.

    But I think redirection is behaving differently too. In previous versions of TCC, there was no need for a pause. I tried replacing the pause with a couple of other commands (unsetarray, setarray, echo something), the code still produced the error, so it seems the pause is needed.
    --
    Peter
     
  23. Peter Bratton

    Joined:
    Jul 1, 2008
    Messages:
    81
    Likes Received:
    0
    Thank you
    --
    Peter
     
  24. dcantor

    Joined:
    May 29, 2008
    Messages:
    505
    Likes Received:
    3
    This appears to have been fixed in build 25. Thanks, Rex.
     

Share This Page