Documentation File reading questions

May 20, 2008
Elkridge, MD, USA
1) Does @FILESEEKL keep track of the current line number? I.e., if the current line is 1000, would %@FILESEEKL[n,1010] be as fast as %@FILESEEKL[n,10,1] ?

2) When DO or FOR is processing a file a line at a time using "DO line in @file" or "FOR %line in (@file)" is there an internal line counter that could be accessed, e.g., via a future internal variable?
May 24, 2010
Northlake, Il

While this almost certainly isn't quite as convenient as it would be if @FileSeekL did what you want it to do (which it clearly does not), it isn't really hard and I've been doing this for some number of years, and I really don't see any significant reason(s) why you couldn't do it this way. Complete, fully tested and working, example:
@Echo Off
On ErrorMsg Goto Fini
On Break Goto Fini
Do X In /P Timer On (Set StartTime=%X)
Set FileName="D:\A Reasonably Large"
Do X In /P Timer On (Set CountTime=%X)
Set Size=%@Inc[%@Lines[%FileName]]
UnSetArray /Q Data
SetArray Data[%Size]
Set Handle=%@FileOpen[%FileName, r]
Set IDX=0
Set Line=%@FileRead[%Handle]
Do While "%Line" != "**EOF**"
  Set Data[%IDX]=%Line
  Set /A IDX+=1
  Set Line=%@FileRead[%Handle]
Echo >NUL: %@FileClose[%Handle]
Do X In /P Timer /S (Set ReadTime=%X)
Do IDX = 0 To %@Dec[%Size] By 1
  @Echo %IDX: %Data[%IDX]
  Set /A IDX+=1
On ErrorMsg
@Echo %Size
Do X In /P Timer Off (Set EndTime=%X)
@Echo      Size: %Size
@Echo Start Time: %StartTime
@Echo Count Time: %CountTime
@Echo  Read Time: %ReadTime
@Echo  End Time: %EndTime
UnSetArray /Q Data
Quit 8
I decided to leave all of the code in, including the "Timers" and "status" messages, etc. (although the "important" code is that between "Set Size=" and "@FileClose" lines); I don't think that they really make the code all that much harder to understand (at least for somebody like you! ;)) and allowed me to measure performance. And in terms of "real world" performance:

"Real world" times when writing the data out to a file with "Echo":
      Size: 3506
Start Time: Timer 1 on:  8:49:53
Count Time: Timer 1 on:  8:49:53
Read Time: Timer 1  Elapsed: 0:00:32.44
  End Time: Timer 1 off:  8:50:31  Elapsed: 0:00:38.25
Which is, if you figure it out, 0.0037 seconds to read each record, which ain't bad by my estimation, and, for what it's worth, 0.00166 seconds to write each record out to a file.

And, for writing it directly to the console:
      Size: 3506
Start Time: Timer 1 on:  8:49:53
Count Time: Timer 1 on:  8:49:53
Read Time: Timer 1  Elapsed: 0:00:32.44
  End Time: Timer 1 off:  8:50:31  Elapsed: 0:00:38.25
Which is 0.012 seconds per record, which also isn't bad, although it does show that there can be a lot of variation in processing times depending on what else is going on in the system, I suppose. And, for reference, it took 12.62 seconds to write the data to the console, or .0036 seconds per record; and the difference between the time to write it out to the console rather than the file really can't be determined in any reliable way given the time differences reading the file. (I will note that when I ran the second test writing the data out to the console the data was presumably already in the disk cache which makes the fact that it took so much longer the second time around rather surprising, I think. Bottom line, elapsed times are not all that relevant except over a fairly large number of "samples". Also, my machine is hardly what one would call fast, particularly the disk drives.)

And this code is simple enough, even with my semi-blindness and bad memory, that I have no "issues" with it whatsoever.

I will also note that reading the file one time ("%@Lines") was not at all a significant performance issue, so trying to guess what the maximum size of the file would be is to set the size of the array is probably not a worthwhile thing to do.

- Dan
May 20, 2008
Elkridge, MD, USA
Since the amount of memory required by an array depends only on the elements actually populated, but not its allocated size, instead of your procedure, I have often allocated an excessively large array, populated it with the @FILEARRAY function, and used the function's value (the number of lines read into the array) for iteration control. Here are some other techniques you might find useful:
- for entities which use offset referencing (i.e., starting with 0) use a variable which is the element count, and another for the biggest offset (they differ by one 1, but I hate to recalculate multiple times)
- when reading lines of a file of known line count, use DO %LINECOUNT - not a special test for EOF
- using @fileread another method to avoid reading past EOF:
do while %@fileseek[%handle,0,1] LT %filesize
note that I saved the file size in a variable so I don't need to access the file system each time, and that this form of @FILESEEK just returns a number without affecting the file buffers in any way
- your loop reading the data from the file into the array (which could have been done with the @FILEARRAY function without a loop) could have used DO IDX = 0 to LASTOFFSET and avoided an extra statement to increase IDX
- in the loop to display the array elements the SET /A command modifies IDX, but the ENDDO loop control immediately following overrides it, making it superfluous.
May 24, 2010
Northlake, Il

Good points all; but number one, I just showed something I've been doing for years pretty much without modification (or even really thinking about it much), and number two, there's far too much in TCC for me (or practically anyone else?) to know it all (although you seem to come close!). However, in the future I'll try to remember to try some of your suggestions (although it's not a high priority since what I've been doing does work just fine and is also quite fast - as I think I indicated, %@Lines takes essentially no time at all for the files I have to deal with).

And where that came from initially is pretty simple: Just the common technique of reading a file until EOF in pretty much any other circumstance.

As far as your comment about the increment being superfluous in the loop, you are, of course, correct there. However, that was just a stupid mistake (as I am so fond of making!) on my part. You see, my habit is to use an explicit "Do While" in such situations (I'm not defending, it's just a habit; maybe it comes from something to do with it works pretty much the same in all of the high-level languages I've used over the years). And, after I made the initial entry of my sample code in the web page, I thought "You know, most people would probably automatically do a "Do" taking that format (so I was totally aware of the "Do" in the form I suggested), and I changed that at the last minute without really checking it (a stupid mistake, and I mean not checking it, not the coding error itself).

- Dan
Similar threads
Thread starter Title Forum Replies Date
R Reading an Unicode file with more than 8191 lines Support 1
J History file load hangs if file has complete garbage in it... Support 1
jdanielp Cannot step into and/or call another .bat file when debugging .bat file Support 2
thorntonpg file explorer - short cut keys Support 1
JohnQSmith New online help file wrong logo Support 7
Jay Sage WAD Possible Bug With OPTION Command With @FILE Support 5
samintz How to? swap endianness in a binary file Support 12
vefatica `Back quotes` - command line vs. batch file Support 5
fpefpe How to? batch file size Support 2
Jay Sage Issues With TCMD File Explorer Support 0
FreezerBurnt Help making a CMD and TCC compatible batch file Support 7
Joe Caverly @FILESIZE and NTFS File Stream Support 1
Joe Caverly Embedding an .EXE (or other file) into a .BTM Support 8
U Command help on file names Support 5
S Concatenate all playlists into 1 file Support 22
Joe Caverly Set a batch variable (%1 - %n) when TCC is executing a batch file Support 5
Joe Caverly Rename file that begins with Emoji Support 7
Joe Caverly PSHELL and NTFS File Streams Support 2
P Kill all other instances of tcc from a btm batch file Support 2
D ffind hangs on large file Support 18
Alpengreis Documentation Minor issue in help file for view /E Support 1
fpefpe batch file ending early Support 18
J Licesne file location Support 1
K robocopy cancelling batch file processing Support 28
Jay Sage TCMD Crashes with "tctoolbar /c /r file" Support 5
A Unable to pass batch file arguments unaltered Support 18
MikeBaas How to...extract a string from a .json-file? Support 10
vefatica LIBRARY and file name completion? Support 4
D What does the file "..\Everything\Run history.csv" store? Support 5
H Takecommand.pdf failed not a pdf file Support 6
Joe Caverly DIRectory Colorization in v25 Help File Support 0
C show file description? with dir? Support 8
D When copying/moving, appending " (2)" to filename when the target file already exist. Support 7
R How to? Dir specific file search patterns with spaces in the pathnames? Support 6
Joe Caverly _EXPANSION in help file Support 0
Stefan Hassel Installation file TC 24.02 Build 47 is actually version 25.00.1 Support 10
vefatica Are @FILEDATE[file] and @FILEDATE[file,u] ever different? Support 2
K_Meinhard Help file error Support 1
Joe Caverly JOBMONITOR in Help File Support 1
D btm file command line augments comparing for string or math. Support 12
A Documentation [Help file] OPTION "//" synopsis formatting issue Support 0
vefatica Can TCC complete this file name? Support 5
G tcedit ist not run in tc 24 file explorer Support 10
M How to set Path variable for non-executable file extensions in 4DOS Support 8
I zip /M fails to delete file after adding to zip file Support 2
Charles Dye Is the "File Associations" installer dialog working? Support 4
J File explorer handling from command line? Support 2
Joe Caverly v24 Help File - DATEMONITOR Support 1
MickeyF Why does this batch file give me 'unknown command "else"' error? Support 17
vefatica TYPE goes crazy with no-BOM Unicode file Support 7

Similar threads