1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Is there an 8MB limit somewhere?

Discussion in 'Support' started by vefatica, Jun 18, 2012.

  1. vefatica

    Joined:
    May 20, 2008
    Messages:
    7,941
    Likes Received:
    30
    Using this alias (to reverse output lines):
    Code:
    v:\> alias tac
    ffind /e"\r\n" /r /v /m /k
    this command correctly produces 8388 1000-byte lines (just under 8MB) and the alias correctly reverses those lines
    Code:
    v:\> (do x=1000001 to 1008388 ( echo %x %@repeat[x,990] ) ) | tac > tac1.txt
    This command attempts the same for 8389 such lines (just over 8MB) and somehow fails
    Code:
    v:\> (do x=1000001 to 1008389 ( echo %x %@repeat[x,990] ) ) | tac > tac2.txt
    The file sizes and the tails of the files show something went awry:
    Code:
    v:\> dir /k /m tac*
    2012-06-18  21:51      8,388,000  tac1.txt
    2012-06-18  21:52      8,388,698  tac2.txt
     
    v:\> tail /n1 tac1.txt
    1000001 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
     
    v:\> tail /n2 tac2.txt
    1004196 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
     
  2. vefatica

    Joined:
    May 20, 2008
    Messages:
    7,941
    Likes Received:
    30
    It's a little interesting how the bad file is corrupt. The first 4194 lines (lines numbered 1008389 through 1004196 from DO's output are correct. The next line of the file is 1008389 (again!) and has two extra x's. Then the lines then go in order (with the correct number of x's) from 1008388 down to 1004196 (again) with the next line, the last line of the file, having no numerical prefix and 694 x's followed by a CRLF.
     
  3. JohnQSmith

    Joined:
    Jan 19, 2011
    Messages:
    559
    Likes Received:
    7
    Upgrade to a new version of tac. Mine works fine. Even for much bigger numbers.
    Code:
    z:\>(do x=1000001 to 1128389 ( echo %x %@repeat[x,990] ) ) | tac > tac2.txt
     
    z:\> *dir /k /m tac*
    6/19/2012  08:34      8,388,000  tac1.txt
    6/19/2012  08:39    128,389,000  tac2.txt
     
  4. Stefano Piccardi

    Joined:
    May 31, 2008
    Messages:
    376
    Likes Received:
    2
    his tac is an alias
     
  5. Stefano Piccardi

    Joined:
    May 31, 2008
    Messages:
    376
    Likes Received:
    2
    Vince, getting rid of pipes in your example results in an apparently infinite loop in FFIND - see size of tac2.txt after breaking out of the apparently stuck FFIND with Ctrl+C.
    Windows 7 SP1 x86 English TCC 14.00.23
    Code:
    C:\>(do x=1000001 to 1008388 ( echo %x %@repeat[x,990] ) ) >itac1.txt
     
    C:\>copy itac1.txt itac2.txt
    C:\itac1.txt => C:\itac2.txt
    1 file copied
     
    C:\>(do x=1008389 to 1008389 ( echo %x %@repeat[x,990] ) ) >>itac2.txt
     
    C:\>dir /k /m itac*
    6/19/2012 16:16 8,388,000 itac1.txt
    6/19/2012 16:22 8,389,000 itac2.txt
     
    C:\>ffind /e"\r\n" /r /v /m /k itac1.txt >tac1.txt
     
    C:\>ffind /e"\r\n" /r /v /m /k itac2.txt >tac2.txt
    ^C
     
    C:\>dir /k /m tac*
    6/19/2012 16:22 8,388,000 tac1.txt
    6/19/2012 16:24 454,461,000 tac2.txt
    
     
  6. vefatica

    Joined:
    May 20, 2008
    Messages:
    7,941
    Likes Received:
    30
    I noticed that also (but I didn't let it go to half a GB). I figured it came from the same limit/problem.
     
  7. vefatica

    Joined:
    May 20, 2008
    Messages:
    7,941
    Likes Received:
    30
    I've been unable to find a TAC.EXE that works at all on WIN7. What's yours?
     
  8. JohnQSmith

    Joined:
    Jan 19, 2011
    Messages:
    559
    Likes Received:
    7
    I use Cygwin and its assorted myriad of utility goodness.
     
  9. JohnQSmith

    Joined:
    Jan 19, 2011
    Messages:
    559
    Likes Received:
    7
    I missed that part of tac being an alias to ffind.
     
  10. vefatica

    Joined:
    May 20, 2008
    Messages:
    7,941
    Likes Received:
    30
    When I use, instead "ffind /t"^r^n" /r /v /m /k" ("/r" is documented to work with "/t") the endless loop problem does not happen (and the order of the lines in the file *is* reversed).
    Code:
    v:\> (do x=1000001 to 1008388 ( echo %x %@repeat[x,990] ) ) > tac1.txt
     
    v:\> (do x=1000001 to 1008389 ( echo %x %@repeat[x,990] ) ) > tac2.txt
     
    v:\> d tac*
    2012-06-21  00:38      8,388,000  tac1.txt
    2012-06-21  00:38      8,389,000  tac2.txt
     
    v:\> ffind /t"^r^n" /r /v /m /k tac1.txt > tac1rev.txt
     
    v:\> ffind /t"^r^n" /r /v /m /k tac2.txt > tac2rev.txt
     
    v:\> d tac*
    2012-06-21  00:38      8,388,000  tac1.txt
    2012-06-21  00:39      8,388,000  tac1rev.txt
    2012-06-21  00:38      8,389,000  tac2.txt
    2012-06-21  00:40      8,389,000  tac2rev.txt
    But when the same FFIND args are used in a pipe, the same (as noted earlier) corruption occurs:
    Code:
    v:\> (do x=1000001 to 1008388 ( echo %x %@repeat[x,990] ) ) | ffind /t"^r^n" /r /v /m /k > tac1.txt
     
    v:\> (do x=1000001 to 1008389 ( echo %x %@repeat[x,990] ) ) | ffind /t"^r^n" /r /v /m /k > tac2.txt
     
    v:\> d tac[12].txt
    2012-06-21  00:47      8,388,000  tac1.txt
    2012-06-21  00:48      8,388,698  tac2.txt
    The file tac2.txt is corrupted in the same way as described earlier (when I was using "/e" instead of "/t").
     
  11. rconn

    rconn Administrator
    Staff Member

    Joined:
    May 14, 2008
    Messages:
    9,855
    Likes Received:
    83
    The FFIND (and LIST) buffer is 8mb. I'm not too worried about this case, as it seems to be a somewhat grotesque misuse of FFIND to do something far more easily done with (many) other tools.
     
  12. Steve Fabian

    Joined:
    May 20, 2008
    Messages:
    3,520
    Likes Received:
    4
    Is there any other simple method to do it with only TCC internal commands?
     
  13. rconn

    rconn Administrator
    Staff Member

    Joined:
    May 14, 2008
    Messages:
    9,855
    Likes Received:
    83
    Not at this time. Why is that important?
     
  14. Steve Fabian

    Joined:
    May 20, 2008
    Messages:
    3,520
    Likes Received:
    4
    Only because of your earlier statement:
    Cf: My only tool for most work is TCC (even when it is not always as simple as using a HLL).
     
  15. Charles Dye

    Charles Dye Super Moderator
    Staff Member

    Joined:
    May 20, 2008
    Messages:
    3,304
    Likes Received:
    39
    Steve, did you try UPEND?
     
  16. Steve Fabian

    Joined:
    May 20, 2008
    Messages:
    3,520
    Likes Received:
    4
    Sorry, I have been very busy away from my computers in the past week.
     
  17. rconn

    rconn Administrator
    Staff Member

    Joined:
    May 14, 2008
    Messages:
    9,855
    Likes Received:
    83
    I don't think there's a compelling need for every possible obscure tool to be built into TCC.
     
  18. Steve Fabian

    Joined:
    May 20, 2008
    Messages:
    3,520
    Likes Received:
    4
    I agree, though I have needed to reverse the order of elements in a file too often to consider it an obscure requirement.
     
  19. Steve Fabian

    Joined:
    May 20, 2008
    Messages:
    3,520
    Likes Received:
    4
    I think this limit should be listed in HELP topic "Limits", along with any other limits of this kind. The more obscure the limit, the more important that it be documented. Most of this thread would not have been posted if that had been done.
     
  20. Frank

    Joined:
    Aug 2, 2011
    Messages:
    258
    Likes Received:
    4
    Code:
    >dir /sfh d:\temp\recurse\ | logparser -i:textline -q:on "select text from stdin order by text desc"
    D:\temp\recurse\dir3\a\b\c
    D:\temp\recurse\dir3\a\b
    D:\temp\recurse\dir3\a
    D:\temp\recurse\dir3
    D:\temp\recurse\dir2\a\b\c
    D:\temp\recurse\dir2\a\b
    D:\temp\recurse\dir2\a
    D:\temp\recurse\dir2
    D:\temp\recurse\dir1\a\b\c
    D:\temp\recurse\dir1\a\b
    D:\temp\recurse\dir1\a
    D:\temp\recurse\dir1
     
    >alias reverse=logparser -i:textline -q:on "select text from stdin order by text desc"
     
    >dir /sfh d:\temp\recurse\ | reverse
    D:\temp\recurse\dir3\a\b\c
    D:\temp\recurse\dir3\a\b
    D:\temp\recurse\dir3\a
    D:\temp\recurse\dir3
    D:\temp\recurse\dir2\a\b\c
    D:\temp\recurse\dir2\a\b
    D:\temp\recurse\dir2\a
    D:\temp\recurse\dir2
    D:\temp\recurse\dir1\a\b\c
    D:\temp\recurse\dir1\a\b
    D:\temp\recurse\dir1\a
    D:\temp\recurse\dir1
    
     
  21. samintz

    samintz Scott Mintz

    Joined:
    May 20, 2008
    Messages:
    1,187
    Likes Received:
    11
    Is there a reason you can't use good ol' fashioned SORT?
    Code:
    > ((do x=1000001 to 1008389 ( echo %x %@repeat[x,990] ) ) | sort /r) > tac2.txt
     
    > dir
    6/26/2012  15:46       8,389,000  tac2.txt
     
    > tail /n1 tac2.txt
    1000001 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
    xxxx
    
    -Scott
     
  22. vefatica

    Joined:
    May 20, 2008
    Messages:
    7,941
    Likes Received:
    30
    The whole point was to produce the lines in reverse order (unsorted). SORT /R will only do that if the file is already sorted. This is the desired behavior:
    Code:
    v:\> echo 2^r^n1^r^n3 | tac
    3
    1
    2
    Not this:
    Code:
    v:\> echo 2^r^n1^r^n3 | sort /r
    3
    2
    1
     
  23. Frank

    Joined:
    Aug 2, 2011
    Messages:
    258
    Likes Received:
    4
    oops, then my example with logparser was rubbish.
     
  24. dcantor

    Joined:
    May 29, 2008
    Messages:
    507
    Likes Received:
    3
    How about
    Code:
    TPIPE /input=infile.txt /output=outfile.txt /simple=34
    
    ? That works for me.
     
  25. JohnQSmith

    Joined:
    Jan 19, 2011
    Messages:
    559
    Likes Received:
    7
    There you go... making sense again.

    edit: yeah, that works like a champ.
    Code:
    C:\temp> (do x=1000001 to 1128389 ( echo %x %@repeat[x,990] ) ) | tpipe /simple=34 > tac.txt
     
    C:\temp> dir tac.txt
     
    Volume in drive C is unlabeled      Serial number is 58b8:a853
    Directory of  C:\temp\tac.txt
     
    6/27/2012  15:08    128,389,000  tac.txt
          128,389,000 bytes in 1 file and 0 dirs    128,389,120 bytes allocated
        24,191,352,832 bytes free
    
     
  26. vefatica

    Joined:
    May 20, 2008
    Messages:
    7,941
    Likes Received:
    30
    I did the same test as JQS with the same satisfactory result. It's quite fast too. Using 8388 lines, which FFIND /R can handle, it took about 30 seconds. With TPIPE it took about 3 seconds.
     

Share This Page