Is there an 8MB limit somewhere?

May 20, 2008
12,176
133
Syracuse, NY, USA
Using this alias (to reverse output lines):
Code:
v:\> alias tac
ffind /e"\r\n" /r /v /m /k
this command correctly produces 8388 1000-byte lines (just under 8MB) and the alias correctly reverses those lines
Code:
v:\> (do x=1000001 to 1008388 ( echo %x %@repeat[x,990] ) ) | tac > tac1.txt
This command attempts the same for 8389 such lines (just over 8MB) and somehow fails
Code:
v:\> (do x=1000001 to 1008389 ( echo %x %@repeat[x,990] ) ) | tac > tac2.txt
The file sizes and the tails of the files show something went awry:
Code:
v:\> dir /k /m tac*
2012-06-18  21:51      8,388,000  tac1.txt
2012-06-18  21:52      8,388,698  tac2.txt
 
v:\> tail /n1 tac1.txt
1000001 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
 
v:\> tail /n2 tac2.txt
1004196 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
 
May 20, 2008
12,176
133
Syracuse, NY, USA
Using this alias (to reverse output lines):
Code:
v:\> alias tac
ffind /e"\r\n" /r /v /m /k
this command correctly produces 8388 1000-byte lines (just under 8MB) and the alias correctly reverses those lines
Code:
v:\> (do x=1000001 to 1008388 ( echo %x %@repeat[x,990] ) ) | tac > tac1.txt
This command attempts the same for 8389 such lines (just over 8MB) and somehow fails
Code:
v:\> (do x=1000001 to 1008389 ( echo %x %@repeat[x,990] ) ) | tac > tac2.txt
The file sizes and the tails of the files show something went awry:
Code:
v:\> dir /k /m tac*
2012-06-18  21:51      8,388,000  tac1.txt
2012-06-18  21:52      8,388,698  tac2.txt
It's a little interesting how the bad file is corrupt. The first 4194 lines (lines numbered 1008389 through 1004196 from DO's output are correct. The next line of the file is 1008389 (again!) and has two extra x's. Then the lines then go in order (with the correct number of x's) from 1008388 down to 1004196 (again) with the next line, the last line of the file, having no numerical prefix and 694 x's followed by a CRLF.
 
Jan 19, 2011
614
15
Norman, OK
Upgrade to a new version of tac. Mine works fine. Even for much bigger numbers.
Code:
z:\>(do x=1000001 to 1128389 ( echo %x %@repeat[x,990] ) ) | tac > tac2.txt
 
z:\> *dir /k /m tac*
6/19/2012  08:34      8,388,000  tac1.txt
6/19/2012  08:39    128,389,000  tac2.txt
 
May 31, 2008
382
2
Vince, getting rid of pipes in your example results in an apparently infinite loop in FFIND - see size of tac2.txt after breaking out of the apparently stuck FFIND with Ctrl+C.
Windows 7 SP1 x86 English TCC 14.00.23
Code:
C:\>(do x=1000001 to 1008388 ( echo %x %@repeat[x,990] ) ) >itac1.txt
 
C:\>copy itac1.txt itac2.txt
C:\itac1.txt => C:\itac2.txt
1 file copied
 
C:\>(do x=1008389 to 1008389 ( echo %x %@repeat[x,990] ) ) >>itac2.txt
 
C:\>dir /k /m itac*
6/19/2012 16:16 8,388,000 itac1.txt
6/19/2012 16:22 8,389,000 itac2.txt
 
C:\>ffind /e"\r\n" /r /v /m /k itac1.txt >tac1.txt
 
C:\>ffind /e"\r\n" /r /v /m /k itac2.txt >tac2.txt
^C
 
C:\>dir /k /m tac*
6/19/2012 16:22 8,388,000 tac1.txt
6/19/2012 16:24 454,461,000 tac2.txt
 
May 20, 2008
12,176
133
Syracuse, NY, USA
Vince, getting rid of pipes in your example results in an apparently infinite loop in FFINDC:\>ffind /e"\r\n" /r /v /m /k itac1.txt >tac1.txt

C:\>ffind /e"\r\n" /r /v /m /k itac2.txt >tac2.txt
^C

C:\>dir /k /m tac*
6/19/2012 16:22 8,388,000 tac1.txt
6/19/2012 16:24 454,461,000 tac2.txt
[/code]
I noticed that also (but I didn't let it go to half a GB). I figured it came from the same limit/problem.
 
May 20, 2008
12,176
133
Syracuse, NY, USA
Upgrade to a new version of tac. Mine works fine. Even for much bigger numbers.
Code:
z:\>(do x=1000001 to 1128389 ( echo %x %@repeat[x,990] ) ) | tac > tac2.txt
 
z:\> *dir /k /m tac*
6/19/2012  08:34      8,388,000  tac1.txt
6/19/2012  08:39    128,389,000  tac2.txt
I've been unable to find a TAC.EXE that works at all on WIN7. What's yours?
 
May 20, 2008
12,176
133
Syracuse, NY, USA
Vince, getting rid of pipes in your example results in an apparently infinite loop in FFIND - see size of tac2.txt after breaking out of the apparently stuck FFIND with Ctrl+C.
When I use, instead "ffind /t"^r^n" /r /v /m /k" ("/r" is documented to work with "/t") the endless loop problem does not happen (and the order of the lines in the file *is* reversed).
Code:
v:\> (do x=1000001 to 1008388 ( echo %x %@repeat[x,990] ) ) > tac1.txt
 
v:\> (do x=1000001 to 1008389 ( echo %x %@repeat[x,990] ) ) > tac2.txt
 
v:\> d tac*
2012-06-21  00:38      8,388,000  tac1.txt
2012-06-21  00:38      8,389,000  tac2.txt
 
v:\> ffind /t"^r^n" /r /v /m /k tac1.txt > tac1rev.txt
 
v:\> ffind /t"^r^n" /r /v /m /k tac2.txt > tac2rev.txt
 
v:\> d tac*
2012-06-21  00:38      8,388,000  tac1.txt
2012-06-21  00:39      8,388,000  tac1rev.txt
2012-06-21  00:38      8,389,000  tac2.txt
2012-06-21  00:40      8,389,000  tac2rev.txt

But when the same FFIND args are used in a pipe, the same (as noted earlier) corruption occurs:
Code:
v:\> (do x=1000001 to 1008388 ( echo %x %@repeat[x,990] ) ) | ffind /t"^r^n" /r /v /m /k > tac1.txt
 
v:\> (do x=1000001 to 1008389 ( echo %x %@repeat[x,990] ) ) | ffind /t"^r^n" /r /v /m /k > tac2.txt
 
v:\> d tac[12].txt
2012-06-21  00:47      8,388,000  tac1.txt
2012-06-21  00:48      8,388,698  tac2.txt

The file tac2.txt is corrupted in the same way as described earlier (when I was using "/e" instead of "/t").
 
Aug 2, 2011
258
4
Berlin, Germany
Code:
>dir /sfh d:\temp\recurse\ | logparser -i:textline -q:on "select text from stdin order by text desc"
D:\temp\recurse\dir3\a\b\c
D:\temp\recurse\dir3\a\b
D:\temp\recurse\dir3\a
D:\temp\recurse\dir3
D:\temp\recurse\dir2\a\b\c
D:\temp\recurse\dir2\a\b
D:\temp\recurse\dir2\a
D:\temp\recurse\dir2
D:\temp\recurse\dir1\a\b\c
D:\temp\recurse\dir1\a\b
D:\temp\recurse\dir1\a
D:\temp\recurse\dir1
 
>alias reverse=logparser -i:textline -q:on "select text from stdin order by text desc"
 
>dir /sfh d:\temp\recurse\ | reverse
D:\temp\recurse\dir3\a\b\c
D:\temp\recurse\dir3\a\b
D:\temp\recurse\dir3\a
D:\temp\recurse\dir3
D:\temp\recurse\dir2\a\b\c
D:\temp\recurse\dir2\a\b
D:\temp\recurse\dir2\a
D:\temp\recurse\dir2
D:\temp\recurse\dir1\a\b\c
D:\temp\recurse\dir1\a\b
D:\temp\recurse\dir1\a
D:\temp\recurse\dir1
 

samintz

Scott Mintz
May 20, 2008
1,557
26
Solon, OH, USA
Is there a reason you can't use good ol' fashioned SORT?
Code:
> ((do x=1000001 to 1008389 ( echo %x %@repeat[x,990] ) ) | sort /r) > tac2.txt
 
> dir
6/26/2012  15:46       8,389,000  tac2.txt
 
> tail /n1 tac2.txt
1000001 xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxx

-Scott
 
May 20, 2008
12,176
133
Syracuse, NY, USA
Is there a reason you can't use good ol' fashioned SORT?

The whole point was to produce the lines in reverse order (unsorted). SORT /R will only do that if the file is already sorted. This is the desired behavior:
Code:
v:\> echo 2^r^n1^r^n3 | tac
3
1
2

Not this:
Code:
v:\> echo 2^r^n1^r^n3 | sort /r
3
2
1
 
Aug 2, 2011
258
4
Berlin, Germany
The whole point was to produce the lines in reverse order (unsorted). SORT /R will only do that if the file is already sorted. This is the desired behavior:
Code:
v:\> echo 2^r^n1^r^n3 | tac
3
1
2

Not this:
Code:
v:\> echo 2^r^n1^r^n3 | sort /r
3
2
1
oops, then my example with logparser was rubbish.
 
May 29, 2008
572
4
Groton, CT
The whole point was to produce the lines in reverse order (unsorted). SORT /R will only do that if the file is already sorted. This is the desired behavior:
Code:
v:\> echo 2^r^n1^r^n3 | tac
3
1
2

Not this:
Code:
v:\> echo 2^r^n1^r^n3 | sort /r
3
2
1

How about
Code:
TPIPE /input=infile.txt /output=outfile.txt /simple=34
? That works for me.
 
Jan 19, 2011
614
15
Norman, OK
How about
Code:
TPIPE /input=infile.txt /output=outfile.txt /simple=34
? That works for me.
There you go... making sense again.

edit: yeah, that works like a champ.
Code:
C:\temp> (do x=1000001 to 1128389 ( echo %x %@repeat[x,990] ) ) | tpipe /simple=34 > tac.txt
 
C:\temp> dir tac.txt
 
Volume in drive C is unlabeled      Serial number is 58b8:a853
Directory of  C:\temp\tac.txt
 
6/27/2012  15:08    128,389,000  tac.txt
      128,389,000 bytes in 1 file and 0 dirs    128,389,120 bytes allocated
    24,191,352,832 bytes free
 
May 20, 2008
12,176
133
Syracuse, NY, USA
I did the same test as JQS with the same satisfactory result. It's quite fast too. Using 8388 lines, which FFIND /R can handle, it took about 30 seconds. With TPIPE it took about 3 seconds.