textsort suggestion for tpipe

Apr 2, 2011
1,404
10
54
North Carolina, USA
VER: TCMD v5.01.52 x64

I have a file gt 2 billion bytes that I would like to sort. Tried DOS's SORT and that errored out. TPIPE doesn't have a /SORT option. I'd love to stay with JPSOFT products but also need to sort this huge file. Hence my suggestion to add a /SORT option to TPIPE or maybe another command?

Thank you ....
 
May 20, 2008
11,300
95
Syracuse, NY, USA
IIRC, TPIPE's "sort" option was removed because it was buggy. But that was a long time ago and since sorting is rather important it would be nice if that option came back (in good working order).
 
May 20, 2008
11,300
95
Syracuse, NY, USA
P.S. How much memory do you have? I'd expect any reasonably fast sorting algorithm to bomb if there were not enough memory.
 
Apr 2, 2011
1,404
10
54
North Carolina, USA
TCMD's MEMORY returns:


65 % Memory load
4,196,941,824 bytes total physical RAM
1,442,693,120 bytes available physical RAM
8,391,979,008 bytes total page file
4,308,668,416 bytes available page file
8,796,092,891,136 bytes total virtual RAM
8,795,816,751,104 bytes available virtual RAM
262,144 characters total alias
262,143 characters free
131,072 characters total function
131,071 characters free
500,000 characters total history

The file in question is:
10/07/2013 21:03 2,772,599,380 tcmd.all
 
May 20, 2008
11,300
95
Syracuse, NY, USA
I just don't know enough about it to say whether that should be sufficient. Have you found a program that will sort it? If it's text, can you open it with WinWord?
 

Ugo

Aug 22, 2013
10
0
VER: TCMD v5.01.52 x64
I have a file gt 2 billion bytes that I would like to sort.
...

I assume that you have a text file, and you want to sort its lines in ASCIIbetical order (alphabetical for characters, plus other ascii symbols).

One viable solution would be to get perl (for instance, the portable perl -no installation- at strawberryperl.com) and use this command:
perl -e "print sort <>" original_file.txt > sorted_file.txt

Note that this code loads the whole file in memory into a list of lines (the <> symbol just does that), sorts the list, and then print every line to the output.
The pipe redirects the output to a file.

I do not know if there is any limitation on the amount of data that can go into a DOS pipe. Neither I can tell you if the piping is efficient with large data (does it use a buffer?).
The bottleneck of this command is likely to be the input/output to the disk.

If you try the command and are happy with it, you can easily create an alias for your convenience, and use it like any other feature of TCMD!

As a slim alternative you can download the free tool "Swiss File Knife" and use its "sort" command. See http://stahlworks.com/dev/swiss-file-knife.html
Again, you can easily wrap this tool into an alias or a batch file.
 
Apr 2, 2011
1,404
10
54
North Carolina, USA
I assume that you have a text file, and you want to sort its lines in ASCIIbetical order (alphabetical for characters, plus other ascii symbols).

One viable solution would be to get perl (for instance, the portable perl -no installation- at strawberryperl.com) and use this command:
perl -e "print sort <>" original_file.txt > sorted_file.txt

Note that this code loads the whole file in memory into a list of lines (the <> symbol just does that), sorts the list, and then print every line to the output.
The pipe redirects the output to a file.

I do not know if there is any limitation on the amount of data that can go into a DOS pipe. Neither I can tell you if the piping is efficient with large data (does it use a buffer?).
The bottleneck of this command is likely to be the input/output to the disk.

If you try the command and are happy with it, you can easily create an alias for your convenience, and use it like any other feature of TCMD!

As a slim alternative you can download the free tool "Swiss File Knife" and use its "sort" command. See http://stahlworks.com/dev/swiss-file-knife.html
Again, you can easily wrap this tool into an alias or a batch file.

I don't understand how do use SFK. Does it work primarily with pipes? How would I sort input.txt and produce input_sorted.txt ?
 

Ugo

Aug 22, 2013
10
0
I don't understand how do use SFK. Does it work primarily with pipes? How would I sort input.txt and produce input_sorted.txt ?
Try
sfk sort
and you'll get the help.
For your specific case just use
sfk filter input.txt +sort > input_sorted.txt
Beware that sfk assumes that every line is no longer than 4000 bytes, while the perl code above does not have this restriction.

You can also use powershell, if you like
Get-Content input.txt | Sort-Object > input_sorted.txt
Now you can easily include your powershell command inside your batch files by typing
PowerShell -Command "& {Get-Content input.txt | Sort-Object > input_sorted.txt}"
 

rconn

Administrator
Staff member
May 14, 2008
12,318
148
IIRC, TPIPE's "sort" option was removed because it was buggy. But that was a long time ago and since sorting is rather important it would be nice if that option came back (in good working order).

It wasn't buggy, it was just slow with large files. That generated enough complaints that I removed the option.
 
Similar threads
Thread starter Title Forum Replies Date
C Question / Suggestion Support 1
fpefpe Suggestion for next release Support 2
C New suggestion Support 1
S Forum suggestion Support 0
rconn Suggestion list for next major version Support 0
J Suggestion for next version Support 0
S Enhancement suggestion: @format and @formatn Support 0
S Suggestion: enhanced automatic quoting of filenames Support 2
dcantor Suggestion for @DATE[] input Support 2
S HELP nit and suggestion Support 2
dcantor Take Command V10 @WINPID suggestion. Support 2
dcantor Syntax files, and suggestion Support 0
samintz How to? TPIPE /simple=61 Support 6
R Bug TPIPE's pdf to text conversions don't work Support 2
Kachupp How to? tpipe Support 4
H TPIPE /simple=4 Support 12
D TPIPE produces unwanted logs. Support 1
Kachupp How to? tpipe Support 8
vefatica Documentation TPIPE, /perl, DotMatchesNewLines Support 1
vefatica Stop TPIPE /EOL removing bad EOLs (and TPIPE changine EOLs)? Support 2
vefatica TPIPE in an alias? Support 2
C How to? TPIPE ?'s Support 2
H History and TPipe Oddness Support 3
Jay Sage Baffling Behavior With TPIPE Support 5
vefatica TPIPE: unbalanced escaped quotes in a regex? Support 5
Joe Caverly Multiple Text Searches at once using FFIND or TPIPE Support 4
Kachupp FFIND TPIPE Support 9
vefatica TPIPE and avarice? Support 5
Peter Murschall TPIPE generate unicode on Piping or redirecting Support 3
x13 TPIPE /REPLACE - Remove line (including EOL) Support 3
Charles Dye Documentation TPIPE /REPLACELIST Support 0
Joe Caverly TPIPE and /inputstring Support 3
vefatica How to? Add real <ESC> characters with TPIPE Support 2
vefatica How to? TPIPE ... see output right away? Support 4
vefatica TPIPE.EXE crashes on Ctrl-C Support 9
O TPIPE: Error -Can´t access textpipeengine.dll Support 24
vefatica Extract fields with TPIPE? Support 10
vefatica TPIPE, crash and question Support 1
MickeyF Another TPIPE question Support 6
MickeyF Can't get my TPIPE /replace to work Support 4
Joe Caverly Printing TPIPE documentation Support 3
C WAD tpipe /simple=28 Support 8
vefatica Remove double-quotes with TPIPE Support 4
vefatica TPIPE crashes on Ctrl-C Support 1
C How to? Add date/time to custom tpipe based BTM Support 1
C tpipe /simple=28 error Support 5
C TPipe /dup Support 5
T TPipe: Replacing String with Double Quotes Support 3
C tpipe and extract email addresses Support 1
vefatica How do I do this with TPIPE? Support 1

Similar threads