WAD Counting things with TPIPE /grep

May 20, 2008
11,547
107
Syracuse, NY, USA
From the help for TPIPE /grep​
Code:
CountMatches - 1 to only output a count of the number of matches
Apparently, this will count the characters (very slowly).
Code:
v:\> type 116402.txt | tpipe /grep=2,0,0,0,1,0,0,0,.
3968262
And this will count the lines:
Code:
v:\> type 116402.txt | tpipe /grep=3,0,0,0,1,0,0,0,.
116402
This counts words, but in the process, outputs 116402 blank lines!
Code:
type 116402.txt | tpipe /grep=2,0,0,0,1,0,0,0,"[^ \t\r\n]*"
(116402 blank lines here)
582010
 
May 20, 2008
11,547
107
Syracuse, NY, USA
From the help for TPIPE /grep​
Code:
CountMatches - 1 to only output a count of the number of matches
Here's a simpler example, showing a blank line output for every line in the file.
Code:
v:\> echo My^r^ndog^r^nhas^r^nfleas. > doggie.txt
 
v:\> tpipe /input=doggie.txt /grep=2,0,0,0,1,0,0,0,"[^ \t\r\n]*"
 
 
 
 
4
 
v:\>
 

rconn

Administrator
Staff member
May 14, 2008
12,406
152
From the help for TPIPE /grep​
Code:
CountMatches - 1 to only output a count of the number of matches
Apparently, this will count the characters (very slowly).
Code:
v:\> type 116402.txt | tpipe /grep=2,0,0,0,1,0,0,0,.
3968262

That's an *incredibly* inefficient way of calling TPIPE ...

And this will count the lines:
Code:
v:\> type 116402.txt | tpipe /grep=3,0,0,0,1,0,0,0,.
116402

See above.

This counts words, but in the process, outputs 116402 blank lines!
Code:
type 116402.txt | tpipe /grep=2,0,0,0,1,0,0,0,"[^ \t\r\n]*"
(116402 blank lines here)
582010

Without your mystery "116402.txt" file, there's probably no point in my even looking at this.

What's with your insistence on TYPE'ing the file and piping the output to TPIPE?
 

rconn

Administrator
Staff member
May 14, 2008
12,406
152
You're inferring a relationship that doesn't exist. There are lots of different kinds of "pipes" -- in this case, I called it "TPIPE" because it's based on the "TextPipe Engine".

I've said in several previous messages that piping input & output is the slowest possible usage. TYPE'ing a file and piping it to TPIPE is at least an order of magnitude slower than simply passing TPIPE the filename.
 
May 20, 2008
11,547
107
Syracuse, NY, USA
Without your mystery "116402.txt" file, there's probably no point in my even looking at this.
There's nothing special about that file or about piping. Whenever you count words (as below), it outputs a blank line for every line in the file (whereas it's supposed to output only the count). Didn't you see my simple example?


Code:
v:\> echo My^r^ndog^r^nhas^r^nfleas. > doggie.txt
 
v:\> tpipe /input=doggie.txt /grep=2,0,0,0,1,0,0,0,"[^ \t\r\n]*"
 
 
 
 
4
 
v:\>
 

rconn

Administrator
Staff member
May 14, 2008
12,406
152
There's nothing special about that file or about piping. Whenever you count words (as below), it outputs a blank line for every line in the file (whereas it's supposed to output only the count). Didn't you see my simple example?


Code:
v:\> echo My^r^ndog^r^nhas^r^nfleas. > doggie.txt
 
v:\> tpipe /input=doggie.txt /grep=2,0,0,0,1,0,0,0,"[^ \t\r\n]*"
 
 
 
 
4
 
v:\>

Your line should be:

tpipe /input=doggie.txt /grep=5,0,0,0,1,0,0,0,"[^ \t\r\n]*"

The "2" is telling TPIPE to extract (and output) all the matching lines; "5" tells it to remove them.

The help for "CountMatches" has a typo; it should read:

CountMatches - 1 to output a count of the number of matches
 
May 20, 2008
11,547
107
Syracuse, NY, USA
Your line should be:

tpipe /input=doggie.txt /grep=5,0,0,0,1,0,0,0,"[^ \t\r\n]*"

The "2" is telling TPIPE to extract (and output) all the matching lines; "5" tells it to remove them.

The help for "CountMatches" has a typo; it should read:

CountMatches - 1 to output a count of the number of matches
That only works because *all* lines match. What if only *some* lines matched?
Code:
v:\> tpipe /input=doggie.txt /grep=5,0,0,0,1,0,0,0,dog
My
has
fleas.
1
 
May 20, 2008
3,515
4
Elkridge, MD, USA
You're inferring a relationship that doesn't exist. There are lots of different kinds of "pipes" -- in this case, I called it "TPIPE" because it's based on the "TextPipe Engine".

I've said in several previous messages that piping input & output is the slowest possible usage. TYPE'ing a file and piping it to TPIPE is at least an order of magnitude slower than simply passing TPIPE the filename.
What's in a name? I am not inferring any relationships. I am inferring that to the average user the name TPIPE means a command relating to pipes - devices allowing contents to flow from one place to another - whether in a tobacco pipe, in the Alaska Pipeline, or in a software mechanism connecting software processes.

It is clear to anyone who understands the underlying mechanisms that reading data from a file by the TYPE command, and directing its standard output to the standard input of another process requires much more processing than would be required if the second process read the file directly, and is much less efficient. This has nothing to do with the intuitive interpretation of the name TPIPE - anyone who speaks English associates the word with pipes. An unfortunate name, even if it is based on the name "TextPipe Engine" - which itself is misnamed.

Possible reasons for sending data to a program using a pipe instead of permitting it to access the source file directly iinclude the need to use a process with access rights to the source file that are not granted to the processing program. Vince's example above does not have such a need...
 
May 20, 2008
3,515
4
Elkridge, MD, USA
Continuing previous post. Many programs coming from the Unix world do not have file access capabilities, accept data only through standard input, and dispose of generated data exlusiively through standard output. Filter utillities like CUT and TR are prime examples. When you try to determine how much of your existing procedure using Unix utilities can be replace by the TPIPE command, it is natural to pipe to TPIPE instead of the more efficient direct reading, requriring much less modification of existing programs.
 
May 20, 2008
11,547
107
Syracuse, NY, USA
Continuing previous post. Many programs coming from the Unix world do not have file access capabilities, accept data only through standard input, and dispose of generated data exlusiively through standard output. Filter utillities like CUT and TR are prime examples. When you try to determine how much of your existing procedure using Unix utilities can be replace by the TPIPE command, it is natural to pipe to TPIPE instead of the more efficient direct reading, requriring much less modification of existing programs.
My two versions of CUT.EXE take a file. Gnu's TR.EXE doesn't; the Thompson Toolkit's TR.EXE (17 years old) does.

Speed varies greatly, with TPIPE not too fast in this comparison.
Code:
v:\> wc IpToCountry.csv
  Lines  Words  Chars
116402  116402 3015512
 
v:\> timer & g:\gnu\grep US IpToCountry.csv > NUL & timer
Timer 1 on: 00:27:03
Timer 1 off: 00:27:03  Elapsed: 0:00:00.61
 
v:\> timer & g:\ttk\grep US IpToCountry.csv > NUL & timer
Timer 1 on: 00:27:34
Timer 1 off: 00:27:34  Elapsed: 0:00:00.07
 
v:\> timer & tpipe /input=IpToCountry.csv /grep=3,0,0,0,0,0,0,0,US > NUL & timer
 
Timer 1 on: 00:27:48
Timer 1 off: 00:27:48  Elapsed: 0:00:00.30
 
Similar threads
Thread starter Title Forum Replies Date
vefatica CD_ENTER makes wacky things happen Support 10
fishman@panix.com V23 cdd /s doing WIERD things. Support 5
M A semi-colon to make things go @WILD? Support 2
Alpengreis Small things to change in installer and after installed Everything Support 0
old coot Interesting TPIPE sort algorithm artifact Support 4
samintz How to? TPIPE /simple=61 Support 6
R Bug TPIPE's pdf to text conversions don't work Support 2
Kachupp How to? tpipe Support 4
H TPIPE /simple=4 Support 12
D TPIPE produces unwanted logs. Support 1
Kachupp How to? tpipe Support 8
vefatica Documentation TPIPE, /perl, DotMatchesNewLines Support 1
vefatica Stop TPIPE /EOL removing bad EOLs (and TPIPE changine EOLs)? Support 2
vefatica TPIPE in an alias? Support 2
C How to? TPIPE ?'s Support 2
H History and TPipe Oddness Support 3
Jay Sage Baffling Behavior With TPIPE Support 5
vefatica TPIPE: unbalanced escaped quotes in a regex? Support 5
Joe Caverly Multiple Text Searches at once using FFIND or TPIPE Support 4
Kachupp FFIND TPIPE Support 9
vefatica TPIPE and avarice? Support 5
Peter Murschall TPIPE generate unicode on Piping or redirecting Support 3
x13 TPIPE /REPLACE - Remove line (including EOL) Support 3
Charles Dye Documentation TPIPE /REPLACELIST Support 0
Joe Caverly TPIPE and /inputstring Support 3
vefatica How to? Add real <ESC> characters with TPIPE Support 2
vefatica How to? TPIPE ... see output right away? Support 4
vefatica TPIPE.EXE crashes on Ctrl-C Support 9
O TPIPE: Error -Can´t access textpipeengine.dll Support 24
vefatica Extract fields with TPIPE? Support 10
vefatica TPIPE, crash and question Support 1
MickeyF Another TPIPE question Support 6
MickeyF Can't get my TPIPE /replace to work Support 4
Joe Caverly Printing TPIPE documentation Support 3
C WAD tpipe /simple=28 Support 8
vefatica Remove double-quotes with TPIPE Support 4
vefatica TPIPE crashes on Ctrl-C Support 1
C How to? Add date/time to custom tpipe based BTM Support 1
C tpipe /simple=28 error Support 5
C TPipe /dup Support 5
T TPipe: Replacing String with Double Quotes Support 3
C tpipe and extract email addresses Support 1
vefatica How do I do this with TPIPE? Support 1
vefatica TPIPE's /perl option? Support 1
D TPIPE help typo Support 0
vefatica TPIPE access violations are back Support 2
Joe Caverly Word Count using TPIPE Support 10
Dan Glynhampton TPIPE in Build 41 Support 4
Phileosophos TPIPE doesn't work at all with v16 Support 3
vefatica TPIPE access violation again Support 1

Similar threads