TPipe /dup

Apr 2, 2011
1,441
10
54
North Carolina, USA
/dup=Type,MatchCase,StartColumn,Length,IncludeOne,Format

Remove or show duplicate lines. The arguments are:

Type:
0 - Remove duplicate lines
1 - Show duplicate lines

MatchCase - If 1, do case-sensitive comparisons

StartColumn - The starting column for comparisons

Length - The Length of the comparison

IncludeOne - Include lines with a count of 1

Format - how the output should be formatted for Type=1. For example, "%d %s" to show the count followed by the string.

=========================

1) Is starting column 0 or 1 based?

2) Not sure what IncludeOne is used for?

I am using "Type=0" so Format is not necessary.

========================
TCC 19.00.15 x64 Windows 7 [Version 6.1.7601]
 
May 31, 2008
62
0
Oklahoma
/dup=Type,MatchCase,StartColumn,Length,IncludeOne,Format

1) Is starting column 0 or 1 based?

2) Not sure what IncludeOne is used for?
tpipe gives me the creeps :rolleyes: but FWIW from the online TextPipe manual
Start column
The comparison can also ignore leading characters if desired by setting the start column higher than 1. This can be used to skip line numbers, which can be used to find duplicates that are not adjacent. To skip line numbers, set the Start Column to 6 (or so), and set the length to 4096, or a length greater than your maximum line length.
. . .
. . .
Include counts of 1
Normally this filter only outputs lines with counts of 2 or more (ie, they are duplicates). When this box is checked,
From this I'd infer that
  1. tpipe columns start at 1
  2. If IncludeOne is 1 then tpipe will also output for those lines with no duplicates [remembering that output for any lines is solely determined by the Format you cleverly choose] :smile:
 
Apr 2, 2011
1,441
10
54
North Carolina, USA
@Bob Chapman -

I am wanting to remove duplicate lines from a file - that contain email address - one per line.

So for lines that :
- only occur once - output that line.
- for lines that occur more then once - output the line only once.

the following is the BTM so far: (not long ...!)

Code:
goto :here

:here
  setlocal
    set fldr=c:\Users\Galloway\Desktop\EMailAddrs\
    rem next set is output file name for SORT /Output...
    set cSrtOut=SortOut.lst
    rem next is work file for email addresses
    set cOut=EmailOut.lst
    if not isdir "%fldr" md /s "%fldr"
    global /h /i /n /q GoSub DoFldr
  endlocal
  quit

:DoFldr
  echo In: %_CWD
  rem before processing current folder
  set nOutSize=%@filesize[%fldr%%cOut]
  rem extract email addresses
  for %fn in (*.eml *.lst *.txt) if ( "%fn" NE "%fldr%%cSrtOut") and ("%fn" NE "%fldr%%cOut") tpipe /input="%fn",0,1 /simple=28 /outputappend=1 /output=%fldr%%cOut
  rem if file has changed = means more email addresses found
  iff nOutSize != %@filesize[%fldr%%cOut] then
    rem make sure file exists
    iff isfile "%fldr%%cOut" then
      rem sort email addresses
      sort /rec 65535 "%%fldr%%cOut" /output=%fldr%%cSrtOut
      rem remove duplicate email addresses
      tpipe /input=%fldr%%cSrtOut /dup=Type,MatchCase,StartColumn,Length,IncludeOne,Format /output=%fldr%%cOut
      rem                                                 0,            0,               0, 65535, ??????????, ?????
    endiff
  endiff
  return
 
May 20, 2008
11,404
99
Syracuse, NY, USA
I am wanting to remove duplicate lines from a file - that contain email address - one per line.
Code:
v:\> type dups.txt
joe@foo.com
bob@foo.com
joe@foo.com
bob@bar.com
bob@foo.com
bob@bar.com
tom@xyz.edu
tom@xyz.com
tom@xyz.edu
moe@foo.com

v:\> tpipe /input=dups.txt /dup=0,0,1,999,1,""
joe@foo.com
bob@foo.com
bob@bar.com
tom@xyz.edu
tom@xyz.com
moe@foo.com

v:\> tpipe /input=dups.txt /dup=0,0,1,999,0,""
joe@foo.com
bob@foo.com
bob@bar.com
tom@xyz.edu
tom@xyz.com
moe@foo.com

I can't get "Type=1" to output anything.

(Edit) The format string needs %% (percent signs doubled).
 
Last edited:
May 20, 2008
11,404
99
Syracuse, NY, USA
I think "Include lines with a count of 1" is for Type=1.
Code:
v:\> type dups.txt
joe@foo.com
bob@foo.com
joe@foo.com
bob@bar.com
bob@foo.com
bob@bar.com
tom@xyz.edu
tom@xyz.com
tom@xyz.edu
moe@foo.com

v:\> tpipe /input=dups.txt /dup=1,0,1,999,1,"%%d %%s"
2 bob@bar.com
2 bob@foo.com
2 joe@foo.com
1 moe@foo.com
1 tom@xyz.com
2 tom@xyz.edu

v:\> tpipe /input=dups.txt /dup=1,0,1,999,0,"%%d %%s"
2 bob@bar.com
2 bob@foo.com
2 joe@foo.com
2 tom@xyz.edu
 
Apr 2, 2011
1,441
10
54
North Carolina, USA
Seems that an example of specifying /input or /output when they are, for example:

tpipe /input="%fn" /simple=28 /outputappend=1 /output=%fldr%%cOut
 
Similar threads
Thread starter Title Forum Replies Date
samintz How to? TPIPE /simple=61 Support 6
R Bug TPIPE's pdf to text conversions don't work Support 2
Kachupp How to? tpipe Support 4
H TPIPE /simple=4 Support 12
D TPIPE produces unwanted logs. Support 1
Kachupp How to? tpipe Support 8
vefatica Documentation TPIPE, /perl, DotMatchesNewLines Support 1
vefatica Stop TPIPE /EOL removing bad EOLs (and TPIPE changine EOLs)? Support 2
vefatica TPIPE in an alias? Support 2
C How to? TPIPE ?'s Support 2
H History and TPipe Oddness Support 3
Jay Sage Baffling Behavior With TPIPE Support 5
vefatica TPIPE: unbalanced escaped quotes in a regex? Support 5
Joe Caverly Multiple Text Searches at once using FFIND or TPIPE Support 4
Kachupp FFIND TPIPE Support 9
vefatica TPIPE and avarice? Support 5
Peter Murschall TPIPE generate unicode on Piping or redirecting Support 3
x13 TPIPE /REPLACE - Remove line (including EOL) Support 3
Charles Dye Documentation TPIPE /REPLACELIST Support 0
Joe Caverly TPIPE and /inputstring Support 3
vefatica How to? Add real <ESC> characters with TPIPE Support 2
vefatica How to? TPIPE ... see output right away? Support 4
vefatica TPIPE.EXE crashes on Ctrl-C Support 9
O TPIPE: Error -Can´t access textpipeengine.dll Support 24
vefatica Extract fields with TPIPE? Support 10
vefatica TPIPE, crash and question Support 1
MickeyF Another TPIPE question Support 6
MickeyF Can't get my TPIPE /replace to work Support 4
Joe Caverly Printing TPIPE documentation Support 3
C WAD tpipe /simple=28 Support 8
vefatica Remove double-quotes with TPIPE Support 4
vefatica TPIPE crashes on Ctrl-C Support 1
C How to? Add date/time to custom tpipe based BTM Support 1
C tpipe /simple=28 error Support 5
T TPipe: Replacing String with Double Quotes Support 3
C tpipe and extract email addresses Support 1
vefatica How do I do this with TPIPE? Support 1
vefatica TPIPE's /perl option? Support 1
D TPIPE help typo Support 0
vefatica TPIPE access violations are back Support 2
Joe Caverly Word Count using TPIPE Support 10
Dan Glynhampton TPIPE in Build 41 Support 4
Phileosophos TPIPE doesn't work at all with v16 Support 3
vefatica TPIPE access violation again Support 1
David McClelland TPIPE matching problem Support 3
vefatica TPIPE: Access violation Support 5
jbanaszczyk TCMD 16 - TPIPE Support 4
P TPipe relaunches TCC window and 4start, etc Support 7
Phileosophos Documentation TPIPE /string type code confusion Support 4
C TPIPE questions.... Support 5

Similar threads