filename - Filename or folder to read. This can be either a disk file, file list (@filename), or CLIP:. If it is not specified, TPIPE will read from standard input.

subfolders - How many subfolders to include (default 0):

0 - no subfolders

1 to 255 - subfolder(s)

255 - all subfolders

action - the action to take (default 1):

1 - include the files

2 - exclude the files

3 - ignore the files

You can specify multiple /input statements.

Filename to write. This can be either a disk file or CLIP:. If it is not specified, TPIPE will write to standard output.

Set the output filter directory.

Determines how binary files are processed. The options are:

0 - Binary files are processed (default)

1 - Binary files are skipped

2 - Binary files are confirmed before processing

size - The sample size in bytes to use for identifying binary files (default 255)

In clipboard mode, determines whether the input is ASCII (0) or Unicode (1). The default is 0.

If 1, the input files will be deleted after processing. USE WITH CAUTION!

If 1, TPIPE will prompt before processing each input files.

If 1, TPIPE will prompt before processing read-only input files.

Process the string as if it were a file and return the result. This option will write the return value to STDOUT; you cannot specify an /output argument.

If n is 1, append to the output file.

Sets the output changed mode. The options are:

0 - Always output

1 - Only output modified files

2 - Delete original if modified

If n=1, TPIPE will open each output file in its associated program upon completion. If there is no association for a file, it will be opened in the default editor.

0 - Output to clipboard (all files are merged)

1 - Output to files

2 - Output to a single merged file

If 1, TPIPE will open each output file in its associated program upon completion.

If n is 1, retain the existing file date for the output file.

Runs the current filter with input from and output to the clipboard.

Name of filter file to load (see /save=filename)

Saves the filter settings defined on the command line to the specified filename, and returns without executing any filters.

The following filters are created as sub filters, until the closing /ENDSUBFILTERS. Sub filters allow a restricted part of the entire text to be operated on by a group of filters without effecting the entire text. For example, a "Restrict to delimited fields" (CSV, Tab, Pipe, etc.) filter can pick out a range of CSV fields, and then a search/replace filter can operate JUST on the text restricted.

End the sub filters defined by the preceding /STARTSUBFILTERS.

Sets the buffer size for the preceding search/replace filter. (The default is 4096.)

Sets the edit distance threshhold for the preceding search/replace filter. (The default is 2.)

Add a comment to a filter file.

 Text - Comment to add

Adds a database-type filter. Database filters will change the output extension to match the format.

Mode

0 Delimited output

1 Fixed width

2 XML

3 Insert script

4 JSON output

GenerateHeader - Generates header information when True.

Timeout - SQL command timeout in seconds.

ConnectionStr - The database connection string.

InsertTable - The name of the insert table.

FieldDelimiter - The string to use between columns.

Qualifier - The string to use around string column values.

Remove or show duplicate lines. The arguments are:

Type:

0 - Remove duplicate lines

1 - Show duplicate lines

MatchCase - If 1, do case-sensitive comparisons.

StartColumn - The starting column for comparisons (the first column is 1).

Length - The Length of the comparison.

IncludeOne - If 1, include lines with a count of 1 (only for Type 1).

Format - how the output should be formatted for Type=1.  Format strings are composed of plain text and format specifiers. Plain text characters are copied as-is to the resulting string.

Format specifiers have the following form:

%[index ":"][-][width][.precision]type

An optional argument index specifier

An optional left justification indicator, ["-"]

An optional width specifier, [width] (an integer). If the width of the number  is less than the width specifier, it will be padded with spaces.

An optional precision specifier [precision] (an integer). If the width of the number is less than the precision, it will be left padded with 0's.

The conversion type character:

d - decimal

s - string

Percent signs in the format string should be doubled (unless you back quote the /dup=`...` argument), and the count argument must appear before the string (unless you use the index specifier). For example, "%%d %%s" shows the count followed by the string. Or, with the index specifier, "string %%1:s count %%0:d".

Add an EOL (end of line) conversion filter. The arguments are:

input:

0 - Unix (LF)

1 - Mac (CR)

2 - Windows (CR/LF)

3 - Auto

If you are unsure of the source, select Auto. The Auto option can detect and modify text files containing a variety of line endings.

4 - Fixed (use the length parameter to specify the length)

If you are converting a mainframe file that contains fixed length records, select "Fixed length" and enter the record length. The maximum record length is 2,147,483,647 characters. Note: If you are converting 132 column mainframe reports, you should set the fixed length to 133, because each line has a prefix character.

output:

0 - Unix

1 - Mac

2 - Windows

3 - None

length - The line length to use if input=4

LFString (optional) - The new line feed string on output when option 4 is chosen for input

Remove (optional) - Whether to remove bad EOLs (default 1)

Add a file-type filter. The arguments are:

type:

0 - Add left margin

1 - Add header

2 - Add footer

3 - Add right margin

4 - Remove lines that match exactly

5 - Retain lines that match exactly

6 - Remove lines matching the Perl pattern

7 - Retain lines matching the Perl pattern

8 - Add text side by side

9 - Add repeating text side by side

10 - Not Used

11 - Not Used

12 - XSLT transform

13 - Restrict to lines from list

14 - Restrict to lines NOT in list

15 - Restrict to lines matching the Perl pattern

16 - Restrict to lines NOT matching the Perl pattern

17 - Restrict to filenames patching the Perl pattern

18 - Restrict to filenames NOT matching the Perl pattern

MatchCase - If 1, do a case sensitive match (where appropriate)

filename - the filename to use

Adds a Grep type line based filter. The arguments are:

Type:

0 Restrict lines matching (for subfilters)

1 Restrict lines NOT matching (for subfilters)

2 Extract matches

3 Extract matching lines (grep)

4 Extract non-matching lines (inverse grep)

5 Remove matching lines

6 Remove non-matching lines

IncludeLineNumbers - 1 to include the line number where the pattern was found

IncludeFilename - 1 to include the filename where the pattern was found

MatchCase - 1 to do a case-sensitive comparison when matching the pattern

CountMatches - 1 to output a count of the number of matches

PatternType

0 Perl pattern

1 Egrep pattern

2 Brief pattern

3 MS Word pattern

UTF8 - 1 to allow matching Unicode UTF8 characters

IgnoreEmpty - 1 to ignore empty matches

Pattern - the (regular expression) pattern to match

Add a head type filter (includes or excludes text at the beginning of the file). The arguments are:

Exclude:

0 - Include the text

1 - Exclude the text

LinesOrBytes:

0 - Measure in lines

1 - Measure in bytes

Count - the number of lines or bytes to include or exclude

Add an insert type filter. The arguments are:

type:

0 - Insert column

Inserts a new column of text. The position the text is inserted is determined by a column count. The leftmost column is column 1 – inserting in this column displaces all other text to the right. If the insert column given is 0, the text is inserted at the end of the line. If the insert column is negative, the text is inserted at the given position relative to the end of the line. If the insert column given is before the start of the line, or beyond the end of the line, then the text is prepended or appended to the line respectively. Note - this filter is designed for ANSI or Unicode UTF-8 data - it will not handle UTF-16 data. If you need to process UTF-16 files, convert them to UTF-8 first and then convert back to UTF-8 after doing the insertion.

1 - Insert bytes

Insert bytes at the given offset (from 0 to the size of the file).

position - the position to insert the string

string - the string to insert

Adds a Line Number filter. The arguments are:

StartNumber - the starting line number

Increment - the amount to add for each new line number

SkipBlankIncrement  - don't increase the line number for blank lines

DontNumberBlank - don't put a line number on blank lines

NumberFormat - The format to use for the line number. The format syntax is:

[-][width][.precision]d

An optional left justification indicator, ["-"]

An optional width specifier, [width] (an integer). If the width of the number  is less than the width specifier, it will be padded with spaces.

An optional precision specifier [precision] (an integer). If the width of the number is less than the precision, it will be left padded with 0's.

The conversion type character:

d - decimal

DontReset - if 1, do not reset the line count at the end of the file. The default is 0.

ResetNewFile - if 1, reset the count at the start of a new file. The default is 0.

Log the TPIPE actions.

 Filename - Name of log file

If n is 1, append to the log file.

Adds a maths type filter.

operation - the operation to perform

0        +

1        -

2        *

3        div (the remainder is ignored)

4        mod (the remainder after division)

5        xor

6        and

7        or

8        not

9        shift left (0 inserted)

10        shift right (0 inserted)

11        rotate left

12        rotate right

operand - the operand to use

Adds a merge type filter (merge into single output filename). The arguments are:

type:

0 Merge into filename

1 Retain lines found in filename

2 Remove lines found in filename

3 Link filter filename

filename - the filename to use

Add a number-type filter. The arguments are:

type:

0 Convert Tabs to Spaces

1 Convert Spaces to Tabs

2 Word wrap (value column width)

3 Pad to width of value

4 Center in width of value

5 Right justify in width of value

6 Restrict CSV field to value

7 Restrict tab-delimited field to value

8 Truncate to width value

9 Force to width value

10 Repeat file value times

11 Restrict to blocks of length

12 Expand packed decimal (with implied decimals)

13 Expand zoned decimal (with implied decimals)

14 Expand unsigned (even-length) packed decimal

15 Expand unsigned (odd-length) packed decimal

Value - the numeric value to use

Sets the Perl matching options for the immediately preceding search/replace filter.

BufferSize - The maximum buffer size to use for matches. Any match must fit into this buffer, so if you want to match larger pieces of text, increase the size of this buffer to suit. Default is 4096.

Greedy - If the pattern finds the longest match (greedy) or the shortest match. Default is false.

AllowComments - Allow comments in the Perl pattern. Default is false.

DotMatchesNewLines - Allow the '.' operator to match all characters, including new lines. Default is true.

Adds a search and replace (find and replace) filter. Search / Replace lists discard blank search terms and terms where the replacement is identical to the search. Search / Replace lists can generate log entries (useful for debugging). Logs can optionally be output only for where replacements occurred.

The arguments are:

Type:

0 Replace

1 Pattern (old style)

2 Sounds like

3 Edit distance

4 Perl pattern

5 Brief pattern

6 Word pattern

MatchCase - Matches case when set to 1, ignores case when set to 0

WholeWord - Matches whole words only when set to 1

CaseReplace - Replaces with matching case when set to 1

PromptOnReplace - Prompts before replacing when set to 1

Extract - If 1, all non-matching text is discarded

FirstOnly - If 1, only replace the first occurrence

SkipPromptIdentical - If 1, don't bother prompting if the replacement text is identical to the original.

Action - the action to perform when found:

0 replace

1 remove

2 send to subfilter

3 send non-matching to subfilter

4 send subpattern 1 to subfilter etc

SearchStr - the string to search for

ReplaceStr - the string to replace it with

Add a search and replace list, using search and replace pairs from the specified file.

Type:

0 Replace

1 Pattern (old style)

2 Sounds like

3 Edit distance

4 Perl pattern

5 Brief pattern

6 Word pattern

MatchCase - Matches case when set to 1, ignores case when set to 0

WholeWord - Matches whole words only when set to 1

CaseReplace - Replaces with matching case when set to 1

PromptOnReplace - Prompts before replacing when set to 1

FirstOnly - If 1, only replace the first occurrence

SkipPromptIdentical - If 1, don't bother prompting if the replacement text is identical to the original.

Simultaneous - If 1, all search strings are scanned for simultaneously instead of consecutively. (This is useful if the search strings and results strings overlap.)

LongestFirst - If 1, searches for long phrases (most specific) before short phrases (least specific) - this is generally used for translations.

Filename - The file to load search/replace pairs from. If the file extension is .XLS or .XLSX, the file is assumed to be Excel format, if the extension is .TAB the file is assumed to have tab-delimited values, and any other extension (including .CSV) is assumed to have Comma-Separated Values. The filename can contain environment variables enclosed in % signs e.g. %TEMP%\myfile.txt. TPIPE corrects any doubled backslashes.

Adds a Run External Program filter. The arguments are:

InputFilename - the filename that TextPipe should read from after the External Program writes to it.

OutputFilename - the filename that TextPipe should write to for the External Program to read in.

CommandLine - the command line of the program to run. Should include double quotes around the entire command line.

Adds an ActiveX script filter.

language: The language of the script

timeout:  The command timeout in seconds

script:  The code

Type - The type of filter to add:

0 – Remove column:

This filter is used to remove columns of text, given a column specification that describes the position of the column relative to the start or end of the line, and the width of the column. There are several ways to specify the columns (Locate,Param1,Param2) to remove:

0 - Start column, End column. This removes all text including and between the specified columns. Useful for removing column in fixed width data files.

1 - Start column, width. Removes Width characters starting from (and including) column Start.

2 - End column, width. Removes Width characters backwards starting from (and including) column End.

3 - Start column to end of line. Removes all characters from the Start column to the very end of the line. Useful for making a file a uniform width.

4 - Width to end of line. Removes Width characters backwards starting from (and including) the last column.

Note - if you are removing more than one column range, it is easiest to remove ranges from right-to-left so that the position of the columns doesn't change.

1 - Restrict lines (restriction filters require sub filters to have any effect)

2 - Restrict columns (restriction filters require sub filters to have any effect)

3 - Restrict to bytes (restriction filters require sub filters to have any effect)

4 - Restrict to delimited fields (CSV, Tab, Pipe, etc.)

6 – Remove lines:

This filter removes a range of lines. There are several ways to specify the lines (Locate,Param1,Param2) to remove:

0 - Start line, End line. This removes all lines including and between the specified lines.

1 - Start line, width. Removes Width lines starting from (and including) line Start.

2 - End line, width. Removes Width lines backwards starting from (and including) line End.

3 - Start line to end of line. Removes all lines from the Start line to the very end of the line.

4 - Width to end of line. Removes Width lines backwards starting from (and including) the last line.

7 – Remove delimited fields (CSV, Tab, Pipe, etc.):

This filter is used to remove fields delimited by a given character. You can choose a predefined delimiter character (Delimiter), or select your own (CustomDelimiter). The trailing delimiter (if any) is also removed. When Comma (.csv) is chosen, TPIPE automatically handles single and double quoted strings, with embedded line feeds.

First Row Contains Field Names

If the first line of the file contains Field Names, set HasHeader to 1 so that TPIPE can count how many fields are expected. It can then determine if a field has embedded CR/LF characters and spans multiple lines. TPIPE can also determine this without a header if the fields are properly double-quoted - TPIPE will notice the missing double quote and continue reading the record from the following line.

Remove Fields

There are several ways to specify the fields (Locate,Param1,Param2) to remove:

0 - Start field, end field. This removes all text including and between the specified fields.

1 - Start field, width. Removes Width fields starting from (and including) field Start.

2 - End field, width. Removes Width fields backwards starting from (and including) field End.

3 - Start field to end of line. Removes all fields from the Start field to the very end of the line.

4 - Width to end of line. Removes Width fields backwards starting from (and including) the last field.

Note - if you are removing more than one field range, it is easiest to remove ranges from right-to-left so that the position of the fields doesn't change.

9 – Move columns:

TPIPE will move columns to a new position on the line. The new position (MoveTo) is specified assuming that the moved columns have been removed from the line.

10 – Move delimited fields (CSV, Tab, Pipe, etc.):

TPIPE will move CSV-delimited fields to a new position on the line. The new position (MoveTo) is specified assuming that the moved fields have been removed from the line. TPIPE ensures that all the delimiters on the line are correctly maintained, both at the end of the line and where the moved fields are inserted. Note - this filter is designed for ANSI or Unicode UTF-8 data - it will not handle UTF-16 data. You will need to convert UTF-16 files to UTF-8 first, do the selection, and then convert back to UTF-16.

12 – Copy columns:

TPIPE will copy columns to a new position (MoveTo) on the line. Note - this filter is designed for ANSI or Unicode UTF-8 data - it will not handle UTF-16 data. You will need to convert UTF-16 files to UTF-8 first, do the selection, and then convert back to UTF-16.

13 – Copy delimited fields (CSV, Tab, Pipe, etc.):

TPIPE will copy CSV-delimited fields to a new position (MoveTo) on the line. TPIPE ensures that all the delimiters on the line are correctly maintained, both at the end of the line and where the copied fields are inserted. Note - this filter is designed for ANSI or Unicode UTF-8 data - it will not handle UTF-16 data. You will need to convert UTF-16 files to UTF-8 first, do the selection, and then convert back to UTF-16.

17 – Remove byte range:

This filter is used to remove a range of bytes. There are several different ways to specify the bytes (Locate,Param1,Param2) to remove:

0 - Start byte, end byte. This removes all text including and between the specified byte.

1 - Start byte, width. Removes Width byte starting from (and including) the start byte.

2 - End byte, width. Removes Width fields backwards starting from (and including) byte End.

3 - Start byte to end of file. Removes all fields from the Start byte to the very end of the file.

4 - Width to end of file. Removes Width fields backwards starting from (and including) the last byte.

Note - if you are removing more than one byte range, it is easiest to remove ranges from right-to-left so that the position of the bytes doesn't change.

Locate - How to determine which areas to affect:

0 - Restrict %d .. %d

1 - Restrict %1:d starting at %0:d

2 - Restrict %1:d starting at END - %0:d

3 - Restrict %d .. END - %d

4 - Restrict END - %d .. END - %d

Param1, Param2 - The integer values for the Locate method.

MoveTo - The integer value where to move or copy the columns or fields to (first columns or field is 1).

Delimiter - The index of the standard delimiter to use:

0 - Comma

1 - Tab

2 - Semicolon

3 - Pipe (|)

4 - Space

5 - Custom

CustomDelimiter - The custom delimiter to use (if Delimiter == 5). This should be a quoted string; if you are not using a custom delimiter then set this field to "".

HasHeader - 1 if the file's first row is a header row, 0 if not.

ProcessIndividually - Whether to apply sub filters to each CSV or Tab field individually (1), or to the fields as one string value (0). The default is 0.

ExcludeDelimiter - Whether or not to include the comma or Tab field delimiter when passing the field to the sub filter. Defaults to 0.

ExcludeQuotes - Whether or not to include the CSV quotes that may surround the field when passing the field to the subfilter. Defaults to 1.

Type - the type of filter to add

0 - Delete column

1 - Restrict lines

2 - Restrict columns

3 - Restrict to bytes

4 - Restrict to delimited fields (CSV, Tab, Pipe etc)

5 - unused

6 - Remove lines

7 - Remove delimited fields (CSV, Tab, Pipe etc)

9 - Move columns

10 - Move delimited fields (CSV, Tab, Pipe etc)

12 - Copy columns

13 - Copy delimited fields (CSV, Tab, Pipe etc)

17 - Remove Byte Range

18 - Extract fields

columnSpec - the double-quoted list of items to remove e.g. "1..10, 16, 20"

moveTo - (integer) where to move or copy the columns or fields to.

processIndividually - whether or not to apply sub filters to each CSV or Tab field individually, or to the fields as one string value.

excludeDelimiter - whether or not to include the comma or Tab field delimiter when passing the field to the sub filter.

excludeQuotes - whether or not to include the CSV quotes that may surround the field when passing the field to the sub filter.

delimiter - (optional) the index of the standard delimiter to use, default 0 for CSV

0 - Comma

1 - Tab

2 - Semicolon

3 - Pipe (|)

4 - Space

5 - Custom

customDelimiter - (optional) the double quoted custom delimiter to use; the default is blank.

hasHeader - (optional) 1 if the file's first row is a header row, default 0.

Adds a simple filter type. n is the type of filter to add, and for those filters that support it, u indicates that the filter will be dealing with Unicode data.

1 – Convert ASCII to EBCDIC

EBCDIC is the character collating sequence commonly used on mainframes. Some characters cannot be converted because they exist in one character set but not the other.

2 – Convert EBCDIC to ASCII

3 – Convert ANSI to OEM

Converts from ANSI to ASCII/OEM. ANSI is an 8-bit character set used by Windows, and it includes all accentuated Roman characters used by non-English languages like French, German and Spanish. (Windows uses UTF-16LE for all of its internal APIs, and converts to ANSI if the user is using raster fonts or ANSI files.) ASCII/OEM is an extension of the original IBM character set where various non-essential characters are replaced by language-specific accentuated characters. Different ASCII/OEM character sets are not compatible. They must be converted to ANSI and then back to the correct ASCII/OEM character set to be readable.

4 – Convert OEM to ANSI

5 – Convert to UPPERCASE

Forces all text to UPPERCASE. To make the conversion, the function uses the current language selected by the user in the system Control Panel. If no language has been selected, TPIPE uses the Windows internal default mapping.

6 – Convert to lowercase

Forces all text to lowercase. To make the conversion, the function uses the current language selected by the user in the system Control Panel. If no language has been selected, TPIPE uses the Windows internal default mapping.

7 – Convert to Title Case

Converts all text to Title Case -- i.e., the first letter of every word is capitalized, and all other letters are forced to lower case. This routine calculates a table of upper and lower case letters on TPIPE startup, and this determination is based on the semantics of the language selected in Control Panel.

8 – Convert to Sentence Case

Converts all text to Sentence case ie the first word in every sentence is capitalized, all other letters are left as is. Sentences start after periods, exclamation marks, colons, question marks, quotes, parentheses and angle brackets (.!:?'"<().

9 – Convert to tOGGLE cASE

tOGGLES tHE cASE of all text -- i.ee, all UPPERCASE characters are converted to lowercase and vice-versa.

10 – Remove blank lines

Removes blank lines. Note, lines with spaces or tabs are not removed. Use the Remove Blanks From Start Of Line filters first to rectify this.

11 – Remove blanks from End of Line

Removes spaces and tabs from the end of every line.

12 – Remove blanks from Start of Line

Removes spaces and tabs from the start of every line.

13 – Remove binary characters

Removes binary characters such as those higher than ASCII code 127, and those less than ASCII code 32 except for carriage returns (ASCII code 13) and line feeds (ASCII code 10).

This filter if very useful if you have a corrupted text file, or if you just want to see what text is inside a binary file. The binary information is removed, leaving you with just the text.

14 – Remove ANSI codes

ANSI (American National Standards Institute) codes are included in various streams of information, to provide a remote computer with control over cursor positioning, text attributes, etc. They are also used in connections between minicomputers and mainframe computers and the terminals connected to them.

The need to use an ANSI filter can be recognized when something like the following example shows up in a file viewed in a text editor:

<[0;1;4mas<[m - MC88000 assembler

In this example, the "as" near the beginning is displayed in a different color than the rest of the line when the ANSI codes are properly processed. The Escape (ASCII 27) codes above have been replaced by the < symbol to make this line printable.

The Remove ANSI Escape Sequences filter can be used to filter out these codes and "clean up" the text so that it can be used in standard fashions such as copying and pasting into a word processor. On Unix machines the man (manual) help utility will only allow page-by-page browsing through a file in a forward direction. By piping the man output to a text file, transferring it to a DOS machine, and running it through the Remove ANSI Escape Sequences filter (and the Convert EOL filter - Unix to DOS if desired), a standard DOS editor can be used for browsing through the file, quoting from it, etc.

15 – Convert IBM drawing characters

IBM drawing characters in the upper ASCII range (128-255) are commonly used to draw lines and boxes, single and double line borders, shaded characters etc. Many devices (such as printers, non-IBM computers etc.) do not support the display of these characters.

This filter converts them to standard ASCII characters (+, - and |) that all computers can display.

16 – Remove HTML and SGML

Use this filter to convert HTML documents to a readable format. This filter removes HTML and XML markup tags i.e. everything including and between <> brackets.

17 – Remove backspaces

Remove backspaces, i.e. all ASCII code 8's.

18 – Resolve backspaces

Resolve backspaces -- i.e., remove both the backspaces and the characters prior to the backspaces that would have been deleted.

19 – Remove multiple whitespace

Removes sequences of multiple spaces or tabs and replaces them with a single space.

20 – UUEncode

Usually used for transmitting binary files inside an email. Files of this type are usually given an extension of .uue. Warning – UUencoded text may be corrupted when passing over a mainframe mail gateway. To avoid corruption, use Mime Base 64 or XXEncode.

21 – Hex Encode

A very simple encoding of a file. Usually used for small files, because it uses a large amount of space. The benefit is that the file is very easy to encode/decode, and the file cannot be corrupted passing through mail gateways.

22 – Hex Decode

Converts a file from its hex representation back to binary. The file to be decoded MUST NOT have any extra characters at the start or end if it is to be successfully processed.

23 – MIME Encode (Base 64)

Used for binary data. Files of this type are usually given an extension of .b64.

24 – MIME Decode (Base 64)

Used for binary data. Files of this type are usually given an extension of .b64. The file to be decoded MUST NOT have any extra characters at the start or end if it is to be successfully processed.

25 – MIME Encode (Quoted printable)

Quoted printable is used for text that is mainly readable, but may contain special characters with accents etc.

26 – MIME Decode (Quoted printable)

The inverse of the above encoding.

27 – UUDecode

Mail attachments can be uuencoded, use this filter to convert the file back to its correct form. Files of this type are usually given an extension of .uue.

28 – Extract email addresses

Extract email addresses. This filter searches for email addresses of the form [email protected], and writes them out one per line (using a DOS line feed, CR/LF). Usually this filter is followed by a filter to remove duplicate lines, and then by a Search and Replace filter, searching for \013\010 and replacing with a comma or semi-colon, depending on the email address separator used by your email software.

29 – Unscramble (ROT13)

This is a simple email encoding usually used to disguise text that some people may find offensive. The encoding is totally reversible (applying it twice removes the encoding). Only alpha characters are affected (A..Z and a..z).

30 – Hex dump

This changes the text to lines consisting of 16 bytes each. Each line has an 8 hex digit file index, 16 bytes (in hex) and the ASCII representation:

00000000 65 67 69 6E 0D 0A 20 20 20 20 20 20 61 64 64 72 egin........addr

00000010 65 73 73 20 3A 3D 20 0D 0A 20 20 20 20 20 20 20 ess.:=..........

00000020 20 64 65 63 54 6F 48 65 78 53 74 72 28 20 28 66 .decToHexStr(.(f

This filter is very useful for identifying special characters to search and replace.

32 – XXEncode

Essentially identical to UUEncode except that the character set used is different to allow it to pass through EBCDIC gateways without corruption. The XXencoding implemented by TPIPE uses the following characters:

+-0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz

33 – XXDecode

Essentially identical to UUDecode except that the character set used is different to allow it to pass through EBCDIC gateways without corruption.

34 – Reverse line order

The order of the input lines is reversed i.e. the last line comes out first and the first line comes out last. A file is read entirely into RAM before being reversed, so be wary of reversing files that are larger than your machine's RAM size.

35 – Remove email headers

This filter removes the email headers that accompany emails exported to a text format. The email headers are the lines such as To:, From:, Subject: and various other message headers added by all the servers through which your email passes before it gets to its destination.

36 – Decimal dump

This changes the text to lines consisting of 10 bytes each. Each line has a 10 decimal digit file index, 10 bytes (in decimal) and the ASCII representation:

0000000000 080 108 101 097 115 101 032 102 101 101 Please fee

0000000010 108 032 102 114 101 101 032 116 111 032 l free to

0000000020 099 111 109 109 101 110 116 013 010 111 comment..o

This filter is very useful for identifying special characters to search and replace.

37 – HTTP Encode

This filter is used to encode text for use in an HTTP header – a (usually) small piece of text that accompanies a web page request to a web server. This filter is very useful for debugging CGI scripts because it can create HTTP requests in the correct form. HTTP encoded text usually looks like the following:

a+%28usually%29+small+piece+of+text+that+accompanies+a+web+page+request+to+a+web+server.+This+filter+is+very+

38 – HTTP Decode

This filter is used to decode text from an HTTP header – a (usually) small piece of text that accompanies a web page request to a web server.

39 – Randomize lines

This filter put lines into random order. This is useful when a random sample of data is required for statistical purposes - just follow this filter with a head/tail of file filter (/head or /tail). The lines output will differ from one run to the next; the order  is determined by a pseudo-random number generator.

40 – Create word list

This filter takes all the incoming words and outputs them one per line. This can be used to generate word lists for Indexes, encryption programs etc. Hyphenated words are recognized as single words, provided that they aren't broken across lines. To get around this limitation, use a Search and Replace filter to replace hyphens followed by line feeds with just a hyphen. Normally you would follow this filter with a remove duplicates filter, or alternatively, a Count Duplicate Lines filter (with Include counts of 1).

catch22 – a word

24-7 – a word

twenty-four – a word

5th – a word

ice cream – two words

Commas or periods after words are treated as word separators.

41 – Reverse each line

Each line is output reversed from left to right. This can be useful to extract domain names from web site log files - use this filter to reverse each line, use an extract matches filter of [\w\d]+\.[\w\d]+ to extract each domain name, then reverse each line again. Note: This filter will NOT work on Unicode or UTF-8 data. It will only work on single-byte data such as ASCII or ANSI.

42 – Convert to RanDOm case

This filter randomly changes the case of characters. This routine calculates a table of upper and lower case letters on TPIPE startup, and this determination is based on the semantics of the language selected in the Windows Control Panel.

Running this filter again will generate different results; for example:

1. ranDoMIze cASe

2. RanDOmIZE case

3. randOMIZE casE

43 – Extract URLs

Extract URLs. This filter lists mailto:, http://, https://, ftp://, ftps://, nntp:, skype:, call:, and gopher:// URLs one per line.

44 – ANSI to Unicode

Converts single byte ANSI characters to double byte Unicode characters. This filter can be useful if you want to send a text file to someone using a language other than your own. This filter is often followed by an Add Header filter, to add a Unicode byte order mark (BOM), \xFF\xFE.

45 – Unicode to ANSI

Converts double byte Unicode characters to single byte ANSI characters. This filter can be useful if you want to send a text file to someone using a language other than your own. This filter is often followed by a Remove start or end of file filter, to either remove the first two bytes of Unicode (before the conversion) or the first byte of ANSI (after the conversion), to remove the leading Unicode byte order mark (BOM).

46 – Display debug window

A debug filter is very handy for debugging filters. When text is passed through this filter, it places the output into a window so that you can see what the text looks like at that stage of the filtering process.

47 – Word concordance

This filter generates a word concordance. A word concordance shows the context or surrounding words for a given set of words in a dictionary.

48 – Remove all

This filter removes all text. Unlike a pattern match filter that matches everything and then throws it away, this filter is far more efficient, especially for large files, as it signals completion back to the input filter so only the first chunk of a multi-gigabyte file will ever get processed.

It is useful in two main situations

1. Inside a subfilter, it prevents any of the subfiltered text from re-entering the text stream. So you could restrict to lines matching a pattern, output the matching lines to a new file, and then remove them.

2. To remove all of the text of a file, then use an Add Header filter with the @fullInputFilename macro to obtain the name of the file.

Note: An Add Left Margin or Add Right Margin filter will not work after a Remove All filter, as they require an actual line to trigger them. Instead, use an Add Header or Add Footer filter.

49 – Restrict to each line in turn

This filter restricts sub filters to operate on each line in turn. This filter is used for its side effect of limiting the matched text to a single line at most.

50 – Convert CSV to Tab-delimited

Converts CSV data (quoted or unquoted) to tab-delimited form. It's preferable to use a file with column headers, because then TPIPE can easily determine if the fields have embedded CR/LFs in them. If the data is properly quoted then TPIPE will determine this automatically. TPIPE will eliminate unnecessary quotes.

51 – Convert CSV to XML

Converts CSV data (quoted or unquoted) to XML form. It's preferable to use a file with column headers, because then TPIPE can easily determine if the fields have embedded CR/LFs in them. If the data is properly quoted then TPIPE will determine this automatically. TPIPE correctly escapes < > " ' and & in the data to the corresponding XML entity. If your data contains invalid XML characters such as ASCII 26 (End-of-file, hex \x1A), follow this filter with a search/replace filter to remove \x1A and replace with nothing.

52 – Convert Tab-delimited to CSV

Converts Tab-delimited data to CSV data. It's preferable to use a file with column headers, because then TPIPE can easily determine if the fields have embedded CR/LFs in them. TPIPE cannot determine this without column headers.

53 – Convert Tab-delimited to XML

Converts Tab-delimited data to XML data. It's preferable to use a file with column headers (/simple=55), because then TPIPE can easily determine if the fields have embedded CR/LFs in them. TPIPE cannot determine this without column headers. TPIPE correctly escapes < > " ' and & in the data to the corresponding XML entity. If your data contains invalid XML characters such as ASCII 26 (End-of-file, hex \x1A), follow this filter with a search/replace filter to remove \x1A and replace with nothing.

54 – Convert CSV (with column headers) to XML

See description for 51 – Convert CSV to XML.

55 – Convert Tab-delimited (with column headers) to XML

See description for 53 – Convert Tab-delimited to XML.

56 – Convert CSV (with column headers) to Tab-delimited

See description for 50 – Convert CSV to Tab-delimited.

57 – Convert Tab-delimited (with column headers) to CSV

See description for 52 – Convert Tab-delimited to CSV.

58 – Restrict to file name

This filter applies its subfilters only to files with filenames (ie drive + path + filename) matching or not matching a pattern or list of patterns. This is very handy for only applying a Convert Word Documents to Text filter only to files matching the pattern

\.DOC$

With the appropriate pattern, this filter can also be used to control subfilters based on filename, folder and drive. Note that this filter uses case-insensitive Perl regular expressions, not Windows wildcards.

59 – Convert Word documents to (UTF8) text

This filter takes ALL incoming documents, opens them with Microsoft Word, and outputs them as text files. This can be used to process a set of Word Documents to text file format. After this filter you can add search and replace filters or any other filters you choose.

This filter requires Microsoft Word 98 or higher to be installed. If you wish to convert documents other than the default .DOC files, you may also need to install Word's conversion filters. If Word cannot be started automatically TPIPE will prompt you to start it manually before continuing.

Unless you know that all documents being processed are Word documents (e.g. by using a wildcard of *.doc in the Files to Process tab), you should restrict this filter to only files matching the pattern:

\.DOC$

60 – Swap UTF-16 word order

This filter swaps pairs of bytes

e.g.

This is commonly used to transform big-endian or little-endian Unicode files so that other programs can use them.

61 – Swap UTF-32 word order

This filter swaps groups of 2-byte words.

e.g.

This is commonly used to transform big-endian or little-endian Unicode files so that other programs can use them.

62 – Remove BOM (Byte Order Mark)

This filter removes the Unicode Byte Order Mark from the start of Unicode files, if it is present.

63 – Make Big Endian

Converts a Little Endian Unicode file into a Big Endian Unicode file

e.g.

Note - the file MUST start with a Byte Order Mark (BOM) for it to be correctly identified.

64 – Make Little Endian

Converts a Big Endian Unicode file into a Little Endian Unicode file

e.g.

Note - the file MUST start with a Byte Order Mark (BOM) for it to be correctly identified.

65 – Compress to Packed Decimal

This filter compresses EBCDIC numeric data (optional leading sign, numbers and periods) to an EBCDIC packed decimal field (also known as Comp-3).

There are several notes to keep in mind when using this filter:

1. You MUST use this filter inside a Restrict to Byte Range filter. The field WIDTH is then set by the containing filter.

2. Compressing a field will decrease your output record length, so ensure you allow for this. A good strategy to avoid problems is to first compress the rightmost field, then work your work back to the leftmost field. This prevents the field column positions from changing and makes the file easier to work with.

This filter will add hex 'B' to negative fields, hex 'C' to positive fields and hex 'F' to unsigned fields. If these codes don't match what your target needs, use a column or CSV restriction to apply a search/replace.

66 – Compress to Zoned Decimal

This filter expands an EBCDIC zoned decimal field to a raw EBCDIC number with a sign. Typically this filter is then followed by a Convert EBCDIC to ASCII filter - after all other fields have been expanded as well.

There are several notes to keep in mind when using this filter:

1. You MUST use this filter inside a Restrict to Byte Range filter. The field WIDTH is then set by the containing filter.

2. Expanding a field will increase your output record length, so ensure you allow for this. A good strategy to avoid problems is to first expand the rightmost field, then work your work back to the leftmost field. This prevents the field column positions from changing and makes the file easier to work with.

67 – Expand Binary Number to EBCDIC

This filter expands a series of digits stored in binary (BIG ENDIAN) form. The maximum width is 8 bytes.

There are several notes to keep in mind when using this filter:

1. You MUST use this filter inside a Restrict to Byte Range filter. The field WIDTH is then set by the containing filter.

2. Expanding a field will increase your output record length, so ensure you allow for this. A good strategy to avoid problems is to first expand the rightmost field, then work your work back to the leftmost field. This prevents the field column positions from changing and makes the file easier to work with.

3. If the data is stored in LITTLE ENDIAN order, use a Reverse filter inside the Restriction prior to the Expand Binary Numbers filter.

68 – Expand Binary Number to ASCII

This filter expands a series of digits stored in binary (BIG ENDIAN) form. The maximum width is 8 bytes.

There are several notes to keep in mind when using this filter:

1. You MUST use this filter inside a Restrict to Byte Range filter. The field WIDTH is then set by the containing filter.

2. Expanding a field will increase your output record length, so ensure you allow for this. A good strategy to avoid problems is to first expand the rightmost field, then work your work back to the leftmost field. This prevents the field column positions from changing and makes the file easier to work with.

3. If the data is stored in LITTLE ENDIAN order, use a Reverse filter inside the Restriction prior to the Expand Binary Numbers filter.

69 – NFC - Canonical Decomposition, followed by Canonical Composition

Applies a Unicode NFC - Canonical Decomposition, followed by Canonical Composition transformation to incoming Unicode text (UTF16-LE). Output is also Unicode UTF16-LE.

70 – NFD - Canonical Decomposition

Applies a Unicode NFD - Canonical Decomposition transformation to incoming Unicode text (UTF16-LE). Output is also Unicode UTF16-LE.

71 – NFKD - Compatibility Decomposition

Applies a Unicode NFKD - Compatibility Decomposition transformation to incoming Unicode text (UTF16-LE). Output is also Unicode UTF16-LE.

72 – NFKC - Compatibility Decomposition, followed by Canonical Composition

Applies a Unicode NFKC - Compatibility Decomposition, followed by Canonical Composition transformation to incoming Unicode text (UTF16-LE). Output is also Unicode UTF16-LE.

73 – Decompose

74 – Compose

Applies a Unicode Compose transformation to incoming Unicode text (UTF16-LE). The output is also Unicode UTF16-LE.

75 – Convert numeric HTML Entities to text

This filter converts decimal/hex numeric HTML/XML entities to plain text. For example:

&#174; → ®

&#xAE; → ®

Typically, the input file is ANSI (single byte) format. This filter will output UTF-8 characters for high-value entities e.g. &#6144; The best approach is to first convert the file from ANSI to UTF-8 (/unicode), then apply this filter.

76 – Convert PDF documents to (UTF8) text

This filter takes ALL incoming documents and converts them from PDF to text. Most of the formatting will be lost.

77 – Restrict to ANSI files

78 – Restrict to Unicode UTF16 files

79 – Restrict to Unicode UTF32 files

80 – Convert Excel spreadsheets to (UTF8) text

This filter takes ALL incoming documents, opens them with Microsoft Excel, and outputs them as CSV (comma-delimited) files (hidden worksheets will be ignored). After running this filter, you can add search and replace filters or any other filters you choose, such as convert the data to Tab-delimited or XML.

This filter requires Microsoft Excel 98 or higher to be installed. If you wish to convert documents other than the default .XLS files, you may also need to install Excel's conversion filters.

Unless you know that all documents being processed are Excel documents (e.g. by using a wildcard of *.xls in the Files to Process tab), you should restrict (/simple=58) this filter to only files matching the pattern

\.XLS$

81 - Shred file

82 - Unicode to escaped ASCII

83 - Restrict to Unicode files

84 - T-filter

The T-Filter allows you to process the same output in multiple ways. You can create a subfilter, and add filters to create the desired output. When this side of the T has finished processing, the data is discarded and the original text continues processing as though the T-filter did not exist.

85 - Convert decimal/hex numeric HTML/XML entities and entity names to text (i.e., &#174 -> ®, or &reg; -> ®). This filter outputs UTF-8 characters for high-value entities.

86 - Convert JSON to Tab

87 - Convert Tab to JSON

88 - Convert Word documents to RTF

/Simple has some redaction filters which are designed to work inside restriction filters.

89 - Remove diacritics

91 - Redact x-over text

92 - Redact x-over digits

93 - Redact x-over all but last 4 digits

94 - Redact x-over non-blanks

95 - Replace with blanks

96 - Redact with pseudo NHS

97 - Redact with pseudo SSN

98 - Redact with pseudo bank number

Sort text files.

Type - the sort type

0 - ANSI sort

1 - ANSI sort (case sensitive)

2 - ASCII sort

3 - ASCII sort (case sensitive)

4 - Numeric sort

5 - Sort by length of line

6 - sort by date and time

7 - sort by date

8 - sort by time

9 - UTF8 sort (case insensitive)

10 - UTF8 sort (case sensitive)

Reverse - If 1, sort in descending order; if 0, sort in ascending order

RemoveDuplicates - If 1, remove duplicate lines; if 0 keep duplicate lines

StartColumn - The column in the line to begin the comparisons

Length - The length of the comparison

Adds a split type filter. The arguments are:

type:

0 Split at a given size

1 Split at a given character

2 Split at a given number of lines

splitSize - the size file to split at

splitChar - the character to split at

splitCharPos

0 Split before the character (it goes into the next file)

1 Split after the character (it remains in the first file)

2 Split on top of the character (remove it)

SplitCharCount - the number of times to see SplitChar before splitting

SplitLines - (optional) split after a given number of lines, default 60

SplitFilename - (optional) the name to give to each output split file. /split will append a "%3.3d" format specifier to the name; i.e. SplitFilename of "foo.txt" will generate output files named "foo.txt.000", "foo.txt.001", etc. If you don't specify a SplitFilename, /split will use the input filename as the base.

FirstFileNumber - (optional) the number of the first file; default is 0

PreventOverload - (optional) true to prevent more than 10,000 files in one folder, default false

The split file filter will remove the last file if it is empty.

Add a string-type filter. The arguments are:

type:

0 - Add left margin

1 - Add header

2 - Add footer

3 - Add right margin

4 - Remove lines that match exactly

5 - Retain lines that match exactly

6 - Remove lines matching the Perl pattern

7 - Retain lines matching the Perl pattern

8 - Add text side by side

9 - Add repeating text side by side

10 - Not Used

11 - Not Used

12 - XSLT transform

13 - Restrict to lines from list

14 - Restrict to lines NOT in list

15 - Restrict to lines matching the Perl pattern

16 - Restrict to lines NOT matching the Perl pattern

17 - Restrict to filenames patching the Perl pattern

18 - Restrict to filenames NOT matching the Perl pattern

matchCase - case sensitive or not (where appropriate)

string - the string to use

Add a tail type filter (includes or excludes text at the end of the file). The arguments are:

Exclude:

0 - Include the text

1 - Exclude the text

LinesOrBytes:

0 - Measure in lines

1 - Measure in bytes

Count - the number of lines or bytes to include or exclude

Convert the file to or from Unicode. input is the encoding for the input file; output is the encoding for the output file. The possible values are:

UTF-16LE

UTF-16BE

UTF-32LE

UTF-32BE

UTF-8

ANSI

ASCII

CPnnn, where nnn is a Windows code page (for example, CP437 or CP1251).

TPIPE handles files internally as UTF-8, so if you want to process a Windows UTF-16LE file, you'll need to convert it to UTF-8 first, then apply the desired filters, and convert it back to UTF-16LE. For example, to wrap a Unicode file at column 80:

tpipe /input=inputname /output=outputname /unicode=UTF-16LE,UTF-8 /number=2,80 /unicode=UTF-8,UTF-16LE

Adds an HTML / XML filter. The arguments are:

Type - the operation to perform:

0 restrict to an element

1 restrict to an attribute

2 restrict to between tags

IncludeText - whether to include the find string in the restriction result (default false)

IncludeQuotes - whether to include surrounding quotes in the attribute result or not (default false)

MatchCase - match case exactly or not (default false)

BufferSize - the maximum expected size of the match (default 32768)

Tag - the element or start tag to find

Attribute - the attribute to find

EndTag - the endTag to find