Is there any way to work around this (apparent) limitation of TCC?

May 20, 2008
11,853
121
Syracuse, NY, USA
Here's yet another way (I like sed)
Code:
sed -n '1!G;h;$p' file
and you can pipe to sed. Notably passing multiple files as arguments reverses the (con)catenation of all lines not each single file.
That's cute but it fails (pipe or file) with enough data (like the names of all dirs on my C drive) going to stdout; it will write 'em to a file but the file contains only \n separators (no \r\n).

Code:
v:\> (do i=1 to 100000 (echo x) ) | g:\gnu\sed.exe -n 1!G;h;$p
G:\gnu\sed.exe: couldn't write 200000 items to stdout: Not enough space
 
v:\> v:\> (do i=1 to 100000 (echo x) ) | g:\gnu\sed.exe -n 1!G;h;$p > x.txt

v:\> type /x x.txt  | head /n3
0000 0000 78 0a 78 0a 78 0a 78 0a  78 0a 78 0a 78 0a 78 0a  x.x.x.x.x.x.x.x.
0000 0010 78 0a 78 0a 78 0a 78 0a  78 0a 78 0a 78 0a 78 0a  x.x.x.x.x.x.x.x.
0000 0020 78 0a 78 0a 78 0a 78 0a  78 0a 78 0a 78 0a 78 0a  x.x.x.x.x.x.x.x.
 
May 20, 2008
11,853
121
Syracuse, NY, USA
Vince, sorry it took me to now to get back to this to answer your question (usual reason(s)), but my "new" "TAC" is 4/20/2005@13:41, and it does, in fact, work OK for me.
Well, that would appear to be the one I just installed, and it doesn't work on the right side of a pipe! It produces no output.
Code:
g:\gnuwin32\bin> tac --version
tac (GNU coreutils) 5.3.0
Written by Jay Lepreau and David MacKenzie.

g:\gnuwin32\bin> dir /k /m tac.exe
2005-04-20  14:41          92,160  tac.exe

g:\gnuwin32\bin> echo 1^r^n2^r^n3^r^n4 | .\tac.exe

g:\gnuwin32\bin>

It works om XP but loses a lot of data:

Code:
f:\> (global /i /q echo %_cwd) | wc
  Lines   Words   Chars
   1595    5294  100875

f:\> (global /i /q echo %_cwd) | \\zz\g$\gnuwin32\bin\tac.exe | wc
  Lines   Words   Chars
     81     316    4136
 
May 24, 2010
855
0
Northlake, Il
Vince,

When I started to type this, I thought I was just going to beat a dead horse, but in doing a little more research it turned out that I was wrong. My TAC (a little more below) works fine when the input is from a specified file or redirected ("<"), but does not work when the input is from a pipe (not how I had used it). However, it does not crash when input is being read from a pipe here under Windows 7, it (somewhat?) rearranges the input data, just not at all correctly. (Again, it works correctly if the data is read directly or by redirection from a file). And to be clear as to what version I am running (the "a little more below"):
Code:
[D:\]"D:\Program Files\GnuWin32\bin\tac.exe" --version
tac (GNU coreutils) 5.3.0
Written by Jay Lepreau and David MacKenzie.
 
Copyright (C) 2005 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

So, I'd have to say it is broken, I have no clue as to where to report that fact. (Redirection works, pipe doesn't; very strange.)

- Dan

P. S. This is literally a 15 minute program (if you're not glacially slow) in C++, if there's interest I'd be OK with writing one (and including the source with it). The only questions are minor ones related to the size of the executable/slight performance issues/being totally "clean" about everything, none of the three being important to me. Specifically, in order to do this the entire file has to be read into memory, and the easiest and most quick solution would to read each line into a reverse-linked list of "gotten" areas of storage, one memory allocation per line read, and not bother to free the gotten storage at any point. Seems to me that would be a 15 minute task with no realistic downside whatsoever. And it would also have the clear advantage (not that there's a strong need given the contents of previous postings in this thread) of being able to read all of and reverse as a group any number input files supplied as the parameter(s).

- Dan
 
May 31, 2008
382
2
That's cute but it fails (pipe or file) with enough data (like the names of all dirs on my C drive) going to stdout; it will write 'em to a file but the file contains only \n separators (no \r\n).
Not here on both accounts. Try replacing your sed.exe with ssed.exe ...
 
May 20, 2008
3,515
4
Elkridge, MD, USA
Gentlemen, since we're working with TC here, why not use FFIND /V /R?

DJ
Sorry, /R does not work when there is no search string, as documented, and there is no "find everything" search string. See HELP FFIND.

The best purely internal method would be to read the pipe into an array (or execute the left side of the pipe in %@execarray[temp_array,left_side]), and ECHO each element in reverse order. An alias could do the job. However, SETDOS/X may be needed...

A plug-in could be written to do both steps, which could overcome the limitations of ECHO when characters of syntactic significance to TCC are part of the data.
 
Apr 13, 2010
318
7
62
The Hague
I'm already using this line in one of my scripts:

Code:
tree /b %RootDir | ffind /R /KM /V /E"^.*:.*$" >! %DirList

and i'm quit happy with the results.

- DJ
 
May 20, 2008
11,853
121
Syracuse, NY, USA
Gentlemen, since we're working with TC here, why not use FFIND /V /R?

DJ
That doesn't give a bottom-up list of directories. Of "/R" the help says
Code:
/R Only meaningful when used in conjunction with the /T "text" or /X options. Searches each file from the end backwards to the beginning. This option is useful if you want to display the last occurrence of the search string in each file instead of the first (the default). It may also speed up searches for information that is normally at the end of a file, such as a signature.
 
Jan 19, 2011
613
15
Norman, OK
What does "TAC --version" report?

I use Cygwin, so my GNU tools stay up to date.

Code:
$ tac --version
tac (GNU coreutils) 8.15
Packaged by Cygwin (8.15-1)
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
 
Written by Jay Lepreau and David MacKenzie.
 
May 20, 2008
11,853
121
Syracuse, NY, USA
That doesn't give a bottom-up list of directories. Of "/R" the help says
Code:
/R Only meaningful when used in conjunction with the /T "text" or /X options. Searches each file from the end backwards to the beginning. This option is useful if you want to display the last occurrence of the search string in each file instead of the first (the default). It may also speed up searches for information that is normally at the end of a file, such as a signature.
OK, I think I didn't fully understand. Indeed, this seems to give a bottom-up list of the directories on my system drive:
Code:
c:\> (global /i /q /h /n echo %_cwd) | ffind /e"\r\n" /r /v /m /k > v:\bottomup.txt
 
c:\> wc v:\bottomup.txt
  Lines  Words  Chars
  17524  22360 1591314

It doesn't seem bound by any limit on what can be output. There are a few directories in which I (even elevated) do not have the "list folder contents" permission, and that would make GLOBAL fail. For example (removing "/i")
Code:
c:\> (global /q /h /n echo %_cwd) | ffind /e"\r\n" /r /v /m /k > v:\bottomup.txt
 
TCC: (Sys) Access is denied.
"C:\ProgramData\Microsoft\WwanSvc\Profiles"

That directory itself (C:\ProgramData\Microsoft\WwanSvc\Profiles) does not appear in the report.
 

Charles Dye

Super Moderator
Staff member
May 20, 2008
4,584
97
Albuquerque, NM
prospero.unm.edu
Mine (textutils 2.1), when used in a pipe, silently does nothing (with no output) on Win7; on XP it crashes.

Would it make sense to have a plugin command to do this?

It dawns on me that one of my plugins already contains about 98% of the code that would be required. The problem with putting a filter command in a plugin is that pipes open a new shell. To use the filter, you'd either have to ensure that the plugin is loaded by the transient shell (e.g. by putting it in the PlugIns directory), or else avoid the pipe by using temporary files or an in-process pipe....
 
May 20, 2008
11,853
121
Syracuse, NY, USA
Would it make sense to have a plugin command to do this?

It dawns on me that one of my plugins already contains about 98% of the code that would be required. The problem with putting a filter command in a plugin is that pipes open a new shell. To use the filter, you'd either have to ensure that the plugin is loaded by the transient shell (e.g. by putting it in the PlugIns directory), or else avoid the pipe by using temporary files or an in-process pipe....
... to do exactly what ... TAC? I doubt it would be used much. This seems pretty good:
Code:
alias tac ffind /e"\r\n" /r /v /m /k
 
May 20, 2008
11,853
121
Syracuse, NY, USA
This seems pretty good:
Code:
alias tac ffind /e"\r\n" /r /v /m /k
But it's not perfect. Apparently it can reverse the output of DIR /S when it's about 7MB:
Code:
v:\> dir /s l:\ | head /n5
 
Volume in drive L is Apps7          Serial number is 5613:1d2f
Directory of  L:\*
 
2011-07-31  21:48        <DIR>    Acrobat 9
 
v:\> dir /s l:\ | tac | tail /n5
2011-07-31  21:48        <DIR>    Acrobat 9
 
Directory of  L:\*
Volume in drive L is Apps7          Serial number is 5613:1d2f
 
 
v:\> dir /s l:\ | wc
  Lines  Words  Chars
155469  609201 6607327

... but not when it's about 16MB:
Code:
v:\> dir /s c:\ | head /n5
 
Volume in drive C is unlabeled      Serial number is b813:1d21
Directory of  C:\*
 
2012-05-19  13:03        <DIR>    aemail
 
v:\> dir /s c:\ | tac | tail /n5
 
Directory of  C:\Windows\Microsoft.NET\Framework\v2.0.50727\ASP.NETWebAdminFiles\Security\Roles\*
 
            37,560 bytes in 4 files and 5 dirs    45,056 bytes allocated
  Total for:  C:\Windows\Microsoft.NET\Framework\v2.0.50727\ASP.NETWebAdminFiles\Security\Permissions\*
 
v:\> dir /s c:\ | wc
  Lines  Words  Chars
298684 1200866 16494682

I wonder what's going on.
 
May 20, 2008
3,515
4
Elkridge, MD, USA
Stefano:
Your example, because of the blank lines, confused me at first. Selecting the line terminator as the search string is CLEVER! And you discovered an undocumented feature of FFIND: /R works not only with /T and /X but also /E.
 
May 20, 2008
3,515
4
Elkridge, MD, USA
That doesn't give a bottom-up list of directories.
Reversing the order of dir/b/s is not the bottom-up list because the original order is not the bottom-down list.
Of "/R" the help says
Code:
/R Only meaningful when used in conjunction with the /T "text" or /X options. Searches each file from the end backwards to the beginning. This option is useful if you want to display the last occurrence of the search string in each file instead of the first (the default). It may also speed up searches for information that is normally at the end of a file, such as a signature.
No, it does not indicate that regular expression search result can be done in reverse order, but Stefano reported it is possible, and this is perhaps a lack of documentation - the /E option was added much later than the /R option...
 
May 31, 2008
382
2
But it's not perfect. Apparently it can reverse the output of DIR /S when it's about 7MB:

... but not when it's about 16MB:

I wonder what's going on.
Maybe determining the exact input size where your example stops yielding correctly reversed output can help finding an answer.
Did you try eliminating the pipe by redirecting DIR to a temp file?

Margin note: in Windows I almost never use pipes in favor of temp files because spawning processes is so time consuming (compared to *nix). This habit of mine pre-dates Windows 7, which possibly can spawn a pipe more quickly than previous Windows versions - but I never did check.
 
May 20, 2008
3,515
4
Elkridge, MD, USA
Would it make sense to have a plugin command to do this?

It dawns on me that one of my plugins already contains about 98% of the code that would be required. The problem with putting a filter command in a plugin is that pipes open a new shell. To use the filter, you'd either have to ensure that the plugin is loaded by the transient shell (e.g. by putting it in the PlugIns directory), or else avoid the pipe by using temporary files or an in-process pipe....
Charles, sorry for such a late (in the thread) response (I think your post missed the email distribution and I found it accidentally while scrolling to the top of the page).

Yes, it would be useful to have a REVERSE command. I wrote such a macro for my editor, but it is awful slow. The command ought to accept input from all the usual sources (specified files, pipe, < and <<) and output to a file or to STDOUT (redirectably to > or | ).
Do not worry about loading the plugin for the right side of a pipe - the issue applies to all plugin commands and functions one might use.
 

Charles Dye

Super Moderator
Staff member
May 20, 2008
4,584
97
Albuquerque, NM
prospero.unm.edu
Yes, it would be useful to have a REVERSE command. I wrote such a macro for my editor, but it is awful slow. The command ought to accept input from all the usual sources (specified files, pipe, < and <<) and output to a file or to STDOUT (redirectably to > or | ).
Do not worry about loading the plugin for the right side of a pipe - the issue applies to all plugin commands and functions one might use.

Steve, please pound on this and see if you can break it. If you can, let me know how!

unm.edu/~cdye/plugins/textutils.html#upend
 
Aug 2, 2011
258
4
Berlin, Germany
hello all,

there was much discussion about TAC and I think all of you are commandline-experts ;) .
Does anyone has a DISTINCT-tool that eliminates double/multiple records?
As Steve said: "The command ought to accept input from all the usual sources (specified files, pipe, < and <<) and output to a file or to STDOUT (redirectably to > or |) ".
Here is an example of what I mean in sql:
Code:
SQL> select gruppe from cd where gruppe like 'D%' order by gruppe;
 
Da,Da,Da
DAF
Davis, Miles
Davis, Miles
Davis, Miles
Davis, Miles
Davis, Miles
Davis, Miles & Coltrane, John
De La Soul
Dean Martin
Dedilhadas de Fernambuco, Cordas
Defunkt
Defunkt
Defunkt
Defunkt
Defunkt
Defunkt
DeWinkel / Hattler
Didley, Bo
DiMeaola, Al
Dixie Dregs
DJ Maxi Jazz and the Soul Food Cafe
D'Rivera, Paquito feat. Arturo Sandoval
Dulfer, Candy
 
SQL> select DISTINCT gruppe from cd where gruppe like 'D%' order by gruppe;
 
Da,Da,Da
DAF
Davis, Miles
Davis, Miles & Coltrane, John
De La Soul
Dean Martin
Dedilhadas de Fernambuco, Cordas
Defunkt
DeWinkel / Hattler
Didley, Bo
DiMeaola, Al
Dixie Dregs
DJ Maxi Jazz and the Soul Food Cafe
D'Rivera, Paquito feat. Arturo Sandoval
Dulfer, Candy

edit:
I just installed v.14 and found out that this could be done with "TPIPE /DUP",
but if someone has a "distinct.exe" (or similar), please let me know.
 
Aug 2, 2011
258
4
Berlin, Germany
just to share it with you: here's another possibility to get directories in reverse order:
logparser is a kinda "Swiss army knife" whitch can work with many input sources:
Code:
>logparser -h
 
Microsoft (R) Log Parser Version 2.2.10
Copyright (C) 2004 Microsoft Corporation. All rights reserved.
 
Usage:  LogParser [-i:<input_format>] [-o:<output_format>] <SQL query> |
                  file:<query_filename>[?param1=value1+...]
                  [<input_format_options>] [<output_format_options>]
                  [-q[:ON|OFF]] [-e:<max_errors>] [-iw[:ON|OFF]]
                  [-stats[:ON|OFF]] [-saveDefaults] [-queryInfo]
 
        LogParser -c -i:<input_format> -o:<output_format> <from_entity>
                  <into_entity> [<where_clause>] [<input_format_options>]
                  [<output_format_options>] [-multiSite[:ON|OFF]]
                  [-q[:ON|OFF]] [-e:<max_errors>] [-iw[:ON|OFF]]
                  [-stats[:ON|OFF]] [-queryInfo]
 
-i:<input_format>  :  one of IISW3C, NCSA, IIS, IISODBC, BIN, IISMSID,
                        HTTPERR, URLSCAN, CSV, TSV, W3C, XML, EVT, ETW,
                        NETMON, REG, ADS, TEXTLINE, TEXTWORD, FS, COM (if
                        omitted, will guess from the FROM clause)
-o:<output_format>  :  one of CSV, TSV, XML, DATAGRID, CHART, SYSLOG,
                        NEUROVIEW, NAT, W3C, IIS, SQL, TPL, NULL (if omitted,
                        will guess from the INTO clause)
-q[:ON|OFF]        :  quiet mode; default is OFF
-e:<max_errors>    :  max # of parse errors before aborting; default is -1 (ignore all)
-iw[:ON|OFF]        :  ignore warnings; default is OFF
-stats[:ON|OFF]    :  display statistics after executing query; default is ON
-c                  :  use built-in conversion query
-multiSite[:ON|OFF] :  send BIN conversion output to multiple files
                        depending on the SiteID value; default is OFF
-saveDefaults      :  save specified options as default values
-restoreDefaults    :  restore factory defaults
-queryInfo          :  display query processing information (does not
                        execute the query)
 
 
Examples:
LogParser "SELECT date, REVERSEDNS(c-ip) AS Client, COUNT(*) FROM file.log
            WHERE sc-status<>200 GROUP BY date, Client" -e:10
LogParser file:myQuery.sql?myInput=C:\temp\ex*.log+myOutput=results.csv
LogParser -c -i:BIN -o:W3C file1.log file2.log "ComputerName IS NOT NULL"
 
Help:
-h GRAMMAR                  : SQL Language Grammar
-h FUNCTIONS [ <function> ] : Functions Syntax
-h EXAMPLES                : Example queries and commands
-h -i:<input_format>        : Help on <input_format>
-h -o:<output_format>      : Help on <output_format>
-h -c                      : Conversion help
here is my example:
Code:
>logparser -i:fs "select distinct extract_path(path) as p from d:\temp\recurse\* order by p desc" -q:on
d:\temp\recurse\dir3\a\b\c
d:\temp\recurse\dir3\a\b
d:\temp\recurse\dir3\a
d:\temp\recurse\dir3
d:\temp\recurse\dir2\a\b\c
d:\temp\recurse\dir2\a\b
d:\temp\recurse\dir2\a
d:\temp\recurse\dir2
d:\temp\recurse\dir1\a\b\c
d:\temp\recurse\dir1\a\b
d:\temp\recurse\dir1\a
d:\temp\recurse\dir1
d:\temp\recurse
 
Jan 19, 2011
613
15
Norman, OK
logparser is a kinda "Swiss army knife" whitch can work with many input sources:
If we're talking "Swiss army knife", take a look at SFK (Swiss File Knife) from http://stahlworks.com/dev/swiss-file-knife.html
It does a bottom up traversal of folders (not reverse sorted) with a simple command. It also does a whole bunch of stuff on the lines of tpipe.
Code:
c:\temp> sfk list -justdirs c:\temp\recurse
c:\temp\recurse\dir1\a\b\c
c:\temp\recurse\dir1\a\b
c:\temp\recurse\dir1\a
c:\temp\recurse\dir1
c:\temp\recurse\dir2\a\b\c
c:\temp\recurse\dir2\a\b
c:\temp\recurse\dir2\a
c:\temp\recurse\dir2
c:\temp\recurse\dir3\a\b\c
c:\temp\recurse\dir3\a\b
c:\temp\recurse\dir3\a
c:\temp\recurse\dir3
c:\temp\recurse
 
Aug 2, 2011
258
4
Berlin, Germany
Thank you, John. I will have a look at it.
( Now I remember that SFK was mentioned before, but I've forgot it )

@ALL
Finally I'm very happy of the friendly and helpful form of use here in this forum.

Keep it up!
:)
 
Nov 18, 2009
22
0
hello all,

there was much discussion about TAC and I think all of you are commandline-experts ;) .
Does anyone has a DISTINCT-tool that eliminates double/multiple records?
As Steve said: "The command ought to accept input from all the usual sources (specified files, pipe, < and <<) and output to a file or to STDOUT (redirectably to > or |) ".

...

edit:
I just installed v.14 and found out that this could be done with "TPIPE /DUP",
but if someone has a "distinct.exe" (or similar), please let me know.

The GNU (hence, Unix) equivalent to what you want is uniq, but it expects the input to be sorted.
 
May 20, 2008
3,515
4
Elkridge, MD, USA
The output of the POSIX (GNU) utility "sort --unique" is DISTINCT; however, IIRC the SQL keyword DISTINCT is independent of the ORDER clause, so sorting may be in a different order than required for uniqueness.
 
Nov 18, 2009
22
0
Sounds like a fun Perl one-liner - though a potential memory hog - implement a uniq that doesn't require a preceding sort...

Looks like sort implements a merge sort of temp spill files if it is going to exceed buffer size - that might take more than one line of Perl :)