How to? Work around an unexpected and very inconvenient behavior of TCC...

[mathewsdw]

Here's the deal. As I have mentioned in this forum in the past, my C: drive was basically at 100% capacity and this was starting to become more than a small annoyance. So I set out to find out what I could "get rid of" on the C: drive (either by moving it to my 2TB external hard drive or deleting it altogether) based on both size and last-accessed date to find the large files that I haven't used in any way in quite a while. So, to do this, I issued the following command:

PDir C:\ /Ta /A-D /Nj /S /Ou /(zc dy/m/d th:m:s fpn) |& Tee CON: >!"Z:\A Text File of Some Kind.txt"

The good news is that this worked perfectly. The bad news is that took more than 12 hours to run (182,117 files when it finished and I don't know exactly how long it took because I went to bed and it was done in the morning when I got up. Maybe this gives you some idea as to why I use a RAM disk on this laptop. And remember 40MB disk drives???? ;)).

And the really bad news is that every other TCC session ran so slowly (I typed an "a" in one session and timed it: it took 7 minutes and 43 seconds to show up on the screen; I timed it with the stop watch in my cell phone) as to be basically unusable, although no other apps (browsers, Word and Notepad documents, an Excel spreadsheet, Microsoft Outlook, whatever) seemed to be even slightly affected. (Since the C: drive is a real, physical, hard disk I would have expected it to be quiescent for regular periods of time waiting for the I/O's to complete; although it is an NTFS drive where small directories (and files) are contained entirely in the MFT.) So the questions are: Why is this happening and what, if anything, can I do about it? (Since, as I have oft mentioned before in these forums, because of my disabilities, particularly issues with my memory and eyesight, this laptop and TCC (since I really can't use the Windows GUI because I really can't see it) are essential parts of my life; and I was effectively unable to do anything useful on this laptop for more than 12 hours; not just a "minor" inconvenience.) I will also add that I used Task Manager to "play around" with the priorities of the various TCC sessions; i.e., raising the priorities of the ones not running the "PDir" to "Real Time"(!!!!) and lowering the priority of the one running the "PDir" to "Low", all with absolutely no effect whatsoever.

- Dan

P. S. BTW, in the "pseudo administrative" account here Windows 7 I can not use the task manager, when run from the task bar, to either cancel or change the priority of a running task ("Access is denied"); rather I have to start it "elevated" from a TCC session ("Start /Elevated TaskMGR") or use Vince's technique (which I have now done; thank you, Vince!) to start it.) Is this true for everyone else, and how do you "work around" it?

 

[rconn]

I corrected your TEE command so it wasn't nonsensical (you were sending two copies of every file to stdout, and then redirecting both of them):

PDir C:\ /Ta /A-D /Nj /S /Ou /(zc dy/m/d th:m:s fpn) |& Tee "Z:\A Text File of Some Kind.txt"

It took 5 minutes 37 seconds on my system (218,016 matching files, 108,583 dirs) and had no perceptible impact on the other TCC tab windows.

 

[Steve Fabian]

Did you move your Windows swap file (in WinXP it is pagefile.sys) to another drive yet? It needs to be on a real drive, not a virtual one, and should be on a drive which can allocate it unfragmented, and the drive have should be rarely accessed by your programs, so as to optimize pagefile accesses. When you switch between "processes" the pagefile stores the dynamic pages of the process that is suspended and that's where the prevously saved process that is swapped in has its dynamic pages reloaded from. But the static pages (i.e. the code and invariant data) are reloaded from their original disk locations; if they are fragmented, it will take a long time to swap. BTW, my own C: drive is had also been badly fragmented, slowing things down, even though my swap file is on a different drive. Defragmenting and compressing the drive resulted in observably better performance.

 

[Charles G]

A good tool I use to remove safely stuff from the computer is CCleaner from www.piriform.com. Haven't read the full thread yet but this will help with removing stuff like Java cache, temporary internet files, browser cookies, and other stuff. You can select or unselect everything. Then to do select more things to remove - I download the following file to the program folder where CCleaner was installed:

http://ini.ccleaner.de/Winapp2.ini

CCleaner also has a very good registry leaneer - thcan selectively backup the registry cleaning....

Here's the deal. As I have mentioned in this forum in the past, my C: drive was basically at 100% capacity and this was starting to become more than a small annoyance. So I set out to find out what I could "get rid of" on the C: drive (either by moving it to my 2TB external hard drive or deleting it altogether) based on both size and last-accessed date to find the large files that I haven't used in any way in quite a while. So, to do this, I issued the following command:

PDir C:\ /Ta /A-D /Nj /S /Ou /(zc dy/m/d th:m:s fpn) |& Tee CON: >!"Z:\A Text File of Some Kind.txt"

...snip...

ow done; thank you, Vince!) to start it.) Is this true for everyone else, and how do you "work around" it?

 

[mathewsdw]

Steve, thank you for your thoughts, but my swap file is the C: drive which is a partition on a real, physical, hard disk; and there is a "defragment" task scheduled in the task scheduler (ScheduledDefrag 3/14/2012 1:00:00 AM Ready) that is supposed to run every Wednesday morning at 1:00 AM as you can see (although I don't really know if it is being run other than the fact that I am virtually always up and using this computer at that time on Wednesdays (as well as pretty much every other day because I am, and have always been, a real "night owl" for whatever reason(s)) so that it is virtually always turned on and not hibernated at that time).

And Rex, while your approach is certainly a bit simpler, I don't see how what I had done was "sending two copies of every file to stdout, and then redirecting both of them", particularly since there was no evidence of that either in the output to the console (which I monitored on a relatively continuos basis) or in the (ultimately sorted on a sort key starting in column 1) final output file. I'm not trying to argue here; just stating the facts as I saw them (and I would be glad to zip up the final sorted output file so that you can see this for yourself). More than that, what I was doing was just a rather slight modification of something I have done many times before: command |& Tee CON: | List. And the amount of time it took do the same on my D: drive (also a partition on a real, physical, hard disk) was 1 hour and 36 minutes and 25.71 seconds and this time around the other running TCC sessions were rather slow but still usable (and, frankly speaking, I understand none of the immediately previous); I formally timed it by putting the "PDir" in a batch file (exactly the same "PDir" as was in my posting (in fact, I copied it directly from my posting into the .btm file to ensure that I didn't make any mistakes) other than the fact that I changed "C:" to "D:" and put an actual, real, file name in place of "Z:\A Text File of Some Kind.txt") and placed a "Timer 1 On" before it and "Timer 1 Off" after it.

A mystery whose solution is no longer important, I think.

- Dan

 

[rconn]

TEE always sends a copy of the program output to CON:. The arguments to TEE are supposed to be filenames -- see the help. So you were asking TEE to send a copy to standard output, another to CON:, and then redirect the TEE output to a file, which TEE would have done anyway (without the redirection) if you had specified a filename.

 

[mathewsdw]

Rex, I'm sorry to be stupid here, but you are really confusing me. To quote the help, "TEE gets its input from standard input (usually the piped output of another command or program), and sends out two copies: one to standard output (hence the pipe to the List command), the other to the file(s) that you specify (the console in my case).". So there are two points here: 1. "Standard output" is not the console ("con:" the last time I checked); it is something that can be "piped" to still another command. And 2. I probably do the following literally a dozen (or more!) times every day: "command |& Tee CON: | List", and I get exactly the results I am looking for - seeing the output of "command" as it is being produced (primarily so I can catch errors that I may made when entering a long-running command early on; and "errors" are not an "every once in a while" occurrence for me because of my blindness and bad memory and even other factors that I have never identified) and then have a scrollable (and saveable if I want to - the "Ins[ert]" key, of course) list to examine the output in detail. And I have absolutely never had two copies of the data appear, anywhere. So where/how are we not understanding each other?

- Dan

 

[samintz]

Dan,

The purpose of TEE is to save the output to a disk file while also seeing it on the console - one input, two outputs. In your use of TEE, you send the output of command to the the file named CON: and the second output is piped into LIST. If you replace LIST with VIEW, you will notice the same output on your console window and in the V window. Since you don't seem to care about the output to CON:, is there a reason you couldn't pipe the output of your command(s) directly to LIST?

E.g.

command |& List

-Scott

 

[Charles Dye]

I think he want to see the output as it's generated; LIST won't display anything until the pipe is closed -- i.e., until the command completes.

 

[rconn]

I think he want to see the output as it's generated; LIST won't display anything until the pipe is closed -- i.e., until the command completes.

Not true; LIST is running in a separate process and will display output as it is generated (unless you're using in-process pipes, or if your parent process is buffering output).

 

[mathewsdw]

Scott, I have to really wonder why everyone is missing the point(s?) I'm trying to make here. #1, I do this for long running commands (it is now a proven fact because I ran a disk-speed benchmark program I downloaded from the web the other day (although I don't have the numbers in front of me at the moment) that the hard disk in this laptop is very slow (why I use a RAM disk) and the external drive is even slower (possibly by an order of magnitude; but I haven't bothered to take the time to test it for a number of reasons; and that's probably because it's connected via a USB port (USB 1.0???) and there is (by default in Windows 7, at least) no write caching for an external drive) and sending the output directly to the console while the command is executing allows me to catch stupid mistakes I may have (probably have?) made early on rather than finding out that I screwed up 20 minutes later, #2. I can then examine the output of the command in detail using the "List" command after it has completed; doing finds, scrolling up and down and left and right, viewing it in hex, and whatever else the "List" command is capable of doing (as far as I am concerned, the "List" command is probably one of TCC's best features; it probably beats cmd.exe's "More" command or using a text editor to view the output of commands by two orders of magnitude), and #3, "View" is basically a GUI app and and I don't do GUI apps unless I literally have no choice because of my blindness - I literally despise GUI apps and I resent them whenever I am forced to use them (this is not a "bias" or "opinion" of some kind, it is due to my vision issues); whereas the "List" command meets my needs perfectly. (I don't think I've ever specifically mentioned this before because I didn't consider it to be all that relevant - I have Take Command set to use a 20pt. Lucida Console font (I just checked); I can't reliably read a console at anything less than the size.)

So the bottom line is that "command |& Tee CON: | List" meets my needs perfectly; end of story.

- Dan

 

[rconn]

Rex, I'm sorry to be stupid here, but you are really confusing me. To quote the help, "TEE gets its input from standard input (usually the piped output of another command or program), and sends out two copies: one to standard output (hence the pipe to the List command), the other to the file(s) that you specify (the console in my case).". So there are two points here: 1. "Standard output" is not the console ("con:" the last time I checked); it is something that can be "piped" to still another command.

Well, yes, it is. Writing to CON: is exactly the same as writing to stdout. You've only got one standard output -- whether you call it CON: or just let TEE write it to the default stdout.

And 2. I probably do the following literally a dozen (or more!) times every day: "command |& Tee CON: | List", and I get exactly the results I am looking for

You will get the same result with a "command |& list", without the (significant) extra overhead of passing everything through TEE.

And I have absolutely never had two copies of the data appear, anywhere.

Yes, you have -- but you've redirected one of them into LIST. TEE is not intended to be used in the middle of a pipe; TEE is intended as a target of a pipe or other redirection operation. You're using (abusing?) TEE in a completely undocumented way; the fact that it *appears* to (mostly) be working for you is more coincidence than design.

 

[mathewsdw]

Rex, boy, I really don't understand what is going on here. Writing to "standard out" is only the same as writing to CON: when standard out is not piped or redirected to a file (i.e. "command |& CON:" is no different than just "command"); and "command |& List" absolutely does not display the output of command as it is being produced, because of buffering I would guess it can take a fairly long time before "List" shows literally anything. And I have been doing this for probably years with no problems whatsoever!!!!!!!!!!



And, BTW, I do not get the "same" results as just "command |& List" for long-running commands!

 

[rconn]

You've invented an imaginary syntax for TEE, and then become annoyed that your imaginary syntax doesn't behave the way you imagine it should! :)

The documented way of doing this in TEE would be:

command |& tee outputfile & list outputfile

And LIST absolutely does display the output of command as it's generated; however, LIST will buffer some of the piped input before it is displayed. This is necessary so that you can scroll back & forth in LIST, as otherwise piped input would be strictly one-way (no scrolling back). If your command doesn't display much output, it may appear that LIST is waiting for "command" to finish, but it's actually just waiting for "command" to write enough to be worth displaying.

 

[rconn]

So the bottom line is that "command |& Tee CON: | List" meets my needs perfectly; end of story.

??? Then why begin this thread complaining that it didn't work?

 

[JohnQSmith]

TEE is not intended to be used in the middle of a pipe; TEE is intended as a target of a pipe or other redirection operation.

I don't understand this statement.

The example from the TEE docs shows exactly that.

ffind /t"Take Command" doc | tee tc.dat | sort > tcs.dat

 

[mathewsdw]

Rex, I pretty much understood what you say about buffering and the "List" command from the outset (I was once a programmer, too). What you don't seem understand is sometimes it can be 10, 15, even 20 or more minutes before "List" gets enough data to actually display anything, and I'd rather not wait for long periods of time before discovering that I've screwed up yet again. And the sequence "command |& Tee CON: | List" shows the output from command essentially immediately as it is being produced, so if I've screwed up (always a pretty likely possibility) I see that almost immediately. I'm going to say something here that's only a little bit off-topic: I have four completely distinct and separate disabilities that are completely independent of each other (specifically, I can't read because of my vision issues, I can't write because I no longer have the coordination to do that, and (by far the worst) I can't remember; a distinct variation of the 3 R's; and one that's kind of personal and most likely due to some combination of the previous three and not at all relevant here; and one more that also is really not something I would like to discuss in a forum of this nature because it is also somewhat personal (although all of my close friends are totally aware of it because it almost always completely explains some aspect of my otherwise unexpected and irrational (mis-)behaviors); and the sum total of all five of these is that I am, as I'm sure I've said before, quite incompetent (and that word is in no sense at all an exaggeration). I have a whole slew of "strategies" to compensate for these things if I actually remember to employ them, which is more likely for those things that happen or would happen most often if I didn't employ one of my compensating strategies. (And my general incompetence is probably not all that evident in the postings I make on these forums; but that is because I am very careful and entering one of these postings can take a very long time before I get it right. I keep an instance of MS Word running at all times to check and correct my spelling, and and an instance of Notepad to copy text into and cut right out of because text in Notepad is straight text with no formatting information of any kind.)

So, going through you comments point-by-point, you say "You've invented an imaginary syntax for TEE, and then become annoyed that your imaginary syntax doesn't behave the way you imagine it should!". Boy, there are many issues here: Given that, as far as I know, from the perspective of an internal command or an external program there is absolutely no difference between a file and a (pseudo?) device (i.e., "CON:", whatever is the difference between "Tee CON:" and "Tee filename"? And given that "CON:" is just a pseudo file name, why would Tee even care???? You say "TEE is intended as a target of a pipe or other redirection operation. You're using (abusing?) TEE in a completely undocumented way; the fact that it *appears* to (mostly) be working for you is more coincidence than design". From the help file: "TEE is normally used to "split" the output of a program so that you can see it on the display". Well, one might assume that "display" means "directly to CON:", but my expectation was and (strong!!!!) experience is that "display" means "standard output", and CON: is where "standard output" normally goes to, and "standard output" is, of course, redirectable and pipeable (Thank God!!! I might add). You say "The arguments to TEE are supposed to be filenames"; please tell me where it uses the plural in the docs. You ask "?? Then why begin this thread complaining that it didn't work?" and I absolutely did not start this thread making a complaint of that nature; I do not have and have never had a complaint of that nature; I started this thread complaining that a PDir command ultimately taking at least 12 hours and all other TCC instances were almost completely unresponsive and therefore totallyh unusable while said PDir command was executing; go back up to the top of this thread and see for yourself!

More: You say "The documented way of doing this in TEE would be: command |& tee outputfile & list outputfile". A: That "this" is not what I am trying to do. Again, I want to see the output of command as it is being produced and never have it go to an explicit file unless I hit the "Ins[ert]" key in the "List" command. B: Your example will "List" nothing (as far as I can tell - more in a moment) because the "List" command will not be executed at all until after the "TEE" command has completed. And C. (The "more in a moment") Your example produced a GPF:

TCC  13.04.52
Module=C:\Windows\system32\KERNELBASE.dll
Address=74BDD36F
Exception=E06D7363
EAX=0168ED98  EBX=10090D10  ECX=00000003  EDX=00000000
ESI=0168F0D0  EDI=016DF69C  EBP=0168EDE8  ESP=0168ED98
CS=0000001B  DS=00000023  ES=00000023  SS=00000023
Flags=00000212
 
Stack:
1 : KERNELBASE.dll 00000001:0000c36f
2 : TakeCmd.dll 00000001:000c4cc9
3 : TakeCmd.dll 00000001:0008fd65

I don't know exactly what command(s) I had entered to cause the crash and there is obviously no "history" file to find out; I believe it was probably a "Dir /S" with some other parameters of some kind ("*.txt"? "/K /M"? "/F"?' I don't really remember.)

And this is Windows 7 (Professional) [Version 6.1.7601].

And note Mr. Smith's response above; as he said, he copied that directly from the docs.

And on a positive note, Rex, I literally could not survive on my own (I am single and live alone and that is by design because I knew many years ago that these things were going to eventually happen to me and I made the (explicit!) decision that I was not going to burden a "loved one" with having to take care of me when they did happen (nor was I going to have biological children to inherit my genetic defects); that is not something you do to people you (at least supposedly!) love) without this computer and, of course, Take Command and particularly TCC. I'll be honest here, this whole discussion has greatly irritated me (particularly your use of the word "nonsensical; you might not have expected anyone to use it the way I am, and have been for many years, using it; but that doesn't at all make it "nonsensical" in my circumstances); but I would guess that it has also greatly irritated you. But I am absolutely standing my ground with no question whatsoever.

- Dan

 

[mathewsdw]

Update: Rex, you said (and again I quote): "Writing to CON: is exactly the same as writing to stdout". To be blunt (and I somewhat apologize for being this blunt; but I am also rather angry), this statement is so silly as to be almost unbelievable. Yes, writing to stdout is the same as writing to CON: if there is no pipe or redirection; and not at all if there is a pipe or redirection. And there's no single command syntax that I am aware of that will let you write both to CON: and redirect to either a pipe or a file at the same time (using TEE after a pipe will allow you to do that.) I am going to be even more blunt here; before I became too disabled to function effectively I was the best programmer I ever met and nobody who knew of my work ever disputed that in any way. I was about an order of magnitude faster than other programmers and there are only two instances in my entire career (I remember them specifically because there were only two of them) when anybody ever found a bug in my code after I had delivered it. And I took a Microsoft C++ competency exam that they used to administer years ago (before they basically gave up on C++ and introduced C#) and it was a two-hour test that I finished in about 45 minutes (it was proctored by a Microsoft employee) and I tied for the second-highest score ever up to that point in time and I know this because the Microsoft employee who had proctored the exam called me personally from Microsoft headquarters in Washington state the day after the exam to tell me this and that all of the questions I had missed were easy in his opinion and if I had spent another ten to fifteen minutes going over my answers before handing in the test I would have had the first perfect score ever up to that point in time. And you, Rex, are the only person I'm even aware of who is almost certainly as good as and quite possibly better than I was. Again to be blunt, yes I do resent your implication(s) that I don't know what I am doing. I may sometimes be stupid; but that stupidity is purely related to my critically bad memory and nothing else.

- Dan

 

[Charles Dye]

Update: Rex, you said (and again I quote): "Writing to CON: is exactly the same as writing to stdout". To be blunt (and I somewhat apologize for being this blunt; but I am also rather angry), this statement is so silly as to be almost unbelievable. Yes, writing to stdout is the same as writing to CON: if there is no pipe or redirection; and not at all if there is a pipe or redirection.

Unfortunately (IMHO), TCC does interpret "CON:" as a handle to stdout, or to stdin -- redirected or not -- in several places. I seem to recall one of your own batch files opening CON: to read text from a pipe.

I personally would prefer something different, e.g. a single minus sign as in Unix; but this design decision seems to be implemented in so many different places throughout the code, and goes so far back (clear back to 4DOS, I think), that changing it would doubtless be an utter fiasco....

 

[mathewsdw]

Charles, two points: You said "TCC does interpret "CON:" as a handle to stdout, or to stdin -- redirected or not -- in several places." If that is true (and I have no reason to doubt you) then it should be documented because it effectively makes "CON:" useless in whatever situations that it is true.

And, again, the syntax "command |& Tee CON: | List" absolutely does work for me (I just tried it yet again - in two completely different circumstances - before "bolding" the word "does", and it worked as I would have hoped in both cases). That being the case, I had never asked a question about it in the first place. And given my oft mentioned level of incompetency due to my bad memory and blindness, it is something that I really need to work.

 

[Charles Dye]

Charles, two points: You said "TCC does interpret "CON:" as a handle to stdout, or to stdin -- redirected or not -- in several places." If that is true (and I have no reason to doubt you) then it should be documented because it effectively makes "CON:" useless in whatever situations that it is true.

Searching the help file for CON, I find that it is documented in some places: FOR, @LINE, @FILEOPEN, @SELECT. Maybe SENDMAIL and SENDHTML -- CON: is mentioned in both of those, but I'm not really clear on whether it means stdin (possibly redirected) or actual console input.

And, again, the syntax "command |& Tee CON: | List" absolutely does work for me (I just tried it yet again - in two completely different circumstances - before "bolding" the word "does", and it worked as I would have hoped in both cases). That being the case, I had never asked a question about it in the first place. And given my oft mentioned level of incompetency due to my bad memory and blindness, it is something that I really need to work.

<shrug> I don't have a problem with that. Rex has coded CON: to mean stdin, or maybe stdout, in several places; but obviously TEE is not among them. I don't understand why he reacted as he did to your usage.

If there's anyplace that you really need console output or console input, and CON: has been recoded to mean stdin or stdout, I imagine you could use CONOUT$ or CONIN$ instead.

 

[JohnQSmith]

Searching the help file for CON, I find that it is documented in some places: FOR, @LINE, @FILEOPEN, @SELECT. Maybe SENDMAIL and SENDHTML -- CON: is mentioned in both of those, but I'm not really clear on whether it means stdin (possibly redirected) or actual console input.

Well there ya go! Problem solved. No need for further argument. Just use CON instead of CON:.

It works as input and output even in cmd.com.

 

[mathewsdw]

Actually, John, it's cmd.exe. ;)

 

[JohnQSmith]

Actually, John, it's cmd.exe. ;)

I'm still getting used to that after so many years of command.com. *sigh*

edit: actually, thinking back, I guess I've spent approximately the same number of years with both. Started with command.com in '89 then switched to cmd.exe in about 2001ish.

 

[Frank]

In my early years I started with cp/m (on an Apple ][+ ?), quickly followed by ms-dos 3.x on an 80286.
And since then I allways use the colon as in NUL:, CON: and so on.
Is this nonsense by now?

 

[Charles Dye]

In my early years I started with cp/m (on an Apple ][+ ?), quickly followed by ms-dos 3.x on an 80286. And since then I allways use the colon as in NUL:, CON: and so on. Is this nonsense by now?

I always use the colon too, but I don't think it matters. As far as I know, TCC treats it the same either way.

 

[Frank]

Puh! So it's not a mannerism of me :) (or should it be "of mine"? I'm not certain in such subtleties)