Welcome!

By registering with us, you'll be able to discuss, share and private message with other members of our community.

SignUp Now!

Slow Pipes

Jay Sage

Jay Sage

Jun
1,092
48
I noticed that my BTM scripts that use pipes run very slowly. When I changed '|' to '|!' (i.e., to use in-process pipes), they ran very fast. I tried various things and eventually realized that the time probably arises from loading all the standard plugins from the "plugins" directory.

How do others handle this? The only thing I have thought of is to change the name of the plugins directory so that the plugins do not load automatically and add code to TCSTART -- after the exit test for pipes and transients -- to load the plugins I want.
 
What are the situations in which you can tell a pipe is slow?
 
I have a number of batch scripts that display file directories with long descriptions. The output from PDIR, ending with the file or directory description that may extend beyond the width of the screen, is piped into a program that wraps each line with a hanging indent to make a neat display.

I'm not sure when I noticed that the response was slower than I remember. It is perhaps worse with the latest build (34), but it probably started earlier. Eventually, I experimented with changing the code from something like

pdir ... | hanging_indent.exe​

to

pdir ... |! hanging_indent.exe​

and I noticed a huge speed-up. Here is some output from a test that was redirecting a typed text file into XSORT. I ran it many times and got more or less the same results each time.

->test.btm​
running with standard pipe​
Timer 1 on: 17:25:03​
Timer 1 off: 17:25:04 Elapsed: 0:00:00.617​
running with in-process pipe​
Timer 1 on: 17:25:04​
Timer 1 off: 17:25:04 Elapsed: 0:00:00.107​

My TCSTART script has code at the beginning to exit promptly if TCC is running in a pipe. But plugins are being loaded. Just now, I put the following at the very beginning: if %_pipe NE 0 quit . It made no difference.

The same behavior occurs in ConPTY and non-ConPTY sessions. I went back to TCMD 35. I could not use XSORT and used TPIPE instead. The time difference between TCMD35 and TCMD36 was not that different (version 35 was slower for both kinds of pipes, but the in-process pipe was dramatically faster.

Here, by the way, is what the output looks like with the piping into the hanging indent program. At first, I used TCC code to do the wrapping, but it could be pretty slow. I figured that compiled code would be much faster. I asked AI to generate Pascal code for me and was amazed that it worked essentially perfectly on the first try.

1776981542147.webp
 
DOS-style pipes are one command redirecting output, followed by another command redirecting input. A real pipe has two processes running simultaneously sending data back & forth. If you are piping to external processes, the real pipes can be very fast. If you are piping to another TCC process to run a TCC command, you've got all the TCC startup overhead to deal with, and it may well be faster to use the DOS pipes and remain in the same session.
 
Well,, I showed the command above. I'm running PDIR and piping the output to an EXE file. Is there something wrong with that command line? I didn't show all the details of the PDIR command, but its just a lot of options to format the data as I wish. The EXE command has the full path for invoking it and two parameters (screen width and hanging indent column).
 
rconn said:
If you are piping to an external app, then you aren't getting a second TCC session involved and your TCSTART & plugins setup is irrelevant.
Click to expand...

That certainly makes sense, but then what is causing the real pipe to be so much slower than the in-process pipe? Could it be that there is something about that program. I guess I should look for a different program to experiment with.

I just pulled out the one line of code with the pipe and ran it from the command line. In that case, the run time is not much different between the two kinds of pipe. When run from the BTM script, the real pipe is much slower. One other peculiar thing: the output appears on the screen rather quickly, but then there is a long delay before a command prompt appears. That suggests that the pipe is doing its work and showing the output, but something is interfering with the completion of the batch file and the presentation of a new command prompt.

With the in-process pipe, there is a slight initial delay. Then the output appears, followed immediately by a new command prompt.

Does that description give anyone an idea?
 
You could do a FOLDERMONITOR in %TEMP to see if hanging_indent.exe itself is using a temp file.
 
The code for the app is very simple and involves no files. The fully commented source code is only 140 lines.

I am beginning to think that the problem is related to the termination of the pipe. As I reported, the output appears on the screen and only then is there a long delay before a new prompt appears. It's as if the pipe is waiting for another line to process and finally gives up and terminates the process. The in-process pipe doesn't have the problem because it knows when it gets to the end of the temporary file.

I am also beginning to suspect a change introduced by Windows. I did not have this problem earlier, and now it happens in TCMD 35 as well. In any case, it does not seem to be a problem related to Take Command or TCC.
 
rconn said:
Is your process doing line buffering for the pipe input? You might want to try turning that off.
Click to expand...

Sorry, but I don't really know what that means, and I certainly do not know how to control it. The command line in a TCC tab in TCMD looks like this:

pdir [options and file spec] | path\wrap.exe​

The exe is compiled from Pascal source code that reads from stdin and writes to stout. It does read the input line by line [ReadLn(Line);] and then processes that string variable into one or more lines that are written out. That code is wrapped in while not Eof(Input) do .
 
Since using an in-process pipe seems to work reasonably well, I'll just do that for now. I might experiment later with modifying the Pascal source to read byte by byte until it sees a line-end instead of reading a whole line at once.

I just experimented with sending PDIR output (pdir /ne /d /o:gd /(-46.46fn 10d:y-m-d i) *) into the Windows sort command, and here's the timing report:

===============================​
Run times:​
Real pipe: 0:00:01.752​
In-process pipe: 0:00:00.102​
===============================​

I imagine that the sort program is designed to be used in a pipe. Nevertheless, the in-process pipe is enormously faster. Here is the batch file that is producing this output.

Code: 
set testcmd=pdir /ne /d /o:gd /(-46.46fn 10d:y-m-d  i) *
set pipecmd=sort

:: Run with a standard pipe
timer /1 on
%testcmd | %pipecmd
set runtime1=%@timer[1]
timer /1 off

:: Run with an in-process pipe
timer /1 on
%testcmd |! %pipecmd
set runtime2=%@timer[1]
timer /1 off

echo.
echo ===============================
echo Run times:
echo `  Real pipe:       `%runtime1
echo `  In-process pipe: `%runtime2
echo ===============================
 
Now I changed pipecmd to tpipe /simple=5. The times were closer, but an in-process pipe was still faster.

===============================
Run times:
Real pipe: 0:00:00.816
In-process pipe: 0:00:00.477
===============================
 
You're not really running a fair comparison - you are using commands (sort, tpipe /simple) that have to read the entire pipe contents before they do anything.

If you were piping input to an app that produced intermediate / continual output as it read the pipe input, you would not be able to use in-process pipes.
 
Curious ... can you send one program's output to another program's input with a named pipe? Would it be any better?
 
rconn said:
If you were piping input to an app that produced intermediate / continual output as it read the pipe input, you would not be able to use in-process pipes.
Click to expand...

Perhaps, but the most frequent examples of piping that I've seen are things like my examples. For my education, could you give an example where a real pipe would be necessary.

For fun, I wrote a Pascal program that reads a character from stdin and writes two copies out to stdout. Then I wrote a BTM script to send 50,000 characters to stdout followed by a trigger character to terminate the Pascal program. Then I timed it with a real pipe and an in-process pipe. As you suggested, the real pipe was faster. The times were about 2.2 seconds for the real pipe and 3.4 seconds for the in-process pipe. However, a lot of the time was taken up writing the characters to the screen. So I modified the Pascal program to read the characters but write nothing out. Then there was only a small difference.
 
The previous entry was made before I finished writing it when my computer went my browser closed. I wanted to report some further experiments.

In order to bypass the time required to write characters to the screen, I changed the Pascal program to just read the piped-in characters and do nothing. Then I wrote a script to make 1000 runs with alternating in-process and real pipes and compiled timing statistics. Here's the output I found on my screen this morning.

Ran 1000 tests with each type of pipe​
​
In-Process Pipes​
Average time.........: 1.374​
Standard deviation...: 0.065​
​
Real Pipes​
Average time.........: 1.311​
Standard deviation...: 0.038​
​
Time Differences​
Average difference...: 0.063​
Standard deviation...: 0.067​
Maximum difference...: 0.530​
Minimum difference...: -0.236​

There is a lot of variation from run to run. I suppose the time depends on what else the computer is doing (though I was not using the computer overnight).

The conclusion I would come to is that real pipes do not have a significant speed advantage over in-process pipes. Of course, they must be used if one wants the piped output to appear as the input is being generated. While my Pascal code was outputting characters, the output appeared immediately with the real pipes but appeared in a burst after a delay with the in-process pipe.

I am having trouble thinking of a case where one would need to see the output as it is processed by the pipe. There would have to be a source of data over which one has no control and wants to see in real time after some processing has been performed.
 
rconn said:
When you have a long-running program writing output continuously, and you want to process the output as it appears.
Click to expand...

I understand the principle; I just can't envision an actual case. Can anyone reading this thread offer a real example? I have had very long-running programs, but they produced their output directly with no need for a pipe to process that output. So I imagine that it would have to be a case of running a program that someone else wrote that does not produce output in the desired format. Even in such a case, it would probably be processing the data line-by-line, not character-by-character, so there would be line buffering.
 
Any kind of monitoring ... that you want to watch in-progress, or even get regular updates, and not have to restart.
 
vefatica said:
Any kind of monitoring ... that you want to watch in-progress, or even get regular updates, and not have to restart.
Click to expand...

I guess that when I've wanted to do that, I would use a batch file to issue queries on my schedule (if, for example, it was something on a website or remote system) or have the program (which I created) write out its own progress updates. I can't recall ever having to use a pipe to process output. Do you actually have processes that you monitor that require a pipe to process the data?

I recall that C and Unix were big on pipes, but I never actually found a use for pipes that I recall, though they seemed like a really clever idea. I never used C (after Fortran, I learned Pascal), and I used Unix only on the computers on which I designed integrated circuits (where I wished that I had TCC instead). Otherwise I used my own Z-System on 8-bit computers or MS-DOS (until 4dos came out).
 
You must log in or register to reply here.
Back
Top