Declined Feature request -- in-process pipes

[dcantor]

I've found that most of the time I use a pipe, the in-process pipe works better for me than the normal pipe. Rex, please consider an option (in the .ini file) to specify that |! is implied by | . Perhaps the option

pipes=normal
pipes=in-process

could be used to select which kind of pipe the "|" character will use by default.

(The down-side would be that to override it, there'd need to be a different syntax, or the normal pipe usage would need to be preceded by "OPTION //pipes=normal" .)

While I'm here talking about pipes, I had trouble finding it in the help file.
It could use index entries for | (there is one for ||, but not |) and for |! .

 

[rconn]

> I've found that most of the time I use a pipe, the in-process pipe
> works better for me than the normal pipe. Rex, please consider an
> option (in the .ini file) to specify that |! is implied by | .
> Perhaps the option
> Code:
> ---------
> pipes=normal
> pipes=in-process
> ---------
> could be used to select which kind of pipe the "|" character will use
> by default.
>
> (The down-side would be that to override it, there'd need to be a
> different syntax, or the normal pipe usage would need to be preceded by
> "OPTION //pipes=normal" .)

That's the sort of option that generates a lot of "bug" reports (where
people forget they set it, and then complain that batch files don't work).
(I was against adding the |! syntax in the first place, but there are
apparently a lot of people nostalgic for DOS.)

The |! syntax is much, much, slower, and I've never found a case where it
was necessary. What sort of things are you doing that you would want that
to be the default?

> While I'm here talking about pipes, I had trouble finding it in the
> help file.
> It could use index entries for | (there is one for ||, but not |) and
> for |! .

I'll add them, if you can convince Microsoft to change their help system.
Currently, you cannot use those characters as keywords.

Rex Conn
JP Software

 

[dcantor]

That's the sort of option that generates a lot of "bug" reports (where
people forget they set it, and then complain that batch files don't work).
(I was against adding the |! syntax in the first place, but there are
apparently a lot of people nostalgic for DOS.)

The |! syntax is much, much, slower, and I've never found a case where it
was necessary. What sort of things are you doing that you would want that
to be the default?

I've found that

pslist avg |! ffind /k /m /v /t"x"

(where pslist is the Sysinternals utility) is MUCH faster using in-process piping than using normal piping. This sort of thing is my typical usage of pipes.

I understand your point of view regarding the bug-generation nature of such things.

I'll add them, if you can convince Microsoft to change their help system.
Currently, you cannot use those characters as keywords.

That's very strange; there's an index entry for "||".

 

[Steve Fabian]

| The |! syntax is much, much, slower, and I've never found a case
| where it was necessary. What sort of things are you doing that
| you would want that to be the default?

In process pipe is the only way to return a result to the main process via a
normal environment variable, which makes it necessary for that purpose. When
the right side contains SORT, in process pipe is probably faster than
multiple process pipe. However, I do not see too much benefit in an option
to make it the default, unless one wants to migrate a lot of 4DOS batch
files to TCC without bulk editing.
--
Steve

 

[rconn]

> | The |! syntax is much, much, slower, and I've never found a case
> | where it was necessary. What sort of things are you doing that
> | you would want that to be the default?
>
> In process pipe is the only way to return a result to the main process
> via a normal environment variable, which makes it necessary for that

purpose.

> When the right side contains SORT, in process pipe is probably faster than
> multiple process pipe.

Unlikely, as the "in process pipe" has to do two file redirections, and
unless you've got a scary fast disk drive it's not apt to be faster than
writing to a real pipe (which goes to a memory buffer).

Rex Conn
JP Software

 

[rconn]

> I've found that
> Code:
> ---------
> pslist avg |! ffind /k /m /v /t"x"
> ---------
> (where pslist is the Sysinternals utility) is MUCH faster using in-
> process piping than using normal piping. This sort of thing is my
> typical usage of pipes.

Hmmm.

---------------

[c:\takecommand11] timer & pslist avg | ffind /k /m /v /t"x" & timer
Timer 1 on: 15:40:09

pslist v1.28 - Sysinternals PsList
Copyright � 2000-2004 Mark Russinovich
Sysinternals

Timer 1 off: 15:40:09 Elapsed: 0:00:00.28

[c:\takecommand11] timer & pslist avg |! ffind /k /m /v /t"x" & timer
Timer 1 on: 15:40:09

pslist v1.28 - Sysinternals PsList
Copyright � 2000-2004 Mark Russinovich
Sysinternals

Timer 1 off: 15:40:09 Elapsed: 0:00:00.29

---------------

How are you defining "MUCH faster"?

Rex Conn
JP Software

 

[dcantor]

Hmmm.

---------------

[c:\takecommand11] timer & pslist avg | ffind /k /m /v /t"x" & timer
Timer 1 on: 15:40:09

pslist v1.28 - Sysinternals PsList
Copyright � 2000-2004 Mark Russinovich
Sysinternals

Timer 1 off: 15:40:09 Elapsed: 0:00:00.28

[c:\takecommand11] timer & pslist avg |! ffind /k /m /v /t"x" & timer
Timer 1 on: 15:40:09

pslist v1.28 - Sysinternals PsList
Copyright � 2000-2004 Mark Russinovich
Sysinternals

Timer 1 off: 15:40:09 Elapsed: 0:00:00.29

---------------

How are you defining "MUCH faster"?

Rex Conn
JP Software

C:\work> timer & pslist avg | ffind /k /m /v /t"x" & timer
Timer 1 on: 17:57:06

pslist v1.28 - Sysinternals PsList
Copyright ⌐ 2000-2004 Mark Russinovich
Sysinternals

Timer 1 off: 17:57:20  Elapsed: 0:00:14.03
avgchsvx           1584   6  43  539  21404     0:21:26.156   104:09:25.791
avgrsx             1592   6  28  242   1648     0:01:02.343   104:09:25.447
avgcsrvx           1924   6   9  186   8504     0:09:33.000   104:09:20.416
avgnsx             2264   6  10  238   8828     0:10:31.484   104:08:18.447
avgcsrvx           3288   6   6  151   4188     0:00:02.046   104:07:40.385
Timer 1 off: 17:57:21  Elapsed: 0:00:14.67

C:\work> timer & pslist avg |! ffind /k /m /v /t"x" & timer
Timer 1 on: 17:57:39

pslist v1.28 - Sysinternals PsList
Copyright ⌐ 2000-2004 Mark Russinovich
Sysinternals

avgchsvx           1584   6  43  539  21404     0:21:26.187   104:09:57.603
avgrsx             1592   6  28  242   1648     0:01:02.343   104:09:57.260
avgcsrvx           1924   6   9  186   8504     0:09:33.015   104:09:52.228
avgnsx             2264   6  10  238   8828     0:10:31.484   104:08:50.260
avgcsrvx           3288   6   6  151   4188     0:00:02.046   104:08:12.197
Timer 1 off: 17:57:39  Elapsed: 0:00:00.19

C:\work>

Very repeatable. Perhaps it's because I have stuff in TCSTART that gets executed even for pipes, but I need that stuff (that's another topic, perhaps).

 

[Steve Fabian]

| Very repeatable. Perhaps it's because I have stuff in TCSTART that
| gets executed even for pipes, but I need that stuff (that's another
| topic, perhaps).

Perhaps the other reason is that the left side of the pipe is an external,
which enumerates processes, and the right side of the standard pipe is an
extra process to be processed by the external on the left side, which does
not exist when you use the in-process pipe.
--
Steve

 

[rconn]

> Very repeatable. Perhaps it's because I have stuff in TCSTART that
> gets executed even for pipes, but I need that stuff (that's another
> topic, perhaps).

You must have to have a helluva LOT in your TCSTART (& TCEXIT?) to make that
kind of difference!

It seems unlikely that you'd need to enable everything for pipes, since they
inherit almost everything from their parent.

Rex Conn
JP Software

 

[rconn]

> | Very repeatable. Perhaps it's because I have stuff in TCSTART that
> | gets executed even for pipes, but I need that stuff (that's another
> | topic, perhaps).
>
> Perhaps the other reason is that the left side of the pipe is an
> external, which enumerates processes, and the right side of the
> standard pipe is an extra process to be processed by the external
> on the left side, which does not exist when you use the in-process pipe.

Adding one process to the list processed by PSLIST is not going to make a
measurable difference.

Rex Conn
JP Software

 

[dcantor]

You must have to have a helluva LOT in your TCSTART (& TCEXIT?) to make that
kind of difference!

It seems unlikely that you'd need to enable everything for pipes, since they
inherit almost everything from their parent.

Rex Conn
JP Software

I have stuff in my TCSTART that I need because TCC is being started as a pipe by a process other than another TCC process. I have code that figures that out and does the right thing.

Anyway, that's the reason for my request. I'd like to see an .ini file option for it, but I'd be quite happy if it was one of those hidden options that doesn't appear in the option dialog.

Thanks for taking the time to understand the problem, and to consider it.

 

[Steve Fabian]

| ---Quote---
|| When the right side contains SORT, in process pipe is probably
|| faster than multiple process pipe.
| ---End Quote---
| Unlikely, as the "in process pipe" has to do two file redirections,
| and unless you've got a scary fast disk drive it's not apt to be
| faster than writing to a real pipe (which goes to a memory buffer).

For my curiosity, and I didn't make any tests, what happens if the left size
of the pipe produces far more data to be sorted than the memory buffer can
hold?
--
Steve

 

[rconn]

> | ---Quote---
> || When the right side contains SORT, in process pipe is probably
> || faster than multiple process pipe.
> | ---End Quote---
> | Unlikely, as the "in process pipe" has to do two file redirections,
> | and unless you've got a scary fast disk drive it's not apt to be
> | faster than writing to a real pipe (which goes to a memory buffer).
>
> For my curiosity, and I didn't make any tests, what happens if the left
> size of the pipe produces far more data to be sorted than the memory

buffer

> can hold?

Doesn't matter. With real pipes, the right side of the pipe is reading it
as the left side is writing it. With "in process" pipes, the right side has
to wait for the left side to finish before it can begin reading.

Rex Conn
JP Software

 

[Jim Cook]

My times are 00.15 for |! and 00.86 for | using the pslist example. The |!
is faster for that.

However, |! is 4.37 versus | of 3.83 when using the following command (~60K
lines listed). In any case, there's not a whole lot of absolute difference.

timer %+ *dir /s %windir% |! ffind /t"ThisTextIsNotFound" %+ timer


I remember having troubles with something involving pipes that would have
been solved by |! but now I can't recall. Drat. I know that for most things
I do, either | or |! will work fine.

On Sun, Apr 18, 2010 at 7:19 PM, rconn <> wrote:

> ---Quote---
> > | ---Quote---
> > || When the right side contains SORT, in process pipe is probably
> > || faster than multiple process pipe.
> > | ---End Quote---
> > | Unlikely, as the "in process pipe" has to do two file redirections,
> > | and unless you've got a scary fast disk drive it's not apt to be
> > | faster than writing to a real pipe (which goes to a memory buffer).
> >
> > For my curiosity, and I didn't make any tests, what happens if the left
> > size of the pipe produces far more data to be sorted than the memory
> ---End Quote---
> buffer
>
>
> ---Quote---
> > can hold?
> ---End Quote---
> Doesn't matter. With real pipes, the right side of the pipe is reading it
> as the left side is writing it. With "in process" pipes, the right side
> has
> to wait for the left side to finish before it can begin reading.
>
> Rex Conn
> JP Software
>
>
>
>
>

--
Jim Cook
2010 Sundays: 4/4, 6/6, 8/8, 10/10, 12/12 and 5/9, 9/5, 7/11, 11/7.
Next year they're Monday.

 

[Steve Fabian]

| My times are 00.15 for |! and 00.86 for | using the pslist example.
| The |! is faster for that.
|
| However, |! is 4.37 versus | of 3.83 when using the following
| command (~60K lines listed). In any case, there's not a whole lot of
| absolute difference.
|
| timer %+ *dir /s %windir% |! ffind /t"ThisTextIsNotFound" %+ timer

I would run the test once for each mode to load everything possible into
cache before running it for measurement. I suspect it would reduce the
difference even more.
--
Steve

 

[Jim Cook]

Actually, I did just that. I ran each four times, and the last three of each
set were consistent.


On Mon, Apr 19, 2010 at 9:33 AM, Steve Fábián <>wrote:

> | My times are 00.15 for |! and 00.86 for | using the pslist example.
> | The |! is faster for that.
> |
> | However, |! is 4.37 versus | of 3.83 when using the following
> | command (~60K lines listed). In any case, there's not a whole lot of
> | absolute difference.
> |
> | timer %+ *dir /s %windir% |! ffind /t"ThisTextIsNotFound" %+ timer
>
> I would run the test once for each mode to load everything possible into
> cache before running it for measurement. I suspect it would reduce the
> difference even more.
> --
> Steve
>
>
>
>
>

--
Jim Cook
2010 Sundays: 4/4, 6/6, 8/8, 10/10, 12/12 and 5/9, 9/5, 7/11, 11/7.
Next year they're Monday.