1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Feature request -- in-process pipes

Discussion in 'Suggestions' started by dcantor, Apr 16, 2010.

  1. dcantor

    Joined:
    May 29, 2008
    Messages:
    507
    Likes Received:
    3
    I've found that most of the time I use a pipe, the in-process pipe works better for me than the normal pipe. Rex, please consider an option (in the .ini file) to specify that |! is implied by | . Perhaps the option
    Code:
    pipes=normal
    pipes=in-process
    
    could be used to select which kind of pipe the "|" character will use by default.

    (The down-side would be that to override it, there'd need to be a different syntax, or the normal pipe usage would need to be preceded by "OPTION //pipes=normal" .)

    While I'm here talking about pipes, I had trouble finding it in the help file.
    It could use index entries for | (there is one for ||, but not |) and for |! .
     
  2. rconn

    rconn Administrator
    Staff Member

    Joined:
    May 14, 2008
    Messages:
    9,860
    Likes Received:
    83
    That's the sort of option that generates a lot of "bug" reports (where
    people forget they set it, and then complain that batch files don't work).
    (I was against adding the |! syntax in the first place, but there are
    apparently a lot of people nostalgic for DOS.)

    The |! syntax is much, much, slower, and I've never found a case where it
    was necessary. What sort of things are you doing that you would want that
    to be the default?


    I'll add them, if you can convince Microsoft to change their help system.
    Currently, you cannot use those characters as keywords.

    Rex Conn
    JP Software
     
  3. dcantor

    Joined:
    May 29, 2008
    Messages:
    507
    Likes Received:
    3
    I've found that
    Code:
    pslist avg |! ffind /k /m /v /t"x" 
    (where pslist is the Sysinternals utility) is MUCH faster using in-process piping than using normal piping. This sort of thing is my typical usage of pipes.

    I understand your point of view regarding the bug-generation nature of such things.

    That's very strange; there's an index entry for "||".
     
  4. Steve Fabian

    Joined:
    May 20, 2008
    Messages:
    3,520
    Likes Received:
    4
    | The |! syntax is much, much, slower, and I've never found a case
    | where it was necessary. What sort of things are you doing that
    | you would want that to be the default?

    In process pipe is the only way to return a result to the main process via a
    normal environment variable, which makes it necessary for that purpose. When
    the right side contains SORT, in process pipe is probably faster than
    multiple process pipe. However, I do not see too much benefit in an option
    to make it the default, unless one wants to migrate a lot of 4DOS batch
    files to TCC without bulk editing.
    --
    Steve
     
  5. rconn

    rconn Administrator
    Staff Member

    Joined:
    May 14, 2008
    Messages:
    9,860
    Likes Received:
    83
    purpose.

    Unlikely, as the "in process pipe" has to do two file redirections, and
    unless you've got a scary fast disk drive it's not apt to be faster than
    writing to a real pipe (which goes to a memory buffer).

    Rex Conn
    JP Software
     
  6. rconn

    rconn Administrator
    Staff Member

    Joined:
    May 14, 2008
    Messages:
    9,860
    Likes Received:
    83
    Hmmm.

    ---------------

    [c:\takecommand11] timer & pslist avg | ffind /k /m /v /t"x" & timer
    Timer 1 on: 15:40:09

    pslist v1.28 - Sysinternals PsList
    Copyright � 2000-2004 Mark Russinovich
    Sysinternals

    Timer 1 off: 15:40:09 Elapsed: 0:00:00.28

    [c:\takecommand11] timer & pslist avg |! ffind /k /m /v /t"x" & timer
    Timer 1 on: 15:40:09

    pslist v1.28 - Sysinternals PsList
    Copyright � 2000-2004 Mark Russinovich
    Sysinternals

    Timer 1 off: 15:40:09 Elapsed: 0:00:00.29

    ---------------

    How are you defining "MUCH faster"?

    Rex Conn
    JP Software
     
  7. dcantor

    Joined:
    May 29, 2008
    Messages:
    507
    Likes Received:
    3
    Code:
    C:\work> timer & pslist avg | ffind /k /m /v /t"x" & timer
    Timer 1 on: 17:57:06
    
    pslist v1.28 - Sysinternals PsList
    Copyright ⌐ 2000-2004 Mark Russinovich
    Sysinternals
    
    Timer 1 off: 17:57:20  Elapsed: 0:00:14.03
    avgchsvx           1584   6  43  539  21404     0:21:26.156   104:09:25.791
    avgrsx             1592   6  28  242   1648     0:01:02.343   104:09:25.447
    avgcsrvx           1924   6   9  186   8504     0:09:33.000   104:09:20.416
    avgnsx             2264   6  10  238   8828     0:10:31.484   104:08:18.447
    avgcsrvx           3288   6   6  151   4188     0:00:02.046   104:07:40.385
    Timer 1 off: 17:57:21  Elapsed: 0:00:14.67
    
    C:\work> timer & pslist avg |! ffind /k /m /v /t"x" & timer
    Timer 1 on: 17:57:39
    
    pslist v1.28 - Sysinternals PsList
    Copyright ⌐ 2000-2004 Mark Russinovich
    Sysinternals
    
    avgchsvx           1584   6  43  539  21404     0:21:26.187   104:09:57.603
    avgrsx             1592   6  28  242   1648     0:01:02.343   104:09:57.260
    avgcsrvx           1924   6   9  186   8504     0:09:33.015   104:09:52.228
    avgnsx             2264   6  10  238   8828     0:10:31.484   104:08:50.260
    avgcsrvx           3288   6   6  151   4188     0:00:02.046   104:08:12.197
    Timer 1 off: 17:57:39  Elapsed: 0:00:00.19
    
    C:\work>
    Very repeatable. Perhaps it's because I have stuff in TCSTART that gets executed even for pipes, but I need that stuff (that's another topic, perhaps).
     
  8. Steve Fabian

    Joined:
    May 20, 2008
    Messages:
    3,520
    Likes Received:
    4
    | Very repeatable. Perhaps it's because I have stuff in TCSTART that
    | gets executed even for pipes, but I need that stuff (that's another
    | topic, perhaps).

    Perhaps the other reason is that the left side of the pipe is an external,
    which enumerates processes, and the right side of the standard pipe is an
    extra process to be processed by the external on the left side, which does
    not exist when you use the in-process pipe.
    --
    Steve
     
  9. rconn

    rconn Administrator
    Staff Member

    Joined:
    May 14, 2008
    Messages:
    9,860
    Likes Received:
    83
    You must have to have a helluva LOT in your TCSTART (& TCEXIT?) to make that
    kind of difference!

    It seems unlikely that you'd need to enable everything for pipes, since they
    inherit almost everything from their parent.

    Rex Conn
    JP Software
     
  10. rconn

    rconn Administrator
    Staff Member

    Joined:
    May 14, 2008
    Messages:
    9,860
    Likes Received:
    83
    Adding one process to the list processed by PSLIST is not going to make a
    measurable difference.

    Rex Conn
    JP Software
     
  11. dcantor

    Joined:
    May 29, 2008
    Messages:
    507
    Likes Received:
    3
    I have stuff in my TCSTART that I need because TCC is being started as a pipe by a process other than another TCC process. I have code that figures that out and does the right thing.

    Anyway, that's the reason for my request. I'd like to see an .ini file option for it, but I'd be quite happy if it was one of those hidden options that doesn't appear in the option dialog.

    Thanks for taking the time to understand the problem, and to consider it.
     
  12. Steve Fabian

    Joined:
    May 20, 2008
    Messages:
    3,520
    Likes Received:
    4
    | ---Quote---
    || When the right side contains SORT, in process pipe is probably
    || faster than multiple process pipe.
    | ---End Quote---
    | Unlikely, as the "in process pipe" has to do two file redirections,
    | and unless you've got a scary fast disk drive it's not apt to be
    | faster than writing to a real pipe (which goes to a memory buffer).

    For my curiosity, and I didn't make any tests, what happens if the left size
    of the pipe produces far more data to be sorted than the memory buffer can
    hold?
    --
    Steve
     
  13. rconn

    rconn Administrator
    Staff Member

    Joined:
    May 14, 2008
    Messages:
    9,860
    Likes Received:
    83
    buffer

    Doesn't matter. With real pipes, the right side of the pipe is reading it
    as the left side is writing it. With "in process" pipes, the right side has
    to wait for the left side to finish before it can begin reading.

    Rex Conn
    JP Software
     
  14. Jim Cook

    Joined:
    May 20, 2008
    Messages:
    604
    Likes Received:
    0
    My times are 00.15 for |! and 00.86 for | using the pslist example. The |!
    is faster for that.

    However, |! is 4.37 versus | of 3.83 when using the following command (~60K
    lines listed). In any case, there's not a whole lot of absolute difference.

    timer %+ *dir /s %windir% |! ffind /t"ThisTextIsNotFound" %+ timer


    I remember having troubles with something involving pipes that would have
    been solved by |! but now I can't recall. Drat. I know that for most things
    I do, either | or |! will work fine.

    On Sun, Apr 18, 2010 at 7:19 PM, rconn <> wrote:




    --
    Jim Cook
    2010 Sundays: 4/4, 6/6, 8/8, 10/10, 12/12 and 5/9, 9/5, 7/11, 11/7.
    Next year they're Monday.
     
  15. Steve Fabian

    Joined:
    May 20, 2008
    Messages:
    3,520
    Likes Received:
    4
    | My times are 00.15 for |! and 00.86 for | using the pslist example.
    | The |! is faster for that.
    |
    | However, |! is 4.37 versus | of 3.83 when using the following
    | command (~60K lines listed). In any case, there's not a whole lot of
    | absolute difference.
    |
    | timer %+ *dir /s %windir% |! ffind /t"ThisTextIsNotFound" %+ timer

    I would run the test once for each mode to load everything possible into
    cache before running it for measurement. I suspect it would reduce the
    difference even more.
    --
    Steve
     
  16. Jim Cook

    Joined:
    May 20, 2008
    Messages:
    604
    Likes Received:
    0
    Actually, I did just that. I ran each four times, and the last three of each
    set were consistent.


    On Mon, Apr 19, 2010 at 9:33 AM, Steve Fábián <>wrote:




    --
    Jim Cook
    2010 Sundays: 4/4, 6/6, 8/8, 10/10, 12/12 and 5/9, 9/5, 7/11, 11/7.
    Next year they're Monday.
     

Share This Page