1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Speeding up redirection to NUL?

Discussion in 'Support' started by vefatica, Aug 24, 2014.

  1. vefatica

    Joined:
    May 20, 2008
    Messages:
    7,939
    Likes Received:
    30
    I have a little app which spits out all the permutations of the characters in its input string. The program uses a single printf() statement. Printing the output is, of course, slow (e.g., there are 3268800 permutations of 10 characters). If I pipe my app to another, it runs in the same time it would if it were not outputting at all. For example,
    Code:
    l:\projects\perms\release> timer & perms.exe abcdefghij | wc -l & timer
    Timer 1 on: 12:51:56
    3628800
    Timer 1 off: 12:51:57  Elapsed: 0:00:01.44
    But if I redirect its output to NUL, it takes a lot longer.
    Code:
    l:\projects\perms\release> timer & perms.exe abcdefghij > nul & timer
    Timer 1 on: 12:53:19
    Timer 1 off: 12:53:46  Elapsed: 0:00:26.82
    This slowness of redirection to NUL does not seem to affect TCC's internal commands. So that makes me wonder if anything can be done to speed it up ... either by TCC, or by my app itself ... maybe by changing the properties of stdout (or something like that).
    Any ideas?
     
  2. Rodolfo

    Joined:
    May 20, 2009
    Messages:
    213
    Likes Received:
    0
    You might add a parameter to Your application that says not to output anything at all.

    Regards

    Rodolfo Giovanninetti
     
  3. JohnQSmith

    Joined:
    Jan 19, 2011
    Messages:
    559
    Likes Received:
    7
    How fast is it if you redirect to a file? or redirecting to nul while running from CMD (for comparison)?

    If he doesn't output anything, why bother running the program at all? I'm guessing he was just doing a test dump to TCC's nul to see what happened and discovered the slowdown.
     
  4. vefatica

    Joined:
    May 20, 2008
    Messages:
    7,939
    Likes Received:
    30
    A file is quite a bit faster than NUL.
    Code:
    p:\perms\release> timer & perms.exe abcdefghij > nul & timer
    Timer 1 on: 20:24:31
    Timer 1 off: 20:25:16  Elapsed: 0:00:27.18
    
    p:\perms\release> timer & perms.exe abcdefghij > ajperms.txt & timer
    Timer 1 on: 20:25:42
    Timer 1 off: 20:25:44  Elapsed: 0:00:01.34
    And the time varies considerably between attempts. Here are five started in an instance of TCC16.
    Code:
    p:\perms\release> timer & perms.exe abcdefghij > nul & timer
    Timer 1 on: 20:34:08
    Timer 1 off: 20:34:51  Elapsed: 0:00:43.68
    
    p:\perms\release> timer & perms.exe abcdefghij > nul & timer
    Timer 1 on: 20:34:57
    Timer 1 off: 20:35:23  Elapsed: 0:00:26.38
    
    p:\perms\release> timer & perms.exe abcdefghij > nul & timer
    Timer 1 on: 20:46:07
    Timer 1 off: 20:47:08  Elapsed: 0:01:00.85
    
    p:\perms\release> timer & perms.exe abcdefghij > nul & timer
    Timer 1 on: 20:47:10
    Timer 1 off: 20:48:11  Elapsed: 0:01:00.57
    
    p:\perms\release> timer & perms.exe abcdefghij > nul & timer
    Timer 1 on: 20:48:18
    Timer 1 off: 20:49:01  Elapsed: 0:00:43.09
    In what testing I've done, it's pretty much the same (slow and variable) in CMD and all versions of TCC back to 4NTv8. Three times seem popular, ~60 sec, ~45 sec, and ~27 sec. Here are three started from 4NTv8.
    Code:
    p:\perms\release> g:\4ntv8\4nt.exe
    
    4NT  8.02.106  Windows Vista [Version 6.1.7601]
    Copyright 1988-2007  Rex Conn & JP Software Inc.  All Rights Reserved
    Registered to Vincent Fatica - 5 System License
    
    p:\perms\release> timer & perms.exe abcdefghij > nul & timer
    Timer 1 on: 21:00:12
    Timer 1 off: 21:00:57  Elapsed: 0:00:45.00
    
    p:\perms\release> timer & perms.exe abcdefghij > nul & timer
    Timer 1 on: 21:01:05
    Timer 1 off: 21:01:32  Elapsed: 0:00:26.88
    
    p:\perms\release> timer & perms.exe abcdefghij > nul & timer
    Timer 1 on: 21:01:45
    Timer 1 off: 21:02:45  Elapsed: 0:01:00.52
     
  5. Joe Caverly

    Joined:
    Aug 28, 2009
    Messages:
    680
    Likes Received:
    8
    How long does it take when redirected to CLIP: ?

    Joe
     
  6. vefatica

    Joined:
    May 20, 2008
    Messages:
    7,939
    Likes Received:
    30
    NUL is the only slow one. Here are three for comparison.
    Code:
    p:\perms\release> timer & perms.exe abcdefghij > nul & timer
    Timer 1 on: 11:00:27
    Timer 1 off: 11:01:26  Elapsed: 0:00:59.83
    
    p:\perms\release> timer & perms.exe abcdefghij > ajperms.txt & timer
    Timer 1 on: 11:01:41
    Timer 1 off: 11:01:42  Elapsed: 0:00:01.31
    
    p:\perms\release> timer & perms.exe abcdefghij > clip: & timer
    Timer 1 on: 11:01:54
    Timer 1 off: 11:01:56  Elapsed: 0:00:01.58
    And one rather peculiar looking workaround (?).

    Code:
    p:\perms\release> timer & perms.exe abcdefghij | > nul & timer
    Timer 1 on: 11:02:40
    Timer 1 off: 11:02:41  Elapsed: 0:00:01.39
     
  7. vefatica

    Joined:
    May 20, 2008
    Messages:
    7,939
    Likes Received:
    30
    FWIW, PERMS.EXE is nothing special ... 25 lines of plain C with one printf() statement. It works like this.
    Code:
    p:\perms\release> perms.exe abc
    abc
    acb
    bac
    bca
    cab
    cba
    It's attached if anyone wants to play with it.
     

    Attached Files:

  8. vefatica

    Joined:
    May 20, 2008
    Messages:
    7,939
    Likes Received:
    30
    But PERMS.EXE does seem a bit unique. By comparison, if I take the output of "perms abcdefghij > ajperms.txt" (43,545,600 bytes) and CAT.EXE it, redirecting to various places, NUL is the fastest!
    Code:
    v:\> timer & cat ajperms.txt > ajperms2.txt & timer
    Timer 1 on: 11:53:49
    Timer 1 off: 11:53:49  Elapsed: 0:00:00.19
    
    v:\> timer & cat ajperms.txt > clip: & timer
    Timer 1 on: 11:54:04
    Timer 1 off: 11:54:04  Elapsed: 0:00:00.36
    
    v:\> timer & cat ajperms.txt > NUL & timer
    Timer 1 on: 11:54:11
    Timer 1 off: 11:54:11  Elapsed: 0:00:00.09
     
  9. Joe Caverly

    Joined:
    Aug 28, 2009
    Messages:
    680
    Likes Received:
    8
    Just an idea, but are you using the default Null device? It is located in;
    Code:
    C:\Windows\System32\drivers\null.sys
    From Device Manager (Windows Vista), I click View, and then Show hidden devices.

    Expand the Non-Plug and Play Drivers.

    Scroll down to the Null driver, right click, and choose Properties.

    On the Driver tab, click Driver Details... which will show you the driver file that you are using for Null.

    Joe
     
  10. Joe Caverly

    Joined:
    Aug 28, 2009
    Messages:
    680
    Likes Received:
    8
    Code:
    timer on & for /l %kount in (1,1,9999) echo %kount > nul & timer off
    Timer 1 on: 16:20:12
    Timer 1 off: 16:20:18  Elapsed: 0:00:05.56
    
    timer on & for /l %kount in (1,1,9999) echo %kount > nul & timer off
    Timer 1 on: 16:20:19
    
    Timer 1 off: 16:20:25  Elapsed: 0:00:05.58
    timer on & for /l %kount in (1,1,9999) echo %kount > nul & timer off
    
    Timer 1 on: 16:20:26
    Timer 1 off: 16:20:32  Elapsed: 0:00:05.79
    Same thing, but bracketing the for loop;

    Code:
    timer on & (for /l %kount in (1,1,9999) echo %kount) > nul & timer off
    Timer 1 on: 16:21:11
    Timer 1 off: 16:21:15  Elapsed: 0:00:04.69
    
    timer on & (for /l %kount in (1,1,9999) echo %kount) > nul & timer off
    Timer 1 on: 16:21:17
    Timer 1 off: 16:21:21  Elapsed: 0:00:04.62
    
    timer on & (for /l %kount in (1,1,9999) echo %kount) > nul & timer off
    Timer 1 on: 16:21:23
    Timer 1 off: 16:21:26  Elapsed: 0:00:03.81
    Joe
     
  11. vefatica

    Joined:
    May 20, 2008
    Messages:
    7,939
    Likes Received:
    30
    Yes, so Win 7 says. But I can't say for sure what shells do when you say "> NUL".

    If my app actually writes to NUL (after FILE *fnul = fopen("nul", ...) or HANDLE hnul=CreateFile("nul", ...)) it all happens quite quickly.
     
  12. vefatica

    Joined:
    May 20, 2008
    Messages:
    7,939
    Likes Received:
    30
    Everything I've said previously, and the time measurements, remain true if I replace my PERMS.EXE program with this one (below) ... ~60 seconds to redirect it to NUL and ~1.5 seconds to redirect it to a file (TCC and CMD alike).
    Code:
    #include <stdio.h>
    
    int main ( int argc, char **argv )
    {
       for ( int i=0; i<3628800; i++ )
         printf("abcdefghij\n");
       return 0;
    }
    So, what's the problem with redirection to NUL, and how can I speed it up?
     
  13. Charles Dye

    Charles Dye Super Moderator
    Staff Member

    Joined:
    May 20, 2008
    Messages:
    3,301
    Likes Received:
    39
    You can change the operation of stdout. (Note that we also must change the line ending from \n to \r\n in this mode.)

    Code:
    #include <stdio.h>
    #include <io.h>
    #include <fcntl.h>
    
    int main ( int argc, char **argv )
    {
        _setmode( _fileno( stdout ), _O_BINARY );
    
       for ( int i = 0; i < 3628800; i++ )
         printf( "abcdefghij\r\n" );
       return 0;
    }
    
    I have no idea why this makes such a dramatic difference to the NUL device, though. Ignoring everything should not be a processor-intensive task.
     
  14. vefatica

    Joined:
    May 20, 2008
    Messages:
    7,939
    Likes Received:
    30
    Yeah, I noticed that by a somewhat different mechanism. The comments in the code below should say it all. That said, I still don't know what TCC is doing when redirecting to NUL or why it's fast for internals or when there's an intervening pipe. And I couldn't Google up any complaints about the Windows nul device.
    Code:
    int main ( int argc, char **argv )
    {
       //freopen("slow.txt", "w", stdout);     // time = 1.5 seconds
       //freopen("slow.txt", "wb", stdout);   // time = 1.4 seconds
       //freopen("nul", "w", stdout);       // time = 58.8 seconds !!!!
       freopen("nul", "wb", stdout);       // time = 3.3 seconds
       for ( int i=0; i<3628800; i++ )
         printf("abcdefghij\n");
       freopen("CON", "w", stdout);
       return 0;
    }
     
  15. vefatica

    Joined:
    May 20, 2008
    Messages:
    7,939
    Likes Received:
    30
    So, opening "nul" in binary (untranslated) mode gets the test in my last post down to 3.3 seconds. A further
    Code:
    setvbuf(stdout, NULL, _IOFBF, 65536);
    gets it down to 1.3 seconds (seemingly about where it belongs). A 32K or 16K buffer is almost as good, but Ican measure the difference.
    If any of this is applicable to how TCC redirects to nul, I hope Rex will implement it. Here's a slightly rewritten test.
    Code:
    int main ( int argc, char **argv )
    {
       //FILE *stream = freopen("nul", "w", stdout);       // time = 58.8 seconds (without setvbuf())
       FILE *stream = freopen("nul", "wb", stdout);       // time = 3.3 seconds
       //_setmode( _fileno( stdout ), _O_BINARY );
       setvbuf(stream, NULL, _IOFBF, 65536);           // time down to 1.3 seconds
       for ( int i=0; i<3628800; i++ )
         printf("abcdefghij\r\n");
       fclose(stream);
       freopen("CON", "w", stdout);
       return 0;
    }
     
  16. AnrDaemon

    Joined:
    Aug 23, 2010
    Messages:
    45
    Likes Received:
    1
    Can you please check if there's a difference between redirecting to "nul" and "nul:" ?
     
  17. mathewsdw

    Joined:
    May 24, 2010
    Messages:
    855
    Likes Received:
    0
    "nul" (without the colon) is just a file named "nul". "nul:" (with the colon) is a device just as is C: (the "C" drive of course), PRN: (the printer), and CON: (the console window).
     
    #17 mathewsdw, Sep 4, 2014
    Last edited: Sep 4, 2014
  18. vefatica

    Joined:
    May 20, 2008
    Messages:
    7,939
    Likes Received:
    30
    I don't know about "nul:" but I've been using "nul" for years (as mentioned under "Redirection" in the help; "nul:" is not mentioned). "Nul" is the Windows null device.

    Check out the contents of HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\DOS Devices.

    P.S., Dan, try naming/renaming a file "nul".
     
  19. vefatica

    Joined:
    May 20, 2008
    Messages:
    7,939
    Likes Received:
    30
    Code:
    v:\> echo %@dosdevice[nul]
    \Device\Null
     
  20. mathewsdw

    Joined:
    May 24, 2010
    Messages:
    855
    Likes Received:
    0
    Well you learn something new every day!!!!! (Obviously I've been using NUL: for many years.)
     
  21. Charles Dye

    Charles Dye Super Moderator
    Staff Member

    Joined:
    May 20, 2008
    Messages:
    3,301
    Likes Received:
    39
    I prefer the trailing-colon format myself, because it "looks" like a device. But it'll work even if you call it NUL.TXT, or NUL.FOO, or C:\WINDOWS\NUL (assuming the C:\WINDOWS directory exists).
     
  22. vefatica

    Joined:
    May 20, 2008
    Messages:
    7,939
    Likes Received:
    30
    It works with garbage before it, too. None of these places (foo, f$, d:) exist.
    Code:
    v:\> echo foo > \\.\foo\nul
    
    v:\> echo foo > \\.\f$\nul
    
    v:\> echo foo > d:\garbage\nul
    
    v:\> echo foo > d:\nul
     
  23. Charles Dye

    Charles Dye Super Moderator
    Staff Member

    Joined:
    May 20, 2008
    Messages:
    3,301
    Likes Received:
    39
    Right; my brain was flashing back to MS-DOS for some reason.
     
  24. vefatica

    Joined:
    May 20, 2008
    Messages:
    7,939
    Likes Received:
    30
    The trailing colons (CON:, et c.) are buried somewhere in my mind. Is that how it was in MS_DOS?
     
  25. Charles Dye

    Charles Dye Super Moderator
    Staff Member

    Joined:
    May 20, 2008
    Messages:
    3,301
    Likes Received:
    39
    Been a while, but I think they were optional there too.
     
  26. vefatica

    Joined:
    May 20, 2008
    Messages:
    7,939
    Likes Received:
    30
    I still have a MD-DOS v5 manual and a 4DOS manual (first edition!, now bound with duct tape). They should jog my memory.
     
  27. AnrDaemon

    Joined:
    Aug 23, 2010
    Messages:
    45
    Likes Received:
    1
    Normally, they are all but the same. You CAN create a file named "nul" or "nul:" but that involving a whole different level of API calls.
     
  28. vefatica

    Joined:
    May 20, 2008
    Messages:
    7,939
    Likes Received:
    30
    What APIs might those be. If you use C's fopen() (and relatives) or WIN32's CreateFile() with the name "nul" the NUL device is opened. And, the colon (:) is a reserved character and not allowed in Windows file names. See, for example,

    http://msdn.microsoft.com/en-us/library/windows/desktop/aa365247(v=vs.85).aspx#naming_conventions
     
  29. vefatica

    Joined:
    May 20, 2008
    Messages:
    7,939
    Likes Received:
    30
    I take part of that back.
    Code:
    v:\> touch /c \\.\h:\workplace\nul
    2014-09-05 20:22:56.125  \\.\h:\workplace\nul
    
    v:\> dir /m /k nu*
    2014-09-05  20:22  0  nul
    
    v:\> del \\.\h:\workplace\nul
    Deleting \\.\h:\workplace\nul
      1 file deleted
     
  30. rconn

    rconn Administrator
    Staff Member

    Joined:
    May 14, 2008
    Messages:
    9,854
    Likes Received:
    83
    Code:
    setvbuf(stdout, NULL, _IOFBF, 65536);
    gets it down to 1.3 seconds (seemingly about where it belongs). A 32K or 16K buffer is almost as good, but Ican measure the difference.
    If any of this is applicable to how TCC redirects to nul, I hope Rex will implement it. Here's a slightly rewritten test.

    TCC already does unbuffered writing for all of its internals. It can't do that for your external app; that's up to you.
     

Share This Page