Welcome!

By registering with us, you'll be able to discuss, share and private message with other members of our community.

SignUp Now!

Speeding up redirection to NUL?

May
12,846
164
I have a little app which spits out all the permutations of the characters in its input string. The program uses a single printf() statement. Printing the output is, of course, slow (e.g., there are 3268800 permutations of 10 characters). If I pipe my app to another, it runs in the same time it would if it were not outputting at all. For example,
Code:
l:\projects\perms\release> timer & perms.exe abcdefghij | wc -l & timer
Timer 1 on: 12:51:56
3628800
Timer 1 off: 12:51:57  Elapsed: 0:00:01.44
But if I redirect its output to NUL, it takes a lot longer.
Code:
l:\projects\perms\release> timer & perms.exe abcdefghij > nul & timer
Timer 1 on: 12:53:19
Timer 1 off: 12:53:46  Elapsed: 0:00:26.82
This slowness of redirection to NUL does not seem to affect TCC's internal commands. So that makes me wonder if anything can be done to speed it up ... either by TCC, or by my app itself ... maybe by changing the properties of stdout (or something like that).
Any ideas?
 
...
So that makes me wonder if anything can be done to speed it up ... either by TCC, or by my app itself ... maybe by changing the properties of stdout (or something like that).
Any ideas?
You might add a parameter to Your application that says not to output anything at all.

Regards

Rodolfo Giovanninetti
 
But if I redirect its output to NUL, it takes a lot longer.
Code:
l:\projects\perms\release> timer & perms.exe abcdefghij > nul & timer
Timer 1 on: 12:53:19
Timer 1 off: 12:53:46  Elapsed: 0:00:26.82

How fast is it if you redirect to a file? or redirecting to nul while running from CMD (for comparison)?

You might add a parameter to Your application that says not to output anything at all.

If he doesn't output anything, why bother running the program at all? I'm guessing he was just doing a test dump to TCC's nul to see what happened and discovered the slowdown.
 
A file is quite a bit faster than NUL.
Code:
p:\perms\release> timer & perms.exe abcdefghij > nul & timer
Timer 1 on: 20:24:31
Timer 1 off: 20:25:16  Elapsed: 0:00:27.18

p:\perms\release> timer & perms.exe abcdefghij > ajperms.txt & timer
Timer 1 on: 20:25:42
Timer 1 off: 20:25:44  Elapsed: 0:00:01.34

And the time varies considerably between attempts. Here are five started in an instance of TCC16.
Code:
p:\perms\release> timer & perms.exe abcdefghij > nul & timer
Timer 1 on: 20:34:08
Timer 1 off: 20:34:51  Elapsed: 0:00:43.68

p:\perms\release> timer & perms.exe abcdefghij > nul & timer
Timer 1 on: 20:34:57
Timer 1 off: 20:35:23  Elapsed: 0:00:26.38

p:\perms\release> timer & perms.exe abcdefghij > nul & timer
Timer 1 on: 20:46:07
Timer 1 off: 20:47:08  Elapsed: 0:01:00.85

p:\perms\release> timer & perms.exe abcdefghij > nul & timer
Timer 1 on: 20:47:10
Timer 1 off: 20:48:11  Elapsed: 0:01:00.57

p:\perms\release> timer & perms.exe abcdefghij > nul & timer
Timer 1 on: 20:48:18
Timer 1 off: 20:49:01  Elapsed: 0:00:43.09

In what testing I've done, it's pretty much the same (slow and variable) in CMD and all versions of TCC back to 4NTv8. Three times seem popular, ~60 sec, ~45 sec, and ~27 sec. Here are three started from 4NTv8.
Code:
p:\perms\release> g:\4ntv8\4nt.exe

4NT  8.02.106  Windows Vista [Version 6.1.7601]
Copyright 1988-2007  Rex Conn & JP Software Inc.  All Rights Reserved
Registered to Vincent Fatica - 5 System License

p:\perms\release> timer & perms.exe abcdefghij > nul & timer
Timer 1 on: 21:00:12
Timer 1 off: 21:00:57  Elapsed: 0:00:45.00

p:\perms\release> timer & perms.exe abcdefghij > nul & timer
Timer 1 on: 21:01:05
Timer 1 off: 21:01:32  Elapsed: 0:00:26.88

p:\perms\release> timer & perms.exe abcdefghij > nul & timer
Timer 1 on: 21:01:45
Timer 1 off: 21:02:45  Elapsed: 0:01:00.52
 
How long does it take when redirected to CLIP: ?

Joe
NUL is the only slow one. Here are three for comparison.
Code:
p:\perms\release> timer & perms.exe abcdefghij > nul & timer
Timer 1 on: 11:00:27
Timer 1 off: 11:01:26  Elapsed: 0:00:59.83

p:\perms\release> timer & perms.exe abcdefghij > ajperms.txt & timer
Timer 1 on: 11:01:41
Timer 1 off: 11:01:42  Elapsed: 0:00:01.31

p:\perms\release> timer & perms.exe abcdefghij > clip: & timer
Timer 1 on: 11:01:54
Timer 1 off: 11:01:56  Elapsed: 0:00:01.58

And one rather peculiar looking workaround (?).

Code:
p:\perms\release> timer & perms.exe abcdefghij | > nul & timer
Timer 1 on: 11:02:40
Timer 1 off: 11:02:41  Elapsed: 0:00:01.39
 
FWIW, PERMS.EXE is nothing special ... 25 lines of plain C with one printf() statement. It works like this.
Code:
p:\perms\release> perms.exe abc
abc
acb
bac
bca
cab
cba
It's attached if anyone wants to play with it.
 

Attachments

  • perms.zip
    23.1 KB · Views: 313
But PERMS.EXE does seem a bit unique. By comparison, if I take the output of "perms abcdefghij > ajperms.txt" (43,545,600 bytes) and CAT.EXE it, redirecting to various places, NUL is the fastest!
Code:
v:\> timer & cat ajperms.txt > ajperms2.txt & timer
Timer 1 on: 11:53:49
Timer 1 off: 11:53:49  Elapsed: 0:00:00.19

v:\> timer & cat ajperms.txt > clip: & timer
Timer 1 on: 11:54:04
Timer 1 off: 11:54:04  Elapsed: 0:00:00.36

v:\> timer & cat ajperms.txt > NUL & timer
Timer 1 on: 11:54:11
Timer 1 off: 11:54:11  Elapsed: 0:00:00.09
 
Just an idea, but are you using the default Null device? It is located in;
Code:
C:\Windows\System32\drivers\null.sys

From Device Manager (Windows Vista), I click View, and then Show hidden devices.

Expand the Non-Plug and Play Drivers.

Scroll down to the Null driver, right click, and choose Properties.

On the Driver tab, click Driver Details... which will show you the driver file that you are using for Null.

Joe
 
Code:
timer on & for /l %kount in (1,1,9999) echo %kount > nul & timer off
Timer 1 on: 16:20:12
Timer 1 off: 16:20:18  Elapsed: 0:00:05.56

timer on & for /l %kount in (1,1,9999) echo %kount > nul & timer off
Timer 1 on: 16:20:19

Timer 1 off: 16:20:25  Elapsed: 0:00:05.58
timer on & for /l %kount in (1,1,9999) echo %kount > nul & timer off

Timer 1 on: 16:20:26
Timer 1 off: 16:20:32  Elapsed: 0:00:05.79

Same thing, but bracketing the for loop;

Code:
timer on & (for /l %kount in (1,1,9999) echo %kount) > nul & timer off
Timer 1 on: 16:21:11
Timer 1 off: 16:21:15  Elapsed: 0:00:04.69

timer on & (for /l %kount in (1,1,9999) echo %kount) > nul & timer off
Timer 1 on: 16:21:17
Timer 1 off: 16:21:21  Elapsed: 0:00:04.62

timer on & (for /l %kount in (1,1,9999) echo %kount) > nul & timer off
Timer 1 on: 16:21:23
Timer 1 off: 16:21:26  Elapsed: 0:00:03.81

Joe
 
Just an idea, but are you using the default Null device? It is located in;
Yes, so Win 7 says. But I can't say for sure what shells do when you say "> NUL".

If my app actually writes to NUL (after FILE *fnul = fopen("nul", ...) or HANDLE hnul=CreateFile("nul", ...)) it all happens quite quickly.
 
Everything I've said previously, and the time measurements, remain true if I replace my PERMS.EXE program with this one (below) ... ~60 seconds to redirect it to NUL and ~1.5 seconds to redirect it to a file (TCC and CMD alike).
Code:
#include <stdio.h>

int main ( int argc, char **argv )
{
   for ( int i=0; i<3628800; i++ )
     printf("abcdefghij\n");
   return 0;
}
So, what's the problem with redirection to NUL, and how can I speed it up?
 
You can change the operation of stdout. (Note that we also must change the line ending from \n to \r\n in this mode.)

Code:
#include <stdio.h>
#include <io.h>
#include <fcntl.h>

int main ( int argc, char **argv )
{
    _setmode( _fileno( stdout ), _O_BINARY );

   for ( int i = 0; i < 3628800; i++ )
     printf( "abcdefghij\r\n" );
   return 0;
}

I have no idea why this makes such a dramatic difference to the NUL device, though. Ignoring everything should not be a processor-intensive task.
 
Yeah, I noticed that by a somewhat different mechanism. The comments in the code below should say it all. That said, I still don't know what TCC is doing when redirecting to NUL or why it's fast for internals or when there's an intervening pipe. And I couldn't Google up any complaints about the Windows nul device.
Code:
int main ( int argc, char **argv )
{
   //freopen("slow.txt", "w", stdout);     // time = 1.5 seconds
   //freopen("slow.txt", "wb", stdout);   // time = 1.4 seconds
   //freopen("nul", "w", stdout);       // time = 58.8 seconds !!!!
   freopen("nul", "wb", stdout);       // time = 3.3 seconds
   for ( int i=0; i<3628800; i++ )
     printf("abcdefghij\n");
   freopen("CON", "w", stdout);
   return 0;
}
 
So, opening "nul" in binary (untranslated) mode gets the test in my last post down to 3.3 seconds. A further
Code:
setvbuf(stdout, NULL, _IOFBF, 65536);
gets it down to 1.3 seconds (seemingly about where it belongs). A 32K or 16K buffer is almost as good, but Ican measure the difference.
If any of this is applicable to how TCC redirects to nul, I hope Rex will implement it. Here's a slightly rewritten test.
Code:
int main ( int argc, char **argv )
{
   //FILE *stream = freopen("nul", "w", stdout);       // time = 58.8 seconds (without setvbuf())
   FILE *stream = freopen("nul", "wb", stdout);       // time = 3.3 seconds
   //_setmode( _fileno( stdout ), _O_BINARY );
   setvbuf(stream, NULL, _IOFBF, 65536);           // time down to 1.3 seconds
   for ( int i=0; i<3628800; i++ )
     printf("abcdefghij\r\n");
   fclose(stream);
   freopen("CON", "w", stdout);
   return 0;
}
 
"nul" (without the colon) is just a file named "nul". "nul:" (with the colon) is a device just as is C: (the "C" drive of course), PRN: (the printer), and CON: (the console window).
 
Last edited:
I don't know about "nul:" but I've been using "nul" for years (as mentioned under "Redirection" in the help; "nul:" is not mentioned). "Nul" is the Windows null device.

Check out the contents of HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\DOS Devices.

P.S., Dan, try naming/renaming a file "nul".
 
I prefer the trailing-colon format myself, because it "looks" like a device. But it'll work even if you call it NUL.TXT, or NUL.FOO, or C:\WINDOWS\NUL (assuming the C:\WINDOWS directory exists).
 
It works with garbage before it, too. None of these places (foo, f$, d:) exist.
Code:
v:\> echo foo > \\.\foo\nul

v:\> echo foo > \\.\f$\nul

v:\> echo foo > d:\garbage\nul

v:\> echo foo > d:\nul
 
The trailing colons (CON:, et c.) are buried somewhere in my mind. Is that how it was in MS_DOS?
 
"nul" (without the colon) is just a file named "nul". "nul:" (with the colon) is a device just as is C: (the "C" drive of course), PRN: (the printer), and CON: (the console window).
Normally, they are all but the same. You CAN create a file named "nul" or "nul:" but that involving a whole different level of API calls.
 
Normally, they are all but the same. You CAN create a file named "nul" or "nul:" but that involving a whole different level of API calls.
What APIs might those be. If you use C's fopen() (and relatives) or WIN32's CreateFile() with the name "nul" the NUL device is opened. And, the colon (:) is a reserved character and not allowed in Windows file names. See, for example,
http://msdn.microsoft.com/en-us/library/windows/desktop/aa365247(v=vs.85).aspx#naming_conventions
http://msdn.microsoft.com/en-us/library/windows/desktop/aa365247(v=vs.85).aspx#naming_conventions
 
I take part of that back.
Code:
v:\> touch /c \\.\h:\workplace\nul
2014-09-05 20:22:56.125  \\.\h:\workplace\nul

v:\> dir /m /k nu*
2014-09-05  20:22  0  nul

v:\> del \\.\h:\workplace\nul
Deleting \\.\h:\workplace\nul
  1 file deleted
 
Code:
setvbuf(stdout, NULL, _IOFBF, 65536);
gets it down to 1.3 seconds (seemingly about where it belongs). A 32K or 16K buffer is almost as good, but Ican measure the difference.
If any of this is applicable to how TCC redirects to nul, I hope Rex will implement it. Here's a slightly rewritten test.

TCC already does unbuffered writing for all of its internals. It can't do that for your external app; that's up to you.
 

Similar threads

Back
Top