How to? Use TEE with .NET output

[Avi Shmidman]

TEE looks like a very useful command! I tried it today for the first time, but did not get useful results. I'm wondering what I'm doing wrong.

After not having success with various .NET programs, I compiled a quick minimal case C# program which simply outputs a set of 100 lines ("This is a Test1", "This is a Test2", etc.). Here's the code:

using System;
using System.Threading;
 
namespace OutputTest {
    class Program {
        static void Main(string[] args) {
            int i = 0;
            Console.OutputEncoding = System.Text.Encoding.UTF8;
 
            for (i = 0; i < 100; i++) {
                Console.WriteLine("This is a test" + i);
                Thread.Sleep(100);
            }
        }
    }
}

The program runs fine by itself at the command line. But when I run it with TEE, I just get eight lines of far eastern characters.

Here's my command line:

OutputTest.exe | tee test.txt

Here's the output:

桔獩椠⁳⁡整瑳ര吊楨⁳獩愠琠獥ㅴ਍桔獩椠⁳⁡整瑳ല
桔獩椠⁳⁡整瑳ള
桔獩椠⁳⁡整瑳ഴ
桔獩椠⁳⁡整瑳വ
桔獩椠⁳⁡整瑳ശ
桔獩椠⁳⁡整瑳ഷ
桔獩椠⁳⁡整瑳സ
桔獩椠⁳⁡整瑳ഹ

If I change the output encoding in the program to Ascii (by setting Console.OutputEncoding to System.Text.Encoding.Ascii), I get the same results.

Note: if I add /D to the TEE invocation, the date does show up properly, but the rest is still in far eastern characters:

2013/02/26 桔獩椠⁳⁡整瑳ര吊楨⁳獩愠琠獥ㅴ਍桔獩椠⁳⁡整瑳ല
2013/02/26 桔獩椠⁳⁡整瑳ള
2013/02/26 桔獩椠⁳⁡整瑳ഴ
2013/02/26 桔獩椠⁳⁡整瑳വ
2013/02/26 桔獩椠⁳⁡整瑳ശ
2013/02/26 桔獩椠⁳⁡整瑳ഷ
2013/02/26 桔獩椠⁳⁡整瑳സ
2013/02/26 桔獩椠⁳⁡整瑳ഹ

What am I doing wrong?

 

[rconn]

It looks like you've configured TCC to expect Unicode input, and then passed it UTF8 (which isn't supported by ANY command processor in Windows) or ASCII. Either pass Unicode or reset TCC to take ASCII input.

 

[Avi Shmidman]

It looks like you've configured TCC to expect Unicode input, and then passed it UTF8 (which isn't supported by ANY command processor in Windows) or ASCII. Either pass Unicode or reset TCC to take ASCII input.

Hi Rex,

The "Unicode Output" setting definitely has an effect: if I turn it off, then TEE properly outputs the output (although, of course, it sends gibberish for all the non-ASCII characters :( )

But the Unicode Output is about the *output* of the pipe, whereas your response explained that there was an issue with the *unicode input* to TCC from my program. So here I am a bit confused, and I would appreciate some help clarifying the issues:

1] If my program sends UTF-8 output (including non-ASCII Hebrew chars) to the console (but without any piping or redirection), the UTF-8 DOES display properly. So it seems that TCC as a command processor *does* support and handle UTF-8 output.

2] The "Unicode Output" option, of course, has no effect on this; whether or not it is checked (and whether or not the "UTF8Output" directive is set to Yes or No), my program's UTF-8 displays properly. It also displays properly regardless of the chcp code page setting. Is there a different setting in TCC that is determining how TCC parses my program's output to the console?

3] What about TEE as a utility? What determines how TEE parses its input (as ASCII or 16-bit unicode)?

4] Is is possible to use TPIPE in the pipeline to convert a program's UTF8 to 16-bit Unicode on the fly?

 

[Avi Shmidman]

Update: I have rewritten the c# program to send it's output as 16-bit unicode. So now, with TCC's UnicodeOutput option set to Yes, TEE gives me the expected results, for both the output to screen as well as for the info saved in the file.

HOWEVER, when I run the program by itself at the command line, the output is garbled (an extra space in between each ASCII letter, with gibberish for any non-ASCII chars).

How can I configure TCC to properly display 16-bit unicode output from console applications? (As noted, when I output UTF8, TCC does properly display unicode characters, so it is not an issue of the fonts).

 

[rconn]

1] If my program sends UTF-8 output (including non-ASCII Hebrew chars) to the console (but without any piping or redirection), the UTF-8 DOES display properly. So it seems that TCC as a command processor *does* support and handle UTF-8 output.

No; this doesn't have anything to do with the command processor. It's strictly up to the Windows console manager.

3] What about TEE as a utility? What determines how TEE parses its input (as ASCII or 16-bit unicode)?

If you have UnicodeOutput set, then TEE assumes that you also want Unicode input. Otherwise, TEE will assume ASCII input.

4] Is is possible to use TPIPE in the pipeline to convert a program's UTF8 to 16-bit Unicode on the fly?

Yes. (That's the only way to do it.)

 

[Avi Shmidman]

Hi Rex,

Thank you for the explanations. This was very informative. At the same time, though, all of this leads to a fairly difficult situation, given that my program needs to output non-ASCII characters as part of its output:

1] If I output UTF-8 from my program, the output will display properly on the console when the program is run straight; however, if I pipe the output to TEE, it will not be processed correctly.

2] If I output UTF-16 from my program, the output will be processed correctly when piped to TEE; however, if I run the program straight on the command line without the pipe, the output will be garbled.

3] If I output ASCII from my program, the non-ASCII characters are not displayed correctly.

Given these restrictions, I'd like to ask if you would please consider providing an option for TEE to accept UTF-8 input.

 

[rconn]

Thank you for the explanations. This was very informative. At the same time, though, all of this leads to a fairly difficult situation, given that my program needs to output non-ASCII characters as part of its output:

1] If I output UTF-8 from my program, the output will display properly on the console when the program is run straight; however, if I pipe the output to TEE, it will not be processed correctly.

2] If I output UTF-16 from my program, the output will be processed correctly when piped to TEE; however, if I run the program straight on the command line without the pipe, the output will be garbled.

3] If I output ASCII from my program, the non-ASCII characters are not displayed correctly.

Given these restrictions, I'd like to ask if you would please consider providing an option for TEE to accept UTF-8 input.

#2 and #3 don't have anything to do with TCC. TCC doesn't have any control over how the Windows console manager displays characters from other applications -- or for that matter, how the Windows console manager displays characters from TCC! (This would require a rather catastrophic hardware & software failure in your CPU and in Windows; I suspect it would have been noticed by now.)

If you want to send UTF8 output to TEE, use TPIPE to preprocess it.