WAD TCC: inconsistent character handling

May 20, 2008
11,285
95
Syracuse, NY, USA
The file in question is ASCII. The character in question is 0xB1(plus/minus). My console font is Consolas, which can handle that character. TCC handles that character rather inconsistently, displaying it in at least 4 different ways.

1580491043682.png
 

rconn

Administrator
Staff member
May 14, 2008
12,315
148
I've mentioned a few dozen times in the past that you will never, ever be satisfied with the results if you convert extended ASCII characters to Unicode and then back to ASCII. (Or even worse, back and forth and back and forth like in your examples.) That's the way Windows works; what you get will depend on your codepage and your font, but it will almost never be what you want. If you're unhappy with the results you should be using UTF-16 or UTF-8 files. Or at the very least, UnicodeOutput or UTF8Output, and/or change your code page to 65001.

In a TCC console window, Windows handles the character display - all TCC does is pass the character and it's up to Windows how it appears.
 
May 20, 2008
11,285
95
Syracuse, NY, USA
I've mentioned a few dozen times in the past that you will never, ever be satisfied with the results if you convert extended ASCII characters to Unicode and then back to ASCII. (Or even worse, back and forth and back and forth like in your examples.) That's the way Windows works; what you get will depend on your codepage and your font, but it will almost never be what you want. If you're unhappy with the results you should be using UTF-16 or UTF-8 files. Or at the very least, UnicodeOutput or UTF8Output, and/or change your code page to 65001.
As I said in Charles Dye's thread about HEAD:

I have a 256-byte file (0255.bin) containing the bytes 0x0 through 0xFF. I wrote a test app to read that file into a buffer, print the decimal values of the bytes in the buffer, use MultiByteToWideChar followed by WideCharToMultiByte (with lpDefaultChar equal to NULL) on the buffer, then print the decimal values again.

I did that for the ANSI, OEM, and THREAD code pages.

In all three cases, the before/after decimal values were identical; i.e., the decimals 0 through 255.

I later did the same (successfully) with CP 866.

This thread started when I simply TYPE'd the file (no pipes). Here's what I saw/see.

1580508717692.png
 
May 20, 2008
11,285
95
Syracuse, NY, USA
Let's start over.

PlusMinusSign is Unicode 0xB1; supported by Consolas; not in my code page (437).
MediumShade is Unicode 0x2592; supported by Consolas; 0xB1 in code page 437. (see it below)

Rex, please explain why/how these are different.

1580532167486.png
 

Charles Dye

Super Moderator
Staff member
May 20, 2008
4,423
85
Albuquerque, NM
prospero.unm.edu
Pretty sure this is the same as AnrDaemon's issue. When HEAD/TAIL read from a pipe, they seem to assume that everything is Unicode. If you have 8-bit text, it becomes 8-bit Unicode — bytes zero-extended to words. So your 0xB1 becomes U+00B1, the plus-minus sign.
 
May 20, 2008
11,285
95
Syracuse, NY, USA
Pretty sure this is the same as AnrDaemon's issue. When HEAD/TAIL read from a pipe, they seem to assume that everything is Unicode. If you have 8-bit text, it becomes 8-bit Unicode — bytes zero-extended to words. So your 0xB1 becomes U+00B1, the plus-minus sign.
Yup! Using codepage 866 and piping to HEAD or TAIL clobbers the entire Cyrillic alphabet, uppercase and lowercase. HEAD and TAIL just seem to ignore codepages altogether (I have no idea why). In contrast, piping to TPIPE seems to respect the current codepage.
 

Charles Dye

Super Moderator
Staff member
May 20, 2008
4,423
85
Albuquerque, NM
prospero.unm.edu
Yup! Using codepage 866 and piping to HEAD or TAIL clobbers the entire Cyrillic alphabet, uppercase and lowercase. HEAD and TAIL just seem to ignore codepages altogether (I have no idea why). In contrast, piping to TPIPE seems to respect the current codepage.

Only from a pipe, though. Using a |! pseudopipe prevents the problem. I think there's a missing MultiByteToWideChar() somewhere.
 
Similar threads
Thread starter Title Forum Replies Date
A Inconsistent return code for TCC scripts Support 4
rconn News Take Command / TCC / CMDebug / TCC-RT v28 Released Support 0
C How to? starting TCC/TCMD v25 Support 2
CWBillow TCC and TCMD in Powershell Support 6
rconn News Take Command / TCC / CMDebug / TCC-RT 27.01.24 uploaded Support 0
rconn News Take Command / TCC / CMDebug / TCC-RT 27.01.23 uploaded Support 0
Alpengreis Fixed Crash after copy dialog with big TCC.exception.log Support 5
rconn News Take Command / TCC / CMDebug / TCC-RT v27.01 Build 22 Uploaded Support 0
L FTYPE in TCC, less quirky than in cmd.exe Support 3
V Set path permanently from within TCC? Support 3
DrusTheAxe Runaway TCC.exception.log filling drive Support 5
rconn News Take Command / TCC / CMDebug / TCC-RT 27.0.21 uploaded Support 0
Joe Caverly tcc.exception.log Support 3
rconn News Take Command / TCC / CMDebug / TCC-RT 27.0.20 uploaded Support 0
rconn News Take Command / TCC / CMDebug / TCC-RT 27.0.19 Support 0
FreezerBurnt Help making a CMD and TCC compatible batch file Support 7
vefatica SCRIPT ... TCC disappears Support 2
rconn News Take Command / TCC / CMDebug / TCC-RT v27.0.18 Support 0
Jesse Heines Access Denied errors in TCC 27 Support 9
fpefpe How to? tcc inifile (%_ininame) Support 3
M How to? Black TCC Prompt Support 2
D Differences between TCC and CMD with COPY /b Support 4
M Oracle batch OPatch.bat.bat not working in TCC but correctly in CMD Support 5
Jay Sage Extra TCC Tasks Support 6
vefatica Can I turn off the minimize to tray mechanism in TCC? Support 5
N Command execution slows down in TCC Prompt after a while Support 6
R Upgrade TCMD 19 to TCC 26 Support 1
Joe Caverly Set a batch variable (%1 - %n) when TCC is executing a batch file Support 5
samintz How to? Is this doable in TCC? Support 2
rconn News Take Command / TCC / CMDebug / TCC-RT 26.02.41 Uploaded Support 0
D TCC tab detaching & restarting Support 3
Ó Recent git output not shown on TCC Support 1
P Kill all other instances of tcc from a btm batch file Support 2
L Using TCC.exe through an SSH connection Support 3
M TCC color issue in ConEmu Support 4
Alpengreis UTF-8 problem in TCC related to Python Support 7
Alpengreis Python support for 3.8.3 in TCC v26.x? Support 2
rconn News Take Command / TCC / CMDebug / TCC-RT 26.01.40 uploaded Support 0
Peter Murschall TCC V26.01.3x Debugging has problems with Alias /R Support 1
rconn News Take Command / TCC / CMDebug / TCC-RT 26.0.37 Uploaded Support 0
vefatica TCC waits for Office apps Support 4
Joe Caverly .BAT works in TCC-RT 25, but not in TCC-RT 26 Support 4
M Strange error messages from TCC in FTP copy Support 7
Kachupp Documentation tcc-rt versions Support 1
V Comments from a first-time user (TCC 26) Support 6
rconn News Take Command / TCC / CMDebug / TCC-RT 26.0.28 Support 0
C TCC 26 BATCOMP BTM problems? Support 5
E TCC/CMD difference Support 5
rconn News Take Command / TCC / CMDebug / TCC-RT Public Beta RC2 Support 0
rconn News Take Command / TCC / CMDebug / TCC-RT v26 Public Beta build 26.0.21 Support 0

Similar threads