Fixed Using codepage 65001 (UTF-8) breaks non-ASCII characters

May 30, 2008
80
0
Gerakas, Greece
When typing any non-US-English character, an empty character (probably 0x00-NUL) is displayed on the right of the typed character. It doesn't overwrite the underlying character (if any), but the string is displayed as if you were pressing `space` after each. The weirder part is that this only manifests itself when typing the non-US English character in two separate lines (which leads me to believe this is a problem with the output stream)

1. Open TCC 20 (either standalone or under Take Command).
2. Type `chcp 65001` to switch to the UTF-8 "codepage".
3. Use any non-US English keyboard (for my example, I'm using Greek, but I've done this with Russian and German keyboards).
4. Type any character that is not 32-127 (my example: τεστ - which means "test" in Greek) and press enter (or type `echo τεστ` and enter, to verify it's not the error stream that has the problem).
5. Repeat step 4.

Actual results:
Ok6czXw.png


Expected results:
(This worked with TCC 19)
GRE3vck.png


Notes:

I've done this with empty settings (they were generated from scratch), just to make sure it wasn't some setting that messed things up.
 
May 30, 2008
80
0
Gerakas, Greece
BTW, isn't the version supposed to be 10.0.14393? Why does TCC go to the fallback version (6.3)?
 
May 30, 2008
80
0
Gerakas, Greece
If it's a Windows issue, why does it work with TCC 19 and it doesn't work with TCC 20?

EDIT: This happens exactly the same under Take Command (I just wanted to isolate the problem).

QznvYHa.png
 
Last edited:

rconn

Administrator
Staff member
May 14, 2008
12,340
149
If it's a Windows issue, why does it work with TCC 19 and it doesn't work with TCC 20?

I keep telling people that Windows does not actually support UTF-8 (other than in a handful of conversion APIs), but nobody wants to listen ...

The reason it behaves differently in v19 vs. v20 is because v20 is using different APIs to fix a different problem with Take Command -- Windows uses different code pages in GUI windows and console windows, and v20 is going to great lengths to try to rationalize those differences.

The reason you're seeing blanks in the output is because TCC is querying Windows for the width of the characters, and Windows is returning "2". I could add a hack to check for codepage 65001 and always assume they're really single-width characters, though that will break Japanese / Korean / Chinese support.

The question I have for you is -- why are you using 65001? Do you think it will provide some benefit?
 
May 30, 2008
235
2
The question I have for you is -- why are you using 65001? Do you think it will provide some benefit?

I use 65001 (but only temporarily, not persistently in a TCC session) when outputting UTF-8 text. Have worked fine so far.
 
May 30, 2008
80
0
Gerakas, Greece
The question I have for you is -- why are you using 65001? Do you think it will provide some benefit?

Well, yes, it does. It's more-or-less required if you're working Python on the console.

I keep telling people that Windows does not actually support UTF-8

Well, it seems to be working in PowerShell/CMD.
 
Similar threads
Thread starter Title Forum Replies Date
Jesse Heines Using Regular Expressions with the REN commanc Support 8
R WAD Unusable state when using Chinese characters Support 3
Joe Caverly Using TYPE with non-English text Support 22
L Using TCC.exe through an SSH connection Support 3
Joe Caverly Using @PSHELL from @EVAL Support 8
Jay Sage Command Will Not Run Using Short Name of Path Support 5
Jay Sage Cannot Postion Cursor in Command Line Using Mouse Support 7
MikeBaas Using the debugger Support 2
Joe Caverly Multiple Text Searches at once using FFIND or TPIPE Support 4
Dick Johnson Using the @instr function Support 5
Joe Caverly Using this CMD technique from TCC Support 17
Joe Caverly What version and Windows OS are you using? Support 3
Craig Fitzgerald Problems using tcstart.btm Support 3
R How to? Append files in multiple subfolders using copy? Support 8
M Ctrl-C when using command line history does nothing Support 2
Joe Caverly Using a Directory Alias with @iniwrite fails Support 14
C French accents using msgbox Support 4
C "Failed to update the system registry. Please try using REGEDIT" Support 3
x13 Problem listing repository files using DIR http(s)://... Support 8
R Regex using ^ Support 2
cxxl WAD Mouse movement sluggish when using TCC list Support 3
jfalch WAD internal "which" should check for FILEEXIST when using AppPaths entry Support 8
D How to? Scale font in TCC using wheel mouse or other means Support 2
fpefpe Documentation copy/move using /c or /u Support 2
D How to? Use typed envars using regex. Support 3
epement Using "everything" Support 10
U Installing the Compaq Visual Fortran 6.0 compiler using Take Command Support 8
Alexander WAD The number of files and dirs are multiplied when using multiple wildcards Support 6
T using bottom line for status bar Support 3
A How to? How do you launch TCC/LE using the already-open instance? Support 5
rps Fixed Using last argument variable ! Support 5
M Incorrect ARGV1 when using linux-style paths Support 4
D How to? Using a custom ini file with TCC from Take Command Support 7
thedave WAD Unable to type various characters using Windows 8's onscreen keyboard Support 7
D Documentation Using "send all input to other tabs" Support 11
Joe Caverly Word Count using TPIPE Support 10
Phileosophos Fixed Option Dialog Hangs when Using Help Support 7
MickeyF problem using COM object in VBScript from v16 x64 TCC but not from v15 32-bit TCC Support 4
S How to? How do I distinguish missing source file from download error when using HTTP? Support 6
Jay Sage Using EVERYTHING Command in TCMD15 Support 5
Charles Dye Odd behavior after using OPTION "Advanced" tab Support 2
T How to? Using FOLDERMONITOR 1 event causes multiple trigger(ing)s Support 8
samintz WAD FILEWRITE using SMOPEN Support 1
ehab aboudaya How to? 2 GUI questions and return HWND using TakeCommandIPC Support 3
R Fixed Using %~dp1 for paths with unavailable drives Support 6
C Using dif. sending addresses with sendmail Support 11
R TC 13 / Using TCDIALOG to exit commands Support 4
epement Video tutorials of using Take Command Support 2
electrotype Error using TCC: "Label not found" Support 6
Phileosophos unzip not using specified path Support 5

Similar threads