Unicode screw-up in IDE

May 20, 2008
11,377
98
Syracuse, NY, USA
Try this.

ECHO foo > foofoo.bat
BDEBUGGER foofoo.bat

In the editor, I see 5 "unprintables".

If I change that experiment to

ECHO foooooooooooooo > foofoo.bat
BDEBUGGER foofoo.bat

it appears correctly in the editor. I'm guessing that in the first case, the file is wrongly identified as Unicode (which is odd since QueryIsFileUnicode() seems to get it right).
 
May 20, 2008
11,377
98
Syracuse, NY, USA
Sorry, make that experiment

Code:
ECHO echo foo > foofoo.bat
bdebugger v:\test.bat
Try this.

ECHO foo > foofoo.bat
BDEBUGGER foofoo.bat

In the editor, I see 5 "unprintables".

If I change that experiment to

ECHO foooooooooooooo > foofoo.bat
BDEBUGGER foofoo.bat

it appears correctly in the editor. I'm guessing that in the first case, the file is wrongly identified as Unicode (which is odd since QueryIsFileUnicode() seems to get it right).
 

rconn

Administrator
Staff member
May 14, 2008
12,340
149
Try this.

ECHO foo > foofoo.bat
BDEBUGGER foofoo.bat

In the editor, I see 5 "unprintables".

If I change that experiment to

ECHO foooooooooooooo > foofoo.bat
BDEBUGGER foofoo.bat

it appears correctly in the editor. I'm guessing that in the first case, the file is wrongly identified as Unicode (which is odd since QueryIsFileUnicode() seems to get it right).

The debugger uses QueryIsFileUnicode to determine if the file is Unicode. As we've discussed several times in the past, very short (and in this case useless?) files are not reliably determined by the Windows IsTextUnicode API.
 
May 20, 2008
11,377
98
Syracuse, NY, USA
On Wed, 30 Dec 2009 20:04:03 -0600, rconn <> wrote:

|The debugger uses QueryIsFileUnicode to determine if the file is Unicode. As we've discussed several times in the past, very short (and in this case useless?) files are not reliably determined by the Windows IsTextUnicode API.

Did you see the corrected test? ... "ECHO echo foo > foofoo.bat".

I have a plugin FCONVERT (to/from Unicode). It only test is QueryIsFileUnicode,
and it apparently gets it right (while the IDE apparently doesn't). It would
seem there has to be more to the story.

v:\> echo echo foo > test.bat

v:\> type /x test.bat
0000 0000 65 63 68 6f 20 66 6f 6f 0d 0a echo foo..

v:\> fconvert test.bat testu.bat

v:\> type /x testu.bat
0000 0000 65 00 63 00 68 00 6f 00 20 00 66 00 6f 00 6f 00 e c h o f o o
0000 0010 0d 00 0a 00 . .

v:\> fconvert testu.bat test2.bat

v:\> type /x test2.bat
0000 0000 65 63 68 6f 20 66 6f 6f 0d 0a echo foo..
--
- Vince
 

rconn

Administrator
Staff member
May 14, 2008
12,340
149
> Did you see the corrected test? ... "ECHO echo foo > foofoo.bat".

Yes.


> I have a plugin FCONVERT (to/from Unicode). It only test is
> QueryIsFileUnicode,
> and it apparently gets it right (while the IDE apparently doesn't). It
> would seem there has to be more to the story.

Trust me, the IDE calls QueryIsFileUnicode. But your results are guaranteed
(by MS) to be highly erratic for anything < 16 bytes.

Having a CR/LF on the end of the file will change everything.

Rex Conn
JP Software
 
Similar threads
Thread starter Title Forum Replies Date
Peter Murschall TEE cannot handle Unicode output Support 2
B Fullwidth Unicode forms display incorrectly Support 5
T @execstr unicode support Support 6
Peter Murschall TPIPE generate unicode on Piping or redirecting Support 3
D Pasting Unicode data has different behavior on TCC and CMD Support 2
vefatica TYPE goes crazy with no-BOM Unicode file Support 7
Charles Dye TCC smashing Unicode quotes Support 9
Peter Murschall UNICODE mixed with ANSI Code Support 11
Joe Caverly Unicode, Codepage 437, and line characters Support 3
B How to? Convert Unicode to ANSI Support 1
StarliteLemming Fileread fails on Unicode file Support 10
vefatica DO ... /P ... and Unicode? Support 3
vefatica Unicode ... I don't understand Support 1
jadaml Echo unicode characters from UTF-8 Batch files? Support 1
Charles Dye @ASCII vs. @UNICODE Support 5
A How to? Filter history list with unicode chars Support 0
vefatica TYPE, Unicode, installer Support 10
A WAD Limitations on display of unicode punctuation marks Support 11
A Include lists and Unicode Support 1
M How to? How do I read a Unicode file through standard-input? Support 4
M WAD A bit of strangeness related to Unicode-marked file not being Unicode Support 2
M @CHAR and UNICODE Support 4
D LIST command wierdness with empty Unicode file Support 1
B Unicode/dword issue in TCC12 Support 4
J dir failure with some unicode characters Support 6
M TCC Unicode support? Support 7
vefatica BOMs in [dir]history / TAIL with Unicode Support 2
vefatica Unicode anomaly Support 0
vefatica Debugger now Unicode? Support 1
vefatica TYPE /X and Unicode files? Support 0
dcantor Convert ASCII to Unicode or vice versa? Support 6
H HISTORY and DIRHISTORY /R can't handle Unicode Support 0
R Reading an Unicode file with more than 8191 lines Support 1
R "The Specified Account Already Exists" - Can't Reinstall To Fix Protection Screw-Up of Last Week Support 8
Kachupp IDE watch references Support 11
T Thanks for the assistance with IDE vs Bdebugger Support 3
vefatica IDE external command syntax color? Support 1
K_Meinhard Small problem in german IDE 26 Support 3
vefatica IDE - when breakpoint is on a blank line? Support 3
Peter Murschall IDE: After undocking the command expansion Window is offended Support 2
Peter Murschall IDE: Ctrl-W is a Harakiri Command ! Support 23
Kachupp ide/bdebugger Support 17
rps IDE/BDEBUGGER ? Support 1
vefatica IDE.EXE's command line? Support 19
Peter Murschall IDE: RTL with non-English resources-problem is back in Build 28 Support 4
Peter Murschall v24 IDE/BDEBUGGER won't start Support 9
vefatica IDE stuck on "Insert tabs as spaces" Support 0
Peter Murschall IDE/BDEBUUGER in TCC 22 didn't run Support 7
T Fixed Problem with use of Batch parameters in the IDE Support 1
T Fixed IDE locks up if edit window for running batch is closed Support 6

Similar threads