Fixed ParameterChar in TCMD.INI no longer working

[Detlef]

I am using
ParameterChar=ø
in the TCMD.INI file

My BTM files use for example this
%mfEXE\COBOL %xN %1ø %xCOpt GNT NOANIM;

This does not work since upgrading to v17. I also changed the ParameterChar to another character ° but this did not work either.

I had to switch back to v16 as thousands of my Batch files don't work any longer.

 

[rconn]

Not reproducible here.

What exactly do you mean by "This does not work" -- is it returning an error? The wrong result?

Are you sure you're using the correct TCMD.INI file? What does SETDOS show?

Can you post a small batch file that demonstrates the problem?

 

[vefatica]

It seems OK here but I noticed something else funny. I use CP 437, and I used char 0xF8 (Latin small letter O with stroke) which looks life the one detlef used.

That's what I see if I look at my INI file with notepad. The same goes for this one-line BTM.

echo %ø

And in both cases, a hex editor confirms it's 0xF8. And, as expected,

v:\> echo %@ascii[%@char[0xf8]]
248
v:\> echo %@char[0xF8]
ø

But LIST, TYPE, and @LINE display a different character.

  args.bat  │ F1 Help │
0000 0000 65 63 68 6f 20 25 f8 0d  0a  echo %°..

v:\> type args.bat
echo %°

v:\> echo %@line[args.bat,0]
echo °

v:\> echo %@ascii[%@line[args.bat,0]]
101 99 104 111 32 176

And VIEW shows something completely different.

 

[Detlef]

@vefatica, thanks for details.

Hi rconn,
I will create the sample batch file later tonight and prepare screen shots from both, v16 and v17 that show the different behaviour.
Regards,
Detlef

 

[vefatica]

In hex mode, VIEW shows it correctly.

 

[rconn]

It seems OK here but I noticed something else funny. I use CP 437, and I used char 0xF8 (Latin small letter O with stroke) which looks life the one detlef used.

Are you using a Unicode font? If not, extended characters will be converted internally (and usually changed) by Windows to Unicode.

And if you're putting this in an ASCII file, Windows will change the extended characters to whatever it thinks suits your selected code page.

 

[rconn]

But LIST, TYPE, and @LINE display a different character.

What does TYPE in CMD display?

 

[vefatica]

Are you using a Unicode font? If not, extended characters will be converted internally (and usually changed) by Windows to Unicode.

And if you're putting this in an ASCII file, Windows will change the extended characters to whatever it thinks suits your selected code page.

I'm using a Unicode font. It is an ASCII file. This routine prints it correctly.

    while ( ReadFile(hIn, pReadBuf, dwReadSize, &dwRead, NULL) && dwRead != 0 )
     {
       dwRead = MultiByteToWideChar(CP_ACP, 0, aBuf, dwRead, uBuf, BUF_SIZE);
       uBuf[dwRead] = 0;
       Printf(L"%s\r\n", uBuf);
     }

 

[rconn]

First, CP_ACP is a disastrous choice, which will only work in an *extremely* limited set of circumstances. TCC queries the current code page and uses that, which is the only way to support non-English & non-CP 437.

Second, if you believe that you can use extended characters from one code page in an ASCII file and Windows will automagically convert them to the correct Unicode characters from another code page, you're doomed to a life of sad disappointment. If you want Unicode characters, use Unicode files. ASCII files belong with the dusty fax machine at the back of your closet.

 

[vefatica]

It also works correctly if I use CP_THREAD_CP. I don't understand the need to change characters (sometimes).

 

[rconn]

It also works correctly if I use CP_THREAD_CP. I don't understand the need to change characters (sometimes).

CP_THREAD_CP is not just disastrous, it's catastrophically disastrous (Google it if you have a strong constitution). And then Google "code pages" and character sets. Then try to invent something that will map a character in one code page to another code page where that character doesn't exist. It's amazing that Windows occasionally *does* manage to get it "right" (whatever "right" means in this context).

And you obviously haven't tried it in CMD yet.

 

[vefatica]

CMD gets it wrong too. IMHO it belongs in "Duplicate DMC bugs".
It may be naive, but I think any given character should always look the same in a console if I haven't changed the console's code page or my locale.

 

[Detlef]

I have attached a ZIP file containing a word document with screen shots of my TestTCC.BTM file with Version 16 and Version 17. 16 Works, 17 does not. Also contained my TCMD.INI as well as the TestTCC.BTM itself.
Again 16 is working! There is no information that this parameters has been touched, changed or anything like this, so I expect it should work like before.

 

[rconn]

CMD gets it wrong too. IMHO it belongs in "Duplicate DMC bugs".
It may be naive, but I think any given character should always look the same in a console if I haven't changed the console's code page or my locale.

If CMD gets it "wrong", then Windows is getting it "wrong" too, because Windows is doing the translations, not CMD (or TCC). You could either pressure Microsoft to change their code page handling to suite your preference, or give up and accept it as-is.

I could change it in TCC to work the way you want, with the minor drawbacks that line drawing characters would no longer be supported, nor would any non-Latin language. And the Latin languages would only be supported for a single code page & character set. Again, if you can convince the 72% of TCMD users outside the USA to switch to US English and codepage 437 ...

 

[Charles Dye]

Ain't Windows character handling wonderful? You can have three different character sets active at the same time (Windows code page, console code page, Unicode BMP). Windows is constantly translating between them. Translation to Unicode works fine; translation to the other two can trash characters. Personally I would love to see all of the 8-bit stuff go bye-bye. But so long as CMD.EXE compatibility is a goal, that just isn't possible.

Incidentally -- anal-retentive pedantry here -- any file containing a character > 127 is ipso facto not an ASCII file.

 

[Detlef]

I dont get it. Sorry.
I am using this stuff since at least 15 years!! Worked from begin on up to version 16. And now everybody blames Microsoft, Code pages, Windows, Ascii, Wide Chars ........ anything else?
I am -- anal-retentive pedantry here -- it is a Bug in TCMD. Sorry. That is how I see it right now. Will this be fixed sooner or later?

 

[myarmor]

@Detlef, well it is true that the codepage is the reason, but there's probably some change
between versions that caused it to differ in v17.

I tried using OPTION to set it to 'ø', and that works..until you close TCC.
Closing TCC, then restarting it and it no longer works (while "°" do).
The value for ParameterChar=ø (4NT section), when viewed in w1252 codepage.

Running OPTION again, and "Parameter" contains a square instead of "ø" (or °).
The hex value for it became 0xFFB0 when I copied and pasted it into a UTF16-LE document.
(I used EditPad Pro's statusbar to see the value).

To test it I used this btm file with the commandline: hello world

@echo off
echo ø = "%ø"
echo ° = "%°"
echo $ = "%$"

and the line in the middle was the one that displayed text between the quotes.

 

[Detlef]

@myarmor, the codepage is a couple decades old and did not change. Given this I don't agree that the codepage is the reason.
But I can agree that a change between version 16 and version 17 makes a difference. And I call such things BUGS. And Bugs have to be solved.
I am myself a developer, my customers would not accept such answers. They would ask exactly for the cause of the disbehaviour and then they will expect an answer how long it takes to fix this problem.

As far as I can interpret the answers I am not feeling as a customer. I did not yet get an answer if and when this will be fixed? And if it does not get fixed, what is the workaround? And why can it not be fixed?

 

[myarmor]

@Detlef, I've been in that business myself for several years. It isn't that surprising that a new major version
have some gotcha's (bugs or whatnot), but they usually get fixed quite rapidly. At least if it is isn't behaving as intended.
This kind isn't easy to detect in QA (specific locale and all), so you can only rely on someone reporting it.

 

[vefatica]

I have confirmed that there's a difference between v16 and v17 here.

In v16, setting the parameter char to 0xF8 (Alt-0248) via the Option dialog works. That character is in the INI file, and it continues to work the next time v16 is started.

In v17 it also works right after being set, BUT it doesn't work the next time v17 is started (although the character is still the same (F8) in the INI file.

 

[vefatica]

I have confirmed that there's a difference between v16 and v17 here.

In v16, setting the parameter char to 0xF8 (Alt-0248) via the Option dialog works. That character is in the INI file, and it continues to work the next time v16 is started.

In v17 it also works right after being set, BUT it doesn't work the next time v17 is started (although the character is still the same (F8) in the INI file.

P.S. All that's true with CP 437 (my default default codepage, i.e., unspecified anywhere).

If I change the default CP for the console thus

Registry\HKCU\Console\Codepage = (REG_DWORD) 1252

then both v16 and v17 get/set ParameterChar correctly upon startup

 

[vefatica]

Furthermore, regardless of TCC version or codepage, revisiting OPTION\Advanced in the same session and, without making any changes, pressing "OK", immediately changes ParameterChar to '?' (0x3F) both in effect and in the INI file.

 

[vefatica]

Furthermore, regardless of TCC version or codepage, revisiting OPTION\Advanced in the same session and, without making any changes, pressing "OK", immediately changes ParameterChar to '?' (0x3F) both in effect and in the INI file.

And, somewhat to my surprise, using a Unicode INI file does not change that behavior.

 

[rconn]

I have attached a ZIP file containing a word document with screen shots of my TestTCC.BTM file with Version 16 and Version 17. 16 Works, 17 does not. Also contained my TCMD.INI as well as the TestTCC.BTM itself.
Again 16 is working! There is no information that this parameters has been touched, changed or anything like this, so I expect it should work like before.

The parameter code has not been touched or changed in v17. It wouldn't be surprising if v16 worked and v17 didn't (even though nothing changed in this code) just because of different configurations.

I need to know what code page you're using, and what keyboard language.

And is this in a TCMD tab window, or a stand-alone console window? (Because the character translations Windows does are completely different between GUI and console apps.)

 

[vefatica]

CP_THREAD_CP is not just disastrous, it's catastrophically disastrous (Google it if you have a strong constitution).

I did Google it and found surprisingly little and nothing outright negative. A few hits mentioned that MFC uses CP_THREAD_ACP (a change from CP_ACP).

 

[myarmor]

@rconn,
Same thing happens in both TC and TCC. TC wraps TCC, so that's given.

The parameter code itself doesn't appear to be the problem, but rather the code for reading/interpreting
the parameterchar value when read from the inifile.
The inifile's parameterchar value is stored correctly (when set to ø using OPTION, the value is ø when viewed as 1252).

The mismatch between 850 and 1252 can explain the difference between what is set (ø) using
OPTION, and what character works (°) after tcc is restarted. The fact that OPTION can't display
a proper value for Advanced/Parameter after restarting tcc also seem to indicate a CP issue between
reading/writing the inifile.

Unless you're using custom functions for writing the inifile, it is apparently written to using an ansi codepage (1252 etc).

Here's mine:
CP according to chcp: 850 - OEM Multilingual Latin 1; Western European (DOS)
Windows ANSI CP seems to be 1252 - ANSI Latin 1; Western European (Windows)
Keyboard: US, NO (US is main, switch with shift-alt).
TCC

 

[rconn]

I ran your batch file through the debugger in both v16 and v17, and there is a difference (and a bug) between the two versions. However, the bug is in v16 (actually in the RTL), not in v17 (which is using an updated RTL).

I made a change to build 59 (already uploaded) which should resolve your issue, though it does break compatibility with CMD.

I do have two questions:

1) Why are you using ParameterChar? It has been obsolete for at least 12 years; it was added many years ago to ease transition between 4DOS and 4NT batch files, and changing it now only makes your batch files incompatible with everybody else (and vice versa).

2) Why did you choose that particular character? It's one of a handful in that character set that don't translate between code pages. (Each code page and font combination have between 0 and 10 characters that are mangled by Windows during translation to & from Unicode; your 0xF8 is one of them.)

 

[vefatica]

Now it's remembered between sessions (I tested CP 437 and 1252). But with either CP, if you (ever) revisit OPTION\Advanced it's not displayed correctly, and pressing "OK" (with or without any changes) changes the ParameterChar to '?' both in the current session and the INI file.

 

[rconn]

Now it's remembered between sessions (I tested CP 437 and 1252). But with either CP, if you (ever) revisit OPTION\Advanced it's not displayed correctly, and pressing "OK" (with or without any changes) changes the ParameterChar to '?' both in the current session and the INI file.

Unavoidable -- the Unicode font used by Windows for that dialog doesn't have a character matching 0xF8. (Another reason why that character is a bad choice for ParameterChar.)

 

[vefatica]

Unavoidable -- the Unicode font used by Windows for that dialog doesn't have a character matching 0xF8. (Another reason why that character is a bad choice for ParameterChar.)

I tried a dozen randomly chosen other characters in 128 - 255 in that option dialog and for every one, when I revisited the dialog it appeared as a empty box and "OK" caused the character to be changed. Are there any in that range that will work, appear correctly in that dialog, and not be changed by "OK"?