WAD BOM printed on command line from BAT file

Feb 23, 2012
240
3
This is really more of a praise-for-TCC post than a bug report, but there is one small item to be addressed, as follows.
I often run .BAT files formatted in UTF-8. In general, my UTF-8 files have a BOM at the top. However, cmd.exe and powershell both balk at such files. Upon encountering the BOM, they throw out the whole first line of the .BAT file as unintelligible (even in cp 65001!). I thus find myself often going back into my editor and manually removing the BOM from these files in order to run them.
I was extremely pleased to find that TCC does not balk at these files at all, and does indeed execute all commands within the batch file as intended.
However, when running the batch file, TCC does still output a few garbage characters to the screen, corresponding to the characters that comprise the BOM at the start of the file. I believe that this is a bug; the BOM is meant only to indicate the format of the file, and should not be output to the screen as if it containing characters to be processed.
 
Nov 2, 2009
294
6
Chile
www.farah.cl
The BOM in a UTF-8-encoded file shouldn't be there in the first place.

Still, it's a ubiquitous bad practice in Windows apps (originated by Microsoft), so I concur with the opinion that robust applications should expect an extraneous BOM and discard it. Perhaps the TCMD debugger could even raise a flag about that...
 

rconn

Administrator
Staff member
May 14, 2008
12,344
149
I often run .BAT files formatted in UTF-8. In general, my UTF-8 files have a BOM at the top.

This is a feature request, not a bug report.

TCC only has limited UTF-8 support (albeit more than CMD or PowerShell). Because Windows has almost no internal UTF-8 support, it has to be coded individually for each command / usage. Over the past 15 years, we've had exactly one request for UTF-8 batch files (yours). It will require a LOT of extra coding (at least several hundred lines of code scattered through several dozen source modules), so you're not likely to see it very soon (unless a few hundred other users chime in and start clamoring for it too!).

I can probably at least filter the BOM marker, but actually converting the UTF-8 file to the internal UTF-16 representation won't happen immediately.
 
Feb 23, 2012
240
3
Hi Rex,
Here's a batch file that executes the command "dir", with a UTF-8 BOM at its start.
I appreciate your swift responses and solutions, as well as your explanation about the difficulties of adding UTF-8 support throughout.
- Avi
 

Attachments

  • test2.bat
    6 bytes · Views: 107
Feb 23, 2012
240
3
Hi Rex,
Thank you for the explanation. I tried now running my batch file in UTF-16 (with BOM) and found that it runs perfectly in TCC. What I'm slowly learning from our exchanges here is that if I just go with UTF-16 instead of UTF-8, all will be well in TCC. This is great!
You seem surprised that I use UTF-8 files for my multi-language work, and, as you've noted that "Windows has almost no internal UTF-8 support". Indeed, from a programming perspective, for instance, the MS Visual suite works almost exclusively in wchars which hold a UTF-16 representation. However, on the flipside, when it comes to text files, my experience has been that most of the programs that I use output UTF-8 formatted files. Indeed, in the MS .NET libraries, the default setting for the streamwriter CLR class is to write files with a UTF-8 representation.
Similarly, when multi-language email messages arrive they are almost always in UTF-8. And when I export a list of files from voidtool's everything, it arrives in UTF-8.
Because of this situation, whenever I work with multilanguage files, my format of choice is UTF-8. And so, as I've been learning TCC, I've been trying each feature with UTF-8. On the other hand, if I can accomplish my objectives more consistently in TCC by using UTF-16 instead, I'll be happy to do so - I'll simply convert any given file to UTF-16 before processing it. I do hope that one day TCC will add full UTF-8 support across the board, so that any file input will be converted on the fly from UTF-8 to the internal UTF-16 as needed; however, as long as UTF-16 files are supported, I'll refrain from pushing the UTF-8 issue further.
- Avi

TCC only has limited UTF-8 support (albeit more than CMD or PowerShell). Because Windows has almost no internal UTF-8 support, it has to be coded individually for each command / usage... I can probably at least filter the BOM marker, but actually converting the UTF-8 file to the internal UTF-16 representation won't happen immediately.
 
Similar threads
Thread starter Title Forum Replies Date
vefatica TYPE goes crazy with no-BOM Unicode file Support 7
vefatica LIST and TYPE show UTF8 BOM Support 4
Charles Dye BOM when UnicodeOutput=Yes Support 11
Stefano Piccardi UTF-8 BOM marker @iniread failure Support 6
Stefano Piccardi detecting BOM, FFIND multibyte regex Support 18
B Printed Take Command Manual Support 6
Dmitry L. Kobyakov How to? How to remove the record of the old version of the Take Command? Support 5
D Can alias pass parameters to for command? Support 3
samintz How to? Search for %(command) in Help? Support 1
T VIEW only works from command line Support 14
rconn News Take Command / TCC / CMDebug / TCC-RT v28 Released Support 0
T Take Command 28: Lua reports version 5.4.0 (5.4.3 expected) Support 0
Jay Sage WAD Possible Bug With OPTION Command With @FILE Support 5
rconn News Take Command / TCC / CMDebug / TCC-RT 27.01.24 uploaded Support 0
J Take command does not start on Windows 10 Support 3
vefatica `Back quotes` - command line vs. batch file Support 5
Jay Sage WAD Failure of Alias Command with /Z Option Support 14
Phileosophos Is there any way to open the Take Command options dialog from the command line? Support 8
Phileosophos Command-line Editing Shortkeys That Fail Support 6
fpefpe How to? free (internal) command Support 0
rconn News Take Command / TCC / CMDebug / TCC-RT 27.01.23 uploaded Support 0
T How to? retain command history across reboot? Support 6
rconn News Take Command / TCC / CMDebug / TCC-RT v27.01 Build 22 Uploaded Support 0
Jay Sage Bug With Flow Control and/or Command Grouping Support 2
GermanDirk How to? EVENTLOG Command with simple User Rights not functional Support 4
Jay Sage Take Command Is Too Fast! Support 4
Jay Sage Comparison of IF with Command Group and IFF Support 6
D Open Windows 10 Photos app from command line Support 11
rconn News Take Command / TCC / CMDebug / TCC-RT 27.0.21 uploaded Support 0
rconn News Take Command / TCC / CMDebug / TCC-RT 27.0.20 uploaded Support 0
U Can not get/start Take Command gui Support 1
rconn News Take Command / TCC / CMDebug / TCC-RT 27.0.19 Support 0
S Increasing Font Point Size in Command Tabs Support 2
mfarah Fixed The DATE command produces the wrong error message. Support 1
vefatica Command name mix-up Support 3
FreezerBurnt "Missing SETLOCAL" when initializing Visual Studio Command Prompt. Support 9
E How to? Closing desktop created by DESKTOP command Support 10
rconn News Take Command / TCC / CMDebug / TCC-RT v27.0.18 Support 0
K Take Command fox XP? Support 7
Joe Caverly Documentation Replace internal command with LIBRARY command Support 0
fpefpe How to? command echo in title line/bar Support 2
U Command help on file names Support 5
vefatica Command line DO with no closing parenthesis? Support 9
vefatica Command history search is off Support 7
H How to? How can I say "if the command separator is not ~ then setdos /C~" Support 8
N Command execution slows down in TCC Prompt after a while Support 6
R Long commands retrieved from command history have blank line when wrapped Support 20
jpeni Take Command Language Examples Support 1
rconn News Take Command / TCC / CMDebug / TCC-RT 26.02.41 Uploaded Support 0
fpefpe winfr command Support 4

Similar threads