LIST and TYPE show UTF8 BOM

May 20, 2008
11,388
99
Syracuse, NY, USA
[Inspired by another thread] It's no big deal to me but ... LIST shows the 3-byte UTF8 BOM (whereas it doesn't show a Unicode BOM). Ditto for TYPE. VIEW does not show the BOM in either case.
 
[Inspired by another thread] It's no big deal to me but ... LIST shows the 3-byte UTF8 BOM (whereas it doesn't show a Unicode BOM). Ditto for TYPE. VIEW does not show the BOM in either case.
In LIST you can use the /8 switch to display the file correctly. I don't see a way to do that with TYPE though.

--
Howard
 

rconn

Administrator
Staff member
May 14, 2008
12,340
149
[Inspired by another thread] It's no big deal to me but ... LIST shows the 3-byte UTF8 BOM (whereas it doesn't show a Unicode BOM). Ditto for TYPE. VIEW does not show the BOM in either case.

Windows doesn't support UTF8 in any meaningful way (and UTF8 BOMs are deprecated at best and actively discouraged in practice). Try it with CMD -- it cannot handle a UTF8 file at all.

LIST and VIEW have the ability to display UTF8 files, but since you cannot actually *do* anything with them in Windows there isn't any point in extending it further.
 
May 30, 2008
67
1
I'd say UTF8 BOM isn't deprecated, but it might be discouraged in certain filetypes, but then in exchange for other ways of telling the same (such as in xml and html)..
The reason is that without the BOM (or similar things such as html charset and xml encoding etc) there is no way to determine 100% accurate whether the file
uses utf8 or an ansi codepage (the latter is often still the case).

Do with them? as in provided by the OS?
Maybe not, but then for most people the OS just sits there, the actual programs people tend to use is more than those. and those often do support utf8 (apparently more often
than not nowadays)
 

Charles Dye

Super Moderator
Staff member
May 20, 2008
4,447
88
Albuquerque, NM
prospero.unm.edu
Windows doesn't support UTF8 in any meaningful way (and UTF8 BOMs are deprecated at best and actively discouraged in practice).

I've been reading the Unicode standard, and I don't see that UTF-8 BOMs are deprecated, required, forbidden, discouraged, or encouraged. You can use a BOM at the start of a UTF-8 file or not; they're valid but not mandatory. Chapter 16.8 of the Standard says:

In UTF-8, the BOM corresponds to the byte sequence <EF BB BF>. Although there are never any questions of byte order with UTF-8 text, this sequence can serve as signature for UTF-8 encoded text where the character set is unmarked.

It seems to me that LIST /8 already does the right thing -- ignores the initial BOM. It would be nice if LIST checked for a UTF-8 BOM at the start of a file and switched to UTF-8 mode automatically. I know that you're tired of LIST and would rather leave further development to VIEW, but detecting a three-byte signature shouldn't be terribly difficult....
 
Similar threads
Thread starter Title Forum Replies Date
dcantor LIST /X and TYPE /X give different results Support 20
FreezerBurnt ANSI Prompt messes up list output. Support 39
Phileosophos PDIR to list filenames without extensions? Support 2
T Is there a way to maintain the position in the environment list, while debugging? Support 10
Alpengreis List command: text truncated in find box (german) Support 9
L LIST's pop-up search loses focus in a Windows Terminal window Support 15
A List windows with titles matching given pattern? Support 2
C How to make list of drive contents when shutdown PC / Everything.exe Support 7
D _USBS list misses flash drives that @USB recognizes. Support 7
Phileosophos LIST edit not invoking my specified editor Support 4
cxxl WAD Mouse movement sluggish when using TCC list Support 3
Oz Solomon Problem with "list" Support 14
P Strange mouse behavior with list Support 2
T Invalid attach tabs list Support 10
M A list of issues .. Support 17
J Documentation New help system: "skip word list" Support 11
WavSlave Fixed LIST /U not working in TCC v19 Support 5
R V19 - At startup, TCC save directory in the directory history list Support 2
C cosmetic issue with LIST's (I) Support 9
Stefano Piccardi list of TCC commands for external editors Support 4
vefatica Get a list of all files with alternate streams? Support 11
Joe Caverly LIST /T (search for Text) Support 2
Joe Caverly ListColors for LIST Support 1
M How to? List path getting short file and directory names... Support 10
C How to? List empty folders Support 21
S How to? Use wild cards in include list without duplicate processing Support 49
vefatica Fixed No text in List View! Support 4
D Folder/List view changed in v15 Support 8
S list seems to loose focus v15 Support 2
A How to? Filter history list with unicode chars Support 0
N WAD Take Command and UNCs for Folder List Support 10
S TCMD No List-View with Libraries? Support 5
M Sync folder/list view with command window? Support 2
vefatica New directory not moved to end of list Support 2
M Problem with the "List" command... Support 19
vefatica DO f in /L <list of file names>? Support 6
C list jpsoft file Support 2
A How to? Filter a list by numeric number within filename Support 28
M Just an out-of-curiosity question the List command vs. the @Lines function... Support 13
Dmitry L. Kobyakov Fixed The Take Command File List Panel shows 0 for all file sizes at DVD Support 4
M WAD A case where the "List" command is clearly not working at all... Support 4
Joe Caverly LIST scrollbar Support 2
R TC 13 / you forgot to list a new feature Support 2
S BDEBUGER: watch list limit? Support 4
T "L" (LIST) not working in TCC's examples.btm Support 3
rconn Suggestion list for next major version Support 0
M FFIND and LIST does not work for UTF-8 fles Support 10
D LIST command wierdness with empty Unicode file Support 1
JohnQSmith List View refresh Support 6
S Using TCC list with KatMouse Support 9

Similar threads