WAD A bit of strangeness related to Unicode-marked file not being Unicode

May 24, 2010
855
0
Northlake, Il
I issued the following command with the "Unicode" option checked:
Code:
PDir * /Nj /a-d /s /(dy/m/d th:m:s fpn) >Z-Drive Files.V2012-02-18.txt
Here is what the "List Z-Drive Files.V2012-02-18.txt" command shows in hex ("x" option):
Code:
0000 0000 ff fe 32 30 31 32 2f 30  32 2f 31 37 20 32 33 3a   ■2012/02/17 23:
0000 0010 35 38 3a 32 38 20 5a 3a  5c 56 69 6e 79 6c 20 43  58:28 Z:\Vinyl C
I don't really think that I have to point out that the "Unicode marker" (0xfffe) is there but the remainder of the file is not Unicode.

And the output is "perfectly normal" when issuing the above command when the "Unicode Output" option is not selected:
Code:
0000 0000 32 30 31 32 2f 30 32 2f  31 37 20 32 33 3a 35 38  2012/02/17 23:58
0000 0010 3a 32 38 20 5a 3a 5c 56  69 6e 79 6c 20 43 61 66  :28 Z:\Vinyl Caf
I'm not absolutely sure at this point that the "Unicode Output" option was checked when I issued the first "PDir" command above, although I am about 90% sure that it, in fact, was, although I don't think it really matters all that much given the below.

Strangely enough the commands "Type Z-Drive Files.V2012-02-18.txt | List" displays output that is "valid" Unicode (although the "list" command did not at least show the 0xfeff marker) when the "Unicode Output" option is selected:
Code:
0000 0000 32 00 30 00 31 00 32 00  2f 00 30 00 32 00 2f 00  2 0 1 2 / 0 2 /
0000 0010 31 00 37 00 20 00 32 00  33 00 3a 00 35 00 38 00  1 7   2 3 : 5 8
And typing Z-Drive Files.V2012-02-18.txt to a file using redirection produces a file that is completely "valid" Unicode (i.e., with the 0xfffe "marker") when the "Unicode Output" option is selected:
Code:
0000 0000 ff fe 32 00 30 00 31 00  32 00 2f 00 30 00 32 00  2 0 1 2 / 0 2
0000 0010 2f 00 31 00 37 00 20 00  32 00 33 00 3a 00 35 00  / 1 7   2 3 : 5
And typing the (incorrect) file to the "List" command produces:
Code:
0000 0000 32 30 31 32 2f 30 32 2f  31 37 20 32 33 3a 35 38  2012/02/17 23:58
0000 0010 3a 32 38 20 5a 3a 5c 56  69 6e 79 6c 20 43 61 66  :28 Z:\Vinyl Caf
which is only different from the "original" file in that the 0xfffe marker is missing.

And, finally, typing the "incorrect" file to the "List" command with the "Unicode Output" option unchecked produces a perfectly valid file in that the Unicode marker is now missing:
Code:
0000 0000 32 30 31 32 2f 30 32 2f  31 37 20 32 33 3a 35 38  2012/02/17 23:58
0000 0010 3a 32 38 20 5a 3a 5c 56  69 6e 79 6c 20 43 61 66  :28 Z:\Vinyl Caf
And typing the "incorrect" file to another file with the "Unicode Output" option" again unchecked produces a perfectly-valid file that is identical to the "incorrect" file other than the fact that the Unicode "marker" (0xfffe) is gone which means, of course, that listing it in "hex" mode produces exactly the same results as immediately above.

You might think from the above that it doesn't really matter if the Unicode marker is there on a file that is not really Unicode because the "Type" command (somewhat strangely, in my opinion) handles it "properly"(?) anyway. Well, while that certainly seems to be true for the "Type" command, is is absolutely not true for the "Find" command, which is how I discovered it. You see, the output of a "Find" command immediately following the PDir command was total garbage.

The truth is that this is not really a major problem in that I've found several ways to "work around" it, but I still think that it's a bug worth investigating.

- Dan
 

rconn

Administrator
Staff member
May 14, 2008
12,345
150
I issued the following command with the "Unicode" option checked:
Code:
PDir * /Nj /a-d /s /(dy/m/d th:m:s fpn) >Z-Drive Files.V2012-02-18.txt

I assume that's not really the command you issued, since you're missing the required double quotes and what you'd actually create is a file called "Z-Drive".

I think it's highly unlikely that there is a bug in the Unicode output code -- it's (very) heavily used by nearly everybody and the code itself has no way of writing the header without also writing everything else in Unicode. (And TCC doesn't know anything about redirected output to a file versus redirected output to a pipe; it's all exactly the same code.)

I tried all of your steps here and couldn't reproduce any problems.
 
May 24, 2010
855
0
Northlake, Il
Rex, I won't argue with you about "reproducing the problem" because I did something just slightly different and got the correct results; and now I'm not sure exactly what I did the first time to get those results. (I have the original file as it came out of a PDir command if you want me to send it to you to look at, and I really don't know how I would have made a file like that "manually" without a hex editor (which I have, somewhere, but it's probably been years since I last had a need to use it). But the "embedded white space" in the file name was just a transcription error because I don't see so good. (And do you really think that I made that all up?)

- Dan
 
Similar threads
Thread starter Title Forum Replies Date
M A little bit of strangeness with @Char... Support 3
Peter Murschall Single-line Do-CMD is a bit uncooperative. Support 6
Joe Caverly VBEEP on 64-bit Support 3
vefatica SETP usually fails with a 32 bit process Support 4
rconn Dropping 32-bit support in Take Command & TCC? Support 14
dcantor How to? Can 32-bit TCC be run on a system with 64-bit TCMD and TCC installed? Support 6
T 32 and 64 bit simultaneous portable versions Support 2
vefatica Make FFIND a bit more friendly? Support 14
CWBillow Everything.exe - 64-bit? Support 8
S 32-bit Take Command v22 install for thumb drive Support 1
Per TCC/LE 14 64-bit won't start on Windows 10 Insider Preview 17063 (171213) Support 12
Joe Caverly SETP and 32-bit process Support 2
gworley How to? Take Command 20 64 bit vs 32 bit Support 1
mikea Documentation Consider expanding the docs for 'Everything' a bit Support 10
T 64 bit TCCLE appears to crash when opening tcc.exe from within tcc.exe window Support 7
vefatica Can a subroutine return a 64-bit integer? Support 4
M 64-bit plugins? Support 1
M An oddity that's a little bit scary... Support 6
rconn News Take Command 16.03.54 32-bit installer Support 0
rconn News Take Command 16.03.54 32-bit installer fix Support 0
MickeyF problem using COM object in VBScript from v16 x64 TCC but not from v15 32-bit TCC Support 4
JohnQSmith Installing TCMD16 on 32 bit XP Support 12
D New 64-bit install goes to Program Files x86 Support 3
F How to install 64-bit after having installed 32-bit on Win7 Support 2
Dan Glynhampton Bad link to 64 bit RC1 download Support 0
M And oddity re the 32-bit TCC on a 64-bit system... Support 4
C Advantages of 32 or 64 bit TCMD in 64 bit Windows 7 Support 3
C How to determine if system is 32 or 64 bit? Support 5
M How to? Identify 64-bit and 32-bit TCC sessions... Support 7
M A bit of a complaint regarding @FileDate and @FileTime Support 3
K_Meinhard Take Command v13 64-bit Support 9
K_Meinhard 64-bit installer Support 3
M Another bit of weirdness.... Support 0
J CTRL-C does not work on Windows 7 64-bit Support 3
S Take Command LE (32 bit) locking up several times a day Support 14
S 64-bit version use? Support 5
S TCMD 12 64-bit locking up frequently Support 7
rconn v12 Release Build test - 32-bit fixed Support 36
Ville 64-bit command line apps invisible in TCC Support 2
gschizas TaskDialog doesn't work in 64-bit Take Command Support 1
D Settings not writting to registry in XP (32 bit) Support 10
D Fixed 20.10 strangeness with external commands Support 9
Jay Sage "New Tab" Strangeness Support 2
M Just an argument-passing "strangeness" that I didn't expect... Support 19
R TC9 FTP strangeness - solved - for KB only Support 1
Alpengreis UTF-8 problem in TCC related to Python Support 7
Jay Sage Help Correction (and Related Question) Support 0
M How to? On an issue closely related to the issue in my previous thread... Support 4
M How to? A seemingly-stupid question related to the "Shift" command. Support 2
M A somewhat humorous minor-request related to "Shift" command... Support 4

Similar threads