DESCRIBE under TCC 11 doesn't work with diacritics

[Esteban]

Hello,

All is said in the title! I am talking about Take Command 11.0 build 40 under Win XP French.

When you enter diacritics (i.e. accentuated characters) from foreign languages (e.g. French éèêëàâùîïôöç...) using DESCRIBE, they are generally converted into question marks.

For instance the sentence:

"La poésie pâle et éphémère"
​

is changed into:

"La po?sie p?le et ?ph?m?re"
​

Sometimes, it is possible to correct these mistakes but as soon as the file is moved or renamed, the infamous bug comes back and kills the accents!

Don't know if other languages than French are affected...

Esteban

 

[rconn]

> All is said in the title! I am talking about Take Command 11.0 build 40
> under Win XP French.
>
> When you enter diacritics (i.e. accentuated characters) from foreign
> languages (e.g. French éèêëàâùîïôöç...) using DESCRIBE, they are
> generally converted into question marks.
>
> For instance the sentence:"La poésie pâle et éphémère"
>
> is changed into:"La po?sie p?le et ?ph?m?re"
>
> Sometimes, it is possible to correct these mistakes but as soon as the
> file is moved or renamed, the infamous bug comes back and kills the
> accents!
>
> Don't know if other languages than French are affected...

This is a font issue, not Take Command or DESCRIBE. Are you using a Unicode
font?

 

[Esteban]

This is a font issue, not Take Command or DESCRIBE. Are you using a Unicode
font?

Hello Rex,

Thanks for answering. I am using Lucida Console, which is not unicode to my knowledge. Anyway, opening the descript.ion file in notepad shows the same question marks. I'm afraid there is something other...

Esteban

 

[rconn]

> ---Quote (Originally by rconn)---
> This is a font issue, not Take Command or DESCRIBE. Are you using a
> Unicode font?
> ---End Quote---
> Hello Rex,
>
> I am using Lucida Console, which is not unicode to my knowledge.
> Anyway, opening the descript.ion file in notepad shows the same
> exclamation marks. I'm afraid there is something other...

Sounds like you have an ASCII description file instead of Unicode.

If you're using NTFS, it's easier to use the NTFS description option, which
won't have any problems with Windows transcribing extended ASCII characters.

 

[Esteban]

Sounds like you have an ASCII description file instead of Unicode.

If you're using NTFS, it's easier to use the NTFS description option, which
won't have any problems with Windows transcribing extended ASCII characters.

OK, but what about the compatibility with old descriptions? I've got thousands of files described in ASCII since 4DOS v3.0 (which I still use for bootable tools), no way to transcribe them all! What's more, NTFS descriptions are fundamentally not compatible with FAT12/16/32 partitions...

And there are some other utilities which also use the same feature, like XnView; you'll understand I really wish to keep compatibility with them all.

Here is perhaps a key to the problem:

• At first, when manually describing a file without description, all goes smoothly, the diacritics are correctly displayed with DIR, the descript.ion file content is fine.
• As soon as the file is renamed or moved, all those éèêëçàù... are changed to question marks.
• Now if from step 1 I try to add some information to a file already described using the alias select /zOu /A describe (%$) the added words are correctly spelled, while the previous ones are changed as soon as I hit Return. To be more specific, when editing the description for the second time, the previously entered accentuated characters are displayed as "?" while they are still correct inside descript.ion, until I press Enter. Unlike the previous alias, the DESCRIBE command alone, with the text inside the command line, works fine!

So there is at least a problem when reading/displaying diacritics from inside descript.ion by the DESCRIBE command, but it seems that all the support of ASCII diacritics is still available.

Hope I've been clear enough! :-) Feel free to ask me doing other tests if need be.

Esteban

 

[Steve Fabian]

I agree that using the traditional DESCRIPT.ION file has advantages.

Your problems may relate to the UnicodeOutput setting.

Since ASCII is a 7-bit code, accented characters have no
representation. Most users refer to the 8-bit code which has ASCII
as its first 128 codes as ASCII. Unfortunately it is "extended
ASCII", and the interpretation of the second 128 character codes
(which include all accented characters, box drawing characters, etc.)
depends on the code page selected, the font selected, the Windows
installation language, and they are different even when those factors
are the same between text-mode and graphics-mode programs. In
particular, when you have a TCC window in a TCMD tab, the display is
different from the same display when TCC has its own window. When TCC
has it own window, the LIST command display is also different from
the TYPE command display.
--
Steve

 

[rconn]

> OK, but what about the compatibility with old descriptions? I've got
> thousands of files described in ASCII since 4DOS v3.0 (which I still
> use for bootable tools), no way to transcribe them all! What's more,
> NTFS descriptions are fundamentally not compatible with FAT12/16/32
> partitions...

It's difficult to imagine that anybody's still using FAT partitions!

It's easy enough to write a batch file to convert DESCRIPT.ION files to NTFS
descriptions; there's been several examples posted here in the past few
years.

> * At first, when manually describing a file without description, all
> goes smoothly, the diacritics are correctly displayed with DIR, the
> descript.ion file content is fine.
> * As soon as the file is renamed or moved, all those éèêëçàù... are
> changed to question marks.

DESCRIBE & the description processing code never convert characters. This
is Windows converting from ASCII files to Unicode (used internally for all
Windows APIs) in your current code page & font, and then converting back
again. Windows will NEVER get this right for all extended characters
(though the right selection of code page & font can reduce the number of
errors) -- which is the whole purpose of using Unicode files, which undergo
no conversions.

 

[Steve Fabian]

| ---Quote---
|| OK, but what about the compatibility with old descriptions? I've got
|| thousands of files described in ASCII since 4DOS v3.0 (which I still
|| use for bootable tools), no way to transcribe them all! What's more,
|| NTFS descriptions are fundamentally not compatible with FAT12/16/32
|| partitions...
| ---End Quote---
| It's difficult to imagine that anybody's still using FAT partitions!

On your hard drive - I agree. However, all those portable devices that
usually connect via USB are a different case. I have a 6GB Seagate ocket
disk drive (it is really a disk drive). I attempted to reformat it to NTFS,
but the WinXP formatting service refused. FAT32/VFAT it had to remain.
Likewise, my "flashdrive" and my SD-card both use FAT32/VFAT alogrithms to
emulate disk drives. I wanted NTFS because of all the junctions and
hard-linked files I have... BTW, I am not sure that my digital camera could
handle an NTFS formatted SD-card anyway. So the need for FAT-style
operations is still here.

Another reason to use FAT partitions is for legacy software. If you
spent $10,000 for some custom DOS software, which takes care of your
business today as well as it did 10 years ago, why spend money on new
software just because you have a newer system?
--
Steve

 

[rconn]

> | ---Quote---
> || OK, but what about the compatibility with old descriptions? I've got
> || thousands of files described in ASCII since 4DOS v3.0 (which I still
> || use for bootable tools), no way to transcribe them all! What's more,
> || NTFS descriptions are fundamentally not compatible with FAT12/16/32
> || partitions...
> | ---End Quote---
> | It's difficult to imagine that anybody's still using FAT partitions!
>
> On your hard drive - I agree. However, all those portable devices
> that usually connect via USB are a different case. I have a 6GB Seagate
> ocket disk drive (it is really a disk drive). I attempted to reformat it
> to NTFS, but the WinXP formatting service refused. FAT32/VFAT it had to
> remain. Likewise, my "flashdrive" and my SD-card both use FAT32/VFAT
> algorithms to emulate disk drives. I wanted NTFS because of all the
> junctions and hard-linked files I have... BTW, I am not sure that my
> digital camera could handle an NTFS formatted SD-card anyway. So the
> need for FAT-style operations is still here.

I remain doubtful that people are using DESCRIPT.ION files on their SD cards
...

> Another reason to use FAT partitions is for legacy software. If you
> spent $10,000 for some custom DOS software, which takes care of your
> business today as well as it did 10 years ago, why spend money on new
> software just because you have a newer system?

True -- but those people are also not buying new versions of Take Command,
so I don't base development decisions on them!

Rex Conn
JP Software

 

[Steve Fabian]

| I remain doubtful that people are using DESCRIPT.ION files on their
| SD cards ...

Only when the SD card is used as a backup device.

| ---Quote---
|| Another reason to use FAT partitions is for legacy software. If
|| you spent $10,000 for some custom DOS software, which takes care of
|| your business today as well as it did 10 years ago, why spend money
|| on new software just because you have a newer system?
| ---End Quote---
| True -- but those people are also not buying new versions of Take
| Command, so I don't base development decisions on them!

I need to agree on this...
--
Steve

 

[Esteban]

I remain doubtful that people are using DESCRIPT.ION files on their SD cards
...


True -- but those people are also not buying new versions of Take Command,
so I don't base development decisions on them!

Rex Conn
JP Software

Cool boys! As I am a reasonable guy, of course I use NTFS for all my large capacity media. But Steve is right when he says that FAT format is still useful, even for people who just bought TCC 11 (at least me). For example I made some bootable tools under 4DOS for some old computers, which were designed at the beginning of this century for a dedicated application, working under Windows NT 4.0 and equipped with a floppy drive and a PCMCIA slot. Yes sir! These devices are supposed to be maintained at least until 2015... I hope the maintenance of my tools will be possible using TCC (on another computer of course) but NTFS descriptions will not make my life easier.

Anyway, I tested for some time TCC v9 then v10, and as far as I can remember there was no problem with diacritics. Only TCC v11 torments me. Well, not sure 100% about TCC v10, but clearly there was a small change somewhere in the code which could perhaps be reversed easily?

Esteban



Feb. 16th : I just tried TCC v10.0.57, which works with no problem. Now I can confirm that the incompatibility with diacritics was introduced between v10 and v11. Rex, help please...

 

[Esteban]

[...]

Anyway, I tested for some time TCC v9 then v10, and as far as I can remember there was no problem with diacritics. Only TCC v11 torments me. Well, not sure 100% about TCC v10, but clearly there was a small change somewhere in the code which could perhaps be reversed easily?

Esteban


Feb. 16th : I just tried TCC v10.0.57, which works with no problem. Now I can confirm that the incompatibility with diacritics was introduced between v10 and v11. Rex, help please...

Hello Rex,

As you didn't answer yet, should I understand you don't intend to correct that bug?

On Feb. 15th, you wrote "those people are also not buying new versions of Take Command, so I don't base development decisions on them!", speaking about me I suppose. Just let me point out that I DID buy your last Take Command 11 on January 17, expecting it to work without regression in comparison to v9 or v10. Did I make a mistake?

I understand it could not be your main preoccupation, but the description feature is the one I am using since the Flood. My external FAT32 hard disks are not compatible with NTFS descriptions, I've got other utilities which access to descript.ion files, no way to change!

Once again Rex, I'm not in a hurry, but the good old description feature shall work with diacritics as it worked in v10. Comparing these 2 versions, I guess it shouldn't be so hard to find what's going wrong. I'm ready to help you for making tests if need be, just tell me.

Thanks in advance,
Esteban.

 

[rconn]

> Hello Rex,
>
> As you didn't answer yet, should I understand you don't intend to
> correct that bug?

I've been on vacation (and out of the country w/no email access) the past
week.

I am unable to reproduce your problem. Can you email me a descript.ion file
that demonstrates the problem you're seeing?

Rex Conn
JP Software

 

[Esteban]

I've been on vacation (and out of the country w/no email access) the past
week.

I am unable to reproduce your problem. Can you email me a descript.ion file
that demonstrates the problem you're seeing?

Rex Conn
JP Software

Hi Rex,

I'm terribly sorry but I didn't read to the forum since February and didn't answer to you last post. Anyway, the problem disappeared since build 46, so I'm happy!

Best,
Georges

 

[gschizas]

Another reason to use FAT partitions is for legacy software. If you
spent $10,000 for some custom DOS software, which takes care of your
business today as well as it did 10 years ago, why spend money on new
software just because you have a newer system?
--
Steve

If you have a custom DOS software, put it on a virtual machine and forget about it :)

 

[Steve Fabian]

| ---Quote (Originally by Steve Fábián)---
| Another reason to use FAT partitions is for legacy software. If you
| spent $10,000 for some custom DOS software, which takes care of your
| business today as well as it did 10 years ago, why spend money on new
| software just because you have a newer system?
| ---End Quote---

Just to clarify matters: I quoted the above from another poster, though
I agree with it. I had made lot of money writing MS-DOS business programs.
Whether or not any is still in use I don't know. However, all would work
from NTFS equally well.

| If you have a custom DOS software, put it on a virtual machine and forget
about it :)

Customers who have such software are often not sufficiently knowledgable
to even know what a VM is. They would need a consultant to perform the
migration, but with proper care, should be able to operate it without
problems.
Regardless, this aspect was well covered in this thread (possibly in
another subthread) months ago.
--
Steve