Need Help with Handling Special Characters

[Jay Sage]

Here is a simplified example of a problem I am having.

I have a file, test.txt, that contains text with some special characters. This is what the file looks like in the TCEDIT editor.

When I operate on that file with various commands and functions, the special characters change. Here are some examples.

>type test.txt
GΘnΘalogie Fundaci≤n

>echo %@line[test.txt,0]
GΘnΘalogie Fundaci≤n

On the other hand, I just found that the following works.

>copy test.txt tmp9: & type tmp9:
C:\Users\Jay\test.txt => tmp9:

     1 file copied
Généalogie Fundación
>echo %@line[tmp9:,0]
Généalogie Fundación

So, I cannot type the file itself, but I can copy it to tmp9: and type it from there. I cannot extract the text from the file using the %@line function, but I can extract the line from tmp9:. That does explain why a script that was working fine when I copied the data into a tmp device and worked on it from there and stopped working when I loaded the data directly from the file.

I imagine that this has something to do with UTF-8 and UTF-16. Perhaps someone can explain it to me so that I have some control over the issue.

 

[rconn]

Those characters could be either UTF-8 or UTF-16.

TCEdit handles everything internally as UTF-8. If you pass it a UTF-16 file it will convert it to UTF-8 before editing, then convert it back to UTF-16 when it is saved.

TCC on the other hand handles everything internally as UTF-16 (because that's what the Windows APIs use).

TCC is pretty good at deciphering the encoding of files it handles; however it can't tell if the font you're using supports the UTF-8 or UTF-16 glyphs you want to display.

If you are going to be using UTF-8, you should switch to the UTF-8 codepage (65001), and make sure you're using a Unicode font.

 

[Jay Sage]

So what happened here?

>set line=Généalogie Fundación

>set line
Généalogie Fundación

>echo %line
Généalogie Fundación

>echo %@ascii[%line]
71 233 110 233 97 108 111 103 105 101 32 70 117 110 100 97 99 105 243 110

>echo %line > tmp8:

>type tmp8:
G,n,alogie Fundaci¢n

I don't see how my choice of fonts is the issue, since the text string displays correctly until I type the contents of tmp8:. So either the ECHO command or the redirection must be the point at which the characters change.

>set line2=%@line[tmp8:,0]

>set line2
G‚n‚alogie Fundaci¢n

>echo %@ascii[%line2]
71 8218 110 8218 97 108 111 103 105 101 32 70 117 110 100 97 99 105 162 110

The characters that were 'e' with an acute accent are now characters with a value of 8218. I know very little about UTF encoding, but my guess is that that character is UTF-16.

Do you think that changing the codepage would help this? Is the codepage a machine-wide setting, or does it apply to each TCC session?

 

[Charles Dye]

If you can re-save the file as either UTF-8 or UTF-16, that will save a lot of confusion.

 

[rconn]

So what happened here?

I don't see how my choice of fonts is the issue, since the text string displays correctly until I type the contents of tmp8:. So either the ECHO command or the redirection must be the point at which the characters change.

By default, redirection takes your UTF-16 internal characters and writes them to ASCII. You can tell redirection to write to UTF-16 or UTF-8 instead.

 

[Joe Caverly]

From the Help file;

Ref: Pipes in TCC

Ref: Redirection in TCC

Joe

 

[Charles Dye]

Incidentally, you can see which encoding TCEDIT is using by selecting File / Encoding. "Default Codepage" means the system ANSI code page, typically 1252 in the United States.

 

[Jay Sage]

Thanks everyone for the help.

I finally gave up and asked AI to write a Free Pascal program to read in the file and convert all non-ASCII characters (plus those with special meaning in HTML) to their equivalent HTML entities. That solved all the problems, since now everything is ASCII and the encoding doesn't matter. I eventually moved a lot of my other TCC script code into the Pascal program. Perplexity even did an excellent job of taking TCC code that I gave it and writing the equivalent Pascal code. Eventually, I even had the Pascal code do the sorting of the data that I had been doing using Excel -- and running into the problem that CSV files do not have data typing, so phone numbers were in some cases being converted to numbers in scientific notation!

I'm still doing the final conversion of the data into a web page with a TCC script. The latter is easier (for me) to write and much easier to debug (with no SETLOCAL/ENDLOCAL all the variables are still there after the program has run or been stopped at some intermediate point). Plus there's the debugger.

Today I did some reading about Unicode encoding techniques and have a better understanding of what was going on.

 

[richieadler]

Today I did some reading about Unicode encoding techniques and have a better understanding of what was going on.

Seminal article on the topic: The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)