AI response decoding?

[vefatica]

v:\> ai why do i sometimes see — (a-circonflex,euro,close-quote) in your responses?
That is **mojibake**: an em dash (`—`) encoded as UTF‑8 is being incorrectly decoded as Windows‑1252 (or a similar legacy encoding).

- Intended character: `—`
- UTF‑8 bytes: `E2 80 94`
- Misread as Windows‑1252: `â€â€`

This usually happens somewhere in the API/client/display pipeline—not because I intentionally produced those three characters. Ensure responses are decoded as UTF‑8 throughout, avoid double encoding, and use an appropriate header such as:

```http
Content-Type: application/json; charset=utf-8
```

 

[Charles Dye]

I like it — an explanation of mojibake, which unintentionally contains further examples of mojibake. Self-referential mojibake.

 

[vefatica]

The default can be pretty messy. Rex, can you do anything about that?

v:\> ai /i trying CHCP 65001; what's the contraction for "do not"?
“Do not†contracts to “don’t.â€
[R]un, [E]dit, [C]ontinue, [Q]uit: C
AI>  I still see “don’t.â€
That’s mojibake. Use the ASCII apostrophe: `don't` instead of `don’t`.
[R]un, [E]dit, [C]ontinue, [Q]uit: C
AI> in another TCC version I put a specific "no unicode" in my default role and that seems to work.  What would Rex Conn have to
 do to implement a fix inside TCC?
Rex Conn would need to fix TCC’s text-decoding path:

1. Treat AI/API JSON responses as UTF‑8.
2. Convert them once with `MultiByteToWideChar(CP_UTF8, ...)`.
3. Output to the console with `WriteConsoleW`.
4. For redirected output, encode explicitly as UTF‑8.

`CHCP 65001` cannot help if TCC first interprets UTF‑8 bytes as Windows‑1252, which produces `“don’tâ€`. The “no Unicode†prompt merely avoids triggering the bug; it is not the real fix.