Skip to main content

UTF-8 at the Windows Command Prompt with Take Command / TCC

Written by Rex Conn on . Posted in Features.

TCC has for the past 20+ years supported Unicode using UTF-16, the format used internally by all Windows versions since NT.  In the past few years there has been a push for UTF-8 support, which is the most common Unicode format in non-Windows environments.

There are some issues with adding UTF-8 support everywhere in Take Command and TCC:

  •   Windows doesn’t have much internal UTF-8 support, other than a handful of APIs to convert ASCII and UTF-16 to & from UTF-8, and the 65001 code page.
  • Most of the Windows APIs still require UTF-16 input, so it’s necessary to convert UTF-8 to UTF-16 before calling an API, and then convert it back again for the result.
  • There are not a lot of Windows apps that support UTF-8. (For example, CMD doesn’t support UTF-8 or UTF-16.)

Because of the lack of general support, in order to extend the UTF-8 support in TCC we’ve had to add the necessary code everywhere UTF-8 is needed. In version 26, we have added UTF-8 support in many more places, including batch files, library files, alias and function files, TCMD.INI, IDE and TCEdit. And TPIPE has been greatly extended to support UTF-8 as its default encoding type (see below).

If you want to use UTF-8 with TCC v26, you can enable it with the OPTION command (“Startup / UTF8”).  You will also probably want to change your code page to 65001.

If you need UTF-8 support in more places in TCC (or Take Command or CMDebug), please let us know on the JP Software Suggestions Forum, and we will try to add them in our next update. 

TPIPE in TCC v26

TPIPE in v26 has been significantly upgraded.  The TextPipe Engine has been updated from 9.9.4 to 11.4, and TPIPE is now fully 64-bit.   Because the new textpipeengine64.dll is larger, the initial load is slightly slower in v26, but it runs your filters much faster.  And the switch to 64-bit means TPIPE can easily handle very large files.

There are dozens of new options for the existing filters, including UTF-8 input and output, JSON support, a new selection filter, enhanced regular expressions, and improved error handling everywhere.  See Whats New in the help for details on TPIPE v26 features. If you’re a TPIPE user, we hope you will be very happy with the v26 updates.  If you’re not yet a TPIPE user, download v26 and see what you’ve been missing!