Skip to main content

Windows Command Line, Tabbed Windows, and JP Software (Part IV)

Written by Rex Conn on . Posted in News.

In my last post I talked about the some of the strategies I tried for displaying console windows in a GUI window during the Take Command 9 design and development.

The first three ideas (the Windows accessibility APIs, injecting a dll into console applications, and allocating consoles from the Take Command session) had all been discarded after encountering fatal flaws. Idea #4 was to hook the Windows console APIs, so I could intercept any call to read from or write to a console window. (ReadConsole, ReadConsoleInput, WriteConsole, WriteConsoleOutput, etc.) In theory, this would provide me with complete coverage of any console I/O, so I could handle them, optionally modifying or redirecting the I/O before passing it on. In practice, it turned out to have all of the disadvantages of #2 (injecting a dll), and a few more:

  • It required dynamically modifying other programs (see: Is Take Command infected with a virus!?).
  • It added a lot of additional complexity.
  • There seemed to be no way to intercept the occasional mysterious API calls that seemed to originate from somewhere deep inside Windows and which called undocumented lower-level console APIs..
  • I encountered another raft of buggy and/or misdocumented APIs. (Some of which I had found back in the NT 3.x days — apparently Microsoft hasn’t put the highest priority on enhancing the console window over the past 15 years.)
  • As a result of Microsoft’s under-documentation of the console APIs, a lot of programs were calling them incorrectly. Do I ignore the bad calls? Try to correct them and pass them on? Or just process them as called, pass them through, and hope they don’t mangle the Take Command display too badly?

On the verge of scrapping the whole idea of tabbed console windows, I thought about reversing idea #3. Instead of Take Command creating the console and then launching the command line application in that console window, what if I launched the command line application (which would then create its own hidden console window) and then connected Take Command to that console window? This idea had some advantages:

  • No antivirus application would complain about this.
  • Take Command could share the console window with the command line application (for example, TCC), and could read from and write directly to that window, without requiring any interprocess communication between Take Command and the command line application.
  • Take Command could easily switch between tab windows by disconnecting from the previous console and attaching to the new one.

Of course, it wasn’t going to be that easy. There were an unusual (even for Windows) number of obstacles:

  • Another seemingly endless series of buggy / mis-documented Windows APIs. For example, some APIs behaved unexpectedly or failed altogether if the console window was hidden.
  • It turned out to be a far from trivial exercise even for something so apparently simple as for Take Command to quickly and reliably get the console window handle of the application it just started.
  • Simply having access to the console window didn’t mean I could update the Take Command window rapidly.
  • I had to monitor all of the console windows continuously (even the ones not visible in a tab window) so I could determine if the app had exited, changed its window and/or buffer size (not uncommon), (re)hide the console window if it had somehow become visible (again, not uncommon), update the tab titles, etc.
  • Catch the instances where the console application started another application, and then exited (leaving the child application running, but with a different process ID).
  • The keyboard input had to be handled in Take Command, filtered appropriately, and then passed to the console windows.
  • The window z-order had to be continually adjusted so the focus would be in the right place.
  • Handling the irksome tendency of DOS apps to resize the console window buffer to 80×25. (The less said about the rest of the problems with running DOS apps, the better.)
  • And so many more that I feel myself slipping into a dangerously depressed state while reliving the long nightmare …

If I had known all of the problems I’d subsequently encounter, I would have continued looking for idea #6, or perhaps taken a job as assistant manager at the local McDonald’s. But I didn’t know, and so I gradually beat the problems one by one into submission. Occasionally through incredibly clever programming, but mostly through intensive heuristic analysis (i.e., trial and error). Turns out there’s a good reason why there are only a tiny handful of other applications that try to implement tabbed windows for console applications (and most of them work slowly & badly).

This approach adopted for Take Command v9 is still in use (streamlined and updated) for v13, and I don’t foresee any major architectural changes in the next version.

So, we’re up to date, and I can get back to talking about new features (and new problems).