How to? Dealing with Ampersand Characters and Piping

[Jay Sage]

I run into problems often because of ampersand characters in material that I want to process in a script. If nothing is being piped, the command setdos /x-5 generally allows one to manage the problem. However, if the output of a command is going to be piped into another command, that does not work, because the /x-5 option turns off piping as well as multiple commands.

Today one of my scripts that displays a file directory with descriptions failed because one of the descriptions had an ampersand it in. It took me a frustratingly long time to figure out that that was the cause of the failure. The solution I came up with was to use setdos /x-5, but instead of trying to pipe the output, I redirected it to a tmp#: pseudo-device. Then I restored normal processing using setdos /x0 and typed the contents of the tmp#: device into a pipe. So the code looked like this:

REM clear the tmp device
tmp /c tmp9:
REM turn off the special meaning of & (and, as a bonus, |)
setdos /x-5
REM run the commands in a loop with the output sent to the tmp device
{command loop} >> tmp9:
REM restore normal processing
setdos /x0
REM use TYPE (which doesn't care about special characters) to send the text to the pipe
type tmp9: |! {pipe process}

Perhaps another solution would have been to change the command separator to a character that was highly unlikely to appear in the command output and then to change it back to '&' after the piping was finished.

 

[Charles Dye]

Perhaps another solution would have been to change the command separator to a character that was highly unlikely to appear in the command output and then to change it back to '&' after the piping was finished.

Or even a noncharacter: setdos /c0xffff

 

[Jay Sage]

Do you know why the help says the following about that option?

NOTE: This option is obsolete, deprecated and not recommended (for compatibility reasons) unless you need to run very old 4DOS batch files!​

Is there some other way to do what setdos /c does? Making a temporary change in the command separator would seem to be a very useful technique.

As I noted above, the technique I described has the additional advantage of allowing the '|' character to be handled in file descriptions. Here's an example of output from my program.

 

[Charles Dye]

Do you know why the help says the following about that option?

At a wild guess... Because misuse of that feature results in a lot of support requests and bogus error reports?

Or maybe just aesthetics. It is kind of a crock.

 

[rconn]

Do you know why the help says the following about that option?

Because users were changing their compound command char to wacky things like $ or @ or #, and then complaining that their batch files didn't work anymore. And they couldn't run anyone else's batch files either.

 

[Jay Sage]

Dealing with '<' and '>' characters is even more challenging. I have one script that needs to allow for them (as well as '&' and '|'), so I turn off both multiple commands and redirection with setdos /x-5-6 and then use the file functions (including @filewrite) to write the data without needing redirection. Then normal processing is reestablished (setdos /x0) and the file can be displayed using the type command without the special characters causing a problem. That approach takes much more work and can sometimes result in orphaned files.

 

[Jay Sage]

Because users were changing their compound command char to wacky things like $ or @ or #, and then complaining that their batch files didn't work anymore. And they couldn't run anyone else's batch files either.

Yes, I can see that. However, people can also change the characters in the TCC option "Advanced" tab, in which case the change is persistent. With SETDOS, at least it affects only the current instance of TCC.

So I hope that you will not take it away from SETDOS.

Would it be difficult to allow SETDOS to disable the special characters on a more specific basis (for example, just multiple commands or just piping)?

 

[Jay Sage]

I've been entertaining myself by reading through the help system. One of the commands I noticed was EXECARRAY, and I realized that it could be used to move data without having to use redirection. So I run code that looks like this.

REM turn off multiple commands and redirection
setdos /x-5-6
REM create display data, which includes special characters, and move it into an array
set code=%@execarray[displayarray,`for %file in (@tmp1:) do ( echo %@gendata[%file] )`]
REM restore normal command line processing
setdos /x0

To make it possible to use the TYPE command to pipe it to my hanging-indent program, I transfer data into a TMP: pseudo-device. The following code takes advantage of the fact the the SET command can display variable data that contains special characters.

REM clear tmp9:
tmp /c tmp9:
REM run through the data saved in the array variable, adding it to tmp9:
do i = 0 to %@lines[tmp1:]
  set displayarray[%i] >> tmp9:
enddo

Now I can pipe the data.

type tmp9: |! WrapDescriptions.exe %_columns %widwrap

Rex, would it be possible in a future version to have an EXECTMP command that would do what EXECARRAY does but put the data into a TMP: pseudo-device (and maybe the CLIP: devices as well)? The pseudo-devices are often more convenient to deal with than arrays.

Here's the output from my program, which handles all sorts of special characters in the description.