Problems with backquotes in filenames

[NeBlackCat]

I'm a lapsed 4NT/TCMD v5 user evaluating the latest TCC with a view to upgrading, mainly due to the new PDIR command.

So far not so good, however. I'm at my wits end trying to make PDIR work with NTFS filenames that contain back quotes.

For example if you do something like this:

pdir /(@right[-2,*])

Then you'll get a "No closing quote" error on filenames containing a back quote, and I can't find a solution except using "setdos /x-7" to turn off quote processing. However that creates other problems with filenames that contain spaces, for example:

setdos /x-7
pdir /(@attrib[*])

Will then print nothing for filenames containing spaces (probably due them being quoted), although most similar functions do still work, such as:

setdos /x-7
pdir /(@sfn[*])

So:

1) how do I change my PDIR command lines so that passing '*' to functions doesn't error on filenames with back ticks?

2) how do I get @attrib to work with "setdos/x-7" in force, with filenames containing spaces? Not using it is not an option.

 

[Steve Fabian]

NeBlackCat wrote:
| So far not so good, however. I'm at my wits end trying to make PDIR
| work with NTFS filenames that contain back quotes.
...

I have no generic answer to this question. However, to report file
attributes and the SFNs of files, you do not need to resort to the @attrib
or @sfn functions. The command below reports both:
pdir/(a fPN)
while the command
pdir/(fPN"="fpn)
reports in the form sfn=lfn.
--
HTH, Steve

 

[NeBlackCat]

NeBlackCat wrote:
| So far not so good, however. I'm at my wits end trying to make PDIR
| work with NTFS filenames that contain back quotes.
...

I have no generic answer to this question. However, to report file
attributes and the SFNs of files, you do not need to resort to the @attrib
or @sfn functions. The command below reports both:
pdir/(a fPN)
while the command
pdir/(fPN"="fpn)
reports in the form sfn=lfn.
--
HTH, Steve

Hi Steve, thanks but I guess you missed the "not using it (@attrib) is not an option" bit! In fact I must only display RHS attributes (don't care about A and the others), so I have to use @attrib and string processing functions.

Similar goes for the filename part, which I have to display in Unix format (forward slashes and no drive letters).

So the above are just simplified examples that still illustrate the issues.

 

[Steve Fabian]

NeBlackCat wrote:
| Hi Steve, thanks but I guess you missed the "not using it (@attrib)
| is not an option" bit! In fact I must only display RHS attributes
| (don't care about A and the others), so I have to use @attrib and
| string processing functions.

I misinterpreted the reference, since you did not include the relevant
information about reporting only the RHS attributes.
|
| Similar goes for the filename part, which I have to display in Unix
| format (forward slashes and no drive letters).

Seems you want to imitate the Unix "ls" program! Now that I understand
the issues better, I can suggest a tedious workaround.
1/ create a file containing both the SFN and the LFN of each file, with
a separator such as vertical bar | between them, e.g. using the command
pdir /(fPN"|"fpn)
2/ process this file one byte at a time, using @filereadb:
2.1/ build the SFN
2.2/ use the SFN and functions like @attrib, @filedate, etc. to obtain
all desired information aboout the file
2.3/ build your output line from the file information collected, and
from reading each character of the LFN from the catalog, performing
appropriate translations, e.g., \ to /.
Note that I when using the @fileopen/@fileread scheme, I found the simplest
way to exit your loop at the end of file is in the style below:

set size=%@filesize[file]
set hndl=%@fileopen[file,r,b]
do while %size LT %@fileseek[%hndl,1,0]
read
work
enddo

BTW, I had a few files from MS-DOS days, when one vendor used the
percent sign % in the last position of the file extension to indicate
compressed files. I had a lot of trouble with them, but since they were just
archived legacy files, I solved the issue by brute force - I changed the
names to Microsoft's choice of characters for the same purpose, the
underscore _. If possible, that's a better method... Please also read about
Charles Dye's plugin Safechars: http://www.unm.edu/~cdye/dl/safechars.zip

--
HTH, Steve

 

[Jim Cook]

I don't see the issue you report. Can you give an example DIR listing
and PDIR syntax you are trying?

TCC 11.00.34 Windows XP [Version 5.1.2600]
TCC Build 34 Windows XP Build 2600 Service Pack 3

D:\test>*pdir /(@right[-2,*])
:\test\Quoted 'name'"
:\test\Quoted `name`"

Here are the hex ASCII representation and filenames I have:
51.75.6F.74.65.64.20.27.6E.61.6D.65.27 Quoted 'name'
51.75.6F.74.65.64.20.60.6E.61.6D.65.60 Quoted `name`


On Sun, Dec 6, 2009 at 9:08 PM, NeBlackCat <> wrote:

> I'm a lapsed 4NT/TCMD v5 user evaluating the latest TCC with a view to upgrading, mainly due to the new PDIR command.
>
> So far not so good, however. I'm at my wits end trying to make PDIR work with NTFS filenames that contain back quotes.
>
> For example if you do something like this:
>
> pdir /(@right[-2,*])
>
> Then you'll get a "No closing quote" error on filenames containing a back quote, and I can't find a solution except using "setdos /x-7" to turn off quote processing. However that creates other problems with filenames that contain spaces, for example:
>
> setdos /x-7
> pdir /(@attrib[*])
>
> Will then print nothing for filenames containing spaces (probably due them being quoted), although most similar functions do still work, such as:
>
> setdos /x-7
> pdir /(@sfn[*])
>
> So:
>
> 1) how do I change my PDIR command lines so that passing '*' to functions doesn't error on filenames with back ticks?
>
> 2) how do I get @attrib to work with "setdos/x-7" in force, with filenames containing spaces? Not using it is not an option.
>
>
>
>
>

--
Jim Cook
2009 Saturdays: 4/4, 6/6, 8/8, 10/10, 12/12 and 5/9, 9/5, 7/11, 11/7.
Next year they're Sunday.

 

[Steve Fabian]

Message below was first posted at 11:09 AM EST (more than 2 hours ago), but
did not post.

E. S. Fabian wrote:
|| NeBlackCat wrote:
||| Hi Steve, thanks but I guess you missed the "not using it (@attrib)
||| is not an option" bit! In fact I must only display RHS attributes
||| (don't care about A and the others), so I have to use @attrib and
||| string processing functions.
||
|| I misinterpreted the reference, since you did not include the
|| relevant information about reporting only the RHS attributes.
|||
||| Similar goes for the filename part, which I have to display in Unix
||| format (forward slashes and no drive letters).
||
|| Seems you want to imitate the Unix "ls" program! Now that I
|| understand the issues better, I can suggest a tedious workaround.
|| 1/ create a file containing both the SFN and the LFN of each
|| file, with a separator such as vertical bar | between them, e.g.
|| using the command pdir /(fPN"|"fpn)
|| 2/ process this file one byte at a time, using @filereadb:
|| 2.1/ build the SFN
|| 2.2/ use the SFN and functions like @attrib, @filedate, etc. to
|| obtain all desired information aboout the file
|| 2.3/ build your output line from the file information collected,
|| and from reading each character of the LFN from the catalog,
|| performing appropriate translations, e.g., \ to /.
|| Note that I when using the @fileopen/@fileread scheme, I found the
|| simplest way to exit your loop at the end of file is in the style
|| below:
||
|| set size=%@filesize[file]
|| set hndl=%@fileopen[file,r,b]
|| do while %size LT %@fileseek[%hndl,1,0]
|| read
|| work
|| enddo
||
|| BTW, I had a few files from MS-DOS days, when one vendor used the
|| percent sign % in the last position of the file extension to indicate
|| compressed files. I had a lot of trouble with them, but since they
|| were just archived legacy files, I solved the issue by brute force -
|| I changed the names to Microsoft's choice of characters for the same
|| purpose, the underscore _. If possible, that's a better method...
|| Please also read about Charles Dye's plugin Safechars:
|| http://www.unm.edu/~cdye/dl/safechars.zip
||
|| --
|| HTH, Steve

 

[NeBlackCat]

I don't see the issue you report. Can you give an example DIR listing
and PDIR syntax you are trying?

TCC 11.00.34 Windows XP [Version 5.1.2600]
TCC Build 34 Windows XP Build 2600 Service Pack 3

D:\test>*pdir /(@right[-2,*])
:\test\Quoted 'name'"
:\test\Quoted `name`"

Here are the hex ASCII representation and filenames I have:
51.75.6F.74.65.64.20.27.6E.61.6D.65.27 Quoted 'name'
51.75.6F.74.65.64.20.60.6E.61.6D.65.60 Quoted `name`


On Sun, Dec 6, 2009 at 9:08 PM, NeBlackCat <> wrote:





--
Jim Cook
2009 Saturdays: 4/4, 6/6, 8/8, 10/10, 12/12 and 5/9, 9/5, 7/11, 11/7.
Next year they're Sunday.

Hi Jim, I suspect the reason you don't see it is because you're using two backquotes not one, so you wouldn't get an error about a missing closing quote. Sorry - I should probably have stated that clearer.

You can reproduce it using the files I zipped and attached to this post. Just a handful of 1 byte text files with strange filenames, a couple of which will trigger this issue.

 

[NeBlackCat]

I misinterpreted the reference, since you did not include the relevant information about reporting only the RHS attributes.

My apologies!

Seems you want to imitate the Unix "ls" program! Now that I understand the issues better, I can suggest a tedious workaround....

Yes, I have to produce output that's directly file-comparable with other tools on both Windows and Linux.

I like the idea of using SFNs to dodge around the problem of characters that can appear in LFNs, but unfortunately I can't guarantee SFNs will be available as some (like me) tend to turn them (and other things, like last access time updating) off as NTFS performance tweaks.

The plugin looks usefull too - I'll play with that and see what can be done.

Thanks again!

 

[Steve Fabian]

NeBlackCat wrote:
| ---Quote (Originally by Steve Fbin)---
| Seems you want to imitate the Unix "ls" program! Now that I
| understand the issues better, I can suggest a tedious workaround....
| ---End Quote---
| Yes, I have to produce output that's directly file-comparable with
| other tools on both Windows and Linux.

Most of the *nix tools are available as freeware, typically under GPL,
including ls, from several sources. Until the arrival of PDIR, I often used
the Cygwin version of ls, esp. when dealing with inodes (to identify
multiply cataloged files). There is still a version on my system, dated
2006-07-05. Since you apparently need to deal with both NTFS and Posix file
systems, you might be better off using ls!
--
HTH, Steve

 

[rconn]

> So far not so good, however. I'm at my wits end trying to make PDIR
> work with NTFS filenames that contain back quotes.
>
> For example if you do something like this:
>
> pdir /(@right[-2,*])
>
> Then you'll get a "No closing quote" error on filenames containing a
> back quote, and I can't find a solution except using "setdos /x-7" to
> turn off quote processing. However that creates other problems with
> filenames that contain spaces, for example:

First, it's a REALLY REALLY REALLY bad idea to use single back quotes in
your filenames. You're going to have endless problems with the parser, and
I strongly urge you to rename your files to more rational names.

Second, this doesn't have anything to do with PDIR; it's the @RIGHT variable
function that's complaining about the single back quote. The functions that
take a single argument (like @ATTRIB) do not complain because they don't
need to parse multiple arguments.

Third, you probably don't need to do any of this with PDIR, since there's
internal PDIR options for everything you've mentioned. If you can give an
example of exactly what you're trying to do, we can provide suggestions on
how to do it without variable functions.

Rex Conn
JP Software

 

[NeBlackCat]

First, it's a REALLY REALLY REALLY bad idea to use single back quotes in
your filenames. You're going to have endless problems with the parser, and
I strongly urge you to rename your files to more rational names.

Second, this doesn't have anything to do with PDIR; it's the @RIGHT variable
function that's complaining about the single back quote. The functions that
take a single argument (like @ATTRIB) do not complain because they don't
need to parse multiple arguments.

Third, you probably don't need to do any of this with PDIR, since there's
internal PDIR options for everything you've mentioned. If you can give an
example of exactly what you're trying to do, we can provide suggestions on
how to do it without variable functions.

Rex Conn
JP Software

First - unfortunately I have no control over what characters appear in file names as I do not create them. There were tens of thousands of them in the first test batch, and only a few using WTF characters, but they are still technically legitimate and still have to work.

Second - I know. I even tried @unicode-ing the filename ("*"), processing that, then @char-ing it back again for printing, but there's no way to avoid @right to chop off drive letters (you can't tell @instr, which takes 2 parameters and therefore works, to return everything after the first two characters due to the way it's position parameters are defined), and @right still fails. (*)

Third - unless I'm missing something, there is no way to tell PDIR to do what I've previously stated (print only the RHS attributes, and show the filename in Unix style ie. slashes converted to forward, and drive letter chopped off), and no way to pass PDIR's invididual output fields to variable functions (just the "*").

(*) if setdos /x had greater granularity of control, ie. allow individual control of each quotes type rather than all of them together, I'd have been ok. Perhaps that's one for the wish list.

 

[Steve Fabian]

NeBlackCat wrote:
| Second - I know. I even tried @unicode-ing the filename ("*"),
| processing that, then @char-ing it back again for printing, but
| there's no way to avoid @right to chop off drive letters (you can't
| tell @instr, which takes 2 parameters and therefore works, to return
| everything after the first two characters due to the way it's
| position parameters are defined), and @right still fails. (*)

Using @INSTR[2,9999,%x] emulates @RIGHT[-2,%x]. However, I suspect it will
fail in the same manner as @RIGHT.

Did you look into Charles Dye's safechars.dll? It may do what you need.

Another alternative method to achieve your goal is to use PDIR to collect
raw data into a binary array, where the "a" report field will have
characters superfluous to your use, but which can be easily discarded.
Likewise, all special characters will be properly stored, and you can just
parse the array a character at a time into your desired format. Tedious, I
know, but not overly complex.
--
HTH, Steve

 

[Jim Cook]

On Tue, Dec 8, 2009 at 6:46 AM, Steve Fábián <> wrote:

> Another alternative method to achieve your goal is to use PDIR to collect
> raw data into a binary array, where the "a" report field will have

Or even use PDIR | perl or the moral equivalent. It ceases to be a
native solution, but sometimes the wrong tool should not be rigged to
work.

--
Jim Cook
2009 Saturdays: 4/4, 6/6, 8/8, 10/10, 12/12 and 5/9, 9/5, 7/11, 11/7.
Next year they're Sunday.

 

[Charles Dye]

Did you look into Charles Dye's safechars.dll? It may do what you need.

I'm not really sure how. Massage the offending filename into a .CMD file, then call CMD.EXE to rename it to something sane?

 

[Steve Fabian]

Charles Dye wrote:
| ---Quote (Originally by Steve Fbin)---
| Did you look into Charles Dye's safechars.dll? It may do what you
| need. ---End Quote---
| I'm not really sure how. Massage the offending filename into a .CMD
| file, then call CMD.EXE to rename it to something sane?

No, renaming is not appropriate for files generated externally, and most
likely to be processed again by software not under the OP's control. Rather,
by using PDIR in a more native mode, e.g., using the "a" report field,
redirecting its output into a temporary file, and processing each line of
the temporary file with @saferead or @safeenv, dropping all unwanted
characters, replacing backslashes with slashes, etc. Of course, the output
must be written using @SAFEWRITE to restore the original codes for the
special characters.

BTW, Charles, there are two more operations desirable in SAFECHARS.DLL,
@safeclip[] ought to have its reverse @safeclipw[], and for Unicode=Yes it
would be desirable to have an EchoSafeUnicode command. This last one might
be very tricky, though - need to know whether or not it is redirected to a
file, and in the latter case whether it is creating the file (need to write
BOM) or apppending to one. Possibly could have two separate commands, one to
create the file with just the BOM, the other to append, with the filename
explicitly specified (as done by TEE command).
--
Steve

 

[Charles Dye]

BTW, Charles, there are two more operations desirable in SAFECHARS.DLL, @safeclip[] ought to have its reverse @safeclipw[],

That would probably be easy.

and for Unicode=Yes it would be desirable to have an EchoSafeUnicode command. This last one might be very tricky, though - need to know whether or not it is redirected to a file, and in the latter case whether it is creating the file (need to write BOM) or apppending to one. Possibly could have two separate commands, one to create the file with just the BOM, the other to append, with the filename explicitly specified (as done by TEE command).

What would that do that %@safewrite doesn't?

 

[Steve Fabian]

Steve wrote:
| an EchoSafeUnicode command

Charles Dye wrote:
| What would that do that %@safewrite doesn't?

Create output files without the cumbersome @file... functions, in other
words, hide the details of the file manipulation froms the user, just like
the "for %line in (@file)", or its equivalent with DO, move the burden of
details from the user's code to the command processor.
--
Steve

 

[Charles Dye]

BTW, Charles, there are two more operations desirable in SAFECHARS.DLL,
@safeclip[] ought to have its reverse @safeclipw[], and for Unicode=Yes it
would be desirable to have an EchoSafeUnicode command. This last one might be very tricky, though - need to know whether or not it is redirected to a file, and in the latter case whether it is creating the file (need to write
BOM) or apppending to one.

All that trickiness seems to be implemented in Take Command already. Just hand a string to QPuts(), and //UnicodeOutput (and the BOM if needed) are handled automatically. Cool.

I'm putting up a new build at http://www.unm.edu/~cdye/plugins/ which I think will do what you want.