1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Problems with backquotes in filenames

Discussion in 'Support' started by NeBlackCat, Dec 7, 2009.

  1. NeBlackCat

    Joined:
    Dec 5, 2009
    Messages:
    9
    Likes Received:
    0
    I'm a lapsed 4NT/TCMD v5 user evaluating the latest TCC with a view to upgrading, mainly due to the new PDIR command.

    So far not so good, however. I'm at my wits end trying to make PDIR work with NTFS filenames that contain back quotes.

    For example if you do something like this:

    pdir /(@right[-2,*])

    Then you'll get a "No closing quote" error on filenames containing a back quote, and I can't find a solution except using "setdos /x-7" to turn off quote processing. However that creates other problems with filenames that contain spaces, for example:

    setdos /x-7
    pdir /(@attrib[*])

    Will then print nothing for filenames containing spaces (probably due them being quoted), although most similar functions do still work, such as:

    setdos /x-7
    pdir /(@sfn[*])

    So:

    1) how do I change my PDIR command lines so that passing '*' to functions doesn't error on filenames with back ticks?

    2) how do I get @attrib to work with "setdos/x-7" in force, with filenames containing spaces? Not using it is not an option.
     
  2. Steve Fabian

    Joined:
    May 20, 2008
    Messages:
    3,520
    Likes Received:
    4
    NeBlackCat wrote:
    | So far not so good, however. I'm at my wits end trying to make PDIR
    | work with NTFS filenames that contain back quotes.
    ...

    I have no generic answer to this question. However, to report file
    attributes and the SFNs of files, you do not need to resort to the @attrib
    or @sfn functions. The command below reports both:
    pdir/(a fPN)
    while the command
    pdir/(fPN"="fpn)
    reports in the form sfn=lfn.
    --
    HTH, Steve
     
  3. NeBlackCat

    Joined:
    Dec 5, 2009
    Messages:
    9
    Likes Received:
    0
    Hi Steve, thanks but I guess you missed the "not using it (@attrib) is not an option" bit! In fact I must only display RHS attributes (don't care about A and the others), so I have to use @attrib and string processing functions.

    Similar goes for the filename part, which I have to display in Unix format (forward slashes and no drive letters).

    So the above are just simplified examples that still illustrate the issues.
     
  4. Steve Fabian

    Joined:
    May 20, 2008
    Messages:
    3,520
    Likes Received:
    4
    NeBlackCat wrote:
    | Hi Steve, thanks but I guess you missed the "not using it (@attrib)
    | is not an option" bit! In fact I must only display RHS attributes
    | (don't care about A and the others), so I have to use @attrib and
    | string processing functions.

    I misinterpreted the reference, since you did not include the relevant
    information about reporting only the RHS attributes.
    |
    | Similar goes for the filename part, which I have to display in Unix
    | format (forward slashes and no drive letters).

    Seems you want to imitate the Unix "ls" program! Now that I understand
    the issues better, I can suggest a tedious workaround.
    1/ create a file containing both the SFN and the LFN of each file, with
    a separator such as vertical bar | between them, e.g. using the command
    pdir /(fPN"|"fpn)
    2/ process this file one byte at a time, using @filereadb:
    2.1/ build the SFN
    2.2/ use the SFN and functions like @attrib, @filedate, etc. to obtain
    all desired information aboout the file
    2.3/ build your output line from the file information collected, and
    from reading each character of the LFN from the catalog, performing
    appropriate translations, e.g., \ to /.
    Note that I when using the @fileopen/@fileread scheme, I found the simplest
    way to exit your loop at the end of file is in the style below:

    set size=%@filesize[file]
    set hndl=%@fileopen[file,r,b]
    do while %size LT %@fileseek[%hndl,1,0]
    read
    work
    enddo

    BTW, I had a few files from MS-DOS days, when one vendor used the
    percent sign % in the last position of the file extension to indicate
    compressed files. I had a lot of trouble with them, but since they were just
    archived legacy files, I solved the issue by brute force - I changed the
    names to Microsoft's choice of characters for the same purpose, the
    underscore _. If possible, that's a better method... Please also read about
    Charles Dye's plugin Safechars: http://www.unm.edu/~cdye/dl/safechars.zip

    --
    HTH, Steve
     
  5. Jim Cook

    Joined:
    May 20, 2008
    Messages:
    604
    Likes Received:
    0
    I don't see the issue you report. Can you give an example DIR listing
    and PDIR syntax you are trying?

    TCC 11.00.34 Windows XP [Version 5.1.2600]
    TCC Build 34 Windows XP Build 2600 Service Pack 3

    D:\test>*pdir /(@right[-2,*])
    :\test\Quoted 'name'"
    :\test\Quoted `name`"

    Here are the hex ASCII representation and filenames I have:
    51.75.6F.74.65.64.20.27.6E.61.6D.65.27 Quoted 'name'
    51.75.6F.74.65.64.20.60.6E.61.6D.65.60 Quoted `name`


    On Sun, Dec 6, 2009 at 9:08 PM, NeBlackCat <> wrote:



    --
    Jim Cook
    2009 Saturdays: 4/4, 6/6, 8/8, 10/10, 12/12 and 5/9, 9/5, 7/11, 11/7.
    Next year they're Sunday.
     
  6. Steve Fabian

    Joined:
    May 20, 2008
    Messages:
    3,520
    Likes Received:
    4
    Message below was first posted at 11:09 AM EST (more than 2 hours ago), but
    did not post.

    E. S. Fabian wrote:
    || NeBlackCat wrote:
    ||| Hi Steve, thanks but I guess you missed the "not using it (@attrib)
    ||| is not an option" bit! In fact I must only display RHS attributes
    ||| (don't care about A and the others), so I have to use @attrib and
    ||| string processing functions.
    ||
    || I misinterpreted the reference, since you did not include the
    || relevant information about reporting only the RHS attributes.
    |||
    ||| Similar goes for the filename part, which I have to display in Unix
    ||| format (forward slashes and no drive letters).
    ||
    || Seems you want to imitate the Unix "ls" program! Now that I
    || understand the issues better, I can suggest a tedious workaround.
    || 1/ create a file containing both the SFN and the LFN of each
    || file, with a separator such as vertical bar | between them, e.g.
    || using the command pdir /(fPN"|"fpn)
    || 2/ process this file one byte at a time, using @filereadb:
    || 2.1/ build the SFN
    || 2.2/ use the SFN and functions like @attrib, @filedate, etc. to
    || obtain all desired information aboout the file
    || 2.3/ build your output line from the file information collected,
    || and from reading each character of the LFN from the catalog,
    || performing appropriate translations, e.g., \ to /.
    || Note that I when using the @fileopen/@fileread scheme, I found the
    || simplest way to exit your loop at the end of file is in the style
    || below:
    ||
    || set size=%@filesize[file]
    || set hndl=%@fileopen[file,r,b]
    || do while %size LT %@fileseek[%hndl,1,0]
    || read
    || work
    || enddo
    ||
    || BTW, I had a few files from MS-DOS days, when one vendor used the
    || percent sign % in the last position of the file extension to indicate
    || compressed files. I had a lot of trouble with them, but since they
    || were just archived legacy files, I solved the issue by brute force -
    || I changed the names to Microsoft's choice of characters for the same
    || purpose, the underscore _. If possible, that's a better method...
    || Please also read about Charles Dye's plugin Safechars:
    || http://www.unm.edu/~cdye/dl/safechars.zip
    ||
    || --
    || HTH, Steve
     
  7. NeBlackCat

    Joined:
    Dec 5, 2009
    Messages:
    9
    Likes Received:
    0
    Hi Jim, I suspect the reason you don't see it is because you're using two backquotes not one, so you wouldn't get an error about a missing closing quote. Sorry - I should probably have stated that clearer.

    You can reproduce it using the files I zipped and attached to this post. Just a handful of 1 byte text files with strange filenames, a couple of which will trigger this issue.
     
  8. NeBlackCat

    Joined:
    Dec 5, 2009
    Messages:
    9
    Likes Received:
    0
    My apologies!

    Yes, I have to produce output that's directly file-comparable with other tools on both Windows and Linux.

    I like the idea of using SFNs to dodge around the problem of characters that can appear in LFNs, but unfortunately I can't guarantee SFNs will be available as some (like me) tend to turn them (and other things, like last access time updating) off as NTFS performance tweaks.

    The plugin looks usefull too - I'll play with that and see what can be done.

    Thanks again!
     
  9. Steve Fabian

    Joined:
    May 20, 2008
    Messages:
    3,520
    Likes Received:
    4
    NeBlackCat wrote:
    | ---Quote (Originally by Steve Fbin)---
    | Seems you want to imitate the Unix "ls" program! Now that I
    | understand the issues better, I can suggest a tedious workaround....
    | ---End Quote---
    | Yes, I have to produce output that's directly file-comparable with
    | other tools on both Windows and Linux.

    Most of the *nix tools are available as freeware, typically under GPL,
    including ls, from several sources. Until the arrival of PDIR, I often used
    the Cygwin version of ls, esp. when dealing with inodes (to identify
    multiply cataloged files). There is still a version on my system, dated
    2006-07-05. Since you apparently need to deal with both NTFS and Posix file
    systems, you might be better off using ls!
    --
    HTH, Steve
     
  10. rconn

    rconn Administrator
    Staff Member

    Joined:
    May 14, 2008
    Messages:
    9,863
    Likes Received:
    83
    First, it's a REALLY REALLY REALLY bad idea to use single back quotes in
    your filenames. You're going to have endless problems with the parser, and
    I strongly urge you to rename your files to more rational names.

    Second, this doesn't have anything to do with PDIR; it's the @RIGHT variable
    function that's complaining about the single back quote. The functions that
    take a single argument (like @ATTRIB) do not complain because they don't
    need to parse multiple arguments.

    Third, you probably don't need to do any of this with PDIR, since there's
    internal PDIR options for everything you've mentioned. If you can give an
    example of exactly what you're trying to do, we can provide suggestions on
    how to do it without variable functions.

    Rex Conn
    JP Software
     
  11. NeBlackCat

    Joined:
    Dec 5, 2009
    Messages:
    9
    Likes Received:
    0
    First - unfortunately I have no control over what characters appear in file names as I do not create them. There were tens of thousands of them in the first test batch, and only a few using WTF characters, but they are still technically legitimate and still have to work.

    Second - I know. I even tried @unicode-ing the filename ("*"), processing that, then @char-ing it back again for printing, but there's no way to avoid @right to chop off drive letters (you can't tell @instr, which takes 2 parameters and therefore works, to return everything after the first two characters due to the way it's position parameters are defined), and @right still fails. (*)

    Third - unless I'm missing something, there is no way to tell PDIR to do what I've previously stated (print only the RHS attributes, and show the filename in Unix style ie. slashes converted to forward, and drive letter chopped off), and no way to pass PDIR's invididual output fields to variable functions (just the "*").

    (*) if setdos /x had greater granularity of control, ie. allow individual control of each quotes type rather than all of them together, I'd have been ok. Perhaps that's one for the wish list.
     
  12. Steve Fabian

    Joined:
    May 20, 2008
    Messages:
    3,520
    Likes Received:
    4
    NeBlackCat wrote:
    | Second - I know. I even tried @unicode-ing the filename ("*"),
    | processing that, then @char-ing it back again for printing, but
    | there's no way to avoid @right to chop off drive letters (you can't
    | tell @instr, which takes 2 parameters and therefore works, to return
    | everything after the first two characters due to the way it's
    | position parameters are defined), and @right still fails. (*)

    Using @INSTR[2,9999,%x] emulates @RIGHT[-2,%x]. However, I suspect it will
    fail in the same manner as @RIGHT.

    Did you look into Charles Dye's safechars.dll? It may do what you need.

    Another alternative method to achieve your goal is to use PDIR to collect
    raw data into a binary array, where the "a" report field will have
    characters superfluous to your use, but which can be easily discarded.
    Likewise, all special characters will be properly stored, and you can just
    parse the array a character at a time into your desired format. Tedious, I
    know, but not overly complex.
    --
    HTH, Steve
     
  13. Jim Cook

    Joined:
    May 20, 2008
    Messages:
    604
    Likes Received:
    0
    On Tue, Dec 8, 2009 at 6:46 AM, Steve Fábián <> wrote:

    Or even use PDIR | perl or the moral equivalent. It ceases to be a
    native solution, but sometimes the wrong tool should not be rigged to
    work.

    --
    Jim Cook
    2009 Saturdays: 4/4, 6/6, 8/8, 10/10, 12/12 and 5/9, 9/5, 7/11, 11/7.
    Next year they're Sunday.
     
  14. Charles Dye

    Charles Dye Super Moderator
    Staff Member

    Joined:
    May 20, 2008
    Messages:
    3,307
    Likes Received:
    39
    I'm not really sure how. Massage the offending filename into a .CMD file, then call CMD.EXE to rename it to something sane?
     
  15. Steve Fabian

    Joined:
    May 20, 2008
    Messages:
    3,520
    Likes Received:
    4
    Charles Dye wrote:
    | ---Quote (Originally by Steve Fbin)---
    | Did you look into Charles Dye's safechars.dll? It may do what you
    | need. ---End Quote---
    | I'm not really sure how. Massage the offending filename into a .CMD
    | file, then call CMD.EXE to rename it to something sane?

    No, renaming is not appropriate for files generated externally, and most
    likely to be processed again by software not under the OP's control. Rather,
    by using PDIR in a more native mode, e.g., using the "a" report field,
    redirecting its output into a temporary file, and processing each line of
    the temporary file with @saferead or @safeenv, dropping all unwanted
    characters, replacing backslashes with slashes, etc. Of course, the output
    must be written using @SAFEWRITE to restore the original codes for the
    special characters.

    BTW, Charles, there are two more operations desirable in SAFECHARS.DLL,
    @safeclip[] ought to have its reverse @safeclipw[], and for Unicode=Yes it
    would be desirable to have an EchoSafeUnicode command. This last one might
    be very tricky, though - need to know whether or not it is redirected to a
    file, and in the latter case whether it is creating the file (need to write
    BOM) or apppending to one. Possibly could have two separate commands, one to
    create the file with just the BOM, the other to append, with the filename
    explicitly specified (as done by TEE command).
    --
    Steve
     
  16. Charles Dye

    Charles Dye Super Moderator
    Staff Member

    Joined:
    May 20, 2008
    Messages:
    3,307
    Likes Received:
    39
    That would probably be easy.

    What would that do that %@safewrite doesn't?
     
  17. Steve Fabian

    Joined:
    May 20, 2008
    Messages:
    3,520
    Likes Received:
    4
    Steve wrote:
    | an EchoSafeUnicode command

    Charles Dye wrote:
    | What would that do that %@safewrite doesn't?

    Create output files without the cumbersome @file... functions, in other
    words, hide the details of the file manipulation froms the user, just like
    the "for %line in (@file)", or its equivalent with DO, move the burden of
    details from the user's code to the command processor.
    --
    Steve
     
  18. Charles Dye

    Charles Dye Super Moderator
    Staff Member

    Joined:
    May 20, 2008
    Messages:
    3,307
    Likes Received:
    39
    All that trickiness seems to be implemented in Take Command already. Just hand a string to QPuts(), and //UnicodeOutput (and the BOM if needed) are handled automatically. Cool.

    I'm putting up a new build at http://www.unm.edu/~cdye/plugins/ which I think will do what you want.
     

Share This Page