Using Regular Expressions to find comments in .btm files

[Joe Caverly]

I have been using the following to find comments in .btm files;

R:\>ffind /kvm /e"^( {0}::| {0}REM )") r:\zoom150.btm
::    File: zoom150.btm
:: Created: 2024-07-19
::   _4ver: 32.10
:: _winver: 10.0
::echo %_startpid
::activate "New tab*"
:: Changed to 13 (was 15) on 2024-07-21
:: Changed to 16 (was 13) on 2024-10-20

However, this neglects comments that do not appear at the beginning of a line.

No biggie, it still gives me the gist of the comments.

A recent conversation has encouraged me to try and find comments that not only appear at the beginning of a line,
but that begin in iff structures and such.

As Regular Expressions and I are not friends,
even after reviewing Beginning Regular Expressions
I gave in and asked DuckAI for assistance.

Link to DuckAI conversation: Shared Duck.ai Chat
(the link expires 30 days from now).

Thus, I have updated my Comments.btm script;

@SETLOCAL
@ECHO OFF
if isdir r:\temp set temp=r:\temp

iff %# eq 0 then
  echo USAGE: %_batchname theFile.btm
  quit
endiff

set theFile=%1

iff exist %theFile then
  ffind /kvm /e"^[ \t]*(::|REM )" %theFile
  ffind /kvm /e"^[ \t]*(::|REM )" %theFile > tmp9:
  echo ^nNumber of comment lines: %@inc[%@lines[tmp9:]]
else
  echo %theFile does not exist.
endiff

:eoj
ENDLOCAL

Output;

R:\>comments r:\zoom150.btm
::    File: zoom150.btm
:: Created: 2024-07-19
::   _4ver: 32.10
:: _winver: 10.0
                :: NEXT SENTENCE
::echo %_startpid
::activate "New tab*"
:: Changed to 13 (was 15) on 2024-07-21
:: Changed to 16 (was 13) on 2024-10-20

Number of comment lines: 9

I re-visted this due to the new LINES command in the FileUtils plugin,
which shows the number of comment lines in a .btm file.

FileUtils plugin: FileUtils Plugin

Joe

 

[Jay Sage]

If you change {0} to {0,}, the expression will allow any number of spaces before the comment designator. Or you could replace {0} with * (which means the same thing).

 

[Joe Caverly]

Thanks @Jay Sage ,
but neither of those caught the line with the leading spaces.

R:\>ffind /kvm /e"^( {0,}::| {0,}REM )") r:\zoom150.btm
::    File: zoom150.btm
:: Created: 2024-07-19
::   _4ver: 32.10
:: _winver: 10.0
::echo %_startpid
::activate "New tab*"
:: Changed to 13 (was 15) on 2024-07-21
:: Changed to 16 (was 13) on 2024-10-20

R:\>ffind /kvm /e"^( *::| *REM )") r:\zoom150.btm
::    File: zoom150.btm
:: Created: 2024-07-19
::   _4ver: 32.10
:: _winver: 10.0
::echo %_startpid
::activate "New tab*"
:: Changed to 13 (was 15) on 2024-07-21
:: Changed to 16 (was 13) on 2024-10-20

R:\>comments.btm r:\zoom150.btm
::    File: zoom150.btm
:: Created: 2024-07-19
::   _4ver: 32.10
:: _winver: 10.0
                :: NEXT SENTENCE
::echo %_startpid
::activate "New tab*"
:: Changed to 13 (was 15) on 2024-07-21
:: Changed to 16 (was 13) on 2024-10-20

Number of comment lines: 9

Joe

 

[Joe Caverly]

Hey @Jay Sage it would seem the line;

                :: NEXT SENTENCE

...is preceeded with two tabs,
not spaces,
which explains why your solution did not work.

I originally missed the fact that the comment line was using tabs,
not spaces.

Joe

 

[Joe Caverly]

Yes @Jay Sage I changed the tabs to spaces,
and your solution works for that comment line.

Thanks for helping me remember that I used tabs instead of spaces.

Joe

 

[Charles Dye]

Perhaps the regex engine has some expression to match whitespace characters in general, instead of just tabs?

 

[Joe Caverly]

From what DuckAI explained to me;

\s includes newline characters in TCC’s default regular-expression syntax.
So \s* can consume past the end of a line—and across blank lines—before matching :: or REM.
That explains why the output includes an empty line.
The documented whitespace class includes \n and \r.

Ref: Extended Windows Wildcards and Windows Regular Expressions in TCC

Ref: DuckAI conversation: Shared Duck.ai Chat

Joe

 

[Jay Sage]

You can use this:

ffind /kvm /e"^\s*(::|REM)"​

That expression is looking for the beginning of a line (^) followed by any amount of white space (\s*) followed by either "::" or "REM".

An easy way to learn about and experiment with regular expressions is to open the Regular Expression Analyzer" tool that is part of Take Command. The help file also includes a lot of detailed information.

 

[Jay Sage]

I just discovered that the expression above is case-sensitive, which is not what one wants (since "rem" or "ReM") can begin a comment). One needs to add the option (?i).

ffind /kvm /e"^\s*(::|(?i)REM)"

The option could be inserted at the beginning of the entire expression to make the case independence apply everywhere in the test.

ffind /kvm /e"(?i)^\s*(::|REM)"

In addition, one should require that "REM" be followed by white space.

ffind /kvm /e"(?i)^\s*(::|REM\s)"

 

[vefatica]

I occasionally do the likes of this.

echo foo & REM comment

This works too (recently tested).

echo foo & :: comment

So how would you find them? You'd need some text, then an unescaped & (whitespace on either side), then REM or ::

I don't feel like trying that one. ^ is just too special.

I thought of suggesting you TEE TMP9: but that apparently doesn't work (sigh!).

How do I find a discussion of TMP0: - TMP9: in the help?

 

[Joe Caverly]

You can use this:

ffind /kvm /e"^\s*(::|REM)"​

That produces a blank line in the search results;

R:\>ffind /kvm /e"^\s*(::|REM)" zoom150.btm
::    File: zoom150.btm
:: Created: 2024-07-19
::   _4ver: 32.10
:: _winver: 10.0
                :: NEXT SENTENCE
::echo %_startpid
::activate "New tab*"

:: Changed to 13 (was 15) on 2024-07-21
:: Changed to 16 (was 13) on 2024-10-20

Refer to Post #7 in this thread for why that happens.
Reference: Using Regular Expressions to find comments in .btm files

Joe

 

[Jay Sage]

OK, I'll look into your comment. Meanwhile, I was worrying about lines beginning with the string REM followed by another character, which makes it not a comment. Does this work? It seems to for me.

ffind /kvm /e"(?i)^\s*(::|rem(\s|$))"

 

[Joe Caverly]

I still get the blank line.

R:\>ffind /kvm /e"(?i)^\s*(::|rem(\s|$))" r:\zoom150.btm
::    File: zoom150.btm
:: Created: 2024-07-19
::   _4ver: 32.10
:: _winver: 10.0
                :: NEXT SENTENCE
::echo %_startpid
::activate "New tab*"

:: Changed to 13 (was 15) on 2024-07-21
:: Changed to 16 (was 13) on 2024-10-20

From DuckAI;

\s includes newline characters in TCC’s default regular-expression syntax.
So \s* can consume past the end of a line—and across blank lines—before matching :: or REM.
That explains why the output includes an empty line.
The documented whitespace class includes \n and \r.

This works for me;

ffind /kvm /e"^[ \t]*(::|REM )"

Joe

 

[Joe Caverly]

OK, I'll look into your comment. Meanwhile, I was worrying about lines beginning with the string REM followed by another character, which makes it not a comment. Does this work? It seems to for me.

ffind /kvm /e"(?i)^\s*(::|rem(\s|$))"

Using this test.btm file;

R:\>type test.btm
@SETLOCAL
@ECHO OFF
if isdir r:\temp set temp=r:\temp

:: Comment
REM Comment
REMbrandt
        REMbrant
        :: Comment
:: Comment
ENDLOCAL

R:\>ffind /kvm /e"(?i)^\s*(::|rem(\s|$))" r:\test.btm

:: Comment
REM Comment
                :: Comment
:: Comment

...works, and it ignores lines where REM is followed by another character.

Note that the lines with pre-ceeding whitespace are tabs,
not spaces.

Using this as a test;

@SETLOCAL
@ECHO OFF
if isdir r:\temp set temp=r:\temp

:: Comment
REM Comment
REMbrandt
        REMbrant
        :: Comment
        
:: Comment
ENDLOCAL

It still emits a blank line,
due to newline characters in TCC’s default regular-expression syntax;

R:\>ffind /kvm /e"(?i)^\s*(::|rem(\s|$))" r:\test.btm

:: Comment
REM Comment
                :: Comment

:: Comment

Joe

 

[epement]

Using this as a test;

@SETLOCAL
@ECHO OFF
if isdir r:\temp set temp=r:\temp

:: Comment
REM Comment
REMbrandt
        REMbrant
        :: Comment
       
:: Comment
ENDLOCAL

It still emits a blank line,
due to newline characters in TCC’s default regular-expression syntax;

R:\>ffind /kvm /e"(?i)^\s*(::|rem(\s|$))" r:\test.btm

:: Comment
REM Comment
                :: Comment

:: Comment

Joe

To omit the blank lines, in the regular expression used for ffind, replace the \s with [ \t] or with [[:blank:]]. The latter is a POSIX character class supported by Perl that means the same thing as [ \t]. It is longer to write, but on the other hand it does not require any spaces in the bracketed character set.

In my opinion, something unexpected is happening with how ffind and fsearch (its successor) handles the input file or standard input. Normally, perl itself works just like grep: it consumes one line at a time, applies the pattern to the single line in the pattern buffer, and prints that line only if the pattern matches. Neither perl nor grep will print blank lines on this pattern, in spite of the \s escape sequence:

:: I changed * to + to focus on the extraneous blank lines
perl -ne "print if /(?i)^\s+(::|rem(\s|$))/" test.btm
pcregrep "(?i)^\s+(::|rem(\s|$))" test.btm

And in the test above, they do not produce blank lines, whereas ffind and fsearch do emit blank lines. This means that ffind are not matching on each line, but are stuffing all the lines of the file (or stdin) into a buffer and applying the expression to the buffer. Observe:

[ 2:30] > echo `one^nblue^nfish` | ffind /km /e"one\s*blue\s*fish"
one

[ 2:30] > echo `one^nblue^nfish` | fsearch /nh /ns /e"one\s*blue\s*fish"
one
blue
fish

Clearly, ffind is a little behind the curve here, and fsearch has fixed a matching problem. Hope this helps.

TLDR: Use [ \t] instead of \s to avoid matching blank lines with the ffind utility.

 

[Jay Sage]

Or just use \w instead of \s.

 

[epement]

Or just use \w instead of \s.

Oh, not \w. In Perl, \w is an escape sequence to match "word characters", defined as alphanumerics plus underscore, matching the same as the character set [A-Za-z0-9_]. Well, normally. Since Perl is Unicode conversant, \w technically matches «alphanumeric plus "_", plus other connector punctuation chars plus Unicode marks». But if the input doesn't contain Unicode characters above 0xFF, then it drops back to its usual meaning.

You may be thinking of \h, a Perl escape sequence to match "horizontal spaces", namely, space or tab. It matches the character set [ \t]. If Unicode input has additional horizontal space characters, those will also match that sequence.

See https:||perldoc.perl.org/perlre#Escape-sequences for the documentation.

Although I have chosen the TCC option to use Perl-style regular expressions, \h is not supported in TCC. This omission is documented here: https:||jpsoft.com/help/regularexpressionsyntax.htm#toggle0186ad_LINK in a part of the documentation I often overlook.

 

[Jay Sage]

Thank you, Eric. That was a moment of confusion. on my part It happened to work with my sample file, but not because it was a correct expression.

Does ( |\t)* work? Or [ \t]*?

 

[epement]

Does ( |\t)* work? Or [ \t]*?

For Joe's case that began this thread, he should use (?i)^[ \t]*(::|rem([ \t]|$)) to match those remarks in batch files.

The regexes ( |\t)* and [ \t]* match the same patterns but work differently.
( |\t)* creates a group matching zero or more occurrences of either spaces or tabs.
[ \t]* is a character set matching zero or more occurrences of either spaces or tabs.

To make this easier to understand, consider the following example, where I use (a|b) because they are visible characters.
In the Perl code, switch -p tells perl to print by default, and -e means expressions or Perl code follow:

[ 8:57] > echo ababbbabbbacde | perl -pe "s/(a|b)*/{$1}/;"      # match (groups) with $1, $2, etc.
{a}cde

[ 8:57] > echo ababbbabbbacde | perl -pe "s/(?:a|b)*/{$&}/;"    # create non-capturing groups
{ababbbabbba}cde

[ 8:57] > echo ababbbabbbacde | perl -pe "s/[ab]*/{$&}/;"       # identical to the match above
{ababbbabbba}cde

Hope this helps!