Command line evaluation question - variable inside parenthesis

[nikbackm]

Hi,

I stumbled upon a (to me) strange thing when I generated some SQL for piping to sqlite3.exe.

These two statements where called in a loop:

set extraWhereCondParsed=%extraWhereCondParsed% OR (%thisCondParsed%)
echo "%extraWhereCondParsed%"

Two outputs from the echo command were:

"("Last Name" LIKE 'b*') OR (%thisCondParsed%)"
"("Last Name" LIKE 'b*') OR (%thisCondParsed%) OR (%thisCondParsed%)"

So the environment variable inside the parenthesis was not evaluated. (But other variables inside parenthesis elsewhere were evaluated, just not this one.

Adding `` around the parenthesis:es fixed the problem.

set extraWhereCondParsed=%extraWhereCondParsed% OR `(`%thisCondParsed%`)`
echo "%extraWhereCondParsed%"

"("Last Name" LIKE 'b*') OR ("First Name" LIKE 'a*')"
"("Last Name" LIKE 'b*') OR ("First Name" LIKE 'a*') OR ("First Name" LIKE 'a*')"

Any idea what's going on? Would it always be a good idea (or even necessary in some cases) to use `` around literal (and not meant for e.g. command grouping) parenthesis:es?

 

[rconn]

I cannot duplicate your results (I suspect you've simplified things in the message to the point that it doesn't show the complete steps to reproduce it).

TCC will not expand variables inside a command group (this is a feature!), but with what you've shown TCC shouldn't think this *is* a command group.

If you think you have a case where it is not expanding variables where it should please try to reduce it to a simple batch file.

 

[nikbackm]

Here is a simplied batch to demonstrate the problem.

Well, as simplified as I could make it with not too much effort. :)

To trigger the (possible) error just run:

err bb -wfn.a* -wfn.b*

Output "will" be:

extraWhereCondParsed: ("First Name" LIKE 'b*')
extraWhereCondParsed: ("First Name" LIKE 'b*') OR (%thisCondParsed%)
Active code page: 1252
.header on
.mode column
.width 6 40 15 15 10 3 3
SELECT Books."BookID","Title","Last Name","First Name","Date read","Owned","Bought Ebook" FROM Books
INNER JOIN DatesRead USING(BookID)
        INNER JOIN AuthorBooks USING(BookID)
        INNER JOIN Authors USING(AuthorID)
WHERE (Title LIKE '%%') AND (("First Name" LIKE 'b%%') OR ())
ORDER BY "Last Name","First Name","Title";

But it "should" be:

extraWhereCondParsed: ("First Name" LIKE 'b*')
extraWhereCondParsed: ("First Name" LIKE 'b*') OR ("First Name" LIKE 'a*')
Active code page: 1252
.header on
.mode column
.width 6 40 15 15 10 3 3
SELECT Books."BookID","Title","Last Name","First Name","Date read","Owned","Bought Ebook" FROM Books
INNER JOIN DatesRead USING(BookID)
        INNER JOIN AuthorBooks USING(BookID)
        INNER JOIN Authors USING(AuthorID)
WHERE (Title LIKE '%%') AND (("First Name" LIKE 'b%%') OR ("First Name" LIKE 'a%%'))
ORDER BY "Last Name","First Name","Title";

"First Name" LIKE 'a*' is simply not expanded from the variable.

Error happens in routine :extractExtraWhereCond [whereCond].
I added some "!!!" markers where it happens.

I

 

[rconn]

Re: Here is a simplied batch to demonstrate the problem.

Well, as simplified as I could make it with not too much effort. :)

Not a bug.

You're running into a problem with nested variable expansion -- your first argument is being expanded to "("First Name" Like 'b*')". TCC then looks to see if it should expand this argument again, and finds that it's now looking at a command group -- so it skips further expansion on the line.

If you do a "setdos /x-4" it should work as you expect.

 

[nikbackm]

Re: Here is a simplied batch to demonstrate the problem.

Not a bug.

You're running into a problem with nested variable expansion -- your first argument is being expanded to "("First Name" Like 'b*')". TCC then looks to see if it should expand this argument again, and finds that it's now looking at a command group -- so it skips further expansion on the line.

If you do a "setdos /x-4" it should work as you expect.

Hm, I don't quite understand why the variable expansion would be nested.

iff defined extraWhereCondParsed then
    set extraWhereCondParsed=%extraWhereCondParsed% OR (%thisCondParsed%)
else
    set extraWhereCondParsed=(%thisCondParsed%)
endiff

echo extraWhereCondParsed: %extraWhereCondParsed%

The gosub routine is called two times.

The first time the else branch is taken and extraWhereCondParsed is set to "(%thisCondParsed%)" The following echo also shows that "thisCondParsed" was expanded correctly and not "got stuck" inside extraWhereCondParsed.

The next time the if-branch is taken, but this time the same "(%thisCondParsed%)" expression fails to expand. Is that just because it came after a preceding (not nested) variable expansion of "%extraWhereCondParsed%" on the same line?

setdos /x-4 DID help, I just don't understand where the nesting comes in. Does it really matter (apparently) where in the line something is?

Changing the order like this also makes it expand correctly.

iff defined extraWhereCondParsed then
    set extraWhereCondParsed=(%thisCondParsed%) OR %extraWhereCondParsed%
else
    set extraWhereCondParsed=(%thisCondParsed%)
endiff

echo extraWhereCondParsed: %extraWhereCondParsed%

I think I prefer using `` around the parenthesis since this expresses the intent best (assuming it's always guaranteed to disable what happens with command grouping here).

 

[rconn]

Re: Here is a simplied batch to demonstrate the problem.

Hm, I don't quite understand why the variable expansion would be nested.

*All* variable expansion is nested unless disabled with SETDOS /X-4. Every time a variable is expanded, the parser runs it through expansion again until it's sure there aren't any more variables. In your case, it expanded the variable, found a command group, and aborted further expansion on the entire command.

 

[nikbackm]

Re: Here is a simplied batch to demonstrate the problem.

*All* variable expansion is nested unless disabled with SETDOS /X-4. Every time a variable is expanded, the parser runs it through expansion again until it's sure there aren't any more variables. In your case, it expanded the variable, found a command group, and aborted further expansion on the entire command.

Ok, I'm starting to get it. Must be a bit slow today. :)

Thanks for taking the time to explain!


So, the second gosub call, when

set extraWhereCondParsed=%extraWhereCondParsed% OR (%thisCondParsed%)

is expanded, then "%extraWhereCondParsed%" will expand to a command group (as set by the else-branch the first call) and this will stop further expansion of variables on the same line. (Adding back quotes around "(" and ")" made them not behave like a command group so in that case it worked as wanted).

Is this behavior documented? The topic "Command Parsing" - "3. Expanding variables" does mention that variable expansion works differently for some internal commands like IF and GLOBAL, but does not mention command groups.
Or maybe that's implied by:

These commands are always followed by another command, so variable expansion takes place separately for the original command and the command that follows it.

Still don't understand how SETDOS /x-4 helped (as well) as that disables nested variable expansion. Which sounds like the opposite of what would be needed. And the variables WERE still expanded.

 

[nikbackm]

Re: Here is a simplied batch to demonstrate the problem.

Ok, I'm starting to get it. Must be a bit slow today. :)

Still don't understand how SETDOS /x-4 helped (as well) as that disables nested variable expansion. Which sounds like the opposite of what would be needed. And the variables WERE still expanded.

Wait, think I just got it.

Disabling nested variable expansion means that "the expander" just expands %extraWhereCondParsed% without further looking at the expanded value. So it does not see that it expands to a command group which would disable further variable expansions on the rest of the command line (and not only of the newly expanded value as setdos /X-4 does).

Initially I just thought of disabling nested variable expansion as having to effect of not further expanding hidden variables becoming visible as a result of expanding an "outer" variable.

Seems it can have other (more subtle) effects too! Well, at least I'm aware of this one now.