Random errors

[Jesse Heines]

I have written a complex script the downloads mutual fund information from fidelity.com, processes that information with TPIPE, and then outputs selected information. The script sometimes works perfectly a shown in the first instance of running it in the attached file. If I run it a second time, however, I sometimes get very strange errors such as:

TCC: D:\E-Drive\Finance\getquotes13.btm [736] Unknown command "&"]"
TCC: D:\E-Drive\Finance\getquotes13.btm [736] No expression "[7,4]"

Line 736 is the end of a large loop. There are absolutely no ampersands in the entire batch file. And [7,4] is an array reference which worked perfectly the previous time I ran the batch file.

What's more, if I continue to run the script repeatedly as shown in the attached file, these errors occur at different places. Very strange.

I thought that this might be some sort of timing problem, that is, something in the script getting ahead of itself. I therefore added a PAUSE statement to slow things down. Sometimes that helped and sometimes it didn't. Again, very strange.

The script sometimes works perfectly, as is proven by the first run in the attached file, but for the life of me I can't figure out what's causing it to fail and print spurious error message in other runs.

Any suggestions would be greatly appreciated.

Jesse Heines, TCMD users since 1998!

 

[vefatica]

A downloaded file with '&' in it's name could trigger the first error. And if you're using SETARRAY /R with with such a filename, unquoted, that might cause an array to be undefined (the second error?).

 

[Joe Caverly]

@vefatica why does it work the first time it is run?

@Jesse Heines Does your script begin/end with SETLOCAL/ENDLOCAL ?

Joe

 

[vefatica]

@vefatica why does it work the first time it is run?

No clue. And if the BTM is over 736 lines, I don't want to see it.

If any arrays/variables are left over from previous runs, things could fail. Array variables aren't protected by SETLOCAL.

 

[Joe Caverly]

Indeed.

Always best to UNSETARRAY before program exit.

Joe

 

[Jesse Heines]

A downloaded file with '&' in it's name could trigger the first error. And if you're using SETARRAY /R with with such a filename, unquoted, that might cause an array to be undefined (the second error?).

Thank you for your response and your suggestions on what might be wrong, but none of the files have an '&' in their names. I do not use SETARRAY /R. I manually set the array values in the script. The only data I read from a file are the current mutual fund values extracted from the Fidelity website and edited by TPIPE. The resultant file being read does not contain any ampersands. I open it with %@FILEOPEN and read its data with %@FILEREAD. I have verified that the data itself is read perfectly by simply using ECHO to display the raw data that is stored in a 2D array. But thank you again for replying. I'm sure that at some point someone will take a stab at a guess and that'll prove to be the solution.

 

[Jesse Heines]

@vefatica why does it work the first time it is run?

@Jesse Heines Does your script begin/end with SETLOCAL/ENDLOCAL ?

Joe

I don't know why it works the first time it is run and not subsequent times. My script did not begin/end with SETLOCAL/ENDLOCAL. Thank you for that tip. I just added these, but unfortunately that did not solve the problem.

 

[Jesse Heines]

No clue. And if the BTM is over 736 lines, I don't want to see it.

If any arrays/variables are left over from previous runs, things could fail. Array variables aren't protected by SETLOCAL.

Yes, the BTM is indeed 918 lines long. This is one reason why I didn't post it. The other reason is that it is complex, and despite the fact that I have documented it extensively for my own edification, someone would have to devote considerable time to understanding it. That said, if someone willing to take a serious look at it I might be willing to make it available to him or her.

 

[Jesse Heines]

Indeed.

Always best to UNSETARRAY before program exit.

Joe

Thank you, Joe, but I do use UNSETARRAY /q funds before SETARRAY funds[22,10], so I don't think that that's the problem.

 

[RogerB]

Since the sensible suggestions made so far haven't worked, here're some ideas and wild speculation that I hope might help you debug the problem, presented in no particular order.

1. Is the file (or files) you're downloading from Fidelity an HTML file? If so, does it contain adverts that change each time it's downloaded? If so, perhaps some adverts contain data that is throwing off your script when it processes the file, but others don't, hence the irregular nature of the failures.

2. If it is an HTML file, then (as I'm sure you're aware) there are lots of characters in HTML that have special meaning to TCC (such as the redirection characters). If you're not already stripping all the HTML characters with TPIPE you could try experimenting with SETDOS to turn off various TCC processing and see if you can find what's causing it, although of course if you turn off something you're using in the batch file you'll create new problems...

3. Although the errors don't occur every time you run it, from your attached output file it looks like the error happens at the same spot in each line - if you count the columns as separated by the colon the error message occurs in the third column every time, so you might want to look closely at the code in that area to see if you can spot anything.

4. Do you use logging? If you put a "Log on" at the start of the file and a "Log off" at the end you'll get a log file generated that will contain all of the commands executed. Make sure you log errors to the same file and then you can examine the resulting log file to look for issues when the error occurs.

5. Have you run the file in TCC's debugger? That will allow you to set breakpoints immediately before the lines causing the problem and inspect the values of variables etc. which might give you a pointer to the problem.

I hope that something there might help you track it down - good luck!

 

[Jesse Heines]

Thank you for all your suggestions, RogerB. As I wrote previously, I have verified that the data is read perfectly from the files processed by TPIPE. Thus, your #1 and #2 are not issues. Your #3 is correct, and I've been looking at that. I've had trouble with your #4: I can turn on logging inside the batch file, but LOG OFF inside the batch file doesn't seem to turn it off. Typing log off on the command line doesn't turn it off, either. Thus, I'm a bit confused about that. Turning logging on via the command line, then running the batch file, then turning logging off again on the command line does seem to work. I have never used TCC's debugger before, but I just tried it out and of course it works well. I'll have to find some time to work through the code with that. Thanks again for your help.