robocopy cancelling batch file processing

May 30, 2008
14
0
I have a set of (.BTM) batch files that I use to update offline copies of my working directories that use the robocopy program. The top level batch files call a common batch file to perform the work in a uniform way. I have noticed that occasionally batch processing terminates prematurely. I have attempted to work around the issue by looping until a flag is set in both the common batch file and the top level batch files, the looping code never gets a chance to continue the processing because it acts like a cancel command was encountered. If I manually rerun the top level batch file it will typically run normally but it may also terminate at another point in the processing.

I recently modified my common batch file to enable command echoing just before running robocopy and got lucky when the issue occurred when synchronizing the first directory. The invocation of robocopy completed and the batch file continued running to next two statements which were on break and a return. At this point the batch file processing was cancelled with a call, 3 gosubs, an iff , and another gosub pending.

This issue occurs on multiple different PCs. All of the PCs are currently running Windows 10, but if I remember correctly the issue also occured under Windows 7. The working directories are on the C: drive and the offline mirror copies are on USB 3.0 drives (SSD or magnetic).

The common batch file can perform the processing using either robocopy or the builtin TCC copy command. I have never seen this issue occur with the copy command but robocopy is about 10 times faster than the copy command.
 
Last edited:
May 20, 2008
11,378
98
Syracuse, NY, USA
I don't know how an external EXE (robocopy) could cause a batch file to terminate unless the batch file were somehow cooperating. Can you give more information?
 
May 30, 2008
235
2
I've noticed a similar issue when using TCC 20 on Windows 10.

Often the batch file just stops prematurely after a ROBOCOPY command completes even though there are many more ROBOCOPY calls left in the file. No error messages are output.

IF you re-run the batch file then it always continues successfully after the ROBOCOPY call where it stopped last time. Probably because there are no changes to copy, and no output from robocopy to display. But a subsequent robocopy call can stop batch processing instead. So just continue re-running until all robocopy calls are completed, lol.

The batch file just calls robocopy directly and does not check the error code from it, no other batch files are called.

This only happens with TCC 20, I also have TCC 15 and there the problem never occurs with the same batch file, same Windows 10 and same computer.

I would guess it has something to do with ANSI processing myself, but have never taken the time to investigate, easier to just re-run the file when not complete.

I attached a simplified version of the batch file.
 

Attachments

  • generic_backup.btm
    567 bytes · Views: 148

rps

Jul 6, 2008
436
6
Have you tried using the START command vs CALL with the ROBOCOPY line?
That should remove any interaction with your batch file.
 
Apr 2, 2011
1,427
10
54
North Carolina, USA
From the CHM help file:

CALL

Purpose: Execute one batch file from within another.

You do not want a batch file to run from the BTM - you want a EXE to run. Have you looked at the DEFER command?

==================
Purpose:Execute a command after the batch file exits

Format:DEFER command

Usage:

A batch file can have multiple DEFER commands. They will be executed in first in, first out order when the batch file exits.

If you have variables on the DEFER command line, they will be expanded before the DEFER command is processed, not when command is executed. To delay variable expansion until command is executed, use single back quotes around the variable names, or double the %'s before the variable names.

If the last argument on the line is a single (, it is interpreted as the beginning of a command group. DEFER will append the following lines (in a batch file) or prompt you for more input (at the command line) until it gets a closing ).



 
May 30, 2008
235
2
I cannot use START since I want the batch file to wait for ROBOCOPY to complete. START will open it in a new separate console and the batch file will continue to the next robocopy directly.

Yes, I can agree that CALL is unnecessary here as an unqualified ROPOCOPY call will run robocopy.exe on my current system.

I often use CALL from batch files anyway, in case I would have a batch file with the same name as the command, in that case a direct call would replace the current batch file, while with CALL it will continue from the current line afterwards.

Haven't seen that using CALL would ever have hurt when used to run executables and aliases.
(Keep in mind that this works fine in TCC 15)

But sure, could test without it next time I use the batch file.
 
May 26, 2008
537
4
Don't use START... or if you do, use START /WAIT. But that is completely unnecessary. Just run robocopy.exe directly with nothing preceding it!
 
May 30, 2008
235
2
Just tested it again with CALL removed.

And what do you know, the problem totally disappeared, everything went through at the first attempt!

So problem solved on my part, thanks for the suggestions everyone!

Apparently the robocopy process somehow returned the CANCEL "code" at times causing the batch processing to end even though it was not a batch file. Guess it happened to work in TCC15 due to different internal implementation details.
 
Aug 23, 2010
636
9
I cannot use START since I want the batch file to wait for ROBOCOPY to complete. START will open it in a new separate console and the batch file will continue to the next robocopy directly.
START "" /B ... will run in the same console and will be nearly identical to DEFER with exception that your batch file will continue after spawning the task, not before.
Be aware that it might mess the screen, if backgrounded task wold output anything.
 
May 30, 2008
14
0
Has anyone noticed that that ROBOCOPY is unusual in that it modifies the window title. I have not found any other external program that modify the window title.

I have continued to periodically debug this problem and have determined that it only happens when returning from a subroutine that invokes ROBOCOPY, with the RETURN command being treated like a CANCEL command. Until today, I was never able to consistently reproduce the problem.

While working on another batch file I encountered a situation that always causes the batch file to terminate early. The batch file uses a subroutine to process one directory at a time using ROBOCOPY. The subroutine generates the source and destination directory names from a GOSUB parameter and other environment variables, invokes ROBOCOPY, and then returns. When encountering the return statement the batch file always terminates (like the return is a CANCEL statement). I determined that if I add a WINDOW command (WINDOW "") to set the window title to the default, the return works correctly with execution continuing after the GOSUB command.

Starting with this new batch file, I created a minimized batch file that consistently reproduces the error with my files. I then enhanced the minimized batch file so that it could create test files, but have not been able to generated a set of test files that will recreate the problem.

Hopefully this additional information (ROBOCOPY modifying the window title causing a RETURN command being treated as a CANCEL command) will enable JP Software to be able to find the root cause.
 
May 30, 2008
235
2
Wrapping the ROBOCOPY calls in @exec[] finally fixed this problem for me.

(The removal of CALL I wrote about earlier in the thread did not help after all)
 
May 30, 2008
14
0
My batch file already had "on break goto ..." statements before invoking ROBOCOPY and "on break" statements after (including in the destination of the "on break goto ..."), shouldn't that have handled ROBOCOPY sending a Ctrl-C / Ctrl-Break?

About 30 days ago, I modified my batch file to wrap the invocations of ROBOCOPY in @exec[] as per nikbackm's comment and have not had the batch file terminate prematurely since.
 
Jan 18, 2011
13
0
Same problem for many, many years. The robocopy is never interrupted, so that helps a lot. I run the thing for variously between 80 and 100 root directories across 6 disks and it almost always stops after just a few roots folders and then only occasionally after restarting till its done. It's annoying but I do like relying on a clean backup for my portable apps and data.

I have fixed the basic batch file so that it can skip the initial checking and start at any of the disks. Redoing a few root folders goes really quickly (partly since they are already mirrored, but maybe memory caching plays a role as well). In fact I have an alias that restarts from the beginning of whatever disk letter I give it up to 4 times. Not elegant, but not unreasonable even if, when it finally finishes, it has backed up a couple of disks two or maybe three times.

Still it makes a 3 to 5 minute job into a 4 to 6 minute one and requires at least one user intervention to complete.

I wonder if the problem occurs with cmd.exe. Might be a good job for jpsoft to check out.
 
May 30, 2008
235
2
Same problem for many, many years. The robocopy is never interrupted, so that helps a lot. I run the thing for variously between 80 and 100 root directories across 6 disks and it almost always stops after just a few roots folders and then only occasionally after restarting till its done. It's annoying but I do like relying on a clean backup for my portable apps and data.

I have fixed the basic batch file so that it can skip the initial checking and start at any of the disks. Redoing a few root folders goes really quickly (partly since they are already mirrored, but maybe memory caching plays a role as well). In fact I have an alias that restarts from the beginning of whatever disk letter I give it up to 4 times. Not elegant, but not unreasonable even if, when it finally finishes, it has backed up a couple of disks two or maybe three times.

Still it makes a 3 to 5 minute job into a 4 to 6 minute one and requires at least one user intervention to complete.

I wonder if the problem occurs with cmd.exe. Might be a good job for jpsoft to check out.

Did you try wrapping ROBOCOPY in @exec?
 
Jan 18, 2011
13
0
I did, just now. Wow! It works and speeds up the process a lot. It must be that @exec raises the priority; I wonder if a high priority start would do the same thing?

Thanks Niklas, you saved me probably ten minutes a week to say nothing of enough of an annoyance to keep me from doing backups as often as I should. And of course you saved the machine a lot more time than than.

This backup routine is supposed to be so easy and quick that I could do it between coffee sips. I timed it when there were only a couple of tiny files to update and it ran in 5 seconds. That's mirroring 74 folders and 130Gb. Such speed was probably colored by it being done not long after I had already done it the old way and so probably caching played a role.
 
May 26, 2008
537
4
@Jack Weeks if you aren't using it yet, try adding the /MT switch to robocopy. In some situations this can improve performance even more, especially when you're doing update synchronizations.
 
Jan 18, 2011
13
0
Thanks I'll try that when I update a backup on a disk that hasn't been backed up to in a while. I might start with say 30 threads and see if that helps.
 
May 26, 2008
537
4
30 threads is probably too aggressive. There is a point where you'll stop seeing speed increases as you increase threads, and it could actually start getting slower. Depends on a lot of variables. The default of 8 might even be too high. I'd probably compare 2, 4, and 8. If 8 is faster than 4, then continue testing above 8.
 
Jan 18, 2011
13
0
30 threads is probably too aggressive. There is a point where you'll stop seeing speed increases as you increase threads, and it could actually start getting slower. Depends on a lot of variables. The default of 8 might even be too high. I'd probably compare 2, 4, and 8. If 8 is faster than 4, then continue testing above 8.
The effect of the MT switch may be great or small, but so far I haven't been able to detect it. When I find some time I'll try to see if I can't devise a bit better way to measuring it. At least figure out how to add up the work done (bytes and files copied and deleted) verses the time to run, and see if I can provide a useful report.
 
Jan 18, 2011
13
0
Its benefit depends on a lot of factors. It sometimes makes no difference. I would just time two runs, one with no MT switch and the other with /MT:2 to see if there is any difference.
Good Plan!

The attached log summarizes the results of a simple copy from one disk to another and so they are not really representative of mirroring 80 to 100 GB over 70 to 80 root folders with tens of thousands of files only a few of which have changed. (My backup runs a separate robocopy for each root folder.)

Anyway for this just-copy-test you are correct in suggesting that lower MT values are likely to be faster. I ran the test doing 2 to 20 and then 20 to 2 threads to make sure the results were not affected by caching. They were identical. Interestingly the highest score turned out to be without an MT switch -- the actual number of threads reported by Task Manager was 3.

BTW You'd think this kind of testing would have already been reported, but while it's been done none the instances I found were very carefully constructed and were mainly just straight copying. Which is to say much like what I am reporting.
 

Attachments

  • MTLogs-20first.txt
    2.8 KB · Views: 20
May 26, 2008
537
4
I'd also compare MT/no MT when doing updates (instead of just the initial data copy). Maybe you don't actually do update "syncs" with robocopy, but I sometimes use it in that way and MT can make a huge difference (sometimes).
 
Similar threads
Thread starter Title Forum Replies Date
K copy /w slow relative to robocopy /mir Support 0
fpefpe robocopy -- copy /w Support 8
vefatica Cancelling selection in the console? Support 7
S How to? Install from same installer copy after cancelling previously Support 6
Ville Fixed Dangerous operation when cancelling the installer! Support 18
vefatica `Back quotes` - command line vs. batch file Support 5
fpefpe How to? batch file size Support 2
FreezerBurnt Help making a CMD and TCC compatible batch file Support 7
M Oracle batch OPatch.bat.bat not working in TCC but correctly in CMD Support 5
Joe Caverly Set a batch variable (%1 - %n) when TCC is executing a batch file Support 5
P Kill all other instances of tcc from a btm batch file Support 2
fpefpe batch file ending early Support 18
R Batch Debugger Command Expansion Window Support 3
A Unable to pass batch file arguments unaltered Support 18
K Batch debugger not stepping correctly Support 4
S incorrect message at start Cancel batch job 4START.bat ? (Y/N/A) : Support 1
MickeyF Why does this batch file give me 'unknown command "else"' error? Support 17
Joe Caverly "Functions" in cmd.exe batch files Support 6
S Batch files Calling Batch Files Support 2
P exit /b in batch files Support 7
P Labels in batch files: TCC vs CMD.exe Support 9
vefatica Batch parameters broken? Support 2
cxxl Set encoding for batch file Support 4
dcantor How to? Batch file to be executed by TCSTART Support 4
J New lexer for TCC batch files. Support 0
T Fixed Problem with use of Batch parameters in the IDE Support 1
T Fixed IDE locks up if edit window for running batch is closed Support 6
T How to? Show the values of batch variables on the TCC screen Support 21
D Alias in batch file Support 3
TT's Help! Uninstalling TCC did not restore cmd.exe as Win10 default batch file processor. Support 4
vefatica Exit code of a batch file? Support 4
J cd command altered in batch file Support 3
MickeyF running a program from cmdline versus in batch file Support 1
B Batch file won't run if there is already an instance of TC 17 running Support 9
nickles Fixed Executing long command lines in a batch generates file name errors Support 16
T batch execution Support 9
rjperrella start /tabna leaves blank windows when running batch scripts Support 10
jadaml Echo unicode characters from UTF-8 Batch files? Support 1
J Files Disappearing During Batch Execution Support 4
nickles Fixed In batch files, %0 now evaluates to the absolute path Support 9
D Installation batch files Support 5
Jay Sage WAD Possible Bug in Batch Parameters Support 7
C How to? run a batch file in administrator mode while debugging in take command software Support 1
R Unexpected 'Cancel Batch Job' prompts Support 13
cgunhouse Fixed bdebugger with not exist batch file causes a crash Support 2
Mezlo menu batch file questions Support 25
D Run all .CMD batch files with CMD.EXE Support 16
T Lua sharing data with batch files Support 2
F Minimising the command window running from a batch file Support 19
C How to? ctrl-c does not prompt "Terminate batch job" Support 2

Similar threads