Welcome!

By registering with us, you'll be able to discuss, share and private message with other members of our community.

SignUp Now!

robocopy cancelling batch file processing

K

kafka

May
14
0
I have a set of (.BTM) batch files that I use to update offline copies of my working directories that use the robocopy program. The top level batch files call a common batch file to perform the work in a uniform way. I have noticed that occasionally batch processing terminates prematurely. I have attempted to work around the issue by looping until a flag is set in both the common batch file and the top level batch files, the looping code never gets a chance to continue the processing because it acts like a cancel command was encountered. If I manually rerun the top level batch file it will typically run normally but it may also terminate at another point in the processing.

I recently modified my common batch file to enable command echoing just before running robocopy and got lucky when the issue occurred when synchronizing the first directory. The invocation of robocopy completed and the batch file continued running to next two statements which were on break and a return. At this point the batch file processing was cancelled with a call, 3 gosubs, an iff , and another gosub pending.

This issue occurs on multiple different PCs. All of the PCs are currently running Windows 10, but if I remember correctly the issue also occured under Windows 7. The working directories are on the C: drive and the offline mirror copies are on USB 3.0 drives (SSD or magnetic).

The common batch file can perform the processing using either robocopy or the builtin TCC copy command. I have never seen this issue occur with the copy command but robocopy is about 10 times faster than the copy command.
 
Last edited:
I don't know how an external EXE (robocopy) could cause a batch file to terminate unless the batch file were somehow cooperating. Can you give more information?
 
Maybe it's quitting if ROBOCOPY's ERRORLEVER is > 0 ?
 
I've noticed a similar issue when using TCC 20 on Windows 10.

Often the batch file just stops prematurely after a ROBOCOPY command completes even though there are many more ROBOCOPY calls left in the file. No error messages are output.

IF you re-run the batch file then it always continues successfully after the ROBOCOPY call where it stopped last time. Probably because there are no changes to copy, and no output from robocopy to display. But a subsequent robocopy call can stop batch processing instead. So just continue re-running until all robocopy calls are completed, lol.

The batch file just calls robocopy directly and does not check the error code from it, no other batch files are called.

This only happens with TCC 20, I also have TCC 15 and there the problem never occurs with the same batch file, same Windows 10 and same computer.

I would guess it has something to do with ANSI processing myself, but have never taken the time to investigate, easier to just re-run the file when not complete.

I attached a simplified version of the batch file.
 

Attachments

Have you tried using the START command vs CALL with the ROBOCOPY line?
That should remove any interaction with your batch file.
 
From the CHM help file:

CALL

Purpose: Execute one batch file from within another.

You do not want a batch file to run from the BTM - you want a EXE to run. Have you looked at the DEFER command?

==================
Purpose:Execute a command after the batch file exits

Format:DEFER command

Usage:

A batch file can have multiple DEFER commands. They will be executed in first in, first out order when the batch file exits.

If you have variables on the DEFER command line, they will be expanded before the DEFER command is processed, not when command is executed. To delay variable expansion until command is executed, use single back quotes around the variable names, or double the %'s before the variable names.

If the last argument on the line is a single (, it is interpreted as the beginning of a command group. DEFER will append the following lines (in a batch file) or prompt you for more input (at the command line) until it gets a closing ).



 
I cannot use START since I want the batch file to wait for ROBOCOPY to complete. START will open it in a new separate console and the batch file will continue to the next robocopy directly.

Yes, I can agree that CALL is unnecessary here as an unqualified ROPOCOPY call will run robocopy.exe on my current system.

I often use CALL from batch files anyway, in case I would have a batch file with the same name as the command, in that case a direct call would replace the current batch file, while with CALL it will continue from the current line afterwards.

Haven't seen that using CALL would ever have hurt when used to run executables and aliases.
(Keep in mind that this works fine in TCC 15)

But sure, could test without it next time I use the batch file.
 
Don't use START... or if you do, use START /WAIT. But that is completely unnecessary. Just run robocopy.exe directly with nothing preceding it!
 
Just tested it again with CALL removed.

And what do you know, the problem totally disappeared, everything went through at the first attempt!

So problem solved on my part, thanks for the suggestions everyone!

Apparently the robocopy process somehow returned the CANCEL "code" at times causing the batch processing to end even though it was not a batch file. Guess it happened to work in TCC15 due to different internal implementation details.
 
nikbackm said:
I cannot use START since I want the batch file to wait for ROBOCOPY to complete. START will open it in a new separate console and the batch file will continue to the next robocopy directly.
Click to expand...
START "" /B ... will run in the same console and will be nearly identical to DEFER with exception that your batch file will continue after spawning the task, not before.
Be aware that it might mess the screen, if backgrounded task wold output anything.
 
Has anyone noticed that that ROBOCOPY is unusual in that it modifies the window title. I have not found any other external program that modify the window title.

I have continued to periodically debug this problem and have determined that it only happens when returning from a subroutine that invokes ROBOCOPY, with the RETURN command being treated like a CANCEL command. Until today, I was never able to consistently reproduce the problem.

While working on another batch file I encountered a situation that always causes the batch file to terminate early. The batch file uses a subroutine to process one directory at a time using ROBOCOPY. The subroutine generates the source and destination directory names from a GOSUB parameter and other environment variables, invokes ROBOCOPY, and then returns. When encountering the return statement the batch file always terminates (like the return is a CANCEL statement). I determined that if I add a WINDOW command (WINDOW "") to set the window title to the default, the return works correctly with execution continuing after the GOSUB command.

Starting with this new batch file, I created a minimized batch file that consistently reproduces the error with my files. I then enhanced the minimized batch file so that it could create test files, but have not been able to generated a set of test files that will recreate the problem.

Hopefully this additional information (ROBOCOPY modifying the window title causing a RETURN command being treated as a CANCEL command) will enable JP Software to be able to find the root cause.
 
Wrapping the ROBOCOPY calls in @exec[] finally fixed this problem for me.

(The removal of CALL I wrote about earlier in the thread did not help after all)
 
My batch file already had "on break goto ..." statements before invoking ROBOCOPY and "on break" statements after (including in the destination of the "on break goto ..."), shouldn't that have handled ROBOCOPY sending a Ctrl-C / Ctrl-Break?

About 30 days ago, I modified my batch file to wrap the invocations of ROBOCOPY in @exec[] as per nikbackm's comment and have not had the batch file terminate prematurely since.
 
Same problem for many, many years. The robocopy is never interrupted, so that helps a lot. I run the thing for variously between 80 and 100 root directories across 6 disks and it almost always stops after just a few roots folders and then only occasionally after restarting till its done. It's annoying but I do like relying on a clean backup for my portable apps and data.

I have fixed the basic batch file so that it can skip the initial checking and start at any of the disks. Redoing a few root folders goes really quickly (partly since they are already mirrored, but maybe memory caching plays a role as well). In fact I have an alias that restarts from the beginning of whatever disk letter I give it up to 4 times. Not elegant, but not unreasonable even if, when it finally finishes, it has backed up a couple of disks two or maybe three times.

Still it makes a 3 to 5 minute job into a 4 to 6 minute one and requires at least one user intervention to complete.

I wonder if the problem occurs with cmd.exe. Might be a good job for jpsoft to check out.
 
Jack Weeks said:
Same problem for many, many years. The robocopy is never interrupted, so that helps a lot. I run the thing for variously between 80 and 100 root directories across 6 disks and it almost always stops after just a few roots folders and then only occasionally after restarting till its done. It's annoying but I do like relying on a clean backup for my portable apps and data.

I have fixed the basic batch file so that it can skip the initial checking and start at any of the disks. Redoing a few root folders goes really quickly (partly since they are already mirrored, but maybe memory caching plays a role as well). In fact I have an alias that restarts from the beginning of whatever disk letter I give it up to 4 times. Not elegant, but not unreasonable even if, when it finally finishes, it has backed up a couple of disks two or maybe three times.

Still it makes a 3 to 5 minute job into a 4 to 6 minute one and requires at least one user intervention to complete.

I wonder if the problem occurs with cmd.exe. Might be a good job for jpsoft to check out.
Click to expand...

Did you try wrapping ROBOCOPY in @exec?
 
I did, just now. Wow! It works and speeds up the process a lot. It must be that @exec raises the priority; I wonder if a high priority start would do the same thing?

Thanks Niklas, you saved me probably ten minutes a week to say nothing of enough of an annoyance to keep me from doing backups as often as I should. And of course you saved the machine a lot more time than than.

This backup routine is supposed to be so easy and quick that I could do it between coffee sips. I timed it when there were only a couple of tiny files to update and it ran in 5 seconds. That's mirroring 74 folders and 130Gb. Such speed was probably colored by it being done not long after I had already done it the old way and so probably caching played a role.
 
@Jack Weeks if you aren't using it yet, try adding the /MT switch to robocopy. In some situations this can improve performance even more, especially when you're doing update synchronizations.
 
Thanks I'll try that when I update a backup on a disk that hasn't been backed up to in a while. I might start with say 30 threads and see if that helps.
 
30 threads is probably too aggressive. There is a point where you'll stop seeing speed increases as you increase threads, and it could actually start getting slower. Depends on a lot of variables. The default of 8 might even be too high. I'd probably compare 2, 4, and 8. If 8 is faster than 4, then continue testing above 8.
 
drwtsn32 said:
30 threads is probably too aggressive. There is a point where you'll stop seeing speed increases as you increase threads, and it could actually start getting slower. Depends on a lot of variables. The default of 8 might even be too high. I'd probably compare 2, 4, and 8. If 8 is faster than 4, then continue testing above 8.
Click to expand...
The effect of the MT switch may be great or small, but so far I haven't been able to detect it. When I find some time I'll try to see if I can't devise a bit better way to measuring it. At least figure out how to add up the work done (bytes and files copied and deleted) verses the time to run, and see if I can provide a useful report.
 
drwtsn32 said:
Its benefit depends on a lot of factors. It sometimes makes no difference. I would just time two runs, one with no MT switch and the other with /MT:2 to see if there is any difference.
Click to expand...
Good Plan!

The attached log summarizes the results of a simple copy from one disk to another and so they are not really representative of mirroring 80 to 100 GB over 70 to 80 root folders with tens of thousands of files only a few of which have changed. (My backup runs a separate robocopy for each root folder.)

Anyway for this just-copy-test you are correct in suggesting that lower MT values are likely to be faster. I ran the test doing 2 to 20 and then 20 to 2 threads to make sure the results were not affected by caching. They were identical. Interestingly the highest score turned out to be without an MT switch -- the actual number of threads reported by Task Manager was 3.

BTW You'd think this kind of testing would have already been reported, but while it's been done none the instances I found were very carefully constructed and were mainly just straight copying. Which is to say much like what I am reporting.
 

Attachments

I'd also compare MT/no MT when doing updates (instead of just the initial data copy). Maybe you don't actually do update "syncs" with robocopy, but I sometimes use it in that way and MT can make a huge difference (sometimes).
 
You must log in or register to reply here.
Back
Top
[FOX] Ultimate Translator
Translate