Create Windows Jobs and optionally attach processes to a job

Joe Caverly · 2019-03-18

I was reviewing Windows Processes and Job Objects in TCC v21 and was wondering if anyone had examples of how this would be used.

I have previously used this method to emulate threads, but how would I convert this to work with the JOBS command?

Joe
TCC 24.01.41 x64 Windows 7 [Version 6.1.7601]

Joe Caverly · 2019-03-19

I've created a job;

Code:

jobs /n=jlcJob

I have added three processes to the job;

Code:

jobs /j=jlcJob 9544
jobs /j=jlcJob 9676
jobs /j=jlcJob 4356

When I query the job;

Code:

c:\users\jlc\utils>jobs /j=jlcJob /i
Total Time: User=936006, Kernel=1092007
Period Time: User=936006, Kernel=1092007
Reads=59 (70016 bytes), Write=0 (0 bytes), Other=3532 (4858556 bytes)
Peak memory used: Process=2719744, Job=7327744
Page Faults=6240

Total Processes=3, Active Processes=3, Terminated Processes=0
ID  9544 - C:\Windows\SysWOW64\cmd.exe
ID  0 - ???
ID  9676 - C:\Windows\System32\notepad.exe

...I see the jobs, but one of the jobs is not identifiable.

Using Process Explorer, I can see the three jobs;

Why can't I see the three jobs from TCC?

Joe
TCC 24.01.41 x64 Windows 7 [Version 6.1.7601]

vefatica · 2019-03-19

I see the same thing on Win10/64 but not on Win7/32. Using this BTM

Code:

jobs /n=job1
do x in /L cmd notepad cmd ( start %x & delay 2 & jobs /j=job1 %_startpid )
jobs /j=job1 /i

On both cases, all three processes are created, but I get this on Win7/32:

Code:

Total Time: User=156001, Kernel=468003
Period Time: User=156001, Kernel=468003
Reads=20 (45678 bytes), Write=0 (0 bytes), Other=912 (9716 bytes)
Peak memory used: Process=2109440, Job=5537792
Page Faults=3194

Total Processes=3, Active Processes=3, Terminated Processes=0
ID  1460 - C:\Windows\System32\cmd.exe
ID  2332 - C:\Windows\System32\notepad.exe
ID  2000 - C:\Windows\System32\cmd.exe

And I get this on Win10/64:

Code:

Total Time: User=468750, Kernel=781250
Period Time: User=468750, Kernel=781250
Reads=1 (60 bytes), Write=0 (0 bytes), Other=317 (7316 bytes)
Peak memory used: Process=4202496, Job=11579392
Page Faults=6189

Total Processes=3, Active Processes=3, Terminated Processes=0
ID  7580 - C:\Windows\System32\cmd.exe
ID  0 - ???
ID  5968 - C:\Windows\System32\notepad.exe

vefatica · 2019-03-19

Joe Caverly said:
I was reviewing Windows Processes and Job Objects in TCC v21 and was wondering if anyone had examples of how this would be used.

I have previously used this method to emulate threads, but how would I convert this to work with the JOBS command?

Joe
TCC 24.01.41 x64 Windows 7 [Version 6.1.7601]

If you're still wanting info on jobs, here's a technique which I have found useful.

If you create a job with "/k"

Code:

jobs /n=job1 /k

and add processes to it like this (for example)

Code:

start /job=job1 cmd

then the instance of CMD will automatically be terminated when the job-creating TCC exits or does

Code:

jobs /j=job1 /c

Joe Caverly · 2019-03-19

Thanks @vefatica for the /K tip.

This does what I want;

Code:

@setlocal
@echo off
::
:: Create a job called job1 that will terminate on handle close
::
jobs /n=job1 /k
::
:: Start three applications, and add them to job1
::
do x in /L cmd notepad cmd ( start /job=job1 %x & delay 2 )
::
:: Display info about job1
::
jobs /j=job1 /i
::
:: Initialize Flag for ActiveProcesses
::
set ActiveProcesses=Y
::
:: Wait until all applications in job1 are finished
::
do while %ActiveProcesses eq Y
  jobs /j=job1 /i | findstr /c:"Active Processes=0"
  if %errorlevel% eq 0 set ActiveProcesses=N
  delay 1
enddo
::
:: All applications in job1 are finished. Close the job handle
::
jobs /j=job1 /c
endlocal

Joe
TCC 24.02.44 x64 Windows 7 [Version 6.1.7601]
TCC Build 44 Windows 7 Build 7601 Service Pack 1

Joe Caverly · 2019-03-20

Here is a re-write of my thread emulating .BTM using JOBS;

Code:

::+-----------------------------------------------------------------------+
::| JobTest.BTM - Show how to run several batch files at the same time,   |
::| and determine when they have all finished.                            |
::|-----------------------------------------------------------------------+
@setlocal
@echo off
::
:: Create five batch files
::
echo delay 7 > proc1.btm
echo delay 2 > proc2.btm
echo delay 3 > proc3.btm
echo delay 4 > proc4.btm
echo delay 5 > proc5.btm
::
:: Create a job called job1 that will terminate on handle close
::
jobs /n=job1 /k
:: Start each batch file in detached mode
::
detach proc1.btm
jobs /j=job1 %_detachpid
detach proc2.btm
jobs /j=job1 %_detachpid
detach proc3.btm
jobs /j=job1 %_detachpid
detach proc4.btm
jobs /j=job1 %_detachpid
detach proc5.btm
jobs /j=job1 %_detachpid
::
:: Display info about job1
::
jobs /j=job1 /i
::
:: Initialize Flag for ActiveProcesses
::
set ActiveProcesses=Y
::
:: Wait until all applications in job1 are finished
::
do while %ActiveProcesses eq Y
  jobs /j=job1 /i | findstr /c:"Active Processes=0"
  if %errorlevel% eq 0 set ActiveProcesses=N
  delay 1
enddo
::
:: All applications in job1 are finished. Close the job handle
::
jobs /j=job1 /c
::
:: Remove the batch files created earlier
::
if exist proc1.btm del proc1.btm > nul
if exist proc2.btm del proc2.btm > nul
if exist proc3.btm del proc3.btm > nul
if exist proc4.btm del proc4.btm > nul
if exist proc5.btm del proc5.btm > nul
endlocal
quit

Sample run;

Code:

c:\users\jlc\utils>jobtest.btm
The process ID is 6176
The process ID is 5548
The process ID is 8072
The process ID is 2036
The process ID is 5832
Total Time: User=0, Kernel=624004
Period Time: User=0, Kernel=624004
Reads=5 (270310 bytes), Write=0 (0 bytes), Other=601 (0 bytes)
Peak memory used: Process=10506240, Job=36646912
Page Faults=10970

Total Processes=5, Active Processes=5, Terminated Processes=0
ID  6176 - C:\Program Files\JPSoft\TCMD24\tcc.exe
ID  0 - ???
ID  5548 - C:\Program Files\JPSoft\TCMD24\tcc.exe
ID  0 - ???
ID  8072 - C:\Program Files\JPSoft\TCMD24\tcc.exe
Total Processes=5, Active Processes=0, Terminated Processes=0

Note that jobs /j=job1 /i still has problems on x64 systems. @vefatica has found that it works on his x32 system.

Joe
TCC 24.02.44 x64 Windows 7 [Version 6.1.7601]
TCC Build 44 Windows 7 Build 7601 Service Pack 1

Charles G · 2019-03-20

Joe Caverly said:
if exist proc1.btm del proc1.btm > nul

this can be done easier by:

if exist proc1.btm del /q proc1.btm

vefatica · 2019-03-20

Joe, does that work ... JOBS /K and DETACH? It doesn't here. According to the help, if you want the processes in the job to terminate when the last handle to the job is closed, you should use START /JOB.

Joe Caverly · 2019-03-20

Yepper, it works here.

I changed the code to provide info every second;

Code:

::
:: Wait until all applications in job1 are finished
::
do while %ActiveProcesses eq Y
  jobs /j=job1 /i | findstr /c:"Active Processes=0"
  if %errorlevel% eq 0 set ActiveProcesses=N
  delay 1
  ::
  :: Display info about job1
  ::
  jobs /j=job1 /i
enddo

...which gives the following;

Code:

The process ID is 8236
The process ID is 8036
The process ID is 7708
The process ID is 5216
The process ID is 2856
Total Time: User=0, Kernel=0
Period Time: User=0, Kernel=0
Reads=4 (219704 bytes), Write=0 (0 bytes), Other=253 (0 bytes)
Peak memory used: Process=4014080, Job=15527936
Page Faults=3463

Total Processes=5, Active Processes=5, Terminated Processes=0
ID  8236 - C:\Program Files\JPSoft\TCMD24\tcc.exe
ID  0 - ???
ID  8036 - C:\Program Files\JPSoft\TCMD24\tcc.exe
ID  0 - ???
ID  7708 - C:\Program Files\JPSoft\TCMD24\tcc.exe
Total Time: User=1092007, Kernel=936006
Period Time: User=1092007, Kernel=936006
Reads=140 (550085 bytes), Write=0 (0 bytes), Other=2745 (19140 bytes)
Peak memory used: Process=14884864, Job=74412032
Page Faults=24078

Total Processes=5, Active Processes=5, Terminated Processes=0
ID  8236 - C:\Program Files\JPSoft\TCMD24\tcc.exe
ID  0 - ???
ID  8036 - C:\Program Files\JPSoft\TCMD24\tcc.exe
ID  0 - ???
ID  7708 - C:\Program Files\JPSoft\TCMD24\tcc.exe
Total Time: User=1092007, Kernel=936006
Period Time: User=1092007, Kernel=936006
Reads=140 (550085 bytes), Write=0 (0 bytes), Other=2900 (24938 bytes)
Peak memory used: Process=15073280, Job=74604544
Page Faults=24143

Total Processes=5, Active Processes=4, Terminated Processes=0
ID  8236 - C:\Program Files\JPSoft\TCMD24\tcc.exe
ID  0 - ???
ID  7708 - C:\Program Files\JPSoft\TCMD24\tcc.exe
ID  0 - ???
Total Time: User=1092007, Kernel=936006
Period Time: User=1092007, Kernel=936006
Reads=140 (550085 bytes), Write=0 (0 bytes), Other=3055 (30736 bytes)
Peak memory used: Process=15073280, Job=74604544
Page Faults=24203

Total Processes=5, Active Processes=3, Terminated Processes=0
ID  8236 - C:\Program Files\JPSoft\TCMD24\tcc.exe
ID  0 - ???
ID  5216 - C:\Program Files\JPSoft\TCMD24\tcc.exe
Total Time: User=1092007, Kernel=936006
Period Time: User=1092007, Kernel=936006
Reads=140 (550085 bytes), Write=0 (0 bytes), Other=3210 (36534 bytes)
Peak memory used: Process=15077376, Job=74604544
Page Faults=24263

Total Processes=5, Active Processes=2, Terminated Processes=0
ID  8236 - C:\Program Files\JPSoft\TCMD24\tcc.exe
ID  0 - ???
Total Time: User=1092007, Kernel=936006
Period Time: User=1092007, Kernel=936006
Reads=140 (550085 bytes), Write=0 (0 bytes), Other=3365 (42332 bytes)
Peak memory used: Process=15077376, Job=74604544
Page Faults=24331

Total Processes=5, Active Processes=1, Terminated Processes=0
ID  8236 - C:\Program Files\JPSoft\TCMD24\tcc.exe
Total Time: User=1092007, Kernel=936006
Period Time: User=1092007, Kernel=936006
Reads=140 (550085 bytes), Write=0 (0 bytes), Other=3365 (42332 bytes)
Peak memory used: Process=15077376, Job=74604544
Page Faults=24331

Total Processes=5, Active Processes=1, Terminated Processes=0
ID  8236 - C:\Program Files\JPSoft\TCMD24\tcc.exe
Total Time: User=1092007, Kernel=936006
Period Time: User=1092007, Kernel=936006
Reads=140 (550085 bytes), Write=0 (0 bytes), Other=3520 (48130 bytes)
Peak memory used: Process=15077376, Job=74604544
Page Faults=24391

Total Processes=5, Active Processes=0, Terminated Processes=0
Total Processes=5, Active Processes=0, Terminated Processes=0
Total Time: User=1092007, Kernel=936006
Period Time: User=1092007, Kernel=936006
Reads=140 (550085 bytes), Write=0 (0 bytes), Other=3520 (48130 bytes)
Peak memory used: Process=15077376, Job=74604544
Page Faults=24391

Total Processes=5, Active Processes=0, Terminated Processes=0

Joe
TCC 24.02.44 x64 Windows 7 [Version 6.1.7601]
TCC Build 44 Windows 7 Build 7601 Service Pack 1

Joe Caverly · 2019-03-20

Hey @vefatica,

Here's what it says in the help file;

Code:

/K All processes associated with the job will terminate when the last handle to the job is closed. If you START other processes, you should use the /job option if you want the START'd processes to exit when the original process exits.

Note that in my .BTM, I am not exiting the original process. Instead, I am using the results of jobs /j=job /i to determine if there are any ActiveProcesses;

Code:

::
:: Initialize Flag for ActiveProcesses
::
set ActiveProcesses=Y

::
:: Wait until all applications in job1 are finished
::
do while %ActiveProcesses eq Y
  jobs /j=job1 /i | findstr /c:"Active Processes=0"
  if %errorlevel% eq 0 set ActiveProcesses=N
  delay 1
  ::
  :: Display info about job1
  ::
  jobs /j=job1 /i
enddo

So, not sure why it is not working on your system.

Joe
TCC 24.02.44 x64 Windows 7 [Version 6.1.7601]
TCC Build 44 Windows 7 Build 7601 Service Pack 1

vefatica · 2019-03-20

It's the automatic termination that's not happening when I use DETACH. You don't seem to be testing automatic termination. Below, PAU.BTM does this:

Code:

do i=1 to 1000 ( delay 1 )

When I run this (J4.BTM)

Code:

jobs /n=job1 /k

iff "%1" == "use_detach" then
    echos DETACH:^s
    detach pau.btm
    jobs /j=job1 %_detachpid
elseiff "%1" == "use_start" then
    start /job=job1 pau.btm
    echo START: The process ID is %_startpid
endiff

tasklist | findstr tcc
pause
jobs /j=job1 /c
tasklist | findstr tcc
if "%1" == "use_detach" taskend %_detachpid

the DETACHed process doesn't terminate; the STARTed process does.

Code:

v:\> j4 use_detach
DETACH: The process ID is 3608
 2940* tcc                24.02.43  [2940]  v:\
 3608  tcc
Press any key when ready...
 2940* tcc                24.02.43  [2940]  v:\
 3608  tcc

v:\> j4 use_start
START: The process ID is 3952
 2940* tcc                24.02.43  [2940]  v:\
 3952  tcc               G:\tc24\TCC.EXE
Press any key when ready...
 2940* tcc                24.02.43  [2940]  v:\

v:\>

vefatica · 2019-03-20

... and that's because START /JOB does not give a job handle to the STARTed process ... closing the original handle is closing the last handle.

In the case of DETACH, the DETACHed process gets a job handle ... closing the original handle is not closing the last handle.

Joe Caverly · 2019-03-20

Hey @vefatica, here's a version of the .BTM that uses JOBMONITOR. It needs a little cleaning up, but is just an example;

Code:

@setlocal
@echo off
if exist jobtest.log del /q jobtest.log
log /w jobtest.log
log /a on
::
:: Create five batch files
::
echo delay 17 > proc1.btm
echo exit >> proc1.btm
echo delay 12 > proc2.btm
echo exit >> proc2.btm
echo delay 13 > proc3.btm
echo exit >> proc3.btm
echo delay 14 > proc4.btm
echo exit >> proc4.btm
echo delay 15 > proc5.btm
echo exit >> proc5.btm
::
:: Create a job called job1 that will terminate on handle close
::
jobs /n=job1 /k
log Starting JobMonitor
jobmonitor job1 process forever Gosub jobenv
:: Start each batch file in detached mode
::
echo %_time
start /pgm "proc1.btm"
jobs /j=job1 %_startpid
delay 1
start /pgm "proc2.btm"
jobs /j=job1 %_startpid
delay 1
start /pgm "proc3.btm"
jobs /j=job1 %_startpid
delay 1
start /pgm "proc4.btm"
jobs /j=job1 %_startpid
delay 1
start /pgm "proc5.btm"
jobs /j=job1 %_startpid
delay 1
::
:: Display info about job1
::
:: Wait until all applications in job1 are finished
::
do until %jobtest eq ActiveProcessZero
  ::
  :: Display info about job1
  ::
  :: echo %_time JobAction: %jobtest
  delay 1
enddo
::
:: All applications in job1 are finished. Close the job handle
::
jobs /j=job1 /c
echo %_time
::
:: Remove the batch files created earlier
::
if exist proc1.btm del proc1.btm > nul
if exist proc2.btm del proc2.btm > nul
if exist proc3.btm del proc3.btm > nul
if exist proc4.btm del proc4.btm > nul
if exist proc5.btm del proc5.btm > nul
::
:: Turn off the Log
::
log off
::
:: Stop monitoring Jobs
::
jobmonitor /c
endlocal
quit

:JobEnv
echo      JobInfo: %_jobaction %_jobpid %_jobprocessname %_jobtime
set jobtest=%_jobaction
Return

Any constructive modifications appreciated.

Joe
TCC 24.02.44 x64 Windows 7 [Version 6.1.7601]
TCC Build 44 Windows 7 Build 7601 Service Pack 1

vefatica · 2019-03-20

Joe Caverly said:
Any constructive modifications appreciated.

I still think your /K is wasted. But I don't think you really want it since you're waiting for the job's processes to exit on their own (nice how you use JOBMONITOR to do that).

AnrDaemon · 2019-03-21

START /B /JOB …

djspits · 2019-03-21

Hello Joe!

Your first post mentioned "emulating threads" but the rest of this discussion focussed on the mechanics of the JOBS command. I feel the need for some additional remarks especially for other TC users who might not be familiar with these topics.

First of all, the commands START, DETACH and JOBS are about processes, not threads, Each process, irrespective of the way it was created, can use multi-threaded processing but that is another level of control and outside the scope of this discussion. Conceptually a JOB is nothing more than a "named set of process constraints". By adding existing processes to a JOB you effectively guarantee that the sum total of resource usage of all processes within the job do not exceed the limits you put on the group.

If your goal is simply to parallelize a bunch of batches you don't necessarily need to concern yourself with jobs. Only if overloads regularly cause your machine to become unresponsive do you need to tweak those parameters.

In that context, using DETACH is not an obvious choice. While it may reduce the memory usage of your processes, it takes away all the other ways that START would allow you to manage the execution of your process. There is a place for DETACH in this universe but this is not it (yet).

So, what other methods are there, then? Well, obviously, if too many processes in parallel are overwhelming your processors, serialize them. The simplest way of course is to use something like the taskmanager to postpone execution until such time that the process would not disrupt any work; outside office hours, when processor idle-time goes up, etc.

The missing concept here (and missing TC command) is a QUEUE. A curious and glaring omission. One would like to be able to SUBMIT a batch to a dispatcher that manages multiple multi-stream queues. Queues are specialized to be high-priority non-interactive (DETACH-ed) number crunchers or low-priority console based dialogs. Queues use START to assign a queue to certain processors/cores in your machine and JOBS to guaratee that all that processing doesn't slow down your word-processing. You can have FOLDERMONITOR monitor a couple of folders on your desktop; "Fast and Furious" and "Slow but Thorough". You just drop your batch into any one of them and they will get executed accordingly. Let your imagination run wild.

One final remark. Remember that BTM files are copied into memory by TCC. That means that the batches you submit for execution can be reused by multiple processes. Try to organize your workloads into a single batch that you lauch multiple times each with a different parameter value.

I hope this helps to put things in perspective.

Regards, DJ.

rconn · 2019-03-22

vefatica said:
I see the same thing on Win10/64 but not on Win7/32. Using this BTM

Code:

jobs /n=job1 do x in /L cmd notepad cmd ( start %x & delay 2 & jobs /j=job1 %_startpid ) jobs /j=job1 /i

The problem is due to a bug in a Windows 10 API. I do not have a workaround for it yet.

Joe Caverly · 2019-03-22

rconn said:
The problem is due to a bug in a Windows 10 API. I do not have a workaround for it yet.

Is there also a bug in a Windows 7 API, as I demonstrated in post #2?

Joe

Search

Welcome!

Create Windows Jobs and optionally attach processes to a job

Joe Caverly

Joe Caverly

vefatica

vefatica

Joe Caverly

Joe Caverly

Charles G

vefatica

Joe Caverly

Joe Caverly

vefatica

vefatica

Joe Caverly

vefatica

AnrDaemon

djspits

rconn

Administrator

Joe Caverly