FOLDERMONITOR missing events??

[Steve Pitts]

I've been messing around with FOLDERMONITOR (prompted by Jang's thread about his problems with it), trying to understand how I might make use of it for handling the synchronisation of folders between the local HD and the NAS, and I came across an anomalous situation that I suspect might be a bug (sorry, undesirable feature).

The sequence of activities that triggered the output below was

• set up duff foldermonitor (the first line of output)
• clear that and set it up properly
• change name of mp3stick.not folder using Windows Explorer
• rename that folder twice more using an elevated command prompt
• edit meminfo_full_dump.out using Kedit for Windows and immediately save it (which you can see produces a .BAK version as well as updating the original)
• create empty_example.txt using an ECHO from the elevated prompt
• edit that empty_example.txt using Kedit again, add some text and then save. This created a backup and updated the original but FOLDERMONITOR only saw the first of those actions
• delete both empty_example.txt and the .bak version from Windows Explorer but FOLDERMONITOR only registered one of the two deletions

I:\websites\Badgers\new>A= C= N= 1= 2=
                        foldermonitor /c

I:\websites\Badgers\new>foldermonitor /s d:\temp\ created deleted modified renamed forever echo A=%%_folderaction C=%%_foldercount N=%%_foldername 1=%
%_folderfile1 2=%%_folderfile2

I:\websites\Badgers\new>foldermonitor
d:\temp\   created deleted modified renamed   (0/FOREVER)   echo A=%_folderaction C=%_foldercount N=%_foldername 1=%_folderfile1 2=%_folderfile2

I:\websites\Badgers\new>A=RENAMED C=1 N=d:\temp\ 1=mp3stick.not 2=mp3stick
                        elevate

I:\websites\Badgers\new>foldermonitor
d:\temp\   created deleted modified renamed   (1/FOREVER)   echo A=%_folderaction C=%_foldercount N=%_foldername 1=%_folderfile1 2=%_folderfile2

I:\websites\Badgers\new>A=RENAMED C=2 N=d:\temp\ 1=mp3stick 2=mp3sticky
A=RENAMED C=3 N=d:\temp\ 1=mp3sticky 2=mp3stick
A=RENAMED C=4 N=d:\temp\ 1=meminfo_full_dump.out 2=meminfo_full_dump.bak
A=MODIFIED C=5 N=d:\temp\ 1=meminfo_full_dump.out 2=
A=CREATED C=6 N=d:\temp\ 1=empty_example.txt 2=
A=RENAMED C=7 N=d:\temp\ 1=empty_example.txt 2=empty_example.bak
A=DELETED C=8 N=d:\temp\ 1=empty_example.txt 2=
                        foldermonitor
d:\temp\   created deleted modified renamed   (8/FOREVER)   echo A=%_folderaction C=%_foldercount N=%_foldername 1=%_folderfile1 2=%_folderfile2

I:\websites\Badgers\new>

It might be worth noting that the meminfo file is over 180MB in size, whereas the empty text file started as 1 byte and was only a dozen or so when I saved it, so I wonder whether there is a timing issue here with events happening too close together??

I know I'm one build behind the curve and will attempt to re-test this later, but for now I have a cricket match to go and umpire :)

TCC 12.10.66 x64 Windows 7 [Version 6.1.7601]
TCC Build 66 Windows 7 Build 7601 Service Pack 1

 

[rconn]

WAD -- FOLDERMONITOR (like all of the other monitoring commands) does not queue multiple rapid events. All of the monitoring commands are handled by a thread that wakes up every 200 ms and checks to see if anything has been triggered; if so, it processes those commands and then goes back to sleep.

This was chosen as the most reasonable compromise between handling events quickly while not overloading the rest of the system.

In your example, although FOLDERMONITOR is seeing all of the events, it's only notifying you about the most recent ones when you fire them off rapidly. IMO that's acceptable, given that in the real world you'd never expect to be able to handle potentially thousands of FOLDERMONITOR notifications per second. (Nor could you expect to actually DO anything with every notification if you did get it.)

 

[Steve Pitts]

This was chosen as the most reasonable compromise between handling events quickly while not overloading the rest of the system

No problem with that decision, although I'm obviously paying the price for having a beast of a machine and therefore being much more likely to hit this situation, but perhaps the behaviour should be documented in the help for each of the monitoring commands??

 

[vefatica]

Steve, I revived an old plugin which provides WATCHDIR. When I build that project, WATCHDIR logs 76 events (compared to FOLDERMONITOR's 8, I don't think it misses any). I've got to do some work on it, but I think I can make one available this evening. Are you interested? So far I have only built it for 32-bit Windows. Do you need x64? Here's it's log file for the build. It can also output these lines to the console.

1 20110613123003.470 DELETE later.obj
2 20110613123003.486 DELETE sysutil.obj
3 20110613123003.486 DELETE utility.obj
4 20110613123003.486 DELETE watchdir.obj
5 20110613123003.486 DELETE watchwin.obj
6 20110613123003.486 DELETE 4threads.dll
7 20110613123003.486 DELETE vc90.pdb
8 20110613123003.486 DELETE vc90.idb
9 20110613123003.486 DELETE 4threads.lib
10 20110613123003.486 DELETE 4threads.exp
11 20110613123003.486 DELETE 4threads.dll.intermediate.manifest
12 20110613123003.486 DELETE mt.dep
13 20110613123003.486 MODIFY BuildLog.htm
14 20110613123003.486 CREATE RSP00001922885200.rsp
15 20110613123003.486 MODIFY RSP00001922885200.rsp
16 20110613123003.548 CREATE vc90.pdb
17 20110613123003.548 MODIFY vc90.pdb
18 20110613123003.548 CREATE vc90.idb
19 20110613123003.548 MODIFY vc90.idb
20 20110613123004.502 MODIFY vc90.idb
21 20110613123004.502 MODIFY vc90.pdb
22 20110613123005.673 MODIFY vc90.pdb
23 20110613123005.689 MODIFY vc90.idb
24 20110613123005.689 CREATE later.obj
25 20110613123005.705 MODIFY later.obj
26 20110613123005.705 CREATE sysutil.obj
27 20110613123005.720 MODIFY sysutil.obj
28 20110613123005.720 CREATE utility.obj
29 20110613123005.720 MODIFY utility.obj
30 20110613123005.720 CREATE watchdir.obj
31 20110613123005.861 MODIFY watchdir.obj
32 20110613123005.861 CREATE watchwin.obj
33 20110613123005.877 MODIFY watchwin.obj
34 20110613123005.892 CREATE RSP00001A22885200.rsp
35 20110613123005.892 MODIFY RSP00001A22885200.rsp
36 20110613123005.908 CREATE 4threads.dll
37 20110613123005.908 MODIFY 4threads.dll
38 20110613123005.955 CREATE 4threads.lib
39 20110613123005.955 MODIFY 4threads.lib
40 20110613123005.955 CREATE 4threads.exp
41 20110613123005.955 MODIFY 4threads.exp
42 20110613123005.955 MODIFY 4threads.exp
43 20110613123005.955 MODIFY 4threads.exp
44 20110613123005.955 MODIFY 4threads.lib
45 20110613123005.955 MODIFY 4threads.lib
46 20110613123005.970 CREATE 4threads.dll.intermediate.manifest
47 20110613123005.970 MODIFY 4threads.dll.intermediate.manifest
48 20110613123005.970 MODIFY 4threads.dll.intermediate.manifest
49 20110613123005.970 MODIFY 4threads.dll.intermediate.manifest
50 20110613123005.970 MODIFY 4threads.dll
51 20110613123005.970 MODIFY 4threads.dll
52 20110613123005.970 CREATE RSP00001B22885200.rsp
53 20110613123005.970 MODIFY RSP00001B22885200.rsp
54 20110613123006.002 CREATE RCX33A4.tmp
55 20110613123006.002 MODIFY RCX33A4.tmp
56 20110613123006.002 MODIFY RCX33A4.tmp
57 20110613123006.002 MODIFY RCX33A4.tmp
58 20110613123006.002 MODIFY RCX33A4.tmp
59 20110613123006.002 MODIFY RCX33A4.tmp
60 20110613123006.002 MODIFY RCX33A4.tmp
61 20110613123006.002 DELETE 4threads.dll
62 20110613123006.002 OLDNAME RCX33A4.tmp
63 20110613123006.002 NEWNAME 4threads.dll
64 20110613123006.002 MODIFY 4threads.dll
65 20110613123006.002 MODIFY 4threads.dll
66 20110613123006.002 MODIFY 4threads.dll
67 20110613123006.002 MODIFY 4threads.dll
68 20110613123006.002 CREATE BAT00001C22885200.bat
69 20110613123006.002 MODIFY BAT00001C22885200.bat
70 20110613123006.017 CREATE mt.dep
71 20110613123006.017 MODIFY mt.dep
72 20110613123006.033 DELETE RSP00001922885200.rsp
73 20110613123006.033 DELETE RSP00001A22885200.rsp
74 20110613123006.033 DELETE RSP00001B22885200.rsp
75 20110613123006.033 DELETE BAT00001C22885200.bat
76 20110613123006.033 MODIFY BuildLog.htm
**EOF**

 

[Steve Fabian]

vefatica wrote:
| Steve, I revived an old plugin which provides WATCHDIR. When I build
| that project, WATCHDIR logs 76 events (compared to FOLDERMONITOR's 8,
| I don't think it misses any). I've got to do some work on it, but I
| think I can make one available this evening. Are you interested?

Yes please.
--
Steve

 

[vefatica]

vefatica wrote:
| Steve, I revived an old plugin which provides WATCHDIR. When I build
| that project, WATCHDIR logs 76 events (compared to FOLDERMONITOR's 8,
| I don't think it misses any). I've got to do some work on it, but I
| think I can make one available this evening. Are you interested?

Yes please.

I uploaded ftp://lucky.syr.edu/4plugins/4threads.zip and ftp://lucky.syr.edu/4plugins/x64/4threads64.zip. They contain WATCHDIR and WATCHWIN (both are old, WATCHDIR got some recent attention, WATCHWIN no attention in years.

WATCHDIR tips:

Use a log file (that's the default) and if it's in the directory being watched, exclude it from the watch with the "/X" option. The log file is opened with FILE_SHARE_READ so you should be able to read it while watching; apps that insist on opening it for writing will choke.

Console output (/O) is messy, like FOLDERMONITOR's.

I have no idea how well TCC can keep up with rapid changes if "/E command" is specified.

To watch the build I mentioned earlier, I used the command

g:\projects\4threads\release> watchdir id1 . /* /i * /x id1.log

. Watch the current dir
id1 a name for the watch object (for /List or /Kill)
/* use all filters (look up "ReadDirectoryChanges" at MS)
/i * include all files (except ...)
/x id1.log exclude the (default-named) logfile

Here's the help message.

ATCHDIR id path filter(s) [/S] [/N n] [/I spec(s)] [/X spec(s)] [/R]
/B] [/O] [/L file] [/E "command"]

   id          string ID, case-insensitive, 15 char limit
               also names variable to get last action, time and filename
   path        directory to watch
   /S          watch subdirectories
   /N n        stop after n changes; n < 65536 (0 = unspec'd = no limit)
   /I specs    include names matching any spec
   /X specs    exclude names matching any spec
               specs = spec[;spec]... wildcards cause wildcard comparison of
               target name only, else case-insensitive substring search on
               \path\name. If /I and /X are both spec'd the one spec'd second
               is seen only as exceptions to the one spec'd first
   /R          use case-insensitive regular expression searches (on
               \path\name) instead of those described above (/I and /X)
   /B          block: wait for changes
   /O          console output (can be messy without /B)
   /L file     log file; default: .\[watch_id]<id>.log - use "/L NUL" for no log
   /E command  quote entire command; use \" for interior quotes

 Filters       (at least one)

   /F          filename
   /D          dirname
   /Z          size
   /W          write time
   /C          create time
   /A          access time
   /*          all of the above

ATCHDIR /K [spec] - kill watch(es); wildcard id spec; default: *

ATCHDIR /P [spec] - pause/resume watch(es), a toggle; ignore events

ATCHDIR /L [spec] - list watch(es); spec as above

</id>

 

[Steve Pitts]

I think I can make one available this evening. Are you interested?

I would certainly be willing to give it a look, although I see Mr. Fabian got in before me so you've gone ahead with it anyway :) All I'm looking for is something that monitors each of half a dozen directories (and all of their subdirectories) and records changes made to an activity file. The intention is to post-process that file to update related directories on a NAS and thus keep those handful of directories in sync across three different PCs (I've been trying to find a tool that makes this easy but all seem to founder on what to do when a file gets updated on two different source machines between syncs).

I am a virgin when it comes to plugins (despite being an active 4whatever user for nearly two decades) but I'm sure it is easy enough to set up.

So far I have only built it for 32-bit Windows. Do you need x64?

As things stand I have two 32 bit Windows XP machines running v7 of 4NT/TC (will your plugin work with v7 under XP??) and the beast running 64 bit Windows 7. If I could have made FOLDERMONITOR work then I would have upgraded the two XP machines to v12 of TC, and I guess if your plugin gives me the ability that I'm looking for but only runs in v12 then that would still be an option.

 

[vefatica]

On Tue, 14 Jun 2011 04:15:37 -0400, Steve Pitts <> wrote:

|---Quote (Originally by vefatica)---
|So far I have only built it for 32-bit Windows. Do you need x64?
|---End Quote---
|As things stand I have two 32 bit Windows XP machines running v7 of 4NT/TC (will your plugin work with v7 under XP??) and the beast running 64 bit Windows 7. If I could have made FOLDERMONITOR work then I would have upgraded the two XP machines to v12 of TC, and I guess if your plugin gives me the ability that I'm looking for but only runs in v12 then that would still be an option.

I doubt that it will work with 4NT v7. Since v7 there have been changes to the
basic 4NT/TCC interaction and to a few of the functions provided by TakeCmd.dll
for plugins to use. XP is no problem; it's the platform on which I develop
plugins.