file matching daemon

[osirisja]

Hi

I am very new to Alpha 5 but have many years of development experience. I am trying to put together a quick solution that will achieve 3 particular goals:

1. Run as a daemon to constantly monitor a Control directory for certain files appearing
2. When a file appears, move it to a working directory and then use the filename to lookup a file mapping table, but the filename/filemapping match string should use a replacement character mask and when it matches the input data file, a set of parameters (fields) are taken from the matching record
3. An external application is called passing the input file path and filename as well as parameters from the record above.

I am sure all of this can be done but I am looking for a little advice on how to best achieve each of these steps? I know it's a lot to ask but some simple guidance on best functions to use would be a great start.

Many thanks

Andy

[fsi]

Re: file matching daemon

Not quite sure about goals #2 and #3, but I may be able to offer some insight into goal #1.

You state you wish to "constantly monitor" a directory for certain files. As you likely know, there are at least two common ways of implementing such monitoring. You could 1) poll the Control directory at some interval, looking for your certain files, or 2) rely upon some "directory watching" feature of any development package you are using. I know Visual Studio does offers this. And I think Alpha Five does, too. And, way back when, there were a number of sources of third-party components providing this functionality. If your background is sufficient, you could write a similar component yourself.

Now...

For approach #1, you've got a sure-fire solution. Upside: complete reliability. Downside: polling is frequently considered a waste of cpu cycles. However, depending upon the chosen interval, and assuming the directory table is already in RAM, this approach may not be too offensive to coding purists. I suppose, too, one might evaluate the overall availability of resources, likely demand, etc. The less-frequent your poll, the less intrusive. I would imagine a 1-second poll may not be unreasonable - and may even be quite sufficient for your needs. And, if you can get away with a 5-second poll (or even less frequent), well, I think I would select this approach. That is, unless you had to look for any one of hundred or more "certain" files. In that case, I think a purist would definitely take offense. All a matter of personal opinion, however.

As to approach #2: Unfortunately, the underlying mechanisms for such an event-driven solution (in Windows, at least) are not entirely reliable. I've recently performed research in the area, discovered test results compiled by others, and run tests myself. It is possible (but, do note, not terribly PROBABLE) that a "certain file" could be created but an appropriate triggering event will not be properly raised and/or communicated to your application. I've seen this happen in a burst-cycle where I created 1,000 files in as tight a loop as possible. Typically, 1-3 notifications were never received. For me, any failure rate greater than zero is unacceptable.

[Al Buchholz]

Re: file matching daemon

Start with file.watch functions....

The rest of the coding will use the file,directory, or string replacement functions - you'll use those functions in your self built xbasic functions.

[osirisja]

Re: file matching daemon

Hi Al and fsi.
thanks for your suggestions. So file.watch is the function I need for the daemon. I also Discovered 'sys_shell_wait' will execute the external application I need, passing parameters and feeding the result back to A5. The required parameters and executable run with 'sys_shell_wait' will be taken from the process table I have created based on the matched file name.

Now the only thing I need to figure out is how to match a physical file name against a 'mask' in a record in the process table I have defined. So, for example I have A field defined as a value of 'my_file*9999???.DAT' (mask) and a data file named 'my_file_january2011sub.dat' is picked up, this will match the record with the field mask above. I can then execute the external routine, passing the path and name of the matched file, and all the necessary parameters on the matched record.

Basically I need any incoming file from many different customers to be matched with a specific record in a process table so I know how to process them, all of which can be processed differently.

Last piece of the jigsaw I think :-)

Thanks all

Andy

[fsi]

Re: file matching daemon

I apologize for being, apparently, not sufficiently clear.

Yes, Alpha Five has some "file watcher" functions. This solution, however, falls under category #2 (as previously discussed). Such functionality is achieved by interfacing with Windows via API callbacks FindFirstChangeNotification and ReadDirectoryChangesW. For such functionality dependent upon the O/S, these functions - regardless of high-level development platform (ie: Alpha Five) - are ALWAYS based upon Windows API callbacks. Doesn't matter whether you are getting directory info, the time, memory info, writing/reading files, etc. It's always an API function. API events are (read: "should be") generated and passed to your application. But as I attempted to explain in my previous post, it has been documented (and I believe I even stumbled upon a Microsoft document in the last two months) that these mechanisms are not 100% reliable. Again, in my own testing, notifications often failed in 1-3 times per burst test of 1,000 new file creations. I had also tested for modifications (ie: Last Write Time) and rename with the same results. Who knows - it may even be machine-dependent, or have some connection to other running software (ie: antivirus software - since it hooks the system's file access functions, etc). Very aggravating.

If you can afford to miss the trigger from time to time (and, granted, if you're dealing with a small number of files it seems less likely to be a problem), then the file.watch sort of functions will suit you well. But if you cannot afford to miss even one - ever - then it may be appropriate to look at other approaches.

As to your file matching against masks, you may wish to take a look at Regex functions available in Alpha Five.

[Mike Wilson]

Re: file matching daemon

Look into filter_string_smatch()
Example:

dim vLdir_files as C
dim vLprocess as C

'process list (equivelent to them being in a table for this example)
vLprocess=<<%p%
my_file*9999???.DAT|process_A
your_file*9999???.DAT|Process_B
his_file*9999???.DAT|Process_C
their_file*9999???.DAT|Process_D
other_file*9999???.DAT|Process_E
%p%

'List of files coming in from the directory on the watch
vLdir_files=<<%files%
my_file_january2011sub.dat
my_file_january2011net.dat
your_file_january2011sub.dat
his_file_february2011sub.dat
other_file_September2010net.dat
their_file_January2011net.dat
%files%


'loop through the incoming files and match the file with the process
	dim vLPO as C
	dim vc_matcher as C
	dim vc_match1 as C
	dim vc_match2 as C

	for each foo in vLdir_files
		'acquiring the first matching component 
		vc_match1=word(foo,1,"_")+"_"+word(foo,2,"_")
	
		'acqiring the file type
		vc_match2=word(foo,2,".")
	
		'generating the mask equivelent with extracted match components and wildcards
		vc_matcher=alltrim(vc_match1)+"*"+alltrim(vc_match2)+"*"          'the second * needed to allow for process in list item
	
		'acquiring the process specific to the incoming filetype
		vLPO=vLPO+word(filter_string_smatch(vLprocess,vc_matcher,crlf()),2,"|")+crlf()
	next

'display the output
	msgbox("",vLPO)

[osirisja]

Re: file matching daemon

Hi Mike

That is exactly what I am looking for! The only slight design change to your suggestion will be that the watcher will (preferably) only process one file at a time, move it to a working directory, then match it from there, then go around again with a 5 second delay or so for the next file, etc, so many thanks to all who provided solutions - I just need to join the jigsaw together now :-)

Cheers all

Andy
--

Look into filter_string_smatch()
Example:

dim vLdir_files as C
dim vLprocess as C

'process list (equivelent to them being in a table for this example)
vLprocess=<<%p%
my_file*9999???.DAT|process_A
your_file*9999???.DAT|Process_B
his_file*9999???.DAT|Process_C
their_file*9999???.DAT|Process_D
other_file*9999???.DAT|Process_E
%p%

'List of files coming in from the directory on the watch
vLdir_files=<<%files%
my_file_january2011sub.dat
my_file_january2011net.dat
your_file_january2011sub.dat
his_file_february2011sub.dat
other_file_September2010net.dat
their_file_January2011net.dat
%files%


'loop through the incoming files and match the file with the process
	dim vLPO as C
	dim vc_matcher as C
	dim vc_match1 as C
	dim vc_match2 as C

	for each foo in vLdir_files
		'acquiring the first matching component 
		vc_match1=word(foo,1,"_")+"_"+word(foo,2,"_")
	
		'acqiring the file type
		vc_match2=word(foo,2,".")
	
		'generating the mask equivelent with extracted match components and wildcards
		vc_matcher=alltrim(vc_match1)+"*"+alltrim(vc_match2)+"*"          'the second * needed to allow for process in list item
	
		'acquiring the process specific to the incoming filetype
		vLPO=vLPO+word(filter_string_smatch(vLprocess,vc_matcher,crlf()),2,"|")+crlf()
	next

'display the output
	msgbox("",vLPO)