Xbasic - How to skin a cat, the best way. Matching a tag.

[KeithW]

Re: Xbasic - How to skin a cat, the best way. Matching a tag.

IRA,

I promise I will look at your TIPS page and I really do appreciate the benefit of your wisdom. At this point in time I cannot worry about whether I have the most efficient version or not. What I have to do needs to perform, but when done the next time for the same customer, 9 to 15 months will have passed. I need it to be easily modified for the next use, as I do not know until I get their data-set just what I will be playing with. YES maybe it could be improved upon and over time it WILL improve, I just need something that will get the job done "today" and get the project off-my-plate. As far as funds go, anything I drop out at the moment is purely out of my pocket and it is not that plush at the moment so I will have to make do until something changes. Again, I DO APPRECIATE the time you take to point out the finer points of A5 programming. For the record, I have v10, v11 and Anywhere at my disposal and I could go back and load older if there was a real need.

Keith

[jacobe]

Re: Xbasic - How to skin a cat, the best way. Matching a tag.

This problem is basically that of a spell checker, except you want to flag words that match.

Pattern matching and string searching make for slow spell checking.

The classic approach for spell checking is to use a hash table. Hash searching is extremely fast. Search speed is pretty much constant no matter what size word or phrase you are looking for.

Basically you want to turn your "dictionary" of words into a hash table and index the hash. For each record to check, you would then turn the record contents into a list of hashes to search against your dictionary.

Alpha 5 has a two built in functions for hashing, *CRC32() is probably good enough for this application, it's fast but you would want to see if you had collisions (it's not really designed for hashing words). MD5() is overkill and slower, the searches are going to be slower (but how slow depends on the size of your dictionary).

[G Gabriel]

Re: Xbasic - How to skin a cat, the best way. Matching a tag.

Keith:
Didn't mean to offend with the covert government thing..disregard.
When I mentioned the spelling I was trying to make a corollary between the possible variations of what you are searching for and misspelling a word as an analogy.
Yes, I think an array is a good way to go. To make it better, you could
1-Read all the terms in an array.
2-The array, of course, will have a reference to the company (an ID for instance)..
3-To make things faster, once you encounter a term that matches in the array, you could flag this company and expunge all the other terms for that company from the array so you do not have to keep reading them again and again down the array.
As I said, I had trouble with matching in an array in v8, but you could test and see how well it works in the newer versions.
And as to the different format of files from these companies, I would just read them as much as possible into a csv file.
And as an aside, Excel is extremely powerful assuming you know how to program in VB. You could do all of the above and more in VB. Only drawback to Excel is the number of records.

[KeithW]

Re: Xbasic - How to skin a cat, the best way. Matching a tag.

NO offense taken, just pointing we are neither the government, nor the bank, we are an independent firm that is taking direction from the bank plus our expertise and doing what needs to be done. Sometimes the client is gracious and other times they are adversarial but in either case we simply have a job to do and regardless of the relationship, it gets done.

I am NOT really worried about the spelling errors, at least not the point of trying to hash terms or do proximity searches, that is overboard for our purposes. I cannot necessarily flush the array on the first hit unless I want to have to review every hit... if I hit multi-terms and the code to store is different, then instead of placing FED, STATE, MUNI I place MULTI and then can search for that specifically and change to what is appropriate.

YES, from the beginning my idea was to load up the array and then ditch the TERMS file as it is no longer needed. Once the pass is thru (array), UPDATE THE Master File if a match exists and then get the next Master File record and respin thru the array again. How many Master File records exist is how many times the array will be spun thru.

Again as Ira said, once thru each file.

Keith

[G Gabriel]

Re: Xbasic - How to skin a cat, the best way. Matching a tag.

And if you want to make things faster, I would sort those search words in a descending order according to the most popular ones.

[KeithW]

Re: Xbasic - How to skin a cat, the best way. Matching a tag.

If you *almost* always need to "go thru the *whole* list" as apposed to "stop on first match", why would it matter, just wondering....

[G Gabriel]

Re: Xbasic - How to skin a cat, the best way. Matching a tag.

It matters.
Say the most popular word is "University"
and say you have 700 elements in the array
if "university", being the most popular, occurs in say 200 of these 700 and if you remove these 200 elements from the array after the first run, then in the second run you will be searching only 500 elements with the second most popular word..the array will shrink much more faster than if you search 700 elements for a word that occurs only once in the average..

[KeithW]

Re: Xbasic - How to skin a cat, the best way. Matching a tag.

Well hopefully, I will only have to run it one time per review. In time I would learn what the most popular terms are however initially, I would not know that and the list has to be exercised at least one time. Again, this is not a daily thing and maybe not even monthly, quarterly and annually yes...