Appending duplicate records to another db

[gkeramidas]

how can i accomplish this? let's say a db has 3 duplicate records. i mark them and only 2 get marked so i can only append the 2 records. i would like to append all 3 records, not just the two duplicates. why you ask, because they are only duplicates in 2 fields. i need to compare some data in a third field. so i want the 3 records it the new db, with 3 fields.

can this be accomplished with dupes or do i need to proceed in another direction?

thanks

gary

[Geoffrey Hollander]

RE: Appending duplicate records to another db

Gary,

I'm currently in the final stages of writing a dupe checker and I can tell you that it works great; you can do this with Alpha Five very neatly, but you have to write some XBasic code. You also will need to add a field to your table in order to find the surviving (unmarked) record in each dupe set.

Add a numeric field called something like: DUPE_GRP(N, 6, 0). Once you have marked the duplicate records you then sort the table in DUPE_GRP + unmarked() order (note: not actual syntax).

Next, go through the table finding the first marked record in each set. When the first one is found, go back up one record, add a number 1 to the DUPE_GRP field and continue to do so until each record in that set has been assigned the same DUPE_GRP number. Repeat the loop, adding 1 to the next set's DUPE_GRP number, using a counter, until you have gone through the whole table.

Once that has been done, you can copy out all the dupes -- survivors and casualties alike -- to another table. Now that you have a DUPE_GRP flag for all records in each set of dupes, you can delete them, sort them, split them into survivor only and casualty only tables, etc.

Good luck.

Regards,

Geoff Hollander

[gkeramidas]

RE: Appending duplicate records to another db

ok, geoffrey, i'll give that a try

[gkeramidas]

RE: Appending duplicate records to another db

geoff:

this worked for me, you may want to see if it will work for your application.

db fields example: date, time, card#, amount. i need to check for duplicate amounts.

1. i marked duplicate amounts.
2. i did an append to a new database, including all fields, of the marked records.
3. then i did an intersect of the new database to give me all of the duplicate amounts in the original database and stored them in a 3rd database.
now i have all duplicate amounts in a separate database that i can do my reporting with.

let me know what you think

gary

[Geoffrey Hollander]

RE: Appending duplicate records to another db

Gary,

I'm glad you got things working. Your approach is a good one in that, whenever changing a table significantly, it's smart to work with outside or temporary tables.

The only problem I would have using this approach for my application would be, because it is a utility program meant to work on any table thrown at it; that I never know, from table to table, what its structure and field characteristics are going to be.

The only way I can do this (that I've thought of so far, anyway) is to copy in the table to a "working" table. The user, based on the selected operation, will then map certain types of fields currently comprising the working table.

If, for example, the user is searching for near duplicate records based on a matchcode of: "first_name + last_name + address + substr(zip,1,5)", the user has to tell the program which fields hold the first and last name, address and ZIP Code information.

The working file is already a temporary file in this case, so I'm kind of free to beat it up a little. Typically in merge/purge operations the steps are to define the matchcode and how closely you want the "suspects" to match, find and then mark the dupes, review and edit what has been found (disposition status and-or data), run a report showing survivor and casualy records and their disposition (after editing), and then saving the unique and survivor records to one file, the duplicate, casualty records to another file and, sometimes, outputting a copy of the complete file with the dupe flags (usually for later processing).

But, of course, this can vary too. Skipping steps and being able to go back to and change previous setup options and rerun operations to get more accurate results all mean that I have to plan to handle record processing sequentially, both backwards and forwards, so that each operation lays some ground work for the probable next step.

I think your solution is innovative and will certainly keep your data clean. (I guess it just goes to show, again, what a great product Alpha Five is.) Good job Gary, writing dupe checking code's not for sissies!

Best regards,

Geoff Hollander