power outage fouls things

[Sandra Anderson]

Don't have time to check past messages - just moving today and got call that an app in A5 will no longer work since a power outage of a few minutes yesterday. I had them reindex everything, but didn't help. Can't search for customers, enter new records, etc. The records entered yesterday have disappeared. Can anyone help quickly?? Sandy

[Jim Rowlen]

RE: power outage fouls things

Try database compact (file on the menu bar). If that doesn't do it I hope you make safeties.

[tlyon]

RE: power outage fouls things

I ran into a similar problem once, before I learned the meaning of 'backup'. (I have that one memorized now!!)

I was able to recover most of my data by creating a new database and attaching the old tables to it. I had to try a lot of different methods to get the job done, including attempting an export/import operation.

Overall, not a pretty sight, but I wish you best of luck. I'd bet everyone here has been there at least once. It's no fun.

PS: if anyone isn't backing up regularly, disaster is just waiting for you to happen. Don't wait for that first huge disaster to strike. Start now, while you are still young.

Tom Lyon

[BRochford]

RE: power outage fouls things

Once you recover, suggest to ALL of your Customers that they invest a couple of $100 & purchase a UPS for their PC's and backup, backup, backup.

[Laura Wisk]

RE: power outage fouls things

Can some of you experts out there advise us on proper "back up" techniques? I have been trying to extract that info out of my literature and HELP.

[Tom Cone Jr]

RE: power outage fouls things

Not that I consider myself *expert*, but here's how I approach it:

1. First I think of the application as basically having two components. The application, including all the layouts (forms, reports, letters), global scripts, operations, and so on is the first. I make at least two copies of the entire database, and store them in separate locations. These will permit me to 'start over', if my pc melts down. In these copies I often include the data tables, memo files and indices, too, but I think of these copies as just templates that will be 'filled in' with more current versions separately if need be.

2. The actual 'data' is stored in the *.dbf and *.fpt files, and the indices are in the *.cdx files. These *.cdx files include the index definitions, too. These data and index files comprise the second component of my backup system. I make safety copies of these daily, rotating disks, tapes, or zip drive media... I store extra copies off-site in case there's a theft or fire at the primary location.

3. If the application is relatively small you could backup both components every time. Otherwise, it seems more efficient to me to just backup the data and indices, after I've made a couple of copies of the entire package.

To me there are several important, but often boring, details:
1) It's easy to skip rotating the backup media. Don't do it, if you use the same one over and over, and then discover you have a problem with it you don't have a backup for your backup! By rotating media you have several layers of backups available, each a bit older than the former.
2) Occasionally you should attempt a restore (to a temporary folder different from your primary database folder), just to verify that the backups are *in fact* usable.
3) Off site storage is a pain in the neck, and may never be needed. If that day should ever come, however, you'll thank your lucky stars that you have a second copy available...
4) The frequency of backups probably should vary depending on how much data entry is occuring. I balance the time and effort to do the backup against the time and effort to rekey the data entered in the system since my last backup. It doesn't make sense to backup after each new record. However, if your data entry is intensive you may want to backup more than once a day.

This is a good topic, and I look forward to seeing how others approach it.

-- tom

[csda1]

RE: power outage fouls things

Backing up is a very good thing to do. Key issues are

1. Backup data integrity over time
2. Choice of Media
3. Backup schedule

Since it is possible that you may not realize that some part of your data or application is corrupt (typically the least used sections or records), you must be able to go back to an old enough copy to find the correct version of the data. Thus, as said before, the backups must be made at progressively longer periods. For most clients, I suggest

1. A backup for every day of the week (or work week), reusing them each week. This will require 5 to 7 backups
2. A backup for every week of the month, reusing them each month. This will require 5 backups.
3. A backup for every month of the year, reusing them each year. This will require 12 backups.
4. A backup for every year that you keep forever.

If you enter a lot of data, you can also do several backups during the course of the day, although I typically make these to a hard drive for speed and convenience.

Keep a set on site and offsite (or at least in a fire/flood resistant location) if you can. You can also use a free or paid internet service to store your backup (but be careful of privacy issues of your files)

I suggest ZIPing the files using WinZIP or similar for 2 reasons. 1st, it compresses to over 90% of the original space most times. 2nd, it has error checking that will tell you if your backup ZIP file has ever been corrupted. This addresses one part of the integrity of the data over time.

Finally, I like ZIP drives (not to be confused with ZIP files) for there storage capability, reliable reading/writing (not the drive, but the disk media), and cost. If your ZIP files are small enough, you could potentially put it on a floppy. Another alternative are CD-R's and CD-RW's, although they write much slower. Some people use a removable hard drive as well.

The one method I abhorr (although needed for the larger backups) is tape systems. The reasons include incompatibility with another tape drive of the same model, slowness, difficult to restore and examine contents, constantly changing standards, formats, etc.

Regards,

Ira J. Perlow
Computer Systems Design & Associates
[email protected]

[Stan Mathews]

RE: power outage fouls things

TC,Jr.

That's "expert" advice if I've ever seen it.

Stan

[Jose de Leon]

RE: power outage fouls things

Here is a link to a good start on Backup strategies.

http://www.hp.com/tape/papers/strategy.html

And here is some advice based on my experience.

I run an ISP and backup is vital to our business. Here is what I have learned during this time.

1) You must be paranoid. Put yourself (or your client) in a state of paranoia when it comes to backups. You will naturally develop a good backup scheme this way.

Having said that,

2) Your backup hardware I prefer DAT Tape drives. They are highly reliable, fast, have built in error control, and proven technology. Don't worry about incompatibilities amoung Tape drives of same technology, the only incompatibilities are when you fail to verify that you are using the same software for backup/restore or if you used hardware compression (more on that later).

Yes, you can use CD Writers, but they lack error control features, Zip drives lack even more error control.

3) Your media Your choice of media will probably dictate the type of hardware. But again, DAT tapes are good and have long shelf life. Don't even consider using floppy disks as their shelf life is 1 year max with no usage. CDroms have a 100 year theoretical shelf life, but that is if you get no finger prints or scratches on it. I don't know what the shelf life of Zip disks are, but I suspect about 5 years or something similar to floppies.

4) Your backup schedule Follow it. Additionally, it is important that you retire old tapes that are no longer reliable. It is also important that you take a tape from your rotation and save it somewhere safe never to be used again unless in an emergency. You should pull a tape from the rotation at least once a month and replace with a fresh tape. Off site storage is best. There are off site storage services. Also read the link above to get an idea about schedules.

TIPS:

a) Never ever use a compression program like WinZip or Arj on MISSION CRITICAL data. These programs alter the contents of your files. Therefore, if there is an error extracting any of your files from the Zip archive, then you can pretty much kiss a large chunk of your data goodbye.

If that doesn't make sense, think about this: You just compressed a 100MB text file to 5MB. That is a 95% compression ratio. Where did that 95MB go? It didn't get "squeezed", it got mathematically encoded into less space. If I had a row of 25,000 letter A's, I could encode that into this format: 'A:25000' which could be interpreted to display a row of 25,000 A's. Suppose one of the zeroes gets lost due to an unrecoverable error, then I just lost 22,500 A's. If compression was not used, it would be much easier to fix the ONE CHAR that was missing than to fix the 22,500 CHARS that got nuked. This is why you never compress your MISSION CRITICAL backup data. For other types of backups, zipping is fine.

b) DO NOT USE the hardware compression feature of your backup device for the same reason above in addition to the fact that your tapes will become dependent on that particular make and model of tape drive. All devices that support hardware compression have a method of disabling this feature either via dip switch or jumpers.

c) DO USE devices that support error control as these can help you recover from an error.

d) USE RAID system for fault tolerance. This is only useful during a hard drive failure. It will allow you to restore instantly or they can switch to a mirrored drive instantly with no loss in productivity. For more info on RAID look at http://www.dpt.com


In conclusion, follow a backup schedule, use quality tape drives, like DAT, as they are more reliable, do not compress your files. Use RAID systems for fault tolerance.

Hope this helps.

Jose

[Jose de Leon]

RE: power outage fouls things

There's more

There is the matter of your database files!

How do you back those up? Do you backup in whole or do you backup the exported data?

It depends on the tools you have available to recover from corrupt database files. A few bad characters can corrupt an entire database file. But a few bad characters in a text file are easily recovered from.

I sometimes backup databases as a template with data or without data. Then I export data to an ASCII file with all the information available to import back into an empty database. Then backup the ASCII file. If you have to do a restore, it is a simple matter of reloading the ASCII file into an empty database.

Jose

[Oran Hamilton]

RE: power outage fouls things

"The best laid plans of mice and men aft gang astray!"

The pointers already presented pretty much cover the why's and how's, but I can not emphasize enough checking the media and the backup routine itself.

I worked for a hospital that was undergoing a change from two seperate mini's with proprietary software to one NT network with proprietary software. One of the mini's was for the laboratory and their test equipment. Two of the lab sections were not happy with the modules that came with the new software, so they elected to continue using the old until something was found.

During the conversion, the sections that were being moved to the new required specal extraction procedures to insure the data for the two remaining sections were left in place.

Prior to, and during, the conversion we had set back up procedures in place. There were daily, weekly, and monthly backups completed. The weekly and monthly were for purge archive data.

To make a long story short, after the conversion, we continued to do the daily, weekly, and monthly backups on schedule. As it turned out, the purge routine worked fine, but somehow the routine that was supposed to create the specific file(s) that we thought we were backing up had somehow been modified during the conversion. This resulted in approximately 4 months of weekly and monthly purge archive tapes that were essentially blank. We discovered this when the lab asked us to restore an encounter (specific lab results) for a patient.

Because we had been successfully using the same procedures for many years, we never checked the tape to insure that we had a correct backup. It was not the fault of the storage media, just the backup routine itself.

My advice - even though you assume you had a successful backup, check it regularly to insure it is viable. Otherwise, you could lose a lot of data especially if you are purging older data to keep the size of the database manageable.

As to the solution for the scrambled data, I have has some success doing the following:

1. If you do not have a template of your database/tables stored seperately, try copying table a table without data or indexes. If you are successful, try appending the data from the scrambled table to it. If you can append, you will have to do it table by table, then check your field rules and finally recreate your indexes.

2. If you developed the app, you should have the original database and all table somewhere. Good practice dictates you develop it in a test area and then place the working copy in the live area. Usually you wind up using the test bed to build enhancements and copy the enhancements to the live area. If so, make a copy of the test database in another directory with field rules and indexes but no data. This will give you a template you can then use. Again, try appending the data table by table. If you are lucky, you will have everything back in place. If this works, backup the scrambled stuff, delete it, then copy the template to the live area.

HTH

Oran

[csda1]

RE: power outage fouls things

Jose,

I welcome your informed opinion, and much of what you say is absolutely true, but I have to also disagree with you on several points you made.

"1) You must be paranoid."
Absolutely true, but if everyone (or everything) is against you, then you are not paranoid!

Whether you are Christian, Moslem, Jewish, Atheist, Agnostic or anything else, it's imperative you become religious about backups!

"2) Your backup hardware Don't worry about incompatibilities amoung Tape drives of same technology, the only incompatibilities are when you fail to verify that you are using the same software for backup/restore or if you used hardware compression (more on that later).

You must make sure your media can be read by 99% of the drives available to read it. When you have a fire or flood or just a mechanical failure, your original hardware drive (tape, zip or otherwise) WILL NOT BE THERE TO READ YOUR BACKUP Zip drives (which claims that any ZIP disk is readable by any working ZIP Drive), CD-R/CD-RW (which have a tight standard) are examples of this. Tape drives are notorious for lack of interchangeability. However there may be some Tape drives and media which are OK.

The other part of this is that the hardware may be no longer available (How many Jumbo 250 tape drives are still being used?) If the format is discontinued/unsupported, you may find that you no longer have a piece of hardware to read your backup. This means that you must constantly change your media storage to one that is "in vogue" and popular.

Yes, you can use CD Writers, but they lack error control features, Zip drives lack even more error control.

I'll argue that this is unimportant. If you have 25 backups, even if one failed, you have others to choose from (yes they may be a slight bit older). Backup is required due to a catastrophic loss, or because of corruption of the data over time. For catastrophic failure, the last 2 backups are most important, for data corruption, you may have to go back quite far in your backups to find good data.

3) Your media Your choice of media will probably dictate the type of hardware. But again, DAT tapes are good and have long shelf life. Don't even consider using floppy disks as their shelf life is 1 year max with no usage. CDroms have a 100 year theoretical shelf life, but that is if you get no finger prints or scratches on it. I don't know what the shelf life of Zip disks are, but I suspect about 5 years or something similar to floppies.

If you are rotating your backups, shelf life is unimportant except for yearly backups. Floppies are cheap (because many user's are on limited budgets). CD's should never have any fingerprints or scratches as you should not be touching them except to do the backup. That makes them better for less frequent backups. and if using CD-R's, you might write multiple sessions, but eventually they would be filled up, and then they should never be physically touched very often.

Never ever use a compression program like WinZip or Arj on MISSION CRITICAL data. These programs alter the contents of your files. Therefore, if there is an error extracting any of your files from the Zip archive, then you can pretty much kiss a large chunk of your data goodbye.

These compression programs don't alter your data, they compress them. I think many would be confused by what you said. The reverse process restores your data EXACTLY. There are many checksums and CRC codes used to allow verification that the data inside is in fact the original contents. Using CRC's and similar error checking codes (whether compressed or not, Tape Drive/or disk) is absolutely required. It's the only way you can be sure your media has not failed on you.

If that doesn't make sense, think about this: You just compressed a 100MB text file to 5MB. That is a 95% compression ratio. Where did that 95MB go? It didn't get "squeezed", it got mathematically encoded into less space. If I had a row of 25,000 letter A's, I could encode that into this format: 'A:25000' which could be interpreted to display a row of 25,000 A's. Suppose one of the zeroes gets lost due to an unrecoverable error, then I just lost 22,500 A's. If compression was not used, it would be much easier to fix the ONE CHAR that was missing than to fix the 22,500 CHARS that got nuked. This is why you never compress your MISSION CRITICAL backup data. For other types of backups, zipping is fine.

You can't fix the code unless you know it's missing. Hence the need for CRC's in all cases. It is also 25000 more likely to have a single character corrupted than when compressed. With compression, you can make several backups in the same space. You only need one to be good. Also, many of the compression formats have Error Correction Codes, which means that, at least a certain class of failures can be repaired, even though they are compressed.

"d) USE RAID system for fault tolerance."

If in your budgets, well worth it. You can use Prommise technologies IDE RAID Controller with 2-4 IDE drives for a relatively low cost solution. Visit them at http://www.promise.com


Finally, despite any controversy in what we have discussed about backups, the most important thing to remember is;

JUST DO IT!


Regards,

Ira J. Perlow
Computer Systems Design & Associates
[email protected]

[Elliott Lester]

RE: power outage fouls things

Ira:

Good points.

We do not even call them backups, but rather safeties.

I'll add my two cents worth to all of the suggestions.
Hard drive space is very cheap right now. We add hard drives to systems as a second or third or 4th drive and zip and copy over the entire folder to that unused or rarely used drive on a often basis. Then open the database to make certain it works and close it. We then add a shortcut
and label it safetie 1, safetie 2. This permits almost instant access to the system incase of any failure--your customers do not care that you CRASHED.

In addition, any time Jim make any change whatsoever, or if we have any major crash--that is defined as a crash that rebuilding of indexes does not solve the problem--we do a
safety prior to attempting a major fix. We do it on one of our development systems, or extra hard drives.

We utilize CD-Rs for daily, weekly archieves as we call them. They are fast, transportable and almost every PC made today has the ability to read them. And we don't just
copy certain files, but rather the entire database--yes
compressed by Winzip. These CD archieves are removed from our office and stored off site. In this way, if there was a fire, we could go down to any PC store and grab one, load up the CD-rom disc and be back in business in the time it took to get there and back.

We have found that many problems that have occurred required us to go back and back to a clean safety. What I mean by this, is that sometimes data gets Scrambled and you do not know this--memo fields for example. You continue your safeties (backups) daily with this scrambled data. Then you have a failure and must reconstruct your data. If you are religious about safeties, you can go back until you find clean data and then rebuild from there.

The key to all backups or whatever you call them is to
make certain that you determine a schedule that will appropriately protect you and require the least amount of
manual work for your staff to reconstruct work. This analysis requires you to bring operations into the equation.
We found that we store hard copy or electronic incoming data in a format that can easily be reproduced. For example, for key punchers we store all items being put into the system by date in a file cabinet. In this way, if we crash like the the first person did, we can fix the problem and easily reconstruct the data by just pulling out the data inputted and reimputting it into the system.

Jim archieves all EDI incoming and outgoing for the same
reason. Yes, we could rerun outgoing, but it is easier to just go to the archieve, unzip what you sent yesterday and resend it. Same for EDI incomming. We extend this to
FAXes and e-mail also. Since all incoming and outgoing
faxes and e-mail might reviewed for data input, we archieve
these as well to a schedule that was developed with a rebuild in mind--how much time and effort and risk can you take if you lose data and need to reconstruct orders or
outgoing information.

While we love zip drives and utilize them for to and from
tasks, CD-Rs are very hardy, easily transportable, pretty
inexpensive and easy to use. They could be locked in a fire proof safe ($100.00 at office max), or taken home or
put into a safe deposit box.

Let me add that you must also clearly mark any safety for what it is, and mark it so that some lay person or other programmer understands. We do it in plain english with
the date and contents.

I suggest that everyone create a disaster recovery plan that will detail all of this including by the way how to
run your system in case of failure--or G-d forbid death or incapacity of the programmer. This should be placed where
it is accessible by operations and each year you need to
go through a disaster recovery test with a lter (low teker)
to make certain they can follow and recover your data.

Finally, with e-mail, you can even zip up and send your
entire database to off site machines almost if not automatically. Just make certain that on a regular basis
(we do it daily) someone checks to make certain that there
is data, that the system can be unzipped and it works.

I know companies that double load onto their CD-Rs zipped up
safeties with the hope that if one does not work, the other will.

Also don't forget about separate accounting systems, Word processing programs and even your outlook. And something very important, if you upgrade your system and no longer utilize the backup medium you have chosen, you must transfer the former medium to what you utilize--two years from now you may not beable to find it, or you will have to pay alot of money for someone else to transfer it.

So:

1. Make a written plan
2. Incorporate operations into the analysis to assure that
hard copy data can be reconstructed easily and assuming
as Ira pointed out you may not have the same PC as you
originally backed up upon or through.
3. Create a schedule that your company can live with
cost, risk all assessed.
4. Clearly label all medium
5. Recognize that LTers might have to unzip and/or recreate
data quickly (shortcut on the desktop example)
6. Keep sufficient safeties to restore non corrupt data.
7. Store off site, or within fireproof and water proof
storeage.
8. Check by unzipping and using on a test basis all
backups to assure that you actually have data.
9. Once a year simulate a disaster and reconstruct your
system. (you should keep copies of licenses #s manuals
and any other literature or documents necessary to
reload from scratch A5 or other backed up software.
10. When in development, create safeties often,

Always know that if a hard drive fails, it often for money can be rebuilt--for around $1200.00 to $2000.00. This means that the data itself--not the database can be recovered off a crashed drive by companies that specialize in this. You can find them on the net.

Good luck