Alpha Software Mobile Development Tools:   Alpha Anywhere    |   Alpha TransForm subscribe to our YouTube Channel  Follow Us on LinkedIn  Follow Us on Twitter  Follow Us on Facebook

Announcement

Collapse

The Alpha Software Forum Participation Guidelines

The Alpha Software Forum is a free forum created for Alpha Software Developer Community to ask for help, exchange ideas, and share solutions. Alpha Software strives to create an environment where all members of the community can feel safe to participate. In order to ensure the Alpha Software Forum is a place where all feel welcome, forum participants are expected to behave as follows:
  • Be professional in your conduct
  • Be kind to others
  • Be constructive when giving feedback
  • Be open to new ideas and suggestions
  • Stay on topic


Be sure all comments and threads you post are respectful. Posts that contain any of the following content will be considered a violation of your agreement as a member of the Alpha Software Forum Community and will be moderated:
  • Spam.
  • Vulgar language.
  • Quotes from private conversations without permission, including pricing and other sales related discussions.
  • Personal attacks, insults, or subtle put-downs.
  • Harassment, bullying, threatening, mocking, shaming, or deriding anyone.
  • Sexist, racist, homophobic, transphobic, ableist, or otherwise discriminatory jokes and language.
  • Sexually explicit or violent material, links, or language.
  • Pirated, hacked, or copyright-infringing material.
  • Encouraging of others to engage in the above behaviors.


If a thread or post is found to contain any of the content outlined above, a moderator may choose to take one of the following actions:
  • Remove the Post or Thread - the content is removed from the forum.
  • Place the User in Moderation - all posts and new threads must be approved by a moderator before they are posted.
  • Temporarily Ban the User - user is banned from forum for a period of time.
  • Permanently Ban the User - user is permanently banned from the forum.


Moderators may also rename posts and threads if they are too generic or do not property reflect the content.

Moderators may move threads if they have been posted in the incorrect forum.

Threads/Posts questioning specific moderator decisions or actions (such as "why was a user banned?") are not allowed and will be removed.

The owners of Alpha Software Corporation (Forum Owner) reserve the right to remove, edit, move, or close any thread for any reason; or ban any forum member without notice, reason, or explanation.

Community members are encouraged to click the "Report Post" icon in the lower left of a given post if they feel the post is in violation of the rules. This will alert the Moderators to take a look.

Alpha Software Corporation may amend the guidelines from time to time and may also vary the procedures it sets out where appropriate in a particular case. Your agreement to comply with the guidelines will be deemed agreement to any changes to it.



Bonus TIPS for Successful Posting

Try a Search First
It is highly recommended that a Search be done on your topic before posting, as many questions have been answered in prior posts. As with any search engine, the shorter the search term, the more "hits" will be returned, but the more specific the search term is, the greater the relevance of those "hits". Searching for "table" might well return every message on the board while "tablesum" would greatly restrict the number of messages returned.

When you do post
First, make sure you are posting your question in the correct forum. For example, if you post an issue regarding Desktop applications on the Mobile & Browser Applications board , not only will your question not be seen by the appropriate audience, it may also be removed or relocated.

The more detail you provide about your problem or question, the more likely someone is to understand your request and be able to help. A sample database with a minimum of records (and its support files, zipped together) will make it much easier to diagnose issues with your application. Screen shots of error messages are especially helpful.

When explaining how to reproduce your problem, please be as detailed as possible. Describe every step, click-by-click and keypress-by-keypress. Otherwise when others try to duplicate your problem, they may do something slightly different and end up with different results.

A note about attachments
You may only attach one file to each message. Attachment file size is limited to 2MB. If you need to include several files, you may do so by zipping them into a single archive.

If you forgot to attach your files to your post, please do NOT create a new thread. Instead, reply to your original message and attach the file there.

When attaching screen shots, it is best to attach an image file (.BMP, .JPG, .GIF, .PNG, etc.) or a zip file of several images, as opposed to a Word document containing the screen shots. Because Word documents are prone to viruses, many message board users will not open your Word file, therefore limiting their ability to help you.

Similarly, if you are uploading a zipped archive, you should simply create a .ZIP file and not a self-extracting .EXE as many users will not run your EXE file.
See more
See less

Append times, reasons, solution?

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

    Append times, reasons, solution?

    I think I have an answer to those of you that have asked why some appends take so long (this was prompted by a post by xxxxxx). I've also seen slow appends, but not for every table. So, I ran a test, using a saved operation and as a script (there was no time difference in those two methods).

    My computer is a P3-500 with 64MB RAM, Win98, using v4.03 build 221 for the test.

    I used a customer's table SP (which I had noticed was slow to append) with about 35000 records. The table has 12 indexes (some of which are rather complex), 30 fields, and LOTS of field rules, some of which are lookups or complex calculations. So, I made a copy of 500 records of the table, and called it SPX. Then I appended SPX to SP.

    It took about 150 seconds for the append portion of the operation, and another 20 seconds for the index rebuild portion (which is about the same time it takes to update indexes).

    Then I eliminated all the indexes. The append portion took the same amount of time, but there were no indexes to rebuild, so the overall operation was faster.

    Next, I eliminated all the field rules. BAM, it was done!! (The index rebuild still took 20 seconds). However, eliminating field rules isn't necessarily the best option. So I looked at the rules, to see which ones were causing the slowdown. I found two that used LOOKUPN. I took those out, and ran the append. It was very nearly as fast.

    This makes sense, since LOOKUPN (and LOOKUPC and LOOKUPD) are going to take a lot longer to execute that simpler rules.

    These results are consistent with my previous experience with other customers, in that some appends are really fast (appending to tables with few field rules), and some are REALLY slow. This is because the append evaluates the field rules (to make sure records that violate the rules are NOT appended, but end up in VIOLATIONS.DBF instead *). As a result, even a small table could take a long time to append if the field rules are sufficiently complex.

    Now that the problem is defined, how about a solution? Something that seems to work quite well is to rename the dictionary files (SP.DDD, SP.DDM, SP.DDX) to something like SP.XDD, SP.XDM, etc., so the field rules are NOT enforced, and then run the append from a script (hint: after creating a saved append, run it with the script recorder on. Then copy the results to a global script. You'll have to change the first line, "table=table.current()" to "table=table.open("sp")"). After running the append, rename the dictionary files back to their original names, to restore the field rules.

    One possible disadvantage is that you might append data that violates the field rules (which would NOT append if the field rules are enforce, but end up in VIOLATIONS.DBF instead). My advice: make sure the data is valid BEFORE the append.

    #2
    RE: Append times, reasons, solution?

    Philip, sorry - didn't mean to leave your name as "xxxxx"

    Comment


      #3
      RE: Append times, reasons, solution?

      I've been called worse. I sent you an email on this.

      Thanks

      Comment


        #4
        RE: Append times, reasons, solution?

        Bill,

        Good information.

        LOOKUPx() will improve in speed in V5 as it will not keep opening and closing the lookup table (it will use a caching method).

        If the number of records are small compared to the size of the database, you may actually want to run the append in a shared mode. In this method, A5 updates the index after each record append. It does not have to rebuild the index at the end.

        So the choice is faster appends with index rebuilds vs slower appends with no index rebuilds. It depends upon your numbers.

        There may be a way to open the table being appended to and not honor field rules (other than renaming the dictionaries), but I'll have to investigate that one. For renaming, I suspect Alpha will try to recreate new ones, so they will have to be deleted after the append.

        Regards,

        Ira J. Perlow
        Computer Systems Design & Associates
        [email protected]
        Regards,

        Ira J. Perlow
        Computer Systems Design


        CSDA A5 Products
        New - Free CSDA DiagInfo - v1.39, 30 Apr 2013
        CSDA Barcode Functions

        CSDA Code Utility
        CSDA Screen Capture


        Comment


          #5
          RE: Append times, reasons, solution?

          Ira, by renaming the three dictionary files, you also eliminate the indexes, so rebuilding isn't an issue.

          You are correct, Alpha will create new files. Therefore, before you rename them to their original names, you'll have to delete the ones created AFTER you renamed the originals.

          Comment


            #6
            RE: Append times, reasons, solution?

            Oops (hm, I seem to be doing a lot of OOPS today), I was wrong about the indexes, since they are in the CDX file, and are NOT eliminated when renaming the dictionary files.

            I look at it this way -- sooner or later, you really SHOULD update indexes, so whether you do it during the append or later, it's going to take the same amount of time.

            Comment


              #7
              RE: Append times, reasons, solution?

              Bill:

              Remember that if you rename the data dictionary and open the table without the dd any fields with long names will be truncated. Voice of experience!

              I'v found that, tedious though it may be, the only safe and secure way to is to remove all validation and autoincrement rules pre-append and replace them post-append.

              Finian
              Finian

              Comment


                #8
                RE: Append times, reasons, solution?

                Finian & Bill

                Then the most appropriate way might be to have two data dictionaries, the 2nd having no field rules (nothing else is required either). Then swap data dictionaries as needed. Be sure to swap the right ones being used (optimized or network copy) as appropriate.

                Regards,

                Ira J. Perlow
                Computer Systems Design & Associates
                [email protected]
                Regards,

                Ira J. Perlow
                Computer Systems Design


                CSDA A5 Products
                New - Free CSDA DiagInfo - v1.39, 30 Apr 2013
                CSDA Barcode Functions

                CSDA Code Utility
                CSDA Screen Capture


                Comment


                  #9
                  RE: Append times, reasons, solution?

                  Finian, I didn't realize this! Thanks for the warning!

                  My personal practice is to keep my table names to 8.3 and field names 10 characters or less, so I never would have run into this.

                  Ira, I think yours is a good idea. Sounds like it would also be a good submission for the soon-to-be Tips section.

                  Bill

                  Comment


                    #10
                    RE: Append times, reasons, solution?

                    in v5 Xbasic it will be possible to modify the field rules
                    through Xbasic. So it should be possible to disable the
                    lookups and then add them back at the end.

                    in v4, however,the simpler solution may just be to
                    add records one at a time by copying them from one table
                    to another, rather than running an append operation. Then
                    I don't believe the field rules will all be evaluated.
                    e.g.
                    t=table.open("master",file_rw_shared)
                    t1=table.open("transaction",file_ro_shared)
                    t1.fetch_first()
                    while .not. t1.fetch_eof()
                    data=t1.record_data_get()
                    t.enter_begin(IGNORE_FIELD_RULES)
                    t.record_data_set(data)
                    t.enter_end(.t.)
                    t1.fetch_next()
                    end while
                    t1.close()
                    t.close()

                    something like this should work, without evaluating field
                    rules.

                    Comment

                    Working...
                    X