Alpha Software Mobile Development Tools:   Alpha Anywhere    |   Alpha TransForm subscribe to our YouTube Channel  Follow Us on LinkedIn  Follow Us on Twitter  Follow Us on Facebook

Announcement

Collapse

The Alpha Software Forum Participation Guidelines

The Alpha Software Forum is a free forum created for Alpha Software Developer Community to ask for help, exchange ideas, and share solutions. Alpha Software strives to create an environment where all members of the community can feel safe to participate. In order to ensure the Alpha Software Forum is a place where all feel welcome, forum participants are expected to behave as follows:
  • Be professional in your conduct
  • Be kind to others
  • Be constructive when giving feedback
  • Be open to new ideas and suggestions
  • Stay on topic


Be sure all comments and threads you post are respectful. Posts that contain any of the following content will be considered a violation of your agreement as a member of the Alpha Software Forum Community and will be moderated:
  • Spam.
  • Vulgar language.
  • Quotes from private conversations without permission, including pricing and other sales related discussions.
  • Personal attacks, insults, or subtle put-downs.
  • Harassment, bullying, threatening, mocking, shaming, or deriding anyone.
  • Sexist, racist, homophobic, transphobic, ableist, or otherwise discriminatory jokes and language.
  • Sexually explicit or violent material, links, or language.
  • Pirated, hacked, or copyright-infringing material.
  • Encouraging of others to engage in the above behaviors.


If a thread or post is found to contain any of the content outlined above, a moderator may choose to take one of the following actions:
  • Remove the Post or Thread - the content is removed from the forum.
  • Place the User in Moderation - all posts and new threads must be approved by a moderator before they are posted.
  • Temporarily Ban the User - user is banned from forum for a period of time.
  • Permanently Ban the User - user is permanently banned from the forum.


Moderators may also rename posts and threads if they are too generic or do not property reflect the content.

Moderators may move threads if they have been posted in the incorrect forum.

Threads/Posts questioning specific moderator decisions or actions (such as "why was a user banned?") are not allowed and will be removed.

The owners of Alpha Software Corporation (Forum Owner) reserve the right to remove, edit, move, or close any thread for any reason; or ban any forum member without notice, reason, or explanation.

Community members are encouraged to click the "Report Post" icon in the lower left of a given post if they feel the post is in violation of the rules. This will alert the Moderators to take a look.

Alpha Software Corporation may amend the guidelines from time to time and may also vary the procedures it sets out where appropriate in a particular case. Your agreement to comply with the guidelines will be deemed agreement to any changes to it.



Bonus TIPS for Successful Posting

Try a Search First
It is highly recommended that a Search be done on your topic before posting, as many questions have been answered in prior posts. As with any search engine, the shorter the search term, the more "hits" will be returned, but the more specific the search term is, the greater the relevance of those "hits". Searching for "table" might well return every message on the board while "tablesum" would greatly restrict the number of messages returned.

When you do post
First, make sure you are posting your question in the correct forum. For example, if you post an issue regarding Desktop applications on the Mobile & Browser Applications board , not only will your question not be seen by the appropriate audience, it may also be removed or relocated.

The more detail you provide about your problem or question, the more likely someone is to understand your request and be able to help. A sample database with a minimum of records (and its support files, zipped together) will make it much easier to diagnose issues with your application. Screen shots of error messages are especially helpful.

When explaining how to reproduce your problem, please be as detailed as possible. Describe every step, click-by-click and keypress-by-keypress. Otherwise when others try to duplicate your problem, they may do something slightly different and end up with different results.

A note about attachments
You may only attach one file to each message. Attachment file size is limited to 2MB. If you need to include several files, you may do so by zipping them into a single archive.

If you forgot to attach your files to your post, please do NOT create a new thread. Instead, reply to your original message and attach the file there.

When attaching screen shots, it is best to attach an image file (.BMP, .JPG, .GIF, .PNG, etc.) or a zip file of several images, as opposed to a Word document containing the screen shots. Because Word documents are prone to viruses, many message board users will not open your Word file, therefore limiting their ability to help you.

Similarly, if you are uploading a zipped archive, you should simply create a .ZIP file and not a self-extracting .EXE as many users will not run your EXE file.
See more
See less

Speeding Up a Loop Inside a User Defined Function

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

    Speeding Up a Loop Inside a User Defined Function

    I created a UDF to test (random selection) a population of 5 million theoretical records. Although I have not attempted to run this sample against the full 5 million, I have tested it against 50,000 and 150,000 records and noticed that my function will process 50,000 records in 70 seconds and 150,000 in 275 seconds. The results or the records selected are captured in an array. I included the basic building blocks of my function in the code section below so you can get an idea on how I structured the UDF. My concern is that the speed of the function is slowing down as more records are processed. I also noticed that my CPU usage for Alpha stays around 8.5% and the memory usage stays around 70 mb. If there is a way to utilize the full capacity of my CPU, I am open to suggestions. If there is a more efficient method to design my UDF, I will appreciate any input. I did try to maximize the efficiency of my UDF through the following:

    • The function type is V
    • The function does not make any calls to any other global UDF
    • While minimizing the use of functions, any necessary functions are embedded in my UDF.
    • I minimized the use of variables unless necessary.


    Code:
    myFunction as V ()
    
    dim sample[] as P
    dim some_other_vars
    
    for i = 1 to 5000000[INDENT]ranNum = my_Embedded_RanNum_Generator()
    
    if ranNum < .01 then[/INDENT][INDENT=2]
    dim a as L[/INDENT][INDENT=2]a = meets_criteria_check()
    
    if a then[/INDENT][INDENT=3]sample[].number = i
    sample[..].other_stuff = other stuff[/INDENT][INDENT=2]end if[/INDENT][INDENT]end if[/INDENT]
    
    next i
    
    file.From_property("test/test.txt", sample)
    
    End Function
    I realize I could break out my criteria check into its own independent process to reduce the loops but the combined process time may be the same. I was hoping someone would have a suggestion on the function as is before I go down this road. Could changing the loop method make a difference? It appears Alpha is only using 1 of my 12 processing cores. Is there a way to increase this?
    Thanks,
    James

    #2
    Re: Speeding Up a Loop Inside a User Defined Function

    What if you try this? Depending on what other variables you have defined, you might want to add it to the other embedded functions.

    Code:
    myFunction as V ([COLOR="#FF0000"]vars as p = local_variables()[/COLOR])
    [COLOR="#FF0000"]WITH vars[/COLOR]
    dim sample[] as P
    dim some_other_vars
    
    for i = 1 to 5000000
    ranNum = my_Embedded_RanNum_Generator()
    
    if ranNum < .01 then
    
    dim a as L
    a = meets_criteria_check()
    
    if a then
    sample[].number = i
    sample[..].other_stuff = other stuff
    end if
    end if
    
    next i
    
    file.From_property("test/test.txt", sample)
    [COLOR="#FF0000"]END WITH[/COLOR]
    End Function
    Robin

    Discernment is not needed in things that differ, but in those things that appear to be the same. - Miles Sanford

    Comment


      #3
      Re: Speeding Up a Loop Inside a User Defined Function

      James,

      I don't see a need to declare (dim) the variable "a" each time through the loop. Consider moving it to the line before "for i = 1 ....."

      Then assign it the value FALSE at the top of the loop, before the ranNum var is populated.

      Two functions are called each time through the loop, neither are shown.
      my_embedded_ran_num_generator()
      meets_criteria_check()

      these should be examined for ways to be improved

      Comment


        #4
        Re: Speeding Up a Loop Inside a User Defined Function

        Tom, good point but this only happens in the code I provided in the message. It does not exist in my actual function. I just did a piss poor example of my code. My original question should also probably be revised to: Why is their a degradation in speed within a for..next loop when you are adding information to a property array? It becomes apparent after 100,000 loops.

        I did find that my seed number (used in my embedded random number generation function) was being created every time I requested a random number which it should not be. Once the seed number is generated, it should not be changed. This will allow me to reproduce the sample as long as I record the original seed number. Changing this helped. Now I can process 150,000 records in 134 seconds instead of 275. 300,000 records takes 460 seconds so my degradation still exists as it takes 326 seconds to process an additional 150,000 records.

        I am setting up several other alternative methods to process this volume and will post back my results. The two other methods I will try are as follows:
        1. Record the results to a text file instead of a property array. I have a feeling the rapid growth in the property array size is causing the degradation but I am not sure. Recording the results to a text variable or directly into a text file may alleviate this issue. I can always pull it into an array from the final text that is generated.
        2. Create a master function that breaks up the loops into 100 sub-loops so that I can delete the property array before the start of each sub-loop. This will keep the array size from growing too large and should keep the speed of each sub-loop close to the results above. Right now I am just throwing s**t on the fan to see if it sticks.


        If anyone else has some suggestions or has come across a similar issue with for..next loops, please let me know. I will post my test results and final random number generator function in case anyone is interested.
        Thanks,
        James

        Comment


          #5
          Re: Speeding Up a Loop Inside a User Defined Function

          Without doing any of the testing myself, it is possible that as you progress into a greater number of records being processed, your system is using virtual memory in
          addition to RAM.
          Have you considered a breakpoint (ex: append the data to a txt file every 75000 records), this would most likely keep you in ram and minimize the writes to disk.
          Gregg
          https://paiza.io is a great site to test and share sql code

          Comment


            #6
            Re: Speeding Up a Loop Inside a User Defined Function

            I think Gregg has a good point.

            Have you told us how much memory is being allocated for each addition to the array? Have you compared the cumulative memory requirements with available RAM? Are other processes running in the background that consume memory resources?

            Comment


              #7
              Re: Speeding Up a Loop Inside a User Defined Function

              Thanks for your input. I did consider the memory but there is no indication that this is causing the problem. When I run, the cumulative memory does not exceed 16% and Alpha is using approx 8.5% of the CPU and 70mb of my 32gb memory. Using msinfo32, I have available physical memory of 26.4gb and virtual of 28.4gb.

              I have an AMD processor with 6 cores (12 virtual). What I find interesting is the process only appears to be using 1 virtual core which is 8.33% of my capacity. Is it possible to force Alpha to use more of the available cores? I have tried using the Alpha function that selects the best processor but it did not make a difference.
              Thanks,
              James

              Comment


                #8
                Re: Speeding Up a Loop Inside a User Defined Function

                I think I have it figured out. As far as I can tell, there is no issue with my code. The speed is what it is. I did do a speed test on the for i loop where I processed data using 4 different methods. Two of the methods displayed severe degradation and one method was the clear winner. I have attached the workspace that has the test function and my random number generator. Feel free to test against your system but the conclusion should not change. Although using a property array, like I did, may make sense at first, recording your data to a table or temporary table and converting it to an array later will speed up your large loops. Below are the test results. Also notice that recording your data to a file does not result in any degradation. Had my test looped 200,000 instead of 100,000, the file method would have come in second place in both speed and relative speed.

                For i Loop Speed on 100,000 Loops (in seconds)
                Increment To Property Array To Character Var To File To Table
                1
                6
                15
                53
                4
                2
                19
                73
                102
                8
                3
                38
                173
                151
                12
                4
                64
                312
                200
                16
                5
                97
                491
                249
                20
                6
                137
                713
                299
                24
                7
                184
                979
                352
                28
                8
                238
                1286
                401
                32
                9
                299
                1636
                450
                36
                10
                365
                2034
                499
                40
                Relative Speed (Indexed)
                Data to Table
                1
                Data to Array
                9.1
                Data to Text File
                12.5
                Data to Character Var
                50.9
                Attached Files
                Thanks,
                James

                Comment

                Working...
                X