Network Operation Performance

[Fletch]

Re: Network Operation Performance

Oh if you are concerned that your virus detection software may be a problem, then switch to NOD32. It has a smaller memory footprint and without a doubt is the best virus detection product on the market.

NOD32 is fast becoming the choice for web servers.

[csda1]

Re: Network Operation Performance

Barry,

If Alpha runs with only 5 users (given a fast server etc), (and assuming no bad network hardware) I absolutely guarantee it's your application code and methods.

In most people's typical A5 application (using Genie's and action scripting), it typically produces quick, but very often, inefficient code. This is almost always related to doing unnecessary queries, queries without LQO, and duplicate queries. Reports written incorrectly can lock the tables for unnecessary length of times.

As an example, 1 client placed a Find prompt on opening a form. Despite the fact that all code had to do was open the form, ask for a key value to find and then do a find of that key value in a preexisting index, the genie created code that was doing a full query of the database every single time the form was open. No one ever said that action scripting is efficient, and while in many cases it does not matter and get's you up and running fast, it can generate a lot of code that eat's up a lot of network bandwidth.

In other cases, people use the wrong techniques. Changing techniques, you can get amazing increases in speeds. In 2 totally unrelated problems I was having, I highly optimized the original code. Then, by changing how I did it, I decreased the execution time to 1/200th of it's original value. Obviously, mileage may vary, but it illustrates how important technique is.

Another example took a form 30 seconds to do it's calculated fields. The equations stayed exactly the same, but how they were run made it run in 0.25 seconds, a 120% improvement in speed.

The reason this happens with Alpha 5, is it is the only database with advanced abilities that a non-programmer can create an application in. Most other databases with advanced abilities require a programmer to get any practical application in, and programmers are more knowledgeable in their methods of coding.

If you take the same type of database (filemaker pro, Access and the like that are not True Client-Server products), you will find that optimized code in both run pretty much at the same speed (basically the limits of the server, network). However, the development time is radically different, with A5 holding a big lead for most applications!

Same thing hold's true for client server operations (including those accessing via SQL like Alpha 5)

[Fletch]

Re: Network Operation Performance

Ira

I cannot deny what you are saying as your expertise is far greater than mine. I'm learning AV and working with a AV deveoper. My comments are impressions. I am finding new ways to make AV run better. I thought that perhaps there were some network "tweaks" or startup routines that I might have missed.

[jkletrovets]

Re: Network Operation Performance

Greetings all,

I am in the process of developing my first "full blown" Alpha app to replace a Foxpro 2.6 app I have had running for 10 years. I have read a lot of the information regarding network optimization and have followed a lot of them. After reading this thread though, I find myself with a couple of questions.

Attached is a screen shot of my main screen. The set behind this screen has one parent with 16 children and 4 grandchildren. All of the links to the child tables are simple character fields. I have populated 6 of the tables with records comparable to what the app will see in real life (about 60,000 to 80,000) to test performance. I must say the performance (both local and on my test network -shadowed) is quite acceptable.

In real life this app will have 5-8 regular users and maybe 15 more "part time" users.

To those of you more experienced I was wondering if:

1. Will I experience significant slow down in the real world scenario of the user situation I mentioned from what I am seeing now?

2. Am I looking for trouble having a set with this many tables? From a long term stability point of view - should I break down this big set into smaller sets and do away with the tabs?

3. Dr. Wayne - based on your copious knowledge of Alpha - I am curious why you say the "Tabs are more trouble than they are worth"? In testing here they have been working great and are convenient but I am sure you know something I don't. All screens are modal and I am controlling the user's data entry(on both regular fields and memo fields) per some of your other suggestions for memo fields that work.

4. Martin - I can definitely confirm what you say about using parentform.refresh_layout(). On one of my tabs I was using this and I noticed a HUGE degradation in performance even on my local machine. I changed the set and did away with that command and the speed came right back.

Regards,

Jeff

[csda1]

Re: Network Operation Performance

Jeff,

The more tables linked in a set, the more network traffic is generated every time a read or write of an entry is made. If most of the children are not being used, it is normally better to separate things into smaller sets using just the tables required for the need.

If all user's sit on a record for a long time, there is not much overhead. However, if users and running queries (implied as in a report or explicit) and reading or writing records, that adds a lot. So you could have 200 people sitting on records being OK, or 5 users running complex sets reports constantly that could potentially bring the system to it's knees.

Anything done on the local client machine has no impact on the shared network tables, so any local temporary files have no system-wide speed impact.

[jkletrovets]

Re: Network Operation Performance

Thanks Ira

Just to help me clarify(in the shadowed environment) - if a user makes a change to a field in the parent(for example adds a date to a date field). Is the entire set's data then set back and forth across the network (all 16 child tables) to refresh that one date field?

My app is not a heavy pounding type of app where there are hundreds of parent records getting added every day. But, users can be fickle about everything as I am sure you know. Foxpro was quite fast and inherently has less overhead than Alpha (but it has a ton less capability) so I am thinking I should probably think about speed a little more.

Hmmm...Sounds like maybe I need to do a little redesign.

On the reporting side of things - has it been your experience it is better to copy the pertinent records to the work station in a temp table and then run your reports? I know there is probably not a clear cut answer. Just curious if you had a general guideline you follow?

Regards,

Jeff

[csda1]

Re: Network Operation Performance

Jeff,

when you make a change, I'm not sure if the data is sent for all, or just the changed tables of a set, but it really doesn't matter. It's not the data that is the overhead (well, not usually), it's getting access and releasing access to the records (file and record locking) that is what is significant, and those happen for every record including each child of every table in the set. with a 1 to many being say 50 children, that could be 60+ locks.

A record copy (with an LQO query/index) is very fast to the local drive, and only causes 1 access to the network per record. Then the local machine can play with it to it's heart's content without impacting anybody else on the network. It is not very easy to do cleanly, but it has less network impact and runs as fast for most small reports, and sometimes much faster for larger reports (and no impact on other users)

[John Gamble]

Re: Network Operation Performance

Thanks Ira

Foxpro was quite fast and inherently has less overhead than Alpha (but it has a ton less capability) so I am thinking I should probably think about speed a little more.

Jeff

Just a comment on Foxpro. Our accounting system is also Foxpro and is blazingly fast.

Until Alpha becomes client server (as Foxpro) I don't believe we can expect similar speeds. I must admit to being very disappointed as client server was promised quite some time ago.

Chief benifits I'm looking forward to with client server:

1. Speed
2. Convenience (program changes to server only). Forget shadowing.
3. Volume - Allow more users at less cost over a given network.
4. Overhead - Significantly less.

Therefore reduced cost.

John


John

[Bob Whitaker]

Re: Network Operation Performance

Good morning all.
Ira in your post a little earlier you mentioned how you had achieved some significant performance increases via your coding methods and I wondered if you could, without taking too much of your time and giving away too many of your secrets, give us some examples of how this was achieved.
Thanks in anticipation.

Bob Whitaker
U.K.

[martinwcole]

Re: Network Operation Performance

ditto

[csda1]

Re: Network Operation Performance

Hi Bob & Martin,

It's a little difficult to really give a handle on how to speed up a generic problem. But I find it interesting that I find speedups of 1/100th or 1/200th of the time rather often.

The 1st thing is to use the correct algorithm. This can be an iterative process based upon what works fast and what doesn't in A5.

Look for the key functions that operate fast (You can use the CSDA Code Utility product to accurately measure the speed of a function). E.g. line_count() is slow, occurs() is much faster, and w_count() is a bit faster than occurs(). So use the w_count to count your lines.

Never pay for the same ground twice (the exception being that each assignment line has a high time penalty, so it is often better to duplicate a calculation, than to pull out a common sub-expression. However, this could change)

Move as much code out of the loops. Use lists, collections, arrays or database initialize as appropriate for the task.

Use high level functions that are efficient in A5, e.g. Crosstab operation rather than XBasic. Don't use functions that are XBasic wrappers for lower level functions. Don't use Action Scripting. Always make sure LQO occurs for queries. Don't call simplistic functions. Instead, embed the same code in a higher level function to save the function calling overhead. Minimize network traffic by doing temporary operations on the local drive (which also run faster on the local drive).

Finally, speeding up code or providing a specialized function for others is one of my key consulting services. Some other consultants on the message board hire me when they get stuck. The case of speeding up the calculations on a form that I spoke of, took a full day. Was it worth it? If it's your main form, I think so.

I'm developing an XDialog front end (an enhancement to the CSDA Code Utility) for selecting scripts (with sort columns by dates/name/type/size/lines/description/created by etc) originally was taking 30 seconds for really big scripts (250k bytes in size) to process, now it takes about less than 0.3 seconds, and much faster for smaller scripts. Average processing time is 30+ scripts/second (Finian that means 100 seconds to bring up the XDialog the 1st time for you!). It went from impractical to very practical. Probably 2 man-weeks in just this XDialog.

Speed costs to develop, and you don't always have the extreme improvements, but in key routines it is definitely worth it!

[martinwcole]

Re: Network Operation Performance

I sincerely hope you get a good ROI!!

[jkletrovets]

Re: Network Operation Performance

Ira

Thanks for the excellent explanation to Bob and Martin's question.

From Ira - It's not the data that is the overhead (well, not usually), it's getting access and releasing access to the records (file and record locking) that is what is significant, and those happen for every record including each child of every table in the set. with a 1 to many being say 50 children, that could be 60+ locks.

This raises an interesting question. I assumed that the locks are not made on modal screens unless the user puts the record into "Edit" mode. Is this erroneous thinking on my part? I take from what you said above the locks are attempted even if the form is just opened? For example, if I have a set that has a one-to-many relationship with 50 children. Are all 50 children locked when a form with an embedded browse showing those 50 children is opened? I have not tested this but this would seem to prove troublesome in a network environment. Perhaps I misunderstood?

Thanks again...

Jeff

[csda1]

Re: Network Operation Performance

This raises an interesting question. I assumed that the locks are not made on modal screens unless the user puts the record into "Edit" mode. Is this erroneous thinking on my part? I take from what you said above the locks are attempted even if the form is just opened? For example, if I have a set that has a one-to-many relationship with 50 children. Are all 50 children locked when a form with an embedded browse showing those 50 children is opened? I have not tested this but this would seem to prove troublesome in a network environment. Perhaps I misunderstood?

On a non-Client Server model, any access of file data requires a locking procedure to get access to the record. These locks happen during a read and a write, although the write locks take a bit longer. These must be atomic operations (meaning they must be completed without interruption from other accesses) to get record access, but don't affect other user's reading the record once the lock is obtained. These locks are sequential for all shared users of a table, and thus is a limiting factor. This is the reason that non Client-Server models don't work well over a Wide Area Network. If a lock takes 1 millisecond on a LAN, it could take 2 seconds over the Internet, holding everyone up during the atomic lock operation. Multiply that by every record access and it slows a network to a crawl. Citrix servers and PCAnywhere/VNC solve this problem by allowing you to access the network from the LAN, and you just see a copy of the screen. All actual program execution is on a computer on the LAN.

So the moment you open a table or set, whether a form, report, browse, whatever, the parent and all children must go through the locking process as you access each parent. Referential integrity puts additional demands on the children as well.

The atomic operation of the locking process applies only to those users sharing the a table (which could be via a set), and thus the lock process is sequential for those users (although very fast). Other tables in use have a separate sequential lock process.

The only speed issue is that if the total data and locking processes for all user's overwhelms the network, then you will experience slowdowns systemwide, as opposed to those just sharing a particular table.

So in general, simpler sets designed for the needed purpose put's a much smaller load on the network and for sequential locks to particular tables. Shadowing (Network Optimization) moves a copy of all layouts to the local drive, as they do not need to be actively shared. This keeps them off the network, saving the network for shared data and locks.

Client-server essentially moves all lock processes to the server, making the speed of the server a limiting factor. Internally, the server side is essentially performing the equivalent of locks to access and change the data, and this moves the lock process off the network to the server, meaning the lock process (which is not really a lock, but sort of works as if there was) is extremely fast since it's all done in the local memory of the server. The downside is that the server is now a limiting factor to speed, not the network.

Incidentally, Queries that have any part of the filter that is not LQO, require table access (and the associated locks) for each of those records. LQO essentially sets a start and end range of which records to access, eliminating many extra accesses.

[jkletrovets]

Re: Network Operation Performance

Ira,

Thank you VERY much for the explanation Alpha's locking methodologies!

I will definitely be changing my app to smaller sets.

I am familiar with client-server and was hoping Version 8 would handle this natively with form development, reports etc. But it looks like that will have to wait until the next version.

For what it is worth, Foxpro 2.6 works very similar to Alpha - but it had none of the great features like sets, data dictionary etc. Although, I always thought it only tried to lock records while they were being edited. Maybe I was wrong about that after reading your post since they are the same underlying data engine.

But, you could go against the tables with SQL statements and return the data you want for queries blazingly fast. One nice thing Foxpro did do was that you could run a query with SQL statements it created a cursor with the result. The cursor was basically a temporary table built from your select statement but contained in memory. You could browse this cursor, run reports on it etc. I miss that.

It seems Alpha's queries are really just filters? I have paid great attention to making use of the LQO capabilities of Alpha. Foxpro's word for LQO was Rushmore Technology.

It would be nice to see a compiler from Alpha. Do you know if there has ever been any talks about doing so? Foxpro had it and it made updates a breeze.

All in all though, I love Alpha. It's a great program.

Thanks again for your explanations...

Jeff