App Server Computer Requring Restart At Least Once A Week. CPU pegged at 99%

[-Jinx-]

Has anyone else observed this problem? At least once per week, sometimes as many as 3 times, my web application stops responding. When I check the server, Alpha's application server has the CPU pegged at 99% usage in the task manager. Restarting the application server doesn't usually solve the problem. I have to restart the entire server to get things going again. Before the recent updates this happened about once every 3 weeks which still isn't good but wasn't enough for me to get too excited over. Now it's getting pretty crazy.

Does anyone have an idea on where to look for the problem? I can't seem to find anything out of the ordinary in any of the server logs and I've tried to pinpoint parts of my application that might be causing it but it seems random. This is getting to the point where it's causing major issues with my users. Any help would be greatly appreciated.

[mikeallenbrown]

Re: App Server Computer Requring Restart At Least Once A Week. CPU pegged at 99%

You'll need to turn the logs on in your app server and then monitor then especially after the phenomenon happens. Also, many of us use a product called AlwaysUp. I've set my AlwaysUp to restart my app servers every morning around 3am.

[Steve Wood]

Re: App Server Computer Requring Restart At Least Once A Week. CPU pegged at 99%

It almost always comes down to usage. Users running reports back-to-back, users and bots downloading large pages or files from your site. You really do just have to observe and look at the logs. Look for the session_folder and see how many sessions are active. If there are hundreds of active sessions, see if those are active users. It could also be your code, but you said you tried looking at that.

[-Jinx-]

Re: App Server Computer Requring Restart At Least Once A Week. CPU pegged at 99%

Thanks guys,
A little more information for you. I'm running this on a company intranet over the LAN. There should be a total of about 200 users with only a fraction of those being active at any given time. So, hopefully there aren't any bots hitting anything.

It certainly could be my code somewhere, but I'm not sure exactly what to look for and so far it seems a little random as to what is killing the server. I can't seem to nail down an application that is killing it consistently. It would help if the errors could be a bit more descriptive in some cases and the XBasic error log doesn't even time stamp anything so I can't trace it back to the access log or error log to see who/what did it. I do know that I can kill the server at will with some Xbasic if I'm doing some long operations; especially moving files. I should say that it effectively kills it for all the other users when I do that but the server will actually start working again after it's finished executing. I don't think that type of thing is happening in this case. I can count the number of users on one hand that have access to anything that could kill the server (namely an "SQL Wizard") that I made for custom queries.

I haven't made any real changes since the update so it does appear that something done in the update is causing things to be less stable.

I've read about people having to use Always Up. I'm confused though, as to to why we should be forced to use that. If so many people are having similar problems (so much so that they need a program like Always Up), shouldn't Alpha be looking into the stability of their application server instead? I had a similar issue with some DNC software and had to restart the server every day to stop it from happening. Eventually though, the makers of that software fixed there problem and I haven't had to restart it since unless some other unrelated network issue happens.

I'm happy to give any suggestions a try.

[Steve Wood]

Re: App Server Computer Requring Restart At Least Once A Week. CPU pegged at 99%

I think Alpha's long term solution to capacity is to move to the IIS server version; although that is to new to have proved more robust. If you can show proof, I would collect your statistics and proof, and submit it as a bug.

[-Jinx-]

Re: App Server Computer Requring Restart At Least Once A Week. CPU pegged at 99%

Can you guys make sense of this?

Wed May 18 10:43:02 2016
 Thread priority reduced.
Thread was busy 95% of the last 21 seconds.
Raw Data: 952288989517061,350796875,952293592642058,352781250,2001358,1984375,22633847,21593750
The script causing this behavior is using a lot of CPU time.   
You may want to review the page to make sure you have optimized any XBasic scripting.
Long running and processor intensive scripts tend to tie up available threads and adversely affect the perceived performance for multiple users.

Execution Stack:
0#1 evaluate_shared_private(           
0#37 OOhttpdServerClientMIME(ptemp,"AppServerMain")
0#2 A5WCallResult = evaluate_a5w_to_response(page_contents,Response,quote(" " + eval_settings.global.Request.ScriptName),eval_settings,EVALUATE_A5W_TIMEOUT,RequestBundle.RequestInfo.Filename.Path)
0#169 x_out = a5_ajax_dialog2(tmpl)
==========================================================

The time in the Xbasic error log doesn't match with anything in the access log and the server error log doesn't have anything around that time either.

I'm having a hard time finding which app is causing this. Where does it say? What script is it? Am I blind? What is the meaning of life? Is red meat really that bad for you?

[Pat Bremkamp]

Re: App Server Computer Requring Restart At Least Once A Week. CPU pegged at 99%

Sounds like you are using a single instance of the web server. In your case, I'd recommend using multiple server instances. That way, you could have the users who have long running routines use an instance separate from the bulk of the users. we are running 7 instances with port 80 dedicated to out clients and the other 6 for internal use over the LAN. We don't have a load balancer, but get some benefit out of having people assigned to specific ports.

Another idea is to run your file movements at night as a cron job. that way, you have plenty of time for it to run without interfering with regular use.

[-Jinx-]

Re: App Server Computer Requring Restart At Least Once A Week. CPU pegged at 99%

Sounds like you are using a single instance of the web server. In your case, I'd recommend using multiple server instances. That way, you could have the users who have long running routines use an instance separate from the bulk of the users. we are running 7 instances with port 80 dedicated to out clients and the other 6 for internal use over the LAN. We don't have a load balancer, but get some benefit out of having people assigned to specific ports.

Another idea is to run your file movements at night as a cron job. that way, you have plenty of time for it to run without interfering with regular use.

TskMgrPic.png
Is this what you mean by multiple instances?

In any case, if one instance has the servers CPU pegged at 99%, I'm not sure it would help to run more. "Usually", that's the mode of failure, though I can't discount that there are other things going on as well. For instance, sometimes when it's not working it's the application server itself that just won't respond, not the server computer that it's running on.

The file movements that I spoke of are only done by admin users and rarely, so I'd know if that was the cause of my issues.

I might have narrowed it down a little though. It appears that some large repeating sections in a couple different UX's may be responsible for killing everything. It seems that when certain users with less RAM and slower computers use these UX's, it will kill the app server. It's hard to track down though because it seems to be the Javascript running on their computers that gives an error (script running too slow) and maybe when they tell their browser to kill the script it messes with something. I know that I can run these UX's no problem and it's hard to believe that a client computer would be at fault for killing something on the app server but it kind of seems like that's the case.

[-Jinx-]

Re: App Server Computer Requring Restart At Least Once A Week. CPU pegged at 99%

I think I've just confirmed that I can kill the app server at will with just a client computer.

1. Open UX or Open UX that has a large number of repeating section rows and skip step 2
2. Attempt to add a large number of repeating section rows programatically
3. Wait for "slow running script error"
4. Click Continue
5. Click Stop Script

Even after closing the client browser, the app server itself will be unresponsive.

I know that I have to work on the UX's in question and optimize them BUT would you guys agree this is still a bug in that a client computer shouldn't be able to kill the server.

The problem with sending it in as a bug is trying to give Selwyn something that he can reproduce. I'm not sure it's possible.

I have a video reproduction of the bug that I'd rather not post the link to publicly. If you guys would like to see it PM me.

[-Jinx-]

Re: App Server Computer Requring Restart At Least Once A Week. CPU pegged at 99%

Another update, this appears to only kill the server when there is a "lookup and fill in fields" (or something similar I'm sure) action javascript in the row. I'm thinking the combination of the unresponsive script and a user exiting that prematurely when there is a callback being made is the culprit. If I disable that action, I still get the unresponsive error (as expected) but the server never get's touched as everything is client side.

[mikeallenbrown]

Re: App Server Computer Requring Restart At Least Once A Week. CPU pegged at 99%

I've heard from sources I trust that repeating sections are not optimized very well and should be avoided if possible. I try not to use them.

[-Jinx-]

Re: App Server Computer Requring Restart At Least Once A Week. CPU pegged at 99%

I've heard from sources I trust that repeating sections are not optimized very well and should be avoided if possible. I try not to use them.

Ugh, I think you're right. I try to use them because they are "out of the box" instead of having to code something to basically do what a repeating section does. Alpha is saying it's a programming issue on my part and not a bug but I'm not using any custom Xbasic, just what alpha provides via "action javascript". I am using custom javascript client side but only to work client side, with no callbacks and when I get rid of the Xbasic ("lookup and fill in fields" action javascript provided by Alpha), my javascript works OK, albeit slowly. Actually, I take that back, MY javascript runs fast, it's the provided methods that are slow. Here is the JavaScript I'm running to take data from a list and input it into the repeating section:

var lstsel = {dialog.Object}.getValue('PERLIST');
if (lstsel != '') {
		lstsel.shift();
		lstsel = lstsel.toString();
		lstsel = lstsel.split(',');
		var PrtNo = []
		var DwgNo = []
		var lcnt=0
		for (i=0;i<lstsel.length/2;i++)
		{
		PrtNo[i] = lstsel[lcnt]
		lcnt = lcnt + 1
		DwgNo[i] = lstsel[lcnt]
		lcnt = lcnt + 1
		}

	var RsRwCnt = {dialog.Object}._getRepeatingSectionRowCount('CONTAINER_1');

	for (i=0;i<PrtNo.length;i++){
		{dialog.Object}.addRepeatingSectionRow('CONTAINER_1');
		{dialog.object}.setValue('PARTNO:',PrtNo[i]);
		{dialog.object}.setValue('DWGNO:',DwgNo[i]);
		RsRwCnt = RsRwCnt +1
	}
}

var lObj = {dialog.object}.getControl('PERLIST');
lObj._deselectAll();

I might try to experiment when I have some time and use the populate method or something to see if I can build everything with a loop and then inject it with one command.

Anyway, Alpha is misbehaving in my opinion and it's gotten worse with the recent updates with no changes on my part. I really think it's a bug.

[frankbicknell]

Re: App Server Computer Requring Restart At Least Once A Week. CPU pegged at 99%

I see where the script error is coming from but what does not make sense is why the server is going wacky.
That is a lot of work going on with the loops and string manipulation.
Wouldn't it be better to export that list to json data then import it into the repeating section.
I don't know how to do it but it seems like that should be possible.
I am going to be doing something like that in the future. Believe it or not I am still using lookup grids. Some day I will get with the program. I hope we can figure out a better way soon.
I agree something is a little weak with the latest updates. I still can't get through an update remotely. I have never had any server issues before. As a matter of fact I could leave my servers up for months without any problems so I agree something is up.

[-Jinx-]

Re: App Server Computer Requring Restart At Least Once A Week. CPU pegged at 99%

I see where the script error is coming from but what does not make sense is why the server is going wacky.
That is a lot of work going on with the loops and string manipulation.
Wouldn't it be better to export that list to json data then import it into the repeating section.
I don't know how to do it but it seems like that should be possible.
I am going to be doing something like that in the future. Believe it or not I am still using lookup grids. Some day I will get with the program. I hope we can figure out a better way soon.
I agree something is a little weak with the latest updates. I still can't get through an update remotely. I have never had any server issues before. As a matter of fact I could leave my servers up for months without any problems so I agree something is up.

Hi Frank,

Actually, I'm doing what you suggested (JSON and the populate method) on another UX that also kills the server. The common denominator is that both UX's have some sort of genie-defined event in the repeating section rows that is what is actually causing the server to go wacky, IMHO. But who knows. This is rather maddening since I'm extremely busy with my other duties at the moment and can't dedicate as much time as I'd like to solving this, or at least solving whatever I can that's not inside of a black box.

[frankbicknell]

Re: App Server Computer Requring Restart At Least Once A Week. CPU pegged at 99%

I have a UX that has 4 repeating sections. Some of those sections can have as many as 30 rows. I add the rows one at a time. I am not experiencing any problems doing it that way.