Server troubles - continuing

[Steve Workings]

We've suffered months (years?) of server problems. Of our 8 instances on two servers, we are nearly every day having to restart one or two instances just to clear whatever's going on.

The symptom is that users usually get a login screen that's not complete, so they can't log in (the load balancers keep putting them on the same server and instance) and complain to us. It seems clear to me the javascript isn't completing when that happens.

A typical stack log entry looks like this:

==========================================================
Mon Sep 20 11:18:50 2021
Exception has been thrown by the target of an invocation.
Object reference not set to an instance of an object.
===== Stack Trace Follows =====
System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.NullReferenceException: Object reference not set to an instance of an object.
at A5XbasicAppServer.AppServerGzipCache.<>c__DisplayClass9_0.<GetGzippedFileName>b__0(AppServerGzipCacheEntry x)
at System.Collections.Generic.List`1.Find(Predicate`1 match)
at A5XbasicAppServer.AppServerGzipCache.GetGzippedFileName(String requestedFileName)
at A5XbasicAppServer.A5ResponseIntegrated.SendFile(String Filename, Boolean PromptToSave, String OutputFilename)
at A5XbasicAppServer.A5ResponseWithLegacy.SendFile(String Filename, Boolean PromptToSave, String OutputFilename)
--- End of inner exception stack trace ---
at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
at Syste
Execution Stack:
0#1 evaluate_shared_private(
0#39 OOhttpdServerClientMIME(ptemp,"AppServerMain5ce4b90327b34494a84ff7b6fa2fdaeb")
==========================================================

In some other detail logging with an error log, this is often preceded by a whole bunch of these before the errors begin:

[Thu Sep 16 20:41:06 2021] [error] [Thread ID: a750330be4dc454bb5c1cfaf0c6addb0 Request Sequence: 2674] (Internal Server Error: ) Error processing a5_application.a5i. Not found
[Thu Sep 16 20:41:06 2021] [error] [Thread ID: a750330be4dc454bb5c1cfaf0c6addb0 Request Sequence: 2674] (Internal Server Error: ) Error processing a5_application.a5i. Not found
[Thu Sep 16 20:41:06 2021] [error] [Thread ID: a750330be4dc454bb5c1cfaf0c6addb0 Request Sequence: 2674] (Internal Server Error: ) Error processing a5_application.a5i. Not found
[Thu Sep 16 20:41:06 2021] [error] [Thread ID: a750330be4dc454bb5c1cfaf0c6addb0 Request Sequence: 2674] (Internal Server Error: ) Error processing a5_application.a5i. Not found

We're currently on 7715-5579, and really hesitant to go through the upgrade process anytime soon. Our last effort to upgrade just landed us stuff like this despite our best testing efforts.

Anyone else having similar problems or have any suggestions for fix?

[Pizzadude]

I am also on 7715-5579, and have had a lot of issues with this release.

I have to re-start the WAS at least once a day. It does occasionally cause Alpha to lock up in a user's browser, but not often. Mostly it starts running slow, to the point of causing timing issues. Re-starting the WAS always fixes it, but boots off all the users.

Re-starting due to slowness have been a problem for me for years. I never have more than 5 users logged on at one time.

I have seen a lot of posts about fixes that are available in the nightly releases, but not in a "stable" release. I wonder when the next stable version will be available.

I tried build 7776 but had to roll back immediately.

[mikeallenbrown]

What's the app server memory usage look like when things go sideways? In my experience once the app server exceeds 1GB (or around that point) it stops responding. I personally haven't had this experience in a very long time however. Like you, I hesitate in my upgrading of versions. More times than not (and despite my testing) something breaks. Recently it's been repeating sections...nightmare.

Have you looked over the HTTP/Requested page logs to see if a particular component might be causing a problem?

Just thought I'd throw these out there.

[Pizzadude]

My memory usage can be under 300k and it will run slow. It's been a while since I tried troubleshooting it. There was a build quite a while ago where the problem did not occur. I thought it was fixed, but then in the next build it was back.

[Steve Workings]

Memory usage isn't usually an indicator. CPU usage is sometimes an indicator, but not reliable. I have to get the use to login (there is a way to manage that) then use a page with A5_Info() to tell me the server and instance.

I've tried matching the access logs but so far can't nail anything down. And, with the user first noticing it on the login component, there's even less obvious cause & effect.

[Pat Bremkamp]

We used to be able to run multiple instances of the standard server. I found with AlwaysUp I and about 11 users, I could run 8 instances and restart during the night, and that eliminated the problem. The WAS continuously grew, but with multiple instances assighed to specific people, usually not enough to get to the severe slowdown state. That all went away when Alpha started the You can only run 4 cores policing. Fortunately, business has been bad and staff has been cut so it's manageable for now.

[peteconway]

re 'Fortunately, business has been bad and staff has been cut so it's manageable for now.' - What can I say.
Pete

[kkfin]

We used to be able to run multiple instances of the standard server. I found with AlwaysUp I and about 11 users, I could run 8 instances and restart during the night, and that eliminated the problem. The WAS continuously grew, but with multiple instances assighed to specific people, usually not enough to get to the severe slowdown state. That all went away when Alpha started the You can only run 4 cores policing. Fortunately, business has been bad and staff has been cut so it's manageable for now.

I wonder if runtime + chrome control + tsplus would be an option in your application. Now you have maybe time to do the switch. User is still using your application like web application but you get benefits of desktop application.

[Tom Cone Jr]

kkfin, could you possibly elaborate on your idea of using RT + chrome control + tsplus a bit? Perhaps in a new thread in a desktop forum? Thanks. -- tom

[Steve Workings]

So, is anyone happy with any server build beyond 7715-5579?

And I'm still here without knowing what I might do to address the problem I show. I sent a bug report a while ago but - unusually - no response.

We're not in a position for daily restarts - our app is in constant use from users around the world.

[peteconway]

I'm in 7715, clients are still running on 7086_5534 Sep 29,2020, I won't let any of them make the jump and they have to wait for updates.
It's not uncommon for to hear nothing just before a the devon where they like to announce a bunch of new stuff, I'm a little worried about it, I had a lot of weird stuff happening while developing, so I copied an entire UX set of controls into a new UX it went from 1.9mb to 2.5mb. whis is what was inside it. - All new Code not in the original.
2021-09-22_10-35-26-New stuff.png2021-09-22_10-37-45.png

[Steve Workings]

Just had to restart two instances. Makes for unhappy users (and co-workers)

[Pizzadude]

I just had to re-start too. Seems like since this thread started, my server problems have been worse.

[Ted Giles]

Steve W.
Are you hosting the app internally, or tied to a different Hosting option?

[kkfin]

kkfin, could you possibly elaborate on your idea of using RT + chrome control + tsplus a bit? Perhaps in a new thread in a desktop forum? Thanks. -- tom

Here is some old example but but you get the idea. You are free to use sql also.

Application(runtime + chrome control) running in desktop here
Same application( runtime + chrome + tsplus) running in web here

Kenneth