Keep-alive warning

[Steve Workings]

We have a lot of these in our Error logs. I can't find a best-practices or any "what to do about this". Any suggestions?

[warning] The server is currently busy waiting for too many Keep-Alive requests. Idle Keep-Alive connections are being closed.

[iRadiate]

Did you change your server Performance settings to something other than default for HTTP keep alive?

Keep Alive.jpg

[kkfin]

Obviously you are using some proxy. in Alpha you could try to disable Http Keep Alive or make it longer(20 or something else)

[Steve Workings]

Thanks. I'm hoping for a bit of an education here. I don't know how to figure out what's best. These are classic server instances. They are under a local load balancer, with Cloudflare out in front. Settings are, I think, the defaults:

KeepAlive.Enable = "true"
KeepAlive.MaxRequests = "10000"
KeepAlive.MaxTimeout = "15"

Does extending (or shortening) the time affect performance? How to choose a different Timeout? What about increasing the MaxRequests?

Sure I could experiment but we *are* having server problems, I see a lot of this in the logs, and would like to approach this intelligently. And, maybe what I'm seeing is OK - I don't know.

[kkfin]

If performance (latency) is bad then...

Cloudflare will keep connection for up to 15 minutes (900 sec) so I think you have two options. Test to totally disable Keep Alive. Your client will get better performance but your classic servers have to work harder but they are ready to serve all the time.

The other option is to put in MaxTimeout a very little value maybe 2-5 (because the CloudFlare timeout is just too big). This means your workers are free to do new jobs again very soon and so the latency should be better for clients.

I think you just have to test but it is normal that server will close keep alive connections with proxy.

[peteconway]

You know what - ask Alpha, in this day and age when Elon Musk has got starlink, space ships, Telsa cars and what ever else going on - why would this problem exit in AA - today. - It blows my mind.

[Lenny Forziati]

When Keep-Alive is in use, clients keep socket connections open to the server to be used for subsequent requests, such as images, style sheets, callbacks, navigating to new pages, and so on. Typically you will see browsers use 4-6 keep-alive connections per server, though there is no strict limit.

On the server side, this dedicates worker threads to those clients. The Classic App Server will run up to 128 worker threads, based on your App Server settings. Assuming 4 connections per client, this means that 32 clients would be enough to tie up all of the worker threads and block out any other clients from making connections. The 33rd client would time out trying to connect. To prevent this situation, the server monitors how many worker threads are dedicated to open but idle keep-alive connections, and it will forcibly close some if there are no other available worker threads.

The log messages you are seeing are an indication that your server is under a high client load and it is taking measures to ensure it can still respond to new clients. This is "normal" operation for the situation, but also means you should add server capacity if it is a regular occurrence because performance is being reduced.

Your situation is slightly muddied by your use of Cloudflare however. With Cloudflare sitting between the end users and your servers, all of those keep-alive connections should be to Cloudflare. What I have been unable to find in Cloudflare's documentation and blog is if they dedicate connections to unique clients, or if they reuse them across clients. If they dedicate them, the explanation above completely applies and you need to consider additional capacity. If they do not dedicate them, and instead will proxy requests from any client on any existing connection, then the situation is still an issue that suggest adding capacity, but it may actually be better to not forcibly close the connections. If you have a support contact at Cloudflare that can definitively answer this, it would be helpful to know. We could potentially add an option to not close the connections in this case.

[mikeallenbrown]

Follow up question Lenny: Worker threads are terminated when the user logs out, right? ...or if a thread is idle longer than the maxTimeout parameter. Does simply closing the browser (not logging out) also terminate worker threads? Is there any reason to not have threads set to the maximum of 128?

Thanks.

[Lenny Forziati]

Worker threads are never terminated because they are relatively expensive to start up. They are not dedicated to a specific client except while they are actively processing a request or they have an idle keep-alive connection open so there is no direct correlation between users and threads.

More worker threads creates more contention for finite server resources - primarily CPU but also system memory. There is a point where too many threads, even if they are not actively processing a request or waiting on a keep-alive connection, will hurt performance. In order to find that point for your system, you would need to do load testing against your application while observing response times.

You can think of the threads as waitstaff at your favorite restaurant. When your server is actively serving you, they cannot do anything for other diners (actively handling a request). When the server is at your table waiting for you to decide what you would like to order, they aren't actually doing anything for you, but they also cannot tend to other diners (keep-alive). When your server is not actively serving you, they can serve other diners (idle threads). When they are done serving that other diner, they are available to serve you again or take on yet another diner that has been recently seated. The restaurant can accommodate more diners by adding waitstaff, but will still be constrained by the size of the kitchen and the number of tables (CPU and RAM). If you keep adding waitstaff anyway, they get in each others way going through doorways, trying to pick up entrees, etc. while overloading the kitchen and not having enough tables to seat diners at.

To complete my analogy and bring it back to the original question by Steve, the Keep-Alive connections being closed would be the same as the restaurant manager going around to the waitstaff that are patiently waiting at tables to take orders, and telling them to move on to other tables instead and come back here once the diners have made their decision.

[mikeallenbrown]

Excellent explanation. Thank you, it's appreciated.

[peteconway]

Nice explanation - good to know - might be an idea to add to the documentation.
Thanks Lenny.

[iRadiate]

Agreed, what a great explanation, even I can understand that analogy

Thanks Lenny

[Pat Bremkamp]

This is another good reason to offload work to stored procedures, especially if the task is a long running query (check the slow query log) or makes a number of server connections. In our case, we have MySQL on a Lynix server and it is not even breating hard. It is multi-threaded and makes fast work of the server load. Another suggestion is to make sure you close connections as soon as possible in your scripts especailly if they are open for a long time. It is easy to open a connection and just leave it open.

[Lenny Forziati]

Worker threads are never terminated because they are relatively expensive to start up. They are not dedicated to a specific client except while they are actively processing a request or they have an idle keep-alive connection open so there is no direct correlation between users and threads.

More worker threads creates more contention for finite server resources - primarily CPU but also system memory. There is a point where too many threads, even if they are not actively processing a request or waiting on a keep-alive connection, will hurt performance. In order to find that point for your system, you would need to do load testing against your application while observing response times.

You can think of the threads as waitstaff at your favorite restaurant. When your server is actively serving you, they cannot do anything for other diners (actively handling a request). When the server is at your table waiting for you to decide what you would like to order, they aren't actually doing anything for you, but they also cannot tend to other diners (keep-alive). When your server is not actively serving you, they can serve other diners (idle threads). When they are done serving that other diner, they are available to serve you again or take on yet another diner that has been recently seated. The restaurant can accommodate more diners by adding waitstaff, but will still be constrained by the size of the kitchen and the number of tables (CPU and RAM). If you keep adding waitstaff anyway, they get in each others way going through doorways, trying to pick up entrees, etc. while overloading the kitchen and not having enough tables to seat diners at.

To complete my analogy and bring it back to the original question by Steve, the Keep-Alive connections being closed would be the same as the restaurant manager going around to the waitstaff that are patiently waiting at tables to take orders, and telling them to move on to other tables instead and come back here once the diners have made their decision.

[kkfin]

In running server instance interactive window you can do interesting things. One is to get info about server. Few interesting functions are SetPrimaryProcessor, PrimaryProcessor, showprocessorassignments, thread_variables and A5_ThreadStackDump. But more info more pain in head.