long-running cURL, poll with sleep()?

[nmarler]

I have a long-running cURL call (15-90 sec) that is running in an ajax callback on WAS instances, and I need to make the interaction as efficient/scalable as possible. It's my understanding that during the cURL call the entire WAS instance will be blocked and unable to respond to other requests.

Alternatively, there's an Async HTTP API whereby I would make an initial cURL call that would kickoff the process and return a token, and then I would poll with the token until the process is complete. The benefit of this approach would be that each cURL interaction would be very quick.

- How can I efficiently handle the polling?
- I'm thinking a polling loop with calls to SLEEP()
- During SLEEP() is the WAS instance available to handle other requests?

Would I need to worry about this if running on IIS? (Iow, is this a WAS-only problem?)

[mikeallenbrown]

Re: long-running cURL, poll with sleep()?

While executing a xbasic script (any xbasic script, cURL or otherwise) that WAS instance is busy and cannot fulfill any other requests. Using sleep() in your script will only make your wait longer. Like the documentation says, it pauses the script but you're still in it. I'm not familiar enough with IIS and Alpha but my understanding is the limitation is the same. It sounds like your alternative is the way to go.

[nmarler]

Re: long-running cURL, poll with sleep()?

Thanks for your insight into the sleep function. Sorry, but why do you think the Async HTTP alternative is the way to go? It seems that with either approach the WAS instance will be blocked until the remote work is complete and I might as well stick with the [simpler] long-running cURL call

[mikeallenbrown]

Re: long-running cURL, poll with sleep()?

You said, "The benefit of this approach would be that each cURL interaction would be very quick." ... maybe I misunderstood. It sounds like this alternative method would be a better option. If you can break up this script/method into smaller chunks (into multiple scripts) then that would free the WAS, in between these calls, to fulfill other requests. You shouldn't lock up the WAS for 90 seconds ...or even 15 seconds. Any other user who is on that instance will see this as an app problem, or click something multiple times, reload the page, etc.

So, I guess the question is: Can you break up this method into smaller chunks? I too have long running processes. Some of which could take minutes or even up to an hour. I use Alpha's runtime for these really long processes. My user's can them be notified by email when it's complete or just return later to see it's status. To use Alpha runtime requires a separate license.

[kkfin]

Re: long-running cURL, poll with sleep()?

Truth is little different like in post #2 explained. Each client consumes threads and there is a limit how many threads WAS can handle. Alpha will create new one until the maximum amount of threads are in work. So WAS can handle many requests at same time. But to create a new thread is also basically a slow process. The client has to wait until one old thread is free or there is a new one available. Now WAS is totally slow. This behavior is easy to test. (How thread safe Alpha is in practice is an other question. From roadmap we get feeling that xbasic is not a thread safe).

Situation is exactly same with Alpha IIS.

This thread behavior is the main reason we now have Asp.Net Core.

Ken

[nmarler]

Re: long-running cURL, poll with sleep()?

I see now how you were thinking. In my case, when user clicks a button they expect a response and are willing to wait, so unfortunately an asynchronous email notification or return visit is not an option.

Turns out I had a conversation with someone at Alpha today that I think wraps up this question. Great news: the long-running cURL call does not block the WAS instance, so there's no need to worry about polling

- the Xbasic engine is the shared resource to worry about, it's locked when a script starts running
- before the script ends, however, the Xbasic lock can be released when certain functions are encountered. Specifically, database access, file I/O, cURL, sleep(), etc.
- when the Xbasic lock is released other requests can be processed
- when the database/cURL/sleep/etc operation completes, the thread gets back in line to have the Xbasic engine finish its script

This all happens the same on WAS and IIS

See 4th paragraph:
https://www.alphasoftware.com/blog/i...ver-utlization

[Davidk]

Re: long-running cURL, poll with sleep()?

Nate, this is very useful information... thanks for sharing.

[mikeallenbrown]

Re: long-running cURL, poll with sleep()?

Nate, this is very useful information... thanks for sharing.

Yes, it is. I think I'll do some testing later for piece of mind. Sleep() will be an easy one. I've always been under the impression when a xbasic script is running no other requests are handled until the script is finished.

[nmarler]

Re: long-running cURL, poll with sleep()?

I've always been under the impression when a xbasic script is running no other requests are handled until the script is finished.

Me too, and what caused me to start this thread. I specifically asked about sleep since we were discussing it earlier. Interested to hear what your testing shows

[mikeallenbrown]

Re: long-running cURL, poll with sleep()?

My simple test using sleep() seems to confirm what Alpha is saying. I've attached my component I made for testing this. In the component are notes.

Always a good idea for someone else to confirm the testing method is sound. I'm running 5667-5254 by the way.

sleep_Test.a5wcmp