A5_DownLoadFile() revisited

[CALocklin]

Re: A5_DownLoadFile() revisited

It helps to read the Help file. In this case, I won't apologize for not looking at it sooner because apparently nobody else did either. I was looking at this command for a different reason than this thread and got very frustrated with the command. However, after reading the Help file I believe I have PART of the solution. The rest will be submitted as a bug report.

The Help file points out that the HTTP_Download() command has a "modal" argument. In all the examples here, it was set to .F. and nobody thought about it. This needs to be set to True (.T.) if you want to use it in a loop. Without that, the script will continue before the file is downloaded. However, the A5_DownLoadFile() command (which I suspect is just a wrapper for the HTTP_Download() function) is "modal" by default. In fact, there is no way to turn off "modal" in the A5_DownLoadFile() function.

So, there was a "timing" issue of our own making when using the HTTP_Download() function but there is not a timing issue when using the A5_DownLoadFile() function.

The original question about the "not an open table" issue seems to be a bug. If I test this using a crlf() string to hold the names of the files to be downloaded, it works like a champ. Whenever I use a table, the table gets closed by whichever function is used - the HTTP_Download() function or the A5_DownLoadFile() function.

FWIW, I've decided that these two functions, Http_download() and A5_DownLoadFile(), are now in my "scary to use" category.

My "scary to use" category is defined as any function that works in some cases but, for no apparent reason, does not work in others.

After quite a bit of testing I've discovered that these commands will download almost any kind of file from some websites but don't seem to work at all on other websites. The real problem is that I don't know why they don't work on certain websites. If I did, then I could probably take them out of the "scary to use" category.

The test app I'm sending with my bug report is attached in case anyone else wants to try it. (The files to be downloaded by my test routines are on my website and will be there for at least a month. No guarantees after that. I purposely picked small files so they won't take long to download even on a phone modem.)

I also included my "scary to use" comments in the bug report so hopefully we can get some clarification on that as well.

[G Gabriel]

Re: A5_DownLoadFile() revisited

Cal:
I didn't open your zip file but I think you are getting ahead of yourself:
1-

In all the examples here, it was set to .F. and nobody thought about it. This needs to be set to True (.T.) if you want to use it in a loop.

Graham already did that:

without the extra parameters ie ",.f.,options" it just loops around launching the first retrieved file until I select close, with the added options it does the first six, then throws up the "NOT OPEN TABLE" error, even though the 7th file is available on the source html page.

2-You make an assumption that:

the A5_DownLoadFile() command (which I suspect is just a wrapper for the HTTP_Download() function) is "modal" by default

The help file says:
A5_DownLoadFile() function downloads a file
If it were a wrapper, one would suspect it would have had arguments for modal/modeless.
More likely it is either:
a-Designed without a built-in retry mechanism or
b-It is not meant to be used in a loop.

The real problem is that I don't know why they don't work on certain websites. If I did, then I could probably take them out of the "scary to use" category.

There was a thread a while back, I believe by Garaham as well, and Lenny Forzati addressed that issue.

[CALocklin]

Re: A5_DownLoadFile() revisited

Cal:
1-
Graham already did that:

Oops! Sorry, I did see that and then forgot it after doing all my other tests.

2-You make an assumption that:
the A5_DownLoadFile() command (which I suspect is just a wrapper for the HTTP_Download() function) is "modal" by default
The help file says:
A5_DownLoadFile() function downloads a file
If it were a wrapper, one would suspect it would have had arguments for modal/modeless.
More likely it is either:
It does not have a built-in retry mechanism or it is not meant to be used in a loop.

I'm not sure where to start here. The fact that it "downloads a file" has nothing to do with either "the A5_DownLoadFile() command is "modal" by default" or "I suspect [the A5_Downloadfile() command] is just a wrapper for the HTTP_Download() function". These are all separate facts and not interrelated at all.

I never mentioned the issue of a "retry" because there was no reason to. The Help file does say that the HTTP_Download() function "does not have a built-in mechanism to retry when transmission errors occur." However, we are not dealing with a transmission error; the original problem was that the table got closed after the first file was retrieved and it was the closed table that crashed the script - not the download. The original routine completed the first download successfully thus no need to retry. Yes, there were issues when people started using the HTTP command without the 'modal' option set to .T. but that's a separate issue that has been resolved.

Where do you get the idea that possibly "it is not meant to be used in a loop"? This is like saying, "The lookupc() function only returns one value therefore it is not meant to be used in a loop." The same logic could be applied to anything - "The File.exists() function only returns a value for one file at a time and, therefore, it is not intended to be used in a loop." or "The <tbl>.delete() function only deletes one record at a time therefore it is not intended to be used in a loop." All that "it downloads a file" means is that the function will only download one file each time that function runs. There are other functions that can work with multiple files as input such as FTP_put_file_multiple() so the fact that this one can only download a file just confirms that you cannot give the function a crlf() list of files to download. Sorry, no relevance here.

Besides, the problem isn't really a looping issue at all. We just got started down the wrong rut and had a hard time getting out of it. The following script should prove that it's not a loop issue. Put the script in any application you have and use any table name you have in the Table.open() command because it has no relevance to the download. Even so, the tp.fetch_next() command will cause an error because the download function somehow closes the table. Sounds a lot like a bug to me.

tp = table.open("<Any_table_you_want")
tp.fetch_first()   [COLOR=blue]'This works fine.[/COLOR]
http_download("www.aimsdc.net/airports_per_mile.xls","C:\airports_per_mile.xls",.t.)
tp.fetch_next()   [COLOR=blue]'This crashes with a "table not open" error. Why?[/COLOR]
tp.close()  
ui_msg_box( "DONE", "" )

Re: If it were a wrapper, one would suspect it would have had arguments for modal/modeless.

Why? If the wrapper function included all the arguments that are in the function being wrapped then there would be no need for the wrapper function. Wrapper functions, at least those created by Alpha, are used to simplify/standardize something so people who only need the basic features don't have to worry about all the other options.

There was a thread a while back, I believe by Garaham as well, and Lenny Forzati addressed that issue.

Could be. The trick is finding these threads. Things like this need to be added to the Help file so people can, theoretically, find out about them when first learning to use the command. (OK, people probably won't read the Help file before trying to use the command but that should be the first place they go when they run into a problem.)

I didn't find any relevant threads on the v8 forum but I did find this one on the the v7 forum. However, I'm pretty sure the site I'm having an issue with doesn't have a robot filter - I'll check on it. Unfortunately, the person who owns the site doesn't want me posting the info here.

[G Gabriel]

Re: A5_DownLoadFile() revisited

This thread started out with a script that has A5_DownLoadFile() to download a file go to next record, download another and so on and then closes the table.
It worked fine for the first record then threw an error that the table is not open.
Either:
a-Like you said, A5_DownLoadFile() has a bug that closes the table. Haven't tested that so I can't comment, or more logically
b-The script ran down the loop and closed the table all while the first file being downloaded.

A5_DownLoadFile() is a logical flag and such flag was never used. Perhaps, the script should have been modified to use this flag.

[G Gabriel]

Re: A5_DownLoadFile() revisited

Maybe the script should be modified to:

source_html = "http://www.combatindex.com/insignia/airforce/graphics/"
target_dir = "d:\my databases\bitmaps\testData" + chr(92)
tbl = table.open("usaf_sqn_data")
tbl.fetch_first()
WHILE .not. tbl.fetch_eof()
gif_name = alltrim(tbl.graphic_name)
x=a5_DownLoadFile(source_html+gif_name,target_dir+gif_name)
while x=.f.
sleep(.5)
end while
tbl.fetch_next()
END WHILE
tbl.close()

Which is not much different from the flag I suggested.

Basically, this whole issue revolves around one question:
Does a5_DownLoadFile() has a bug and it closes the table?
or
Does the script itself closes the table?
I don't know the answer.
An easy way to find out:
remove that last line of the script that closes the table and see what happens.

[G Gabriel]

Re: A5_DownLoadFile() revisited

Cal:
Let's know the response to the bug report. I doubt VERY seriously that this is a bug, it would be weird even by bug standards.

[CALocklin]

Re: A5_DownLoadFile() revisited

Let us know the response to the bug report. I doubt VERY seriously that this is a bug, it would be weird even by bug standards.

It is weird but I'm betting it's a bug. I'll let you know when I hear from Alpha. If it's not a bug, I'll be really upset because this means that all open tables must be closed before running either of these download functions - regardless of whether or not the open table has anything to do with the download itself.

x=a5_DownLoadFile(source_html+gif_name,target_dir+gif_name)
while x=.f.
   sleep(.5)
end while

If you think about this a bit, it won't work. Once the script gets to the WHILE statement, the variable "x" has already been set and won't change unless the script continues to run. So, if "x" is false when the WHILE loop starts, the WHILE loop will be endless because there is nothing inside the loop to reset "x". There is no magic that allows a previously set variable to get reset unless that line of code, or another line, runs again to reset it. Even if, as in the case of HTTP_Downloadfile(), the function is not modal, once the script has gone past that line of code, the variable has been set and won't be reset just because the original action finally finishes.

From Alpha's Help file about HTTP_Downloadfile(): If Modal is FALSE, Status_Flag will always be TRUE, because the function call returns before the download is complete.
Note that nothing is said about the value changing once the download finishes.

I verified that the script in my zip file (the one that attempts to loop through the table and uses the HTTP_download() with modal set to .T.) does NOT continue to run through the loop while the download is still happening by simply adding a ui_msg_box() (a very powerful debugging tool!) immediately after the line with the a5_downloadfile() command. The message box is not triggered until after the download has completed and the user has pressed the Close button. Even pressing the Launch button does not allow the script continue.

I haven't bothered to check out the reason one person was able to get 6 files to download because that script was using .F. for the modal parameter and that really isn't the right way to do it. I doubt that anybody would consider it good practice to download 6 files at a time and that technique (setting modal to .F.) would theoretically allow an unlimited number of "simultaneous" downloads. Since that's not something most people would consider acceptable - at least not by today's standards - I don't see the value in taking the time to check it out.

[Tim Kiebert]

Re: A5_DownLoadFile() revisited

Cal,

It would be intersting to know what happens if you throw <tbl>.Persist() into the mix.

I was recently playing with some code in the interactive window and the table I had opened kept getting closed by a code snippet I was running. I think I was populating a collection from a table. I used persist() after the table open. and it stayed open till I used close()

[G Gabriel]

Re: A5_DownLoadFile() revisited

because this means that all open tables must be closed before running either of these download functions - regardless of whether or not the open table has anything to do with the download itself.

That's why I find it so weird.

[CALocklin]

Re: A5_DownLoadFile() revisited

Cal,

It would be intersting to know what happens if you throw <tbl>.Persist() into the mix.

I was recently playing with some code in the interactive window and the table I had opened kept getting closed by a code snippet I was running. I think I was populating a collection from a table. I used persist() after the table open. and it stayed open till I used close()

Hooray! We have a solution! I take back everything bad I ever said about you guys from down under. Using ".persist()" solves the problem.

However, I still think that's a work-around that shouldn't be required and there is a bug in the download function(s).

Besides, the help text for the .persist() command says its purpose is to keep the table open when going from one script to another. In this case we are only running one script with a function call - but there are lots of function calls in most scripts so that shouldn't be an issue.

(BTW: I don't recall ever saying anything bad about you guys. I was down there for 2 weeks on a business trip many years ago. Everybody was great and I was treated very well. I even tape recorded my recollections when the trip was over so I'd have a permanent record. The only problem with the whole trip was the flight back. The guy next to me was fairly tall and muscular but courteous about how much space he was taking up - until the next guy sat down and he must have weighed about 400 lbs. He was barely able to get in the seat. I ended up sitting almost sideways pressed into the corner for 14 hours unable to move at all. Then I figured I'd get some relief on the last 4 hours from Los Angeles to Detroit but no, it turned out that this plane was also full and the really big guy from the first plane had the seat right next to me. To make things worse, he was starting to get very ripe. I was really glad when we finally landed in Detroit. It was a terrible way to end what was otherwise a great trip.)

[Tim Kiebert]

Re: A5_DownLoadFile() revisited

Hooray! We have a solution! I take back everything bad I ever said about you guys from down under. Using ".persist()" solves the problem.

I didn't realise there was that much riding on my suggestion. ;)

However, I still think that's a work-around that shouldn't be required and there is a bug in the download function(s).

Besides, the help text for the .persist() command says its purpose is to keep the table open when going from one script to another. In this case we are only running one script with a function call - but there are lots of function calls in most scripts so that shouldn't be an issue.

Agreed. If not a bug it is at least an affect or consequence that needs documenting. Like, 'when using this function in a loop with an open table also use <tbl>.Persist'.

It depends on the circumstances of course but I would favour getting a list of values from the table and then looping the list. (which I think you have demonstrated worked ok) That way the table is not open for what may be a long period of time.

[CALocklin]

Re: A5_DownLoadFile() revisited

It depends on the circumstances of course but I would favour getting a list of values from the table and then looping the list. (which I think you have demonstrated worked ok) That way the table is not open for what may be a long period of time.

As you said, it depends on the circumstances. For instance, you may have a table open for another purpose. And, yes, looping through a list works fine.

[G Gabriel]

Re: A5_DownLoadFile() revisited

The question has yet to be answered:
What closes the table?

It's the opposite of the norm. More often than not, people are complaining that tables stay open when they should be closed.

[CALocklin]

Re: A5_DownLoadFile() revisited

Selwyn's reply:

I agree that it is a bug that on some servers this function does not work.
I don't know what the characteristics are of the servers on which it does not work.

We have so much on our plate that it is not possible for us to put this bug on a priority list.
If you need to download files using Xbasic, you will have to use some 3rd party COM object.

The function was developed for internal use at alpha so that the help files, the Oracle Thin Client and other files could be downloaded automatically in Alpha Five from our own servers (or from filekicker, where we host many of our downloadable files). It is working fine for our needs.

Note that Tim Kiebert's suggestion above seems to be a suitable work-around - use <tbl>.persist() before running the download command.

[G Gabriel]

Re: A5_DownLoadFile() revisited

That's a great non-answer answer, but for understandable reasons.

That's aside:
If you use table.persist() before a5_DownLoadFile():
a-What good will that do? according to you, a5_DownLoadFile closes the table, and
b-When and how are you going to close the table?

If you put it in the loop after a5_DownLoadFile(), the table will remain open for the first loop only.

Has anyone tested any of these scenarios? or are all of these just abstract thinking?
Quite frankly, I am in no mood to test any of these theories as it doesn't do me any good nor do I have the time, but for any one who cares, I suggest puttng these theories to test.