A5_DownLoadFile() revisited

[Graham Wickens]

trying to download a lot of gif files from various websites using the following code:

source_html = "http://www.combatindex.com/insignia/airforce/graphics/"
target_dir = "d:\my databases\bitmaps\testData" + chr(92)
tbl = table.open("usaf_sqn_data")
tbl.fetch_first()
WHILE .not. tbl.fetch_eof()
gif_name = alltrim(tbl.graphic_name)
a5_DownLoadFile(source_html+gif_name,target_dir+gif_name)
tbl.fetch_next()
END WHILE
tbl.close()

when I ran it, the script downloaded the first gif file from the table "usaf_sqn_data" just fine, but then threw up error message:

Not an Open Table

on the tbl.fetch_next() statement

presumably A5_downloadfile() does more than just download source file as it seems (to me anyway) that it is either "losing" the tbl.pointer or is closing the table for some reason.

anyone got any suggestions

[Stan Mathews]

Re: A5_DownLoadFile() revisited

The documentation indicates that the function

A5_DownLoadFile()

is to be used to download a web page into a text file. I don't see mention of retrieving files.

No telling what is happening the way you are trying to use it.

Have you tried FTP_GET_FILE()? or HTTP_DOWNLOAD()?

[Graham Wickens]

Re: A5_DownLoadFile() revisited

1) A5_DownLoadFile does download a .gif file in the correct format and the result is usable.

2) HTTP_Download() does the download but saves the .GIF file as a TEXT file with a .GIF filetype suffix !! I also got the same "not open table" error.

3) FTP_GET_FILE() assumes that the source site is an FTP site and I have a username and password, which it isnt and I havent

[Stan Mathews]

Re: A5_DownLoadFile() revisited

What does your HTTP_Download() statement have as parameters. The docs example is with a pdf file.

This works for me...

http_download("http://msgboard.alphasoftware.com/alphaforum/images/alpha/logo_sm_gray.gif", "c:\test.gif")

As does this (didn't see the gif section).

http_download("http://www.combatindex.com/insignia/airforce/graphics/12_airlift_flight.jpg", "c:\test.jpg")

[Graham Wickens]

Re: A5_DownLoadFile() revisited

seems that HTTP_Download works fine for just one file but does not like being in a loop

dim options as P
options.close_on_complete = .t.
options.launch_on_complete = .f.

source_html = "http://www.combatindex.com/insignia/airforce/graphics/"
target_dir = "d:\my databases\bitmaps\testData" + chr(92)
tbl = table.open("usaf_sqn_data")
tbl.fetch_first()
WHILE .not. tbl.fetch_eof()
gif_name = alltrim(tbl.graphic_name)
http_download(source_html+gif_name,target_dir+gif_name,.f.,options)
tbl.fetch_next()
END WHILE
tbl.close()

without the extra parameters ie ",.f.,options" it just loops around launching the first retrieved file until I select close, with the added options it does the first six, then throws up the "NOT OPEN TABLE" error, even though the 7th file is available on the source html page.

As you can see, I have made all references to the parent table using a variable so that the table is not referenced by the HTTP_DOWNLOAD function at all!!

[Doug Page]

Re: A5_DownLoadFile() revisited

Graham,

I have sometimes seen where combining items within the function can cause a problem. So how about trying this:

S_Name=source_html+gif_name
T_Dir=target_dir+gif_name
http_download(S_Name,T_Dir,.f.,options)

I would also try it with the A5_DownloadFile() code you used earlier to see if it helps.

[G Gabriel]

Re: A5_DownLoadFile() revisited

Graham:
I believe this is all a timing issue.
Your secript goes to:
tbl.fetch_next()
before it has finished downloading and saving the file using the current records parameters.

There was a thread with a similar issue not long ago.

There is no earthly way that your computer or any computer known to man could download and save a file at the same speed it takes the script to move to the next line of code fetching the next record and actually going to the next record and starting a new loop. In reality, all that happen before the file is downloaded and saved.

You have to give alpha enough time to download, save the file and trigger a flag showing that the file was downloaded and saved before moving to the next record.

[Graham Wickens]

Re: A5_DownLoadFile() revisited

OK I take your point:
HTTP_DOWNLOAD() has a logical result posted

dim options as P
options.close_on_complete = .t.
options.launch_on_complete = .f.
'
source_html = "http://www.combatindex.com/insignia/airforce/graphics/"
target_dir = "d:\my databases\bitmaps\testData" + chr(92)
tbl = table.open("usaf_sqn_data")
tbl.fetch_first()
WHILE .not. tbl.fetch_eof()
gif_name = alltrim(tbl.graphic_name)
flag = http_download(source_html+gif_name,target_dir+gif_name,.f.,options)
tbl.fetch_next()
END WHILE
tbl.close()

I am at a loss as how to use this logical flag in my WHILE loop. Could someone tell me how please?

[Graham Wickens]

Re: A5_DownLoadFile() revisited

I found the WAIT_UNTIL() function, is this what I should be using? I cannot get the syntax correct:

flag = http_download(source_html+gif_name,target_dir+gif_name,.f.,options)
continue = wait_until("flag=.t.",6000)

[Tim Kiebert]

Re: A5_DownLoadFile() revisited

I think it should be

[COLOR=Red]     continue = wait_until(flag,1,6000)  [/COLOR]

This will check flag every second and finish waiting when flag is true or 6000 seconds have elapsed. You probably need to declare flag at the beginning of the script as it may not have been implicitly set the first time through the loop. You also need to set flag to false again after you are finished with it so it is ready for the next time through the loop.

You might also try get a list of values from the table using

list = table.external_record_content_GET("usaf_sqn_data","graphic_name")

and then using For...Next to loop through the names. That would eliminate the possibility of the download interfering with table fetches.

[G Gabriel]

Re: A5_DownLoadFile() revisited

Graham:
That flag does not serve much. It should be something like:

dim options as P
options.close_on_complete = .t.
options.launch_on_complete = .f.
'
source_html = "http://www.combatindex.com/insignia/airforce/graphics/"
target_dir = "d:\my databases\bitmaps\testData" + chr(92)
tbl = table.open("usaf_sqn_data")
tbl.fetch_first()
WHILE .not. tbl.fetch_eof()
gif_name = alltrim(tbl.graphic_name)
x=http_download(source_html+gif_name,target_dir+gif_name,.f.,options)
[COLOR="Blue"]While
x.exists()=.f.
sleep(.1) 'change the interval here as practical.
end while[/COLOR]
tbl.fetch_next()
END WHILE
tbl.close()

[Graham Wickens]

Re: A5_DownLoadFile() revisited

Still having problems.

While
x.exists()=.f.
sleep(.1) 'change the interval here as practical.
end while

error message x.exists method not found

also when using "wait_until(x=.t.,600000)" I still get table not open errors

[G Gabriel]

Re: A5_DownLoadFile() revisited

Try:
While
file.exists(x)=.f.
sleep(.5) 'change the interval here as practical.
end while

[CALocklin]

Re: A5_DownLoadFile() revisited

On the surface this sounds like a timing issue but I'm not convinced. Http_download() is a function and therefore should finish running before the rest of the script continues.

Note that I said should because I'm not positive it does. (I'm not sure whether "finished running" also means "completed the download" in this case.) To test it, I'd do something like this:

   ......
   gif_name = alltrim(tbl.graphic_name)
[COLOR=blue]   start_time = now()
[/COLOR][COLOR=red]   flag = http_download(source_html+gif_name,target_dir+gif_name,.f.,options)[/COLOR]
[COLOR=blue]   etime = ltrim(str(now()-start_time,10,3))[/COLOR]
[COLOR=#0000ff]   IF file.exists( target_dir+gif_name)[/COLOR]
[COLOR=#0000ff]      ui_msg_box("EXISTS", etime)[/COLOR]
[COLOR=#0000ff]   ELSE[/COLOR]
[COLOR=#0000ff]      ui_msg_box("FILE NOT DOWNLOADED", etime)[/COLOR]
[COLOR=#0000ff]   END IF[/COLOR]
   tbl.fetch_next()
END WHILE
tbl.close()

You might also need to check the file size as well as "exists" in case the file is only partly downloaded.

I'm not saying it isn't a timing issue but I think it's worth verifying first.

[Tom Cone Jr]

Re: A5_DownLoadFile() revisited

The absence of any error checking sequences, or error recovery sequences, could be causing the script to fail. What if there is no GIF at the source with the same name as is retrieved from the local table?

Has anyone run the script in debug mode?

Both are often useful in figuring out why a script that runs through a loop is failing.