Programmable Web Crawler

[Peter.Wayne]

Re: Programmable Web Crawler

John,
I don't think there's a flaw in the activeX. As you can see, I posted two versions of a solution, one by modifying your original PinellasClick script, and the second with an Xdialog. In both cases the script navigates through all the pages. The second solution is more "robust" in that it uses the documentcomplete() event of the activex object. Otherwise it's necessary to fiddle with xbasic_wait_for_idle() to be sure that there's enough of a pause in between attempts to read the pages.

[jkukuda]

Re: Programmable Web Crawler

I see after running your code that concatenating the string worked just fine. From here I think I can handle it and as I said when I started, something for the code archive or your web site. Thank you ever so much for the solution, I have certainly learned a lot.

[Marcel Kollenaar]

Re: Programmable Web Crawler

Hi,

I found some other info at link. These are Ruby and Python examples under linux but it may ignite some ideas for Xbasic.

[Elke]

Re: Programmable Web Crawler

Marcel, I get this from the dom inspector

document.getElementById("dgResults").childNodes.item(0).childNodes.item(21).childNodes.item(0).childNodes.item(4)

It is button 3 on the web page

I then add
.click()
to simulate a mouse click

Hope this helps

John,

WOW! I read, read, and READ but I simply cannot even get to how to simulate a mouse click. Which tools do you use from I use http://www.ieinspector.com

Any chance you can walk me through how to do this?

[jkukuda]

Re: Programmable Web Crawler

I'll try to put something together for you and do a short avi. Give me a few days as I just got your message

[jkukuda]

Re: Programmable Web Crawler

I use the webdeveloper. I'll try to put it no the avi.

[jkukuda]

Re: Programmable Web Crawler

document.getElementById("dgResults").childNodes.item(0).childNodes.item(21).childNodes.item(0).childNodes.item(4).click()

[jkukuda]

Re: Programmable Web Crawler

I understand the reading, You don't know how sore my head got from beating it against the wall trying to get this to work. After that I got my life back.

[Elke]

Re: Programmable Web Crawler

Thanks John,

I finally got how to click, ugh. Peter Wayne's example had

doc.getElementById("q").click()

changed it to

doc.forms.quote.childNodes.item(0).childNodes.item(1).childNodes.item(0).click()

since the Yahoo Finance changed the button name it is no longer q.

Now I'm stuck on Function documentcomplete, I cannot get it to fire.

- Thanks again, Elke

[jkukuda]

Re: Programmable Web Crawler

I could never get it to fire from a form. I have had great difficulty using properties and events on a form. The syncronization between alpha and activex seems to be a problem and most web sites have some differences.
Here is a code snippit that works most of the time.

x=.t.
while x <> .f.
UI_YIELD()
x=actx.busy
end while

[jkukuda]

Re: Programmable Web Crawler

I am now trying to figure out how to pass the parameters for this event.

function newwindow2 as v (ppDisp as OLE::shell.explorer.2::IWebBrowser,Cancel as L)
	ui_msg_box("Congrats","New Window")
end function

This is from the activex events on a form.

I got the following from this link http://msdn2.microsoft.com/en-us/lib...wb_event_sinks

Perhaps you'll want all links to open in your current WebBrowser Control instance. Or perhaps you'll want to open a link in a new WebBrowser Control instance under your control, with your user interface and with your branding.

You can intercept an event, DWebBrowserEvents2::NewWindow2, in your IDispatch implementation to control this. Your control needs to connect to the DWebBrowserEvents2 connection point to intercept this event.

Once you're connected to DWebBrowserEvents2, implement your IDispatch::Invoke so that it handles DISPID_NEWWINDOW2. During the IDispatch::Invoke function call for DISPID_NEWWINDOW2, the array pDispParams contains two parameters. The first one, at index zero, is a Boolean value that tells the WebBrowser Control whether to cancel the new window or not. By default, it is FALSE and a new window will open. If you want to cancel new window creation completely, set the flag to TRUE.

I don't know how to pass back the parameters for this function. I need to override the defaults to prevent a new window from opening so it will use the current window.

All help and suggestions are appreciated.