Alpha Software Mobile Development Tools:   Alpha Anywhere    |   Alpha TransForm subscribe to our YouTube Channel  Follow Us on LinkedIn  Follow Us on Twitter  Follow Us on Facebook

Announcement

Collapse

The Alpha Software Forum Participation Guidelines

The Alpha Software Forum is a free forum created for Alpha Software Developer Community to ask for help, exchange ideas, and share solutions. Alpha Software strives to create an environment where all members of the community can feel safe to participate. In order to ensure the Alpha Software Forum is a place where all feel welcome, forum participants are expected to behave as follows:
  • Be professional in your conduct
  • Be kind to others
  • Be constructive when giving feedback
  • Be open to new ideas and suggestions
  • Stay on topic


Be sure all comments and threads you post are respectful. Posts that contain any of the following content will be considered a violation of your agreement as a member of the Alpha Software Forum Community and will be moderated:
  • Spam.
  • Vulgar language.
  • Quotes from private conversations without permission, including pricing and other sales related discussions.
  • Personal attacks, insults, or subtle put-downs.
  • Harassment, bullying, threatening, mocking, shaming, or deriding anyone.
  • Sexist, racist, homophobic, transphobic, ableist, or otherwise discriminatory jokes and language.
  • Sexually explicit or violent material, links, or language.
  • Pirated, hacked, or copyright-infringing material.
  • Encouraging of others to engage in the above behaviors.


If a thread or post is found to contain any of the content outlined above, a moderator may choose to take one of the following actions:
  • Remove the Post or Thread - the content is removed from the forum.
  • Place the User in Moderation - all posts and new threads must be approved by a moderator before they are posted.
  • Temporarily Ban the User - user is banned from forum for a period of time.
  • Permanently Ban the User - user is permanently banned from the forum.


Moderators may also rename posts and threads if they are too generic or do not property reflect the content.

Moderators may move threads if they have been posted in the incorrect forum.

Threads/Posts questioning specific moderator decisions or actions (such as "why was a user banned?") are not allowed and will be removed.

The owners of Alpha Software Corporation (Forum Owner) reserve the right to remove, edit, move, or close any thread for any reason; or ban any forum member without notice, reason, or explanation.

Community members are encouraged to click the "Report Post" icon in the lower left of a given post if they feel the post is in violation of the rules. This will alert the Moderators to take a look.

Alpha Software Corporation may amend the guidelines from time to time and may also vary the procedures it sets out where appropriate in a particular case. Your agreement to comply with the guidelines will be deemed agreement to any changes to it.



Bonus TIPS for Successful Posting

Try a Search First
It is highly recommended that a Search be done on your topic before posting, as many questions have been answered in prior posts. As with any search engine, the shorter the search term, the more "hits" will be returned, but the more specific the search term is, the greater the relevance of those "hits". Searching for "table" might well return every message on the board while "tablesum" would greatly restrict the number of messages returned.

When you do post
First, make sure you are posting your question in the correct forum. For example, if you post an issue regarding Desktop applications on the Mobile & Browser Applications board , not only will your question not be seen by the appropriate audience, it may also be removed or relocated.

The more detail you provide about your problem or question, the more likely someone is to understand your request and be able to help. A sample database with a minimum of records (and its support files, zipped together) will make it much easier to diagnose issues with your application. Screen shots of error messages are especially helpful.

When explaining how to reproduce your problem, please be as detailed as possible. Describe every step, click-by-click and keypress-by-keypress. Otherwise when others try to duplicate your problem, they may do something slightly different and end up with different results.

A note about attachments
You may only attach one file to each message. Attachment file size is limited to 2MB. If you need to include several files, you may do so by zipping them into a single archive.

If you forgot to attach your files to your post, please do NOT create a new thread. Instead, reply to your original message and attach the file there.

When attaching screen shots, it is best to attach an image file (.BMP, .JPG, .GIF, .PNG, etc.) or a zip file of several images, as opposed to a Word document containing the screen shots. Because Word documents are prone to viruses, many message board users will not open your Word file, therefore limiting their ability to help you.

Similarly, if you are uploading a zipped archive, you should simply create a .ZIP file and not a self-extracting .EXE as many users will not run your EXE file.
See more
See less

Programmable Web Crawler

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

    #16
    Re: Programmable Web Crawler

    What a wonderful idea!
    Dave Mason
    [email protected]
    Skype is dave.mason46

    Comment


      #17
      Re: Programmable Web Crawler

      Yep. Found something in the helpfiles for using "Using HTML in an Xdialog". And this link to the Xdialog, HTML, and JavaScript - a Powerful Combination

      Maybe it shines some more light on the subject. Mozilla has a DOM inspector build in.
      Last edited by Marcel Kollenaar; 12-27-2006, 07:34 PM. Reason: Links added.
      Marcel

      I hear and I forget. I see and I remember. I do and I understand.
      ---- Confusius ----

      Comment


        #18
        Re: Programmable Web Crawler

        Installed Mozilla FireFox beside IE. Real neat and fast.

        Marcel

        I hear and I forget. I see and I remember. I do and I understand.
        ---- Confusius ----

        Comment


          #19
          Re: Programmable Web Crawler

          Marcel, I get this from the dom inspector

          document.getElementById("dgResults").childNodes.item(0).childNodes.item(21).childNodes.item(0).childNodes.item(4)

          It is button 3 on the web page

          I then add
          .click()
          to simulate a mouse click

          Hope this helps

          Comment


            #20
            Re: Programmable Web Crawler

            Thanks John. It took some time but the pieces are getting together. :)

            I visited Dr. Peter Waynes website and he has added a new article about "Using Javascript, CSS, and AHAH to Make Your Web Pages More Interactive". I saw the function getElementById() passing by in a javascript.

            I see it as an addition of information to your subject.

            It is worth to pay it a visit.
            Marcel

            I hear and I forget. I see and I remember. I do and I understand.
            ---- Confusius ----

            Comment


              #21
              Re: Programmable Web Crawler

              Your right, his web site has a lot of good info, thanks for leading me to it. I just try, try, try to unserstand this stuff but most of it is over my head.

              Comment


                #22
                Re: Programmable Web Crawler

                Yes, I can understand that some stuff is way over your head. I'm a picture thinker and must draw a picture of the problems I encounter. If I can't draw a picture of it I can't understand it. This is a disadvantage over the theoretical - read fast - thinkers. I try to start always at the beginning, the base and the basics. That is the only way I can understand what I'm doing. I keep reading, trying testing and buy me a book when necessary.

                Maybe the code you produced will not win the beauty contest but your code is a remarkable piece of art-work and I appreciate that very much. This will give us all a new view on doing things.

                Keep reading, reading and reading.

                If you can point us in the direction of the parts you do not understand maybe we can fill in the gaps.
                Marcel

                I hear and I forget. I see and I remember. I do and I understand.
                ---- Confusius ----

                Comment


                  #23
                  Re: Programmable Web Crawler

                  Here is another good link
                  http://msdn.microsoft.com/library/default.asp?url=/workshop/browser/webbrowser/reference/properties/busy.asp

                  Comment


                    #24
                    Re: Programmable Web Crawler

                    Thanks John.
                    Marcel

                    I hear and I forget. I see and I remember. I do and I understand.
                    ---- Confusius ----

                    Comment


                      #25
                      Re: Programmable Web Crawler

                      John,
                      In working with your web crawler I discovered one problem at the line where you acquire click a button on page 8:

                      Code:
                      '___________________Get Page 8__________________________________________
                      on error goto 0
                      topparent:ACTIVEX1.activex.document.getElementById("dgResults").childNodes.item(0).childNodes.item(21).childNodes.item(0).childNodes.item(14).click()
                      If you notice I put in an "on error goto 0" before this line so you can see the error. Your page reference doesn't exist in that document, according to the error I get back. I'm not sure what you're attempting to click on in this page.
                      - Peter

                      Comment


                        #26
                        Re: Programmable Web Crawler

                        John,
                        Oops, I realized that the on error goto DONE was intentional. Your script was set up to error out when the site ran out of pages (assuming there are no more than 10 pages on the site).
                        Anyhow, I played around with your script and got it to work properly. I'm attaching a reworked copy of the Pinellas Click script.
                        - Peter

                        Comment


                          #27
                          Re: Programmable Web Crawler

                          John (and Marcel),
                          I discovered that there's a synchronization issue between when the web browser activeX component loads itself and when the activeX component allows Xbasic to manipulate it: that is, the Microsoft web browser reports back that the document has loaded and the page is not busy, but attempts to press buttons and read from the page fail. That's the reason why the xbasic_wait_for_idle() has seemed necessary. I was unable to get the activeX events to work properly but I corresponded with Selwyn who told me that the activeX event functions share the same local variables as the script they're in; what this means is that it's possible to use the activeX web browser's documentcomplete() event to do all the text capture and button-clicking. To shorten this narrative, I was able to do that and made an Xdialog out of your form and I think it does what you need. Here it is (and I'll also attach it as a script to this msg):
                          Code:
                          'Date Created: 29-Dec-2006 07:58:56 PM
                          'Last Updated: 30-Dec-2006 10:16:29 PM
                          'Created By  : Peter Wayne
                          'Updated By  : Peter Wayne
                          ' web window
                          
                          vCounty="PINELLAS"
                          url="http://www.pcsoweb.com/Inmate/inquiry.aspx"
                          dim mayRead as L=.f.
                          dim vStopDate as D=date()-1
                          
                          dim dlg_title as c="Who's in Jail - Pinellas"
                          dim page as n=0
                          dim search_text as c=""
                          dim nAnchors as n=0
                          
                          dim web as p
                          dim web.object as p
                          dim web.class as c ="shell.explorer.2"
                          
                          web.object.addressbar = .T.
                          web.object.fullscreen = .F.
                          web.object.menubar = .T.
                          web.object.offline = .F.
                          web.object.registerasbrowser = .F.
                          web.object.registerasdroptarget = .T.
                          web.object.resizable = ""
                          web.object.silent = .F.
                          web.object.statusbar = .T.
                          web.object.statustext = ""
                          web.object.theatermode = .F.
                          web.object.toolbar = 1.000000
                          web.object.visible = .T.
                          
                          web.events=<<%code%
                          function documentcomplete as v (pDisp as p, url as A)
                          select
                          case page=0
                          	mayRead=.t.
                          	page=page+1
                          case page=1
                          	doc=web.object.document
                          	tbl=doc.getElementById("dgResults")
                          	rows=tbl.getElementsByTagName("tr")
                          	nRows=rows.length
                          	for i=1 to nRows-2
                          		search_text=search_text+rows.item(i).innerText+crlf()
                          	next
                          	nChildren=1+doc.getElementById("dgResults").childNodes.item(0).childNodes.item(21).childNodes.item(0).childNodes.length
                          	nAnchors=nChildren/2
                          	if page<nAnchors
                          		doc.getElementById("dgResults").childNodes.item(0).childNodes.item(21).childNodes.item(0).childNodes.item(page*2).click()
                          		page=page+1
                          	else
                          		process(search_text)		
                          	end if 
                          case page>1
                          	doc=web.object.document
                          	tbl=doc.getElementById("dgResults") 
                          	rows=tbl.getElementsByTagName("tr")
                          	nRows=rows.length
                          	for i=1 to nRows-2
                          		search_text=search_text+rows.item(i).innerText+crlf()
                          	next
                          	if page<nAnchors
                          		doc.getElementById("dgResults").childNodes.item(0).childNodes.item(21).childNodes.item(0).childNodes.item(page*2).click()
                          		page=page+1
                          	else
                          		process(search_text)		
                          	end if 
                          end select
                          end function
                          %code%
                          
                          ui_modeless_dlg_box(dlg_title,<<%dlg%
                          {font=Tahoma,10}
                          {units=f}
                          {can_exit=Close}
                          {startup=Go}
                          {region}
                          Pick any day before today:| 
                          [%DATE;P=popup.calendar(dtoc(vStopDate));I=popup.calendar%.14vStopDate!vStopDate_*]
                          |<Get Bookings!Read?mayRead>|{sp}<Clear>{sp=12}<Close>;
                          {endregion};
                          {line=1,0};
                          {region}
                          url: {text=48,1url};
                          {endregion};
                          {region}
                          {activex=92,18web}
                          {endregion};
                          %dlg%,<<%code%
                          if a_dlg_button="Go" then
                              web.object.navigate(url)
                          end if
                          if a_dlg_button="Read" then 
                          	doc=web.object.document
                          	doc.forms.Form1.txtBookingDate=vStopDate
                          	doc.forms.Form1.btnBookingDate.click()
                          	mayRead=.f.
                          end if
                          if a_dlg_button="Clear" then
                          	page=0
                          	search_text=""	
                          	web.object.navigate(url)
                          	mayRead=.t.
                          end if
                          if a_dlg_button="Close" then
                              ui_modeless_dlg_close(dlg_title)
                          end if 
                          %code%)
                          
                          function process as v (txt as c)
                          ' here you do whatever you want with the results
                          ' i'll just display them here
                          	showvar(txt,"Search Text")
                          end function
                          What had me flummoxed before was that normal xbasic functions do not see the local variables defined outside of them, but these activeX functions do. Well, if you want consistency in a programming language, you'll probably have to go back to Pascal....

                          Comment


                            #28
                            Re: Programmable Web Crawler

                            Peter,

                            This is a beautiful solution.

                            About the ActiveX Events. I think the use of Events of the embedded dialog has priority one above 'event' like possibilities the DOM provides. Not that I knew there would be a synchronization issue but it is just a logical conclusion of working with these parts. I never encountered this because I always used the 'normal' Events A5 created when an ActiveX is drawn on a form and indeed there you can use every variable collective to the form or ActiveX.

                            About your Xdialog solution. I discovered that implementing ActiveX elements in Xdialogs are 'running' smoother than their version on a form. I'm not a frequent user of Xdialog but this discovery will speed up my Xdialog use. :)

                            Still strange that building the Xdialog version 'seems' to take less time after all than the form build version with all the button stuff. Disadvantage is that you need to know more about programming in Xbasic and Xdialog. The form environment scatters all the actions needed over the different parts of the form objects while the Xdialog version concentrate the code on one location.

                            Thanks for the nice solution.
                            Last edited by Marcel Kollenaar; 12-31-2006, 03:22 PM. Reason: Minor text changes
                            Marcel

                            I hear and I forget. I see and I remember. I do and I understand.
                            ---- Confusius ----

                            Comment


                              #29
                              Re: Programmable Web Crawler

                              Marcel,
                              You should look at the Xdialog genie. It really can help you get started with an Xdialog. If you start with the genie (there's even a genie for a Web browser), then most Xdialogs will take only slightly more effort than forms.
                              - Peter

                              Comment


                                #30
                                Re: Programmable Web Crawler

                                I had looked at the xdialog before but it was over my head. You can create a browser with the genie. I'm Curious now if converting a form to xdialog will reap the same benefits. BTW Peter I really appreciate your help, I knew if you looked at it you could provide some insite as I've looked at some of your other activex articles. I'll have to spend more time learning xdialog. One of these days I hope to make a profit from my web site and I'll then invest in some training. It would be nice if you did training, but I understand this stuff is not your career.
                                One other question(problem) with my code was after the fourth page of downloading the string could not acquire any more data. I observed that after running and looking at the variable that it was changed from C to L. I think alpha has a flaw(activex) that causes this.

                                Comment

                                Working...
                                X