trouble with google indexing

[martin horzempa]

hi
i need a little help with a problem my site isnt being indexed by google and it is something that i think that we really need my site www.jollygreenthumb.com has been up since mid june and google has attempted to crawl the site 9-10 times and it has yet to index the first page

i have a sitemap built and google reports no errors in it yet every time it tries to crawl this site it records an error Network unreachable in the details it gives Unreachable URL

the site is being hosted by firmsupport and have made changes in an attempt to correct the problem but to no avail

of course i have not received any answers to my emails to google concerning my problems - and i hope someone here to shed a little light on them

this error has happened on .html pages and .a5w pages

i have included an xl file that i downloaded from googles sitemap site that list the errors and pages

in an effort to attempt to find out the problem i have built a simple html file and served it up on my little server https://www.npwas.com/text.html

and i have used the wordtool on googles adsense site to make the googlebot attempt to crawl the page

i think that you must have an adsense account to use this but the url is https://adwords.google.com/select/KeywordTool
and if you select the 2nd tab "Site-Related Keywords " google will send the googlebot to the page and attempt to scan it for keywords
and each time i use this tool both for jgt and npwas it returns an error:

Unable to access https://www.npwas.com/text.html.
We're sorry, but we can't seem to access the URL https://www.npwas.com/text.html. Please make sure you have entered the URL correctly and that the site is currently available. Or, begin the keyword generation process again by entering a different URL.

i am assuming that the adsense tool is a good proxy for the regular googlebot and that the 2 errors are the same


i have included my server logs from www.npwas.com test

it seems that the was is attempting to serve the page but google still
reports an error 3 issues came to mind

1. does google recognize the .a5w extension as a valid script extension? i had to list it before the sitemap generator recognized it
   i sent an email to google with this question and they blew me off and sent me to forum hell
   
   i used google search to check for sites and pages using .a5w as the page extension and i found a few but i didnt find any instances that i could say for sure that the .a5w page was crawled - and new links were found - just listing in the index i dont think means that the page was crawled - it could be the destination of a link
   
2. is this an issue of having a session id in the url? i understand ,reading some forums, that this may be an issue with google and i cant see what the was returned to the googlebot but if its the same as whats in the logs this might be an issue
   
   accord to Lenny in this post
   http://msgboard.alphasoftware.com/al...76503#poststop
   
   

   The server will always add the sesison ID to the URL for the first request because it does not yet knwo if the client has a cookie.

   but i have used the google search for "inurl": and "site:" and i can find many instances of url with variations of sessid, sessionid, etc
   that have been indexed but this doesnt mean that the googlebot actually crawled these pages they could just be links that werent followed - cant tell
   
3. i am currently wondering if the request "Accept-Encoding: gzip" might have anytthing to do with it

is it possible to change the settings in the WAS to not send the sessionid in the url??
is it possible to change this setting just for bots and spiders?

does anyone know how to find out if google can crawl and .a5w page ?

thanx in advance

[Lenny Forziati]

Re: trouble with google indexing

Your log files show the request from Google being received and a valid response being sent by the server. Once the response is sent, there isn't anything we can do about it. You would have to work with Google to find out if they are getting the response or not. And if they are getting it, only they could tell you why they are not recognizing it as a valid response.

[martin horzempa]

Re: trouble with google indexing

Thanks for the reply Lenny

yes the logs indicate that the rquest was handed with no errors but the session_id was appended to the response url and this may be a problem
dealing with the googlebot

from their sitemaps site -Webmaster guidelines page
http://www.google.com/support/webmas...l=en#design%22

Use a text browser such as Lynx to examine your site, because most search engine spiders see your site much as Lynx would. If fancy features such as JavaScript, cookies, session IDs, frames, DHTML, or Flash keep you from seeing all of your site in a text browser, then search engine spiders may have trouble crawling your site.

is it possible to change the settings in the WAS to not send the sessionid in the url??
is it possible to change this setting just for bots and spiders by useing the "User-Agent:" property to identify bots and spiders?


it is my ubderstanding that google supports the following media

Web Media Support:
HTML,SHTML,PDF,ASP,JSP,PHP,XML,CFM,DOC,XLS,PPT,RTF,WKS,LWP,WRI, and SWF can all be crawled and indexed.

and i am wondering if you folks at alpha have any idea whether google will
crawl and index a page with a .A5W extension as it does for other script languages?

have you folks at alpha requested google to support crawling and indexing .A5w pages?
if not is there a way you know of that i might do this?

thanks for your assistance

[Lenny Forziati]

Re: trouble with google indexing

is it possible to change the settings in the WAS to not send the sessionid in the url??

The incoming request already had the session id in the URL, the server didn't add it.

is it possible to change this setting just for bots and spiders by useing the "User-Agent:" property to identify bots and spiders?

The user agent string is available to you in the A5W environment and you can conditionalize your code upon it if desired.

it is my ubderstanding that google supports the following media

and i am wondering if you folks at alpha have any idea whether google will
crawl and index a page with a .A5W extension as it does for other script languages?
have you folks at alpha requested google to support crawling and indexing .A5w pages?
if not is there a way you know of that i might do this?

Google and other spiders should be agnostic about the scripting language. Your A5W page is generating output with a MIME type of "text/html", so it is just an HTML page from the spider's point of view

[martin horzempa]

Re: trouble with google indexing

Thanks again Lenny

i am still wondering if i can stop appending the
A5W_Sess_ID to the url on the first request from a client

and if so - how would that be accomplished

i just received an acknowledgement from google
so i am attaching it
thanx again

[Antony]

Re: trouble with google indexing

I am getting this exact problem with google webmaster tools.

It has no probems with my sitemap/robot files yet complains that pages are "unreachable URLs" !!!

The last succesful time it crawled my site was over a year ago - and that was before I used any of the webmaster tools/site maps etc.

The only thing I can think of is that at the time is was sucessful, I was using an index.html as the entry page and that was the only page it every indexed. all .a5w pages are completely ignored - even though they are in the sitemap.

Any ideas please - we are loosing a lot of potentional customers.

Antony