On my forum it seems the Google spider likes to follow my movements - always viewing the same threads I just visited. Problem is, I have a private section of the forum where only Admin and Mods are allowed to view and post. So the Google spider often gets the error message when trying to follow me into the hidden forums.Is this a bad thing? Will google associate all those clicks to the error page with my site, or will it just ignore them? I know I can write a txt file that will tell the spider not to follow me into those certain forum numbers, but if I don't have to it would give me more time to focus on other aspects of the site.[edit:] oops, thought I was in the google forum when posting - feel free to move if that forum is more appropriate.When you say the Google Spider follows your movements, are you sure you're not talking about the AdSense bot?How do I know the difference? Sometimes it says Google Spider in my "Who's Online" and other times it says Guest, but its the same IP. I just assumed it was still the Google Spider, I didn't know there was an AdSense Bot... is it following me to ensure I don't click on any of my own ads? lolNo, the AdSense bot responds to people loading a page containing an AdSense ad. If it's a page the bot hasn't seen before, they'll come visit, grab the page's code, work out what ads to display, then display them.That could be why it seems like it's following you around have you tried configuring your robots.txt file?The adsense bot will also revisit a page everytime if you use a dynamic url:ie: htp://myforum.com/thread.php?thread_id=17&sid=123456if for example the sid keeps changing the bot follows everytime it assumes the page may change as well.Look in your logs googles bot and googles adsense bot are easily identifiable as the use different useragent stringsAlan Lastufka wrote:
Being chased by a huge nasty spider!
- Thread starter mr.metro
- Start date