How to Restrict Bots/Spiders Access ?

M

miki86

New Member
Hey all,
I was wondering if someone knows how can i restrict spiders access to specific category, or forum ?
Is there a mod or some other method for that purpose ?

BTW i'm running vb3.7.3 and those categories can be seen by guests but can't be accessed.
I was threatened once about posting warez and i dont want that stuff to be seen on google.

Thanks in advance.
 
For all my forums guest need to be registered to have access, as i said they can be seen but not accessed.
About a year and something ago when google spider had access he was indexing my VIP section, and regular users did not have access there, so i'm confused, in which usergroup spiders belong?
If i understood you right, you said that if guests can't access, spiders can't too?
And one more thing, about chatbox, guest couldn't see it too and i saw in google cache whole chatbox ?!?
 
well spiders belong to the guest usergroup unless you installed the hack to treeat them as members, and is strange about the chatbox o-ô i'm pretty sure you installed that kind of hack
 
Nope, i didn't use any hack for that, and i'm still not clear how could this happen.
It doesn't matter for now, i'll wait couple of days and let spiders do their job and than we'll see.

Thank you very much for your help.
Regards.
 
Yes but robots.txt is for file/folder permissions, not for forum access permissions.

Itsuki Minami said:
well spiders belong to the guest usergroup unless you installed the hack to treeat them as members
Click to expand...

Does anybody knows which mod is that, since guest have no access at all i'm going to need this mod.

EDIT: Found it, no need.
 
miki86 said:
Yes but robots.txt is for file/folder permissions, not for forum access permissions.
Click to expand...


You asked for:
miki86 said:
I was wondering if someone knows how can i restrict spiders access to specific category, or forum ?
Is there a mod or some other method for that purpose ?
Click to expand...
Just tell in robots.txt that its only allowed to visit certain forums.
 
You mean by typing full URL ?
I installed this Member Bots 1.0 and created a new usergroup with specific permissions, so i'll wait and see.

Thanks.
 
miki86 said:
About a year and something ago when google spider had access he was indexing my VIP section, and regular users did not have access there, so i'm confused, in which usergroup spiders belong?
If i understood you right, you said that if guests can't access, spiders can't too?
And one more thing, about chatbox, guest couldn't see it too and i saw in google cache whole chatbox ?!?
Click to expand...
You have nothing to worry about. Spiders try to access no permission areas all the time but if there treated as a guest, and a guest has no access to that forum then neither does google.

You should go to yoursite.com/online.php and you will see that if some one or a bot does try to access such a section they will have a yellow triangle or a stop sign next to there name showing they did not actually get into that forum.
 
Yes i know that, with this mod installed google index pages even though guest's have no permission to access, although i have a slight problem with this mod:

Member Bots 1.0 - Allow bots to act as members

It's made for 3.6.x so my guess is incompatibility or something, i have managed to add Bots to Search Spiders group, but they are still diplayed as guests and won't change color. And i see i'm not the only one with this problem.
One more thing, the guy there mentioned vbulletin_spiders.xml, i couldn't find it and if you know what is it for please tell me.

Thank you.
 
You must log in or register to reply here.
Back
Top