cbroomitzoom
New Member
In Joomla if I set robots.txt to:
Allow: /components/com_content/
Disallow: /
would this allow crawlers access articles and categories only and will the crawler get the SEF URL or the system URL please? Hi,
Code: Allow: /components/com_content/The Allow directive is not standard and it is not understood by all web robots. It is better not to use it.
Code: Disallow: /This one disallows all robots in all pages. It is understood by all robots. So if you use it, your site will not be indexed by search engines.
Jean-Luc You can edit the robots txt with the robots.txt generator from Webmaster Tools account. I wouldnt use the ROBOTS text why are you trying to use it anyway? I would just use the NOINDEX attribute on pages you didnt want it to INDEX. Quote: Originally Posted by williamgomes http://www.v7n.com/forums/seo-forum/229141-robots-txt-help-please.html#post1567905 & http://forum.joomla.org/viewtopic.php?p=2094048 My advice would be with little knowledge modifying the robots text could have a horrible impact.
It is also not clear why he wants to change the robots text for that purpose in the first place and whatever reason that is could be misguided. Google is your friend...
Check this -> http://en.wikipedia.org/wiki/Robots_Exclusion_Standard
Allow: /components/com_content/
Disallow: /
would this allow crawlers access articles and categories only and will the crawler get the SEF URL or the system URL please? Hi,
Code: Allow: /components/com_content/The Allow directive is not standard and it is not understood by all web robots. It is better not to use it.
Code: Disallow: /This one disallows all robots in all pages. It is understood by all robots. So if you use it, your site will not be indexed by search engines.
Jean-Luc You can edit the robots txt with the robots.txt generator from Webmaster Tools account. I wouldnt use the ROBOTS text why are you trying to use it anyway? I would just use the NOINDEX attribute on pages you didnt want it to INDEX. Quote: Originally Posted by williamgomes http://www.v7n.com/forums/seo-forum/229141-robots-txt-help-please.html#post1567905 & http://forum.joomla.org/viewtopic.php?p=2094048 My advice would be with little knowledge modifying the robots text could have a horrible impact.
It is also not clear why he wants to change the robots text for that purpose in the first place and whatever reason that is could be misguided. Google is your friend...
Check this -> http://en.wikipedia.org/wiki/Robots_Exclusion_Standard