Hi
Can you tell me if a site has robots.txt and Htaccess then what a search engine do first when it visit the site?
Does it check the HTaccess file first Or robots.txt? Quote: Originally Posted by busymaster
Hi
Can you tell me if a site has robots.txt and Htaccess then what a search engine do first when it visit the site?
Does it check the HTaccess file first Or robots.txt?
Easy, it is the robots.txt first. No search engine and no visitor can read your .htaccess file and everybody can read your robots.txt file. .htaccess is a file used internally by Apache when it processes requests from browsers or robots.
In other words, when a search engine or a visitor sends a request to your web server, Apache asks .htaccess if the request deserves some special treatment and then it processes it.
So .htaccess is processed before any file is accessed.
Also note that robots.txt is an informative file. Search engines may respect it or not.
Jean-Luc Quote: Originally Posted by error 404 No search engine and no visitor can read your .htaccess file and everybody can read your robots.txt file. .htaccess is a file used internally by Apache when it processes requests from browsers or robots.
In other words, when a search engine or a visitor sends a request to your web server, Apache asks .htaccess if the request deserves some special treatment and then it processes it.
So .htaccess is processed before any file is accessed.
Also note that robots.txt is an informative file. Search engines may respect it or not.
Jean-Luc
Bingo and +1 for this post. Normally I would not post just to say something like this but your first reply is accurate but murky.
Can you tell me if a site has robots.txt and Htaccess then what a search engine do first when it visit the site?
Does it check the HTaccess file first Or robots.txt? Quote: Originally Posted by busymaster
Hi Can you tell me if a site has robots.txt and Htaccess then what a search engine do first when it visit the site?
Does it check the HTaccess file first Or robots.txt?
Easy, it is the robots.txt first. No search engine and no visitor can read your .htaccess file and everybody can read your robots.txt file. .htaccess is a file used internally by Apache when it processes requests from browsers or robots.
In other words, when a search engine or a visitor sends a request to your web server, Apache asks .htaccess if the request deserves some special treatment and then it processes it.
So .htaccess is processed before any file is accessed.
Also note that robots.txt is an informative file. Search engines may respect it or not.
Jean-Luc Quote: Originally Posted by error 404
No search engine and no visitor can read your .htaccess file and everybody can read your robots.txt file. .htaccess is a file used internally by Apache when it processes requests from browsers or robots.In other words, when a search engine or a visitor sends a request to your web server, Apache asks .htaccess if the request deserves some special treatment and then it processes it.
So .htaccess is processed before any file is accessed.
Also note that robots.txt is an informative file. Search engines may respect it or not.
Jean-Luc
Bingo and +1 for this post. Normally I would not post just to say something like this but your first reply is accurate but murky.