They are still valid if you don't want the search spiders to crawl certain pages. This is what the robot.txt file is for. They didn't really have any seo purpose anyway and most people don't bother with them.They are still valid if you don't want the search spiders to crawl certain pages. This is what the robot.txt file is for. They didn't really have any seo purpose anyway and most people don't bother with them.
To be honest I do not know about robot.txt because I am not really well versed with SEO yet. I am glad to encounter this discussion to broaden my knowledge with the search engine crawlers. My question now is the robot.txt is a necessity of the site otherwise the site will not be inspected by the crawler as if it is not updated with the contents because dormant sites are not checked by the crawlers. Correct me if I’m wrong.To be honest I do not know about robot.txt because I am not really well versed with SEO yet. I am glad to encounter this discussion to broaden my knowledge with the search engine crawlers. My question now is the robot.txt is a necessity of the site otherwise the site will not be inspected by the crawler as if it is not updated with the contents because dormant sites are not checked by the crawlers. Correct me if I’m wrong.
Bert
Are you sure you want to delete this post?