From what I understand with the robot crawlers of search engines, they take on regular sites with no problem. Regular site means you have not robot.tx in your site because that file is an instruction to the robot crawlers on what to do. If there is no robot.txt file then that means there is no instruction so the robot crawlers will do the default which is to crawl on your web pages.From what I understand with the robot crawlers of search engines, they take on regular sites with no problem. Regular site means you have not robot.tx in your site because that file is an instruction to the robot crawlers on what to do. If there is no robot.txt file then that means there is no instruction so the robot crawlers will do the default which is to crawl on your web pages.
rannt
User-agent: *
Disallow:
Thank you
Gašper Put this into your robot.txt file: User-agent: * Disallow: Thank you Gašper
Are you sure you want to delete this post?