Please read carefully. While this looks tough, it is very easy as you will see. Here is and example of the URL to access the…

You can use the /robots.txt file to give instructions about your site to web robots; called The Robots Exclusion Protocol. How it works: a robot wants to vists a…