Various search engines and bots will scan your website pages to index them.

This can consume server resources quickly if you have lots of pages.

We recommend adding a crawl to help throttle those bots.


User-agent: SeznamBot
User-agent: DotBot
User-agent: bingbot
User-agent: YandexBot
Crawl-Delay: 30


Keywords: bots, crawling, indexing

Click Here for how to create robots.txt file.