User-agent: * Disallow: /shovel/ Crawl-delay: 5 User-agent: Yandex Disallow: / Crawl-delay: 500