# Guide Webcrawler Robots # Syntax explanation see http://www.robotstxt.org/wc/norobots.html User-agent: * Disallow: /counter Disallow: /test_ Disallow: /priv User-agent: Slurp Crawl-delay: 10