User-agent: * Disallow: /all spiders User-agent: * Disallow: /cgi-bin/