# All robots will spider the domain User-agent: * Disallow: /cgi-bin/ Disallow: /includes/ Disallow: /Menustuff/ Disallow: /adimages/ Disallow: /adserver/ Disallow: /gbookmx/ Disallow: /lists/ Disallow: /flash/ Disallow: /contents/ Disallow: /codes/ Disallow: /old011/ Disallow: /webalizer/ Disallow: /awstats/ Disallow: /site/ Disallow: /tsep/ Disallow: /sendpage/ Disallow: /adserver3/ Disallow: /dining/ Disallow: /gallery/ Disallow: /old_lists/ Disallow: /pad/ Disallow: /Scripts/ Disallow: /ta.html Disallow: /maillist/