# See the following site for a turtorial # http://www.searchengineworld.com/robots/robots_tutorial.htm # All robots are allowed to go into my site, but # disallow them from searching these directories User-agent: * Disallow: /cgi_bin/ Disallow: /admin/ Disallow: /phpMyAdmin/ Disallow: /includes/ Disallow: /update_cur* Disallow: /katrash* # IF YOU DO NOT WISH TO HAVE THE GOOGLE IMAGE BOT SCAN YOUR DOMAIN FOR IMAGES # THEN YOU CAN INCLUDE THE FOLLOWING IN YOUR ROBOTS FILE. # I FOUND THAT MY BANDWIDTH USAGE DROPPED BY A MASSIVE AMOUNT AFTER I GOT RID # OF THE GOOGLE IMAGE BOT. ALL I HAD WAS IMAGE HUNTERS STEALING PRODUCT SHOTS # AND NOT EVEN BROWSING THE SITE. User-agent: Googlebot-Image Disallow: /