A spider, also known as a crawler or a bot, is the software Proximic uses to visit and access the content of webpages.
Mozilla/5.0 (compatible; proximic; +http://www.proximic.com/info/spider.php)
User-agent: proximic Disallow:
www.forum.com/showthread.php?t=123is rendered into something like this
User-agent: proximic Disallow: /path/Make sure that the robots.txt is in the correct location. It must be in the top directory, e.g. www.domain.com/robots.txt.