Standard for Robot Exclusion

Show Summary Details

Quick Reference

A number of the early bots and spiders used by search engines caused problems when they visited Web sites, for example, they would reduce the access time of other users. This standard specifies a number of measures which attempt to alleviate these problems.

Subjects: Computing.

Reference entries

Users without a subscription are not able to see the full content. Please, subscribe or login to access all content.