robots exclusion standard

Show Summary Details

Quick Reference

A standard which is used to define data stored in a text file associated with a Web site. The text in the file describes the access privileges of robots visiting the page: for example, the text could forbid all access to the Web site by robots or restrict it to a small number of pages.

Subjects: Computing.

Reference entries

Users without a subscription are not able to see the full content. Please, subscribe or login to access all content.