The Role Of The Robot Exclusion In Copyright Defenses

Law360, New York (January 21, 2014, 6:54 PM EST) -- The robots.txt protocol, also known as the robot exclusion standard, is a nearly 20-year-old voluntary Web-programming convention that communicates to Web-crawling or scraping software programs (i.e., "spiders" or "bots") permission, or the lack thereof, to access all or part of a publicly available website.[1] The protocol generally uses simple programming instructions that define which portions of a website robots (or any particular robot) are "disallowed" from accessing — e.g., crawlers may be disallowed from accessing any portion of a website's server whose URL begins with a "/" (effectively the entire server) or only those whose URL begins with a more specific prefix such as "/tmp" or "/log".[2]...

Law360 is on it, so you are, too.

A Law360 subscription puts you at the center of fast-moving legal issues, trends and developments so you can act with speed and confidence. Over 200 articles are published daily across more than 60 topics, industries, practice areas and jurisdictions.

A Law360 subscription includes features such as

  • Daily newsletters
  • Expert analysis
  • Mobile app
  • Advanced search
  • Judge information
  • Real-time alerts
  • 450K+ searchable archived articles

And more!

Experience Law360 today with a free 7-day trial.

Start Free Trial

Already a subscriber? Click here to login

Hello! I'm Law360's automated support bot.

How can I help you today?

For example, you can type:
  • I forgot my password
  • I took a free trial but didn't get a verification email
  • How do I sign up for a newsletter?
Ask a question!