How to keep robots out of your web site
Below is a MRR and PLR article in category Internet Business -> subcategory SEO.
How to Keep Robots Out of Your Website
Summary
Learn how to use the robots.txt file and the Robots Exclusion Protocol to manage search engine indexing and protect your content.Keywords
robots, robots.txt, robots exclusion protocol, internet marketing, home business---
Understanding the Robots.txt File
Search engines are designed to help people quickly find information online. To do this, they use robots, also known as spiders or crawlers, to explore and collect data from web pages. These robots start with user-submitted URLs, links from websites, sitemaps, or the top level of a site. They then access and index information such as titles, keywords, and text.
However, there might be times when you want to prevent search engines from indexing certain pages, like news postings or specific marked pages (e.g., affiliate pages). Compliance with this is voluntary for individual robots.
Using the Robots Exclusion Protocol
To keep robots from indexing certain pages, you can place a robots.txt file in the root directory of your website. For example, if you have a directory called "e-books" that you want to restrict, your robots.txt file should read:
```
User-agent: *
Disallow: /e-books/
```
If you don’t have server access to set up a robots.txt file, you can add a META tag in the head section of your HTML documents. Here’s an example:
```html
```
While support for this META tag is less common than for the robots.txt file, most major web indexes recognize it.
Protecting News Postings
To prevent search engines from archiving your news postings, include an "X-no-archive" line in the headers of your posts:
```
X-no-archive: yes
```
Keep in mind that not all clients allow you to add this line, and some search engines treat all found information as public unless specified otherwise. While exclusion standards can help keep your content out of major search engines, not all robots respect these rules.
If privacy is a major concern for your emails and Usenet postings, consider using anonymous remailers and PGP encryption. These resources can provide more information:
- [Anonymous Remailers](http://www.well.com/user/abacard/remail.html)
- [PGP Information](http://world.std.com/~franl/pgp/)
Even if privacy isn’t your primary concern, remember that anything posted may be indexed and archived indefinitely. Use the robots.txt file as needed to protect your content.
Written by Dr. Roberto A. Bonomi
You can find the original non-AI version of this article here: How to keep robots out of your web site.
You can browse and read all the articles for free. If you want to use them and get PLR and MRR rights, you need to buy the pack. Learn more about this pack of over 100 000 MRR and PLR articles.