I was using Google Web Master tools and noticed the number of 404 pages found on my blog had skyrocketed over the past few weeks, so I did a little digging and found that when I had installed many of the wordpress plug-ins on dragonblogger.com that the Google Index bots no longer can find them and trigger the 404 pages.
This made me realize that no blogger should have Google or any other search engine indexing their wordpress admin stuff, so I decided to modify my robots.txt to prevent this from happening. It is easier than you think and you can follow these instructions to prevent the search engines from indexing your wp-admin, wp-content, and plug-ins for your wordpress blog.
All you do is have physical access to your server, (IE be able to upload a file or ssh into your box)
create a file called robots.txt in your root directory of your web server
Make your robots.txt include these statements:
User-Agent: *
Disallow: /wp-admin/
Disallow: /wp-content/
Allow: /
save the file
Now your wp-admin, wp-content and all subfolders including plug-ins will not be indexed by the search engines.
This should help increase your PR of your blog overall and prevent 404 errors from displaying in Google indexes when you remove plug-ins and other admin related content.
-Dragon Blogger
