StudioPress Community Forums
  StudioPress Community Forums > Forums > General Discussion
For help and support, access to your downloads, or to manage your account please log into My StudioPress.

These forums have been set to read-only so you can browse the existing topics for any questions you may have.

For general discussion on WordPress, CSS and design (NOT for support) visit the new Community Forums.
 
 
Thread Tools Display Modes
Prev Previous Post   Next Post Next
  #1  
Old 02-21-2010, 04:55 PM
Rhett Rhett is offline
Registered User
Genesis Member
 
Join Date: Sep 2009
Posts: 13
Default Using Robots.txt File to Stop Duplicate Content

In order to avoid duplicate content on our WP sites I found where some recommend using the robots.txt file. Does that sound right? I found the following code that one person recommends to stop duplicate content:

User-agent: *

Sitemap: http://yourblog.com/sitemap.xml
Allow: /wp-content/uploads/
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /wp-login.php
Disallow: /wp-includes/
Disallow: /wp-content/
Disallow: /wp-content/plugins/
Disallow: /wp-content/cache/
Disallow: /wp-content/themes/
Disallow: /*?*
Disallow: /*?
Disallow: /archives/
Disallow: /feed/
Disallow: /comments/feed/
Disallow: /comments
Disallow: /feed/$
Disallow: /*/feed/$
Disallow: /*/feed/rss/$
Disallow: /*/trackback/$
Disallow: /*/*/feed/$
Disallow: /*/*/feed/rss/$
Disallow: /*/*/trackback/$
Disallow: /*/*/*/feed/$
Disallow: /*/*/*/feed/rss/$
Disallow: /*/*/*/trackback/$
Disallow: /tmp/
Disallow: /tag/
Disallow: /category/
Disallow: /category/*/*
Disallow: /author/
Disallow: /wp-*
Disallow: /tag/
Disallow: /*.js$
Disallow: /*.inc$
Disallow: /*.css$
Disallow: /*.gz$
Disallow: /*.wmv$
Disallow: /*.cgi$
Disallow: /*.php$
Disallow: /trackback/
Disallow: */trackback
Disallow: */feed
Disallow: */comments
Disallow: /*?*
Disallow: /*?
Disallow: /z/j/
Disallow: /z/c/
Disallow: /stats/
Disallow: */comment-page/
Disallow: /*/feed/
Disallow: /*feed/

This is just one example of a robots.txt file. It seems as if everyone has their own version and I have no idea which is best.

I definitely have a duplicate content issue on my site. WebMaster Tools says I have 595 URLs in my sitemap but now I only have 88 of them indexed. The number of pages indexed keeps going down.

I have the tag.php file set for excerpt and I'm in the process of writing excerpts for all of my posts. That should take care of the tags issues but isn't it possible that duplicate content can also appear through feeds, archives, etc.?

Any advice would be greatly appreciated. It seems as if my site is dying right before my very eyes.

Thanks.
 

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
elusive robots.txt file tmalcolm General Discussion 5 02-02-2011 03:58 PM
robots.txt file? valentinachistova General Discussion 2 01-07-2010 08:26 AM
problem with robots.txt file Calleigh General Discussion 6 09-23-2009 10:34 AM
Robots.txt file jessiegoh General Discussion 3 09-17-2009 07:55 PM


All times are GMT -5. The time now is 02:25 AM.

Powered by vBulletin® Version 3.8.4
Copyright ©2000 - 2013, Jelsoft Enterprises Ltd.