![]() |
|
||||||
|
|
Thread Tools | Display Modes |
|
#1
|
|||
|
|||
|
In order to avoid duplicate content on our WP sites I found where some recommend using the robots.txt file. Does that sound right? I found the following code that one person recommends to stop duplicate content:
User-agent: * Sitemap: http://yourblog.com/sitemap.xml Allow: /wp-content/uploads/ Disallow: /cgi-bin/ Disallow: /wp-admin/ Disallow: /wp-login.php Disallow: /wp-includes/ Disallow: /wp-content/ Disallow: /wp-content/plugins/ Disallow: /wp-content/cache/ Disallow: /wp-content/themes/ Disallow: /*?* Disallow: /*? Disallow: /archives/ Disallow: /feed/ Disallow: /comments/feed/ Disallow: /comments Disallow: /feed/$ Disallow: /*/feed/$ Disallow: /*/feed/rss/$ Disallow: /*/trackback/$ Disallow: /*/*/feed/$ Disallow: /*/*/feed/rss/$ Disallow: /*/*/trackback/$ Disallow: /*/*/*/feed/$ Disallow: /*/*/*/feed/rss/$ Disallow: /*/*/*/trackback/$ Disallow: /tmp/ Disallow: /tag/ Disallow: /category/ Disallow: /category/*/* Disallow: /author/ Disallow: /wp-* Disallow: /tag/ Disallow: /*.js$ Disallow: /*.inc$ Disallow: /*.css$ Disallow: /*.gz$ Disallow: /*.wmv$ Disallow: /*.cgi$ Disallow: /*.php$ Disallow: /trackback/ Disallow: */trackback Disallow: */feed Disallow: */comments Disallow: /*?* Disallow: /*? Disallow: /z/j/ Disallow: /z/c/ Disallow: /stats/ Disallow: */comment-page/ Disallow: /*/feed/ Disallow: /*feed/ This is just one example of a robots.txt file. It seems as if everyone has their own version and I have no idea which is best. I definitely have a duplicate content issue on my site. WebMaster Tools says I have 595 URLs in my sitemap but now I only have 88 of them indexed. The number of pages indexed keeps going down. I have the tag.php file set for excerpt and I'm in the process of writing excerpts for all of my posts. That should take care of the tags issues but isn't it possible that duplicate content can also appear through feeds, archives, etc.? Any advice would be greatly appreciated. It seems as if my site is dying right before my very eyes. Thanks. |
| Thread Tools | |
| Display Modes | |
|
|
Similar Threads
|
||||
| Thread | Thread Starter | Forum | Replies | Last Post |
| elusive robots.txt file | tmalcolm | General Discussion | 5 | 02-02-2011 03:58 PM |
| robots.txt file? | valentinachistova | General Discussion | 2 | 01-07-2010 08:26 AM |
| problem with robots.txt file | Calleigh | General Discussion | 6 | 09-23-2009 10:34 AM |
| Robots.txt file | jessiegoh | General Discussion | 3 | 09-17-2009 07:55 PM |
© Copyright 2012 Copyblogger Media LLC · StudioPress™ is a trademark of Copyblogger Media LLC
Privacy Policy | Refund Policy | Terms of Service | Affiliate Program | Contact Us