Page 1 of 2 12 Last
  • Jump to page:
    #1
  1. No Profile Picture
    Registered User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    Apr 2009
    Posts
    5
    Rep Power
    0

    Is this Robots.txt for my wordpress Blog SEO Friendly?


    My Blog is Wordpress Based and I created a Robots.txt file as my last couple of new Posts were not even being indexed.

    Code:
    User-agent: Googlebot
    Disallow: /wp-content/
    Disallow: /trackback/
    Disallow: /wp-admin/
    Disallow: /feed/
    Disallow: /archives/
    Disallow: /sitemap.xml
    Disallow: /index.php
    Disallow: /*?
    Disallow: /*.php$
    Disallow: /*.js$
    Disallow: /*.inc$
    Disallow: /*.css$
    Disallow: */feed/
    Disallow: */trackback/
    Disallow: /page/
    Disallow: /tag/
    Disallow: /category/
    
    User-agent: Googlebot-Image
    Disallow: /wp-includes/
    
    User-agent: Mediapartners-Google*
    Disallow:
    
    User-agent: ia_archiver
    Disallow: /
    
    User-agent: duggmirror
    Disallow: /
    Could you please advice regarding my Wordpress Robots.txt file
  2. #2
  3. SEO Insultant
    SEO Chat Mastermind (5000+ posts)

    Join Date
    Oct 2005
    Location
    sharing a room with my ego
    Posts
    5,797
    Rep Power
    1822
    No it is not.

    Why disallow access to all that?
    ...Hey you! FREE SEO report card ...I really my hosting company...
  4. #3
  5. No Profile Picture
    Registered User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    Apr 2009
    Posts
    5
    Rep Power
    0
    Originally Posted by dzine
    No it is not.

    Why disallow access to all that?
    Could you please suggest a good Robots.txt file sir. I am new to Robots files. I believed that would avoid duplicate content and hence disallowed them.
  6. #4
  7. No Profile Picture
    Contributing User
    SEO Chat Discoverer (100 - 499 posts)

    Join Date
    Mar 2009
    Location
    Oregon
    Posts
    354
    Rep Power
    0
    Just use
    user-agent: *
    Allow: /

    this allows access to root, there is no need to disallow anything, it will get indexed anyways.

    Comments on this post

    • ClickyB : allow is not an approved robots.txt command (you use it to disallow, not allow)!
  8. #5
  9. No Profile Picture
    Registered User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    Apr 2009
    Posts
    5
    Rep Power
    0
    But that would also include posts under category which may be the same as the original post itself. So that would lead to duplicate post indexing and may ban me. So I have decided to block categories and a fews as shown in my robots.txt . Please feedback
  10. #6
  11. SEO Insultant
    SEO Chat Mastermind (5000+ posts)

    Join Date
    Oct 2005
    Location
    sharing a room with my ego
    Posts
    5,797
    Rep Power
    1822
    What have the search engines indexed so far?
  12. #7
  13. No Profile Picture
    Contributing User
    SEO Chat Discoverer (100 - 499 posts)

    Join Date
    Mar 2009
    Location
    Oregon
    Posts
    354
    Rep Power
    0
    There is a plugin that disallows duplicate indexing of categories and archive pages, just use that.
  14. #8
  15. No Profile Picture
    Registered User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    Apr 2009
    Posts
    5
    Rep Power
    0
    I have made the changes to my Robots.txt

    User-agent: *
    Allow: /

    and used the "Robots Meta Configuration" Plugin

    Then I ran the Tool to check my Robots.txt Syntax using some Robots.txt tool validators. I got Errors. Could you please advice.
  16. #9
  17. SEO Insultant
    SEO Chat Mastermind (5000+ posts)

    Join Date
    Oct 2005
    Location
    sharing a room with my ego
    Posts
    5,797
    Rep Power
    1822
    Well, for starters, the Allow: directive is not part of the original robots.txt spec.

    But... has the plug-in made alterations to the robots.txt file itself?
  18. #10
  19. No Profile Picture
    Contributing User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    May 2006
    Location
    Canada
    Posts
    89
    Rep Power
    9
    Originally Posted by webhush

    [CODE]
    User-agent: Googlebot
    ....
    Disallow: /sitemap.xml
    ....
    What do Google Webmaster Tools *say* about that ?

    Can it retrieve the file ?

    Do you have the sitemap.xml added there (Google Webmaster Tools/Central) ?
  20. #11
  21. No Profile Picture
    Registered User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    Apr 2009
    Posts
    5
    Rep Power
    0
    I guess the Plugin has not made alterations to the Robots.txt File. Now I just have the following in my Robot.txt

    User-agent: *
    Allow: /

    and used the "Robots Meta Configuration" Plugin to prevent spidering the rest
  22. #12
  23. No Profile Picture
    Contributing User
    SEO Chat Discoverer (100 - 499 posts)

    Join Date
    Nov 2004
    Location
    Madrid
    Posts
    209
    Rep Power
    17
    Originally Posted by webhush
    I guess the Plugin has not made alterations to the Robots.txt File. Now I just have the following in my Robot.txt

    User-agent: *
    Allow: /

    and used the "Robots Meta Configuration" Plugin to prevent spidering the rest
    I disagree with much of this thread I'm afraid.

    Using robots.txt to block much of a Wordpress site has a big impact on speed and SERPs. Why index all your tag pages, archives, etc when they are simply duplicates.

    I've spent a lot of time getting the perfect robots.txt file and it has speeded up my site, given me better rankings, and stopped my shared host complaining that my sites were crashing their server.

    Comments on this post

    • dzine agrees : Good points - I guess if depends on the site structure and server performance in the end
  24. #13
  25. SEO Insultant
    SEO Chat Mastermind (5000+ posts)

    Join Date
    Oct 2005
    Location
    sharing a room with my ego
    Posts
    5,797
    Rep Power
    1822
    Why disallow "duplicates" if they are all on your own site? Could well be that one has better on-page, if only by accident, than another, and it might well rank just a bit higher on occasion:

    1) less work
    2) slight advantage on occasion
  26. #14
  27. SEO Insultant
    SEO Chat Mastermind (5000+ posts)

    Join Date
    Oct 2005
    Location
    sharing a room with my ego
    Posts
    5,797
    Rep Power
    1822
    If you do this - depending on your settings - lots of single posts, pages and other things might go unnoticed.

    Also: a large part of that code is non-standard: it doesn't respect basic robots.txt rules. Hm
  28. #15
  29. Contributing User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    Sep 2012
    Location
    WA
    Posts
    81
    Rep Power
    12
    I've read about robot.txt but I thought that it was only for what you didn't want google crawlers or bots, and bing bots to see on your page.

    I had no idea you could use it to direct google/bing bots to pages.

    So let me get this straight in my mind.

    If I add this code into my blogger html
    it will help google find parts of my blog which it may otherwise overlook?

    code [ user-agent: *
    Allow: / ]

    if this is true, Where do I put it? Does it go in the body, or between the <head></head>?

    can you show me some examples of where to place it?

    thanks
    DS
Page 1 of 2 12 Last
  • Jump to page:

Similar Threads

  1. Wordpress blog & Sitemaps
    By cirrus_blue in forum Google Optimization
    Replies: 1
    Last Post: Jan 1st, 2007, 12:54 AM
  2. Free Search Engine Friendly blog hosting
    By vinyl in forum SEO Help (General Chat)
    Replies: 10
    Last Post: Aug 23rd, 2004, 03:47 PM

IMN logo majestic logo threadwatch logo seochat tools logo