#1
  1. No Profile Picture
    Contributing User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    Jan 2005
    Posts
    59
    Rep Power
    10

    Yet another robots.txt question


    Hello,

    I've been surfing the net and the forum but I'm still confused. In the past someone advised me it was better to use an empty robots.txt file to get rid of the 404 loads in my logs. Ok, this works fine. But now I've been reading it is better to:
    1. not have a robots.txt if you don't want to exclude anything
    2. have one, but empty
    3. have one and use this:
    User-agent: *
    Disallow:
    Now, what is true and what isn't?

    I prefer to allow all robots everything, if necessary I can exclude a file to make them happy with the file, a stupid image or something like that.

    Kind regards, Jef

    My site can be found at http://jefpatat.freefronthost.com for the moment the robots.txt is still empty
  2. #2
  3. http://tinyurl.com/cz56g
    SEO Chat Mastermind (5000+ posts)

    Join Date
    Sep 2004
    Location
    D0RDRECHT NL
    Posts
    6,063
    Rep Power
    27
    Hi Jef,

    All three methods are valid and have the same effect, although only method 2 and 3 get rid of the 404 errors in your logs. Choose any method you like...
    ...please help me w/ the real Redscowl Bluesingsky...how2 check backlinks...now postin' @ SEO Refugee ...
    <`)~ LOL now that I finally have a paypal account, I'm charging 19,- for SEO advice via PM. Seriously...
  4. #3
  5. No Profile Picture
    Contributing User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    Jan 2005
    Posts
    59
    Rep Power
    10
    I've read somewhere that an empty robots.txt file is not valid, which is true if you check it in a validator. Won't this harm? Don't robots hate invalid robots.txt?
  6. #4
  7. No Profile Picture
    Contributing User
    SEO Chat Adventurer (500 - 999 posts)

    Join Date
    Sep 2004
    Posts
    758
    Rep Power
    11
    The most safe method is the number 3, that is a robots.txt file containing just:

    User-agent: *
    Disallow:

    This way, it will not generate a 404 (not found) error and it will give spiders full access to the site.
    Please, please, help tsunami survivors making a donation: UNICEF - Red Cross
  8. #5
  9. No Profile Picture
    Registered User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    Dec 2012
    Location
    banglore
    Posts
    3
    Rep Power
    0
    robots usefull to seo how it is u can install robotics is automatically used

    Comments on this post

    • Test-ok : Learn English if you want to answer questions.
    • tstolber : This thread is almost 8 years old and your answer doesn't make sense

Similar Threads

  1. A question about robots.txt and links
    By weee in forum Search Engine Optimization
    Replies: 2
    Last Post: Jan 10th, 2005, 02:55 PM
  2. robots.txt question
    By jngabi in forum SEO Help (General Chat)
    Replies: 3
    Last Post: Sep 21st, 2004, 04:00 AM
  3. A quick question on robots.txt
    By cyberil in forum BING/Yahoo Search Optimization
    Replies: 1
    Last Post: Feb 19th, 2004, 01:29 PM

IMN logo majestic logo threadwatch logo seochat tools logo