#1
  1. No Profile Picture
    Registered User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    May 2013
    Posts
    11
    Rep Power
    0

    Question Which is better Robots.txt file for HTML website


    Hi Every1

    My website in HTML format 15-20 Pages
    Please suggest robot.txt for my website...
    Please check following A or B so i can put in my website

    A:
    User-agent: *
    Allow: /
    Sitemap: Sitemap URL

    B:
    User-agent: *
    Allow: /


    A is with sitemap url and B is without sitemap url and please suggest more advice as per urs expert point of views...


    Thank You
  2. #2
  3. SEO Insultant
    SEO Chat Mastermind (5000+ posts)

    Join Date
    Oct 2005
    Location
    sharing a room with my ego
    Posts
    5,568
    Rep Power
    1773
    This is a moot point.

    What are you hoping to achieve?
  4. #3
  5. Seo Control Panel
    SEO Chat Discoverer (100 - 499 posts)

    Join Date
    Feb 2014
    Location
    India
    Posts
    148
    Rep Power
    9
    Second one is correct one. Sitemap can be directly submitted to google using webmaster tools.
  6. #4
  7. SEO Insultant
    SEO Chat Mastermind (5000+ posts)

    Join Date
    Oct 2005
    Location
    sharing a room with my ego
    Posts
    5,568
    Rep Power
    1773
    Um no. None of them are 'correct' because they are both non-standard.

    My question remains: what are you hoping to achieve?
  8. #5
  9. Seo Control Panel
    SEO Chat Discoverer (100 - 499 posts)

    Join Date
    Feb 2014
    Location
    India
    Posts
    148
    Rep Power
    9
    User-Agent: *
    Allow: /
    It is the default robots.txt provided by google webmaster tools.
  10. #6
  11. SEO Insultant
    SEO Chat Mastermind (5000+ posts)

    Join Date
    Oct 2005
    Location
    sharing a room with my ego
    Posts
    5,568
    Rep Power
    1773
    It is non-standard, and also it is the equivalent of having no robots.txt file at all.
  12. #7
  13. Seo Control Panel
    SEO Chat Discoverer (100 - 499 posts)

    Join Date
    Feb 2014
    Location
    India
    Posts
    148
    Rep Power
    9
    Okay. Please tell me which is the standard format of robots.txt? it will help all.
  14. #8
  15. SEO Insultant
    SEO Chat Mastermind (5000+ posts)

    Join Date
    Oct 2005
    Location
    sharing a room with my ego
    Posts
    5,568
    Rep Power
    1773
    www.robotstxt.org
  16. #9
  17. No Profile Picture
    Newbie
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    Feb 2014
    Posts
    1
    Rep Power
    0
    We create site,we must add robot.txt file or not. without it site can run?
  18. #10
  19. SEO Insultant
    SEO Chat Mastermind (5000+ posts)

    Join Date
    Oct 2005
    Location
    sharing a room with my ego
    Posts
    5,568
    Rep Power
    1773
    There is no need for this file unless you plan to make use of it to hide certain files or folders
  20. #11
  21. Editor-in-Chief
    SEO Chat Adventurer (500 - 999 posts)

    Join Date
    Sep 2008
    Location
    South Florida
    Posts
    717
    Rep Power
    399
    Moved to appropriate section.

Similar Threads

  1. What can use website robots.txt file
    By sanjan in forum Google Optimization
    Replies: 3
    Last Post: Aug 8th, 2012, 08:27 PM
  2. Dynamic Website - robots.txt file Help Required
    By peterson in forum SEO Help (General Chat)
    Replies: 4
    Last Post: Mar 10th, 2011, 06:07 AM
  3. HTML and Robots.txt
    By Elven6 in forum Web Design, Coding and Programming
    Replies: 4
    Last Post: Nov 8th, 2009, 05:16 PM
  4. Robots.txt file
    By kilburn2000 in forum Google Optimization
    Replies: 1
    Last Post: Sep 28th, 2009, 06:00 AM
  5. Robots.txt File
    By Hilary in forum Google Optimization
    Replies: 3
    Last Post: Jan 16th, 2009, 11:58 AM

IMN logo majestic logo threadwatch logo seochat tools logo