Thread: Robot.txt file

    #1
  1. No Profile Picture
    Newbie
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    Jun 2017
    Posts
    2
    Rep Power
    0

    Robot.txt file


    hello guys,

    What is robots.txt file in seo..hows robot.txt file worked in seo.explain in details?
  2. #2
  3. Dinosaur
    SEO Chat High Scholar (3500 - 3999 posts)

    Join Date
    Jun 2011
    Location
    UK
    Posts
    3,672
    Rep Power
    6289
    blocks most friendly bots such as google.
    Others don't take any notice of robots.txt file.

    It has no benefit as far as SEO is concerned except for blocking some pages from being indexed.
    Owner of Page Explorer the page onsite SEO checker
    Useful Tools: Site Statistics: SEM Rush | Site Crawler: Screaming Frog
  4. #3
  5. No Profile Picture
    Banned
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    Jun 2017
    Posts
    3
    Rep Power
    0
    robots.txt is a text file webmasters create to instruct web robots (typically search engine robots) how to crawl pages on their website.

    The robots.txt file is part of the the robots exclusion protocol (REP), a group of web standards that regulate how robots crawl the web, access and index content, and serve that content up to users.

    The Basic Format of robots.txt File is :


    User-agent: [user-agent name]
    Disallow: [URL string not to be crawled]
    Last edited by dzine; Jun 17th, 2017 at 06:55 AM. Reason: added one final 's' to make things complete :--)

Similar Threads

  1. what is robot.txt file?
    By ronaldotson45 in forum Search Engine Optimization
    Replies: 1
    Last Post: Dec 14th, 2012, 01:50 AM
  2. Robot.txt file
    By nikolayjs in forum Search Engine Optimization
    Replies: 3
    Last Post: Jan 9th, 2012, 09:49 PM
  3. What is robot.txt file
    By chandrakant in forum Search Engine Optimization
    Replies: 7
    Last Post: Jan 8th, 2010, 07:36 AM
  4. Robot.txt file
    By ali786seo in forum Search Engine Optimization
    Replies: 3
    Last Post: Dec 18th, 2007, 04:01 AM
  5. Robot.txt file
    By search_pink in forum Google Optimization
    Replies: 4
    Last Post: Aug 20th, 2007, 11:15 AM

IMN logo majestic logo threadwatch logo seochat tools logo