1. No Profile Picture
    Contributing User
    SEO Chat Discoverer (100 - 499 posts)

    Join Date
    Feb 2017
    Rep Power

    Perfectly Optimized Robots.txt

    Curious what everyone's robots.txt file looks like. Personally, I don't really disallow anything (see below). Any opinions on this?

    User-agent: *
    Disallow: /wp-admin/
    Disallow: /wp-includes/
    Sitemap: http://website.com/sitemap.xml
  2. #2
  3. Contributing User
    SEO Chat Discoverer (100 - 499 posts)

    Join Date
    Jun 2012
    Rep Power
    That looks good, from my point of view
    "...using this: sages act out of constructs but they do not rely on them"
  4. #3
  5. No Profile Picture
    Registered User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    Mar 2014
    Rep Power
    You really don't require to block any directory in your robots.txt file, because Google don't crawl password protected directory like wp-admin, they crawl just links.

    The best thing you should try is use Fetch and Render tool from Google search console, and test out your site, and checkout how Googlebot view your site. If you think they render properly then you're good to go. But Personally I don't believe in blocking any directory on wordpress. Checkout yoast robots.txt, they are pretty good in WP SEO, and they also don't block it. Rest of Wordpress article on robots.txt are BS.
  6. #4
  7. Contributing User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    Jun 2017
    Ghaziabad, New Delhi
    Rep Power
    There is no need to provide the reference of sitemap in the robots.txt file.
    I don't think there is any issue in the code you written in your thread, that seems good. You don't disallow anything that's your way to work and someone who disallows something may be he/she do not want to allow crawlers to crawl those directories or web pages or may be nothing is important in those directories.
    But it is a good point for those who do not about the robots.txt file and do the copy paste job for their blogs or website.

Similar Threads

  1. Someone has screwed me perfectly!
    By Novice in forum Search Engine Optimization
    Replies: 10
    Last Post: May 30th, 2013, 06:53 AM
  2. 301 redirect perfectly...Then
    By kalwinmack in forum Search Engine Optimization
    Replies: 4
    Last Post: Jul 28th, 2009, 02:34 PM
  3. Scientists: 115-year-old's brain worked perfectly (AP)
    By RSS_News_User in forum Science News
    Replies: 0
    Last Post: Jun 13th, 2008, 12:03 PM
  4. Is this considered perfectly ok?
    By benjamieson in forum Google Optimization
    Replies: 1
    Last Post: Dec 6th, 2005, 08:23 AM
  5. Let's see the perfectly optimized site
    By Enginyr in forum Google Optimization
    Replies: 31
    Last Post: Jun 1st, 2004, 10:32 AM

IMN logo majestic logo threadwatch logo seochat tools logo