#1
  1. No Profile Picture
    Link building genie.
    SEO Chat Discoverer (100 - 499 posts)

    Join Date
    Apr 2005
    Posts
    216
    Rep Power
    14

    Robots disallow question


    I was just checking my robots.txt on a site that my friend has created and I noticed that he has a LOT of disallows.

    Some of them make sense.. e.g

    User-agent: Mediapartners-Google*
    Disallow:
    User-agent: NDSpider
    User-agent: *
    Disallow: /admin/

    I can understand why he would not want the spider going through the admin section.

    Disallow: /includes/ is this wise?

    Disallow: /themes/ ?????? what is google going to see if the theme of the site is blocked?

    Disallow: /blocks/
    Disallow: /modules/


    There are more as well. Has he made a mistake here? is he trying to block everything except the text? will the text even show on a phpnuke site if he blocks the themes and includes?

    Weird
  2. #2
  3. No Profile Picture
    Link building genie.
    SEO Chat Discoverer (100 - 499 posts)

    Join Date
    Apr 2005
    Posts
    216
    Rep Power
    14
    EDIT: Found a tool.

    p.s is there a tool that will crawl your site and show you what your site would look like to the google bot? (But a tool that follows the rules from the robots text)
    Last edited by deano6410; Aug 4th, 2006 at 06:14 AM.

Similar Threads

  1. Must use this SEO tips!
    By simple-biz in forum Search Engine Optimization
    Replies: 323
    Last Post: Jan 7th, 2008, 11:23 AM
  2. BIG Trouble
    By huntz in forum Google Optimization
    Replies: 15
    Last Post: Nov 1st, 2005, 09:44 AM
  3. robots.txt dissallow ip's
    By cozza in forum Google Optimization
    Replies: 10
    Last Post: Mar 24th, 2005, 05:34 PM
  4. Replies: 19
    Last Post: Oct 24th, 2004, 10:40 PM

IMN logo majestic logo threadwatch logo seochat tools logo