Thread: Robot no index

    #1
  1. No Profile Picture
    Contributing User
    SEO Chat Good Citizen (1000 - 1499 posts)

    Join Date
    Jun 2006
    Location
    England
    Posts
    1,208
    Rep Power
    255

    Robot no index


    <html xmlns="http://www.w3.org/1999/xhtml"', $context['right_to_left'] ? ' dir="rtl"' : '', '><head>
    <meta http-equiv="Content-Type" content="text/html; charset=', $context['character_set'], '" />
    <meta name="description" content="', $context['page_title'], '" />', empty($context['robot_no_index']) ? '' : '
    <meta name="robots" content="noindex" />', '
    <meta name="keywords" content="PHP, MySQL, bulletin, board, free, open, source, smf, simple, machines, forum" />

    I found this bit of code in my forum.

    http://www.whitbyseaanglers.co.uk/forum/index.php*removethisandthestar

    I dont like the look of the words robot no index

    Does this mean what I think it does ?

    Should I remove that code ??

    Would anyone be kind enough to look at my forum and tell me if the pages are blocked from the robots ?

    Many thanks - Glenn
    Last edited by Glenn Kilpatric; Feb 2nd, 2007 at 02:09 PM.
  2. #2
  3. SEO Insultant
    SEO Chat Mastermind (5000+ posts)

    Join Date
    Oct 2005
    Location
    sharing a room with my ego
    Posts
    6,470
    Rep Power
    3420
    It depends on what you want.

    It means that the search engines should not index THIS particular page with THIS particular URL. But if you have a similar page with almost the same content and with ANOTHER URL - but which does not have the robots meta - then it's likely that the search engines will index THAT one instead. For example, you could want the bots to spider your archive pages instead of your dynamic forum pages....

    If all that's not the case, then remove the line (or comment it out) and test without it.

    But erm.... who put that in your code in the first place One of the admin settings?
    ... Hey you! FREE SEO report card ... I really my hosting company ...
  4. #3
  5. No Profile Picture
    Contributing User
    SEO Chat Good Citizen (1000 - 1499 posts)

    Join Date
    Jun 2006
    Location
    England
    Posts
    1,208
    Rep Power
    255
    Hi dzine.

    The company that wrote the forum software put it in the code. I'm a bit worried this means the bots wont read my forum.

    Is there a way to test if the bots can read the forum topics?
  6. #4
  7. SEO Insultant
    SEO Chat Mastermind (5000+ posts)

    Join Date
    Oct 2005
    Location
    sharing a room with my ego
    Posts
    6,470
    Rep Power
    3420
    Well, if you have access to your logs, you may find lots of bot visits to your robots.txt file - and stopping there.

    Also, if the forum has been up for some time and if you have pointed a couple of external links to it -- and it still doesn't show up on Google, Yahoo and Live.com when you use a site: search -- then I think there's only one conclusion.

    Live.com shows something, but when I looked at the cache of that page, the "robots" meta wasn't there. I guess it's not in the forum home page template, is it?


    But: can't you just take it out of the script(s)?
    I'm 99.9% sure there is a setting somewhere on your SMF admin panel that controls this (by setting 'robot_no_index'). I don't know exactly where, because I don't have a forum running the same version scripts as yours.



    Last edited by dzine; Feb 3rd, 2007 at 03:00 AM.
  8. #5
  9. SEO Insultant
    SEO Chat Mastermind (5000+ posts)

    Join Date
    Oct 2005
    Location
    sharing a room with my ego
    Posts
    6,470
    Rep Power
    3420
    Follow-up:

    I have found out that SMF automatically adds the noindex bit on pages that have message IDs etc. in the url. On your forum, that means that URLs like

    http://www.whitbyseaanglers.co.uk/forum/index.php/topic,154.0.html

    will be spidered, but URLs like

    http://www.whitbyseaanglers.co.uk/forum/index.php/topic,154.msg429.html#new

    won't. So if a bot visits your homepage and follows a link to one of the boards, then it may also follow a regular link to any of the threads on that board. I just checked it and if the bots follow the "normal" flow of things on your forum, they are NOT blocked.

    Next step: blogging about your forum and linking to some of its inner pages (using the correct URL, of course)

    Good luck.
  10. #6
  11. No Profile Picture
    Contributing User
    SEO Chat Good Citizen (1000 - 1499 posts)

    Join Date
    Jun 2006
    Location
    England
    Posts
    1,208
    Rep Power
    255
    Thanks dzine,

    Much appreciated mate. I have a blog so will add some links there today.

    Many thanks for your help, it is really apreciated.

    Kindest regards - Glenn

Similar Threads

  1. What is best robot txt coding to insure spiders index all pages?
    By EandA in forum Search Engine Optimization
    Replies: 8
    Last Post: Dec 17th, 2008, 04:10 PM
  2. Can Google index Flash?
    By AltherrWeb in forum Google Optimization
    Replies: 18
    Last Post: Jul 23rd, 2005, 12:42 PM
  3. Link to index (PR5) or the forum (PR6)
    By iKwak in forum Google Optimization
    Replies: 6
    Last Post: Aug 26th, 2004, 01:30 PM
  4. My index page is not in the index... but my internal pages are!
    By mariobox in forum Google Optimization
    Replies: 0
    Last Post: Sep 15th, 2003, 03:37 PM

IMN logo majestic logo threadwatch logo seochat tools logo