#1
  1. No Profile Picture
    Registered User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    May 2006
    Posts
    7
    Rep Power
    0

    Crawled but not indexed?


    I just recently started doing google sitemaps. In the Diagnostic section under the HTTP errors there are some 404 errors from May 7, 10, 11 and 12th (I have since fixed these problems). Can I assume this means that I am being crawled by google semi-regularly?

    And - why if i'm getting crawled so often are no pages except my homepage indexed (which has been for the 2 months my site has been live)?

    Also - since my homepage (Dynamic: www.myteespot.com) goes straight to a dynamic page, i'm using my robots.txt file to only allow google to crawl my static site (Static: www.myteespot.com/catalog/index.html). I am since getting "We can't currently access your home page because of a robots.txt restriction." under the 'Potential Indexing Problems' in the Sitemaps Diagnostics summary. I don't care if my dynamic homepage is indexed - but will this problem affect indexing/pagerank of my static pages?

    Any answers/feedback/hints/suggestions would be greatly apreciated.

    Thanks.
    Last edited by ClickyB; May 15th, 2006 at 09:49 AM. Reason: Removed live links
  2. #2
  3. Contributing User
    SEO Chat Discoverer (100 - 499 posts)

    Join Date
    Nov 2005
    Location
    On da Net
    Posts
    195
    Rep Power
    14
    Originally Posted by MyTeeSpot
    I just recently started doing google sitemaps. In the Diagnostic section under the HTTP errors there are some 404 errors from May 7, 10, 11 and 12th (I have since fixed these problems). Can I assume this means that I am being crawled by google semi-regularly?

    And - why if i'm getting crawled so often are no pages except my homepage indexed (which has been for the 2 months my site has been live)?

    Also - since my homepage (Dynamic: www.myteespot.com) goes straight to a dynamic page, i'm using my robots.txt file to only allow google to crawl my static site (Static: www.myteespot.com/catalog/index.html). I am since getting "We can't currently access your home page because of a robots.txt restriction." under the 'Potential Indexing Problems' in the Sitemaps Diagnostics summary. I don't care if my dynamic homepage is indexed - but will this problem affect indexing/pagerank of my static pages?

    Any answers/feedback/hints/suggestions would be greatly apreciated.

    Thanks.
    You have errors in your robots.txt file since robots.txt file shouldn't say what files/directories you want to allow but just what you want to disallow.

    Check this url for more information

    http://www.robotstxt.org/wc/norobots.html

Similar Threads

  1. My site is de indexed today ......
    By Weblink in forum BING/Yahoo Search Optimization
    Replies: 4
    Last Post: Jan 17th, 2006, 08:09 AM
  2. Crawled to Ranked Timeframes
    By dakman in forum Google Optimization
    Replies: 0
    Last Post: Oct 4th, 2005, 01:49 AM
  3. Site crawled 5 days in a row but new pages not indexed
    By maxscan in forum Google Optimization
    Replies: 8
    Last Post: Sep 28th, 2004, 03:39 PM
  4. What happens to pages crawled not showing in the SERPS?
    By Johno in forum Google Optimization
    Replies: 3
    Last Post: Jul 23rd, 2003, 12:28 PM

IMN logo majestic logo threadwatch logo seochat tools logo