#1
  1. Contributing User
    SEO Chat Discoverer (100 - 499 posts)

    Join Date
    Sep 2003
    Posts
    144
    Rep Power
    16

    want my 404 trap to be spidered


    i have set up a site that has a directory structure like dmoz.org. Im using a 404 error php script to output code based on the url entered. eg. www.123.com/cars/ or www.123.com/bikes/ . (no static html pages exist)
    it works fine from a user standpoint, but i dont think bots will actually be able spider the 404.

    the reason I say this is because i use poodle predictor ( http://www.gritechnologies.com/tools/spider.go )and for each subdirectory it was returning a 404 , rather than the page and data based on the url. (where as it is fine navigating as a user through the site)

    can anyone suggest a step ive missed in setting this up?
  2. #2
  3. iTamer
    SEO Chat Discoverer (100 - 499 posts)

    Join Date
    Apr 2003
    Location
    Auckland, New Zealand
    Posts
    170
    Rep Power
    17
    you need this at the top of your page

    Header("HTTP/1.1 200 OK");

    Now, I do the same sort of thing at one of my sites but I also have a genuine missing page and I leave that as a 404 to stop the spiders from constantly hitting it.

    I also use Xenu to test my pages. http://home.snafu.de/tilman/xenulink.html
  4. #3
  5. Contributing User
    SEO Chat Discoverer (100 - 499 posts)

    Join Date
    Sep 2003
    Posts
    144
    Rep Power
    16
    cheers

Similar Threads

  1. After being spidered....
    By neil_rutherford in forum Search Engine Optimization
    Replies: 1
    Last Post: Jan 29th, 2004, 11:35 PM
  2. Would iframes stop my site getting spidered, read in!!
    By Brad123 in forum Google Optimization
    Replies: 3
    Last Post: Oct 5th, 2003, 06:49 AM
  3. Still only getting home page spidered, how do I get more?
    By adrian28uk in forum Google Optimization
    Replies: 6
    Last Post: May 7th, 2003, 09:23 AM
  4. pages not spidered?
    By kruger in forum Google Optimization
    Replies: 9
    Last Post: Mar 8th, 2003, 10:13 AM
  5. GoogleBot FAQ
    By darkroom in forum Google Optimization
    Replies: 6
    Last Post: Mar 7th, 2003, 02:39 AM

IMN logo majestic logo threadwatch logo seochat tools logo