#1
  1. No Profile Picture
    Contributing User
    SEO Chat Discoverer (100 - 499 posts)

    Join Date
    May 2003
    Posts
    282
    Rep Power
    16

    Question About Giving Slurp Instructions


    Is there a way I command Slurp specifically to crawl my entire site?

    For instance, would the following code work/help?
    <META NAME="Slurp" CONTENT="INDEX, FOLLOW">
    <meta name="Slurp" content="ALL">

    I already have the following on my home page:
    <meta name="ROBOTS" content="index,follow">
    <meta name="RATING" content="General">
    <meta name="ROBOTS" content="ALL">

    Slurp does not seem to want to crawl my entire site however, hence my thought process about adding instructions specificly for slurp.

    Comments/Suggestions?
  2. #2
  3. Around...The
    SEO Chat Discoverer (100 - 499 posts)

    Join Date
    Jun 2004
    Location
    Amsterdam
    Posts
    456
    Rep Power
    16
    No, unfortunately that doesn't help.

    Those tags are only meant to give spiders instructions to NOT search/index your site, by default they search/index everything they encounter.

    Mind you it can take a while before Slurp indexes your entire site, just try to get a few backlinks deep into your site, that should help a bit. And offcourse some high quality backlinks don't hurt either.

    Just heard that Slurp is coming by faster and deeper when your search results are climbing and this is exactly what is happening to me now. But this offcouse is a viscious circle......

    So just try to get links deep into your site from high quality websites. Slurp will then start indexing your site from that location and therefore should be able to find more/different links.
  4. #3
  5. No Profile Picture
    SEO Chat Skiller (1500 - 1999 posts)

    Join Date
    Jul 2004
    Location
    St. James Gate
    Posts
    1,988
    Rep Power
    18
    Don't forget to also develop a good sitemap one click off the homepage. You can then get a couple of quality links to to the sitemap to further speed up the slurp.
  6. #4
  7. No Profile Picture
    Contributing User
    SEO Chat Discoverer (100 - 499 posts)

    Join Date
    May 2003
    Posts
    282
    Rep Power
    16
    I've got backlinks (Google PR6) and a sitemap linked to from every page. Any ideas on what else I can do to get Slurp to crawl the entire site?
  8. #5
  9. No Profile Picture
    SEO Chat Skiller (1500 - 1999 posts)

    Join Date
    Jul 2004
    Location
    St. James Gate
    Posts
    1,988
    Rep Power
    18
    Originally Posted by Sweat
    I've got backlinks (Google PR6) and a sitemap linked to from every page. Any ideas on what else I can do to get Slurp to crawl the entire site?
    If you really want to use your robots file you could do this:


    Code:
     
    <meta name="ROBOTS" content="index,follow">
    <meta name="RATING" content="General">
    <meta name="ROBOTS" content="ALL">
    <META NAME="Slurp" CONTENT="INDEX, FOLLOW">
    <meta name="Slurp" content="ALL">
    <meta name="Bribe" content="$20">
  10. #6
  11. No Profile Picture
    Contributing User
    SEO Chat Discoverer (100 - 499 posts)

    Join Date
    May 2003
    Posts
    282
    Rep Power
    16
    Originally Posted by Jasontnyc
    If you really want to use your robots file you could do this:


    Code:
     
    <meta name="ROBOTS" content="index,follow">
    <meta name="RATING" content="General">
    <meta name="ROBOTS" content="ALL">
    <META NAME="Slurp" CONTENT="INDEX, FOLLOW">
    <meta name="Slurp" content="ALL">
    <meta name="Bribe" content="$20">

    If increase "Bribe" to $50, will it work better.
  12. #7
  13. Resident Black Hat
    SEO Chat Good Citizen (1000 - 1499 posts)

    Join Date
    Aug 2004
    Location
    Woodstock, GA
    Posts
    1,437
    Rep Power
    20
    Heck, it could be like that simplied tax form:

    1. How much did you make last year?

    2. Send it in.

    G-Man
  14. #8
  15. Banned
    SEO Chat Discoverer (100 - 499 posts)

    Join Date
    Sep 2004
    Location
    Vancouver, Calgary , Ontario
    Posts
    259
    Rep Power
    0
    Originally Posted by tfbpa
    No, unfortunately that doesn't help.

    Those tags are only meant to give spiders instructions to NOT search/index your site, by default they search/index everything they encounter.

    Mind you it can take a while before Slurp indexes your entire site, just try to get a few backlinks deep into your site, that should help a bit. And offcourse some high quality backlinks don't hurt either.

    Just heard that Slurp is coming by faster and deeper when your search results are climbing and this is exactly what is happening to me now. But this offcouse is a viscious circle......

    So just try to get links deep into your site from high quality websites. Slurp will then start indexing your site from that location and therefore should be able to find more/different links.
    Well slurp finally found my sitemap yet still doesn't seem to want to index my entire site. I have many deep links for multiple anchors in the link command yet I have a hard time getting more than 15 pages indexed. They keep going up weekly one or two but is their anyway to get them to index every link on my sitemap?
  16. #9
  17. Around...The
    SEO Chat Discoverer (100 - 499 posts)

    Join Date
    Jun 2004
    Location
    Amsterdam
    Posts
    456
    Rep Power
    16
    Sorry, have no great solutions...

    The only thing I can add is that you have to be sure that the PAGES linking deep into your site ARE indexed by Yahoo. Otherwise there is offcourse no point in getting those links. Well, there is no immediate benefit, over time they most probably WILL get indexed but if you want immediate results check if those pages you get links from are indexed. Just type in the url in Yahoo to check.
  18. #10
  19. No Profile Picture
    Contributing User
    SEO Chat Discoverer (100 - 499 posts)

    Join Date
    Oct 2004
    Posts
    122
    Rep Power
    15
    Are site maps critical tpo getting large sites indexed?
  20. #11
  21. Banned
    SEO Chat Discoverer (100 - 499 posts)

    Join Date
    Sep 2004
    Location
    Vancouver, Calgary , Ontario
    Posts
    259
    Rep Power
    0
    Originally Posted by healthyjoe
    Are site maps critical tpo getting large sites indexed?
    actually I think that the search engines recognize sitemap.html or site-map.html or some similar variety and treat them a certain way. I should say I hope they do anyway.

    But think of it as making good sense, providing a page with every link on your site at least gives the search engine the opportunity to index your entire site.

    I'd imagine creating a few backlinks to your sitemap would help as well.

    * important note - make sure to provide some text on your site map and not just links I heard that the search engines don't link pages with just html links and no text as well a opening paragraph and closing paragraph just makes good sense to your visitors and makes your links worth more I am sure
  22. #12
  23. No Profile Picture
    SEO Chat Skiller (1500 - 1999 posts)

    Join Date
    Jul 2004
    Location
    St. James Gate
    Posts
    1,988
    Rep Power
    18
    Originally Posted by bargainfinds
    * important note - make sure to provide some text on your site map and not just links I heard that the search engines don't link pages with just html links and no text as well a opening paragraph and closing paragraph just makes good sense to your visitors and makes your links worth more I am sure
    Do you have any proof of this? I have never had any problems getting a sitemap indexed quick along with all the pages linked to without opening and closing paragraphs.

    Recently I had a forum that was only getting indexed a few pages at a time so I built a site map with links to each individual post (some forums will do this for you) and within two weeks Google indexed around 300 pages (about what I had between the two sitemaps).

    I would be interested to hear what the paragraphs add.

Similar Threads

  1. Slurp Outta Control?
    By shawna in forum BING/Yahoo Search Optimization
    Replies: 8
    Last Post: Oct 21st, 2004, 09:41 AM
  2. Question on Yahoo spiders
    By whitey in forum BING/Yahoo Search Optimization
    Replies: 1
    Last Post: May 25th, 2004, 06:52 PM
  3. Just a quick question
    By didjital1 in forum Google Optimization
    Replies: 1
    Last Post: Aug 10th, 2003, 12:03 PM

IMN logo majestic logo threadwatch logo seochat tools logo