Forum: Search Engine Spiders

Search Engine Spiders forum discussing how to know which search engines are visiting your site by identifying their spiders. Share their IPs and spidering habits, and plan for them to get better SEO results.

Threads 1 to 40 of 529
Page 1 of 14 12311 ... Last
  • Jump to page:
    • Replies: 4
    • Views: 1,955
    Feb 28th, 2013, 06:58 AM
  1. Why Google will store links along with the content ?

    Started by naganna, Feb 15th, 2013 04:59 AM
    Tag List
    • Replies: 0
    • Views: 1,733
    Feb 15th, 2013, 04:59 AM
  2. Nofollow Over Use

    Started by ragecrush, May 15th, 2012 08:45 AM
    • Replies: 1
    • Views: 1,722
    Feb 13th, 2013, 02:58 AM
  3. How do I create a spider-friendly site map?

    Started by riya15, Aug 29th, 2011 06:13 AM
    • Replies: 3
    • Views: 3,222
    Feb 13th, 2013, 01:48 AM
  4. Restricted site and search engine crawling

    Started by ciclonpr, Aug 19th, 2012 09:32 AM
    • Replies: 1
    • Views: 2,677
    Feb 12th, 2013, 01:16 PM
  5. How can I create a robot.txt file using google?

    Started by suprity, Dec 3rd, 2012 08:29 PM
    • Replies: 5
    • Views: 2,157
    Feb 12th, 2013, 01:07 PM
  6. Google Indexing problem Magento Store

    Started by vickydevries, Jul 16th, 2012 04:38 AM
    • Replies: 1
    • Views: 2,575
    Feb 7th, 2013, 01:36 PM
  7. Xml sitemap submit

    Started by Sharon Lisa, Dec 3rd, 2012 03:53 AM
    • Replies: 2
    • Views: 2,044
    Feb 7th, 2013, 01:26 PM
  8. Multiple Sitemaps

    Started by jamesone, Dec 20th, 2012 02:29 PM
    • Replies: 4
    • Views: 2,424
    Jan 1st, 2013, 02:33 AM
  9. Yet another robots.txt question

    Started by Jef Patat, Jan 19th, 2005 09:35 AM
    • Replies: 4
    • Views: 2,740
    Dec 23rd, 2012, 12:53 AM
  10. Question Search engine spider crawling?

    Started by siva_lingam, Sep 28th, 2005 05:14 AM
    • Replies: 3
    • Views: 1,699
    Dec 21st, 2012, 03:47 AM
  11. Forum Tracker

    Started by mchilly, Oct 29th, 2006 11:01 PM
    • Replies: 3
    • Views: 2,093
    Dec 19th, 2012, 03:10 AM
  12. Are Mobile Redirects Harmful to SEO

    Started by thomop, Nov 22nd, 2012 05:04 AM
    • Replies: 4
    • Views: 766
    Nov 23rd, 2012, 04:42 PM
  13. Is this Robots.txt for my wordpress Blog SEO Friendly?

    Started by webhush, Apr 28th, 2009 11:39 AM
    2 Pages
    1 2
    • Replies: 21
    • Views: 5,339
    Nov 11th, 2012, 10:36 AM
  14. Indexing went down to 0

    Started by Deepali, Aug 30th, 2012 02:20 AM
    • Replies: 4
    • Views: 912
    Oct 29th, 2012, 12:26 PM
  15. Content examination in lynx text base browser

    Started by seodip, Jul 24th, 2012 06:54 AM
    • Replies: 0
    • Views: 1,052
    Jul 24th, 2012, 06:54 AM
  16. Thumbs up Sitemap

    Started by Stela Wilson, May 1st, 2012 07:33 PM
    • Replies: 6
    • Views: 1,559
    Jul 20th, 2012, 01:23 AM
  17. What is search engine spider?

    Started by olloboy, Jul 6th, 2012 11:26 PM
    • Replies: 3
    • Views: 1,181
    Jul 17th, 2012, 06:27 AM
  18. Google Robot

    Started by im.amideb, Jul 11th, 2012 07:52 AM
    • Replies: 2
    • Views: 1,062
    Jul 13th, 2012, 05:54 AM
  19. What is Spiders and How They Effect the website

    Started by ilmkidunya, Jan 17th, 2012 12:01 AM
    • Replies: 4
    • Views: 2,036
    Jun 12th, 2012, 04:38 AM
  20. Use of robot.txt!!

    Started by Seo picker, Apr 4th, 2012 05:35 AM
    • Replies: 3
    • Views: 1,472
    May 29th, 2012, 10:30 AM
  21. What is sitemap?

    Started by robert4u, May 29th, 2012 02:46 AM
    • Replies: 1
    • Views: 1,019
    May 29th, 2012, 03:43 AM
  22. Help desk software

    Started by boomiurandai, Feb 23rd, 2008 07:08 AM
    • Replies: 2
    • Views: 2,314
    May 21st, 2012, 10:11 PM
  23. DoCoMo/1.0/P502i/c10 (Google CHTML Proxy/1.0)

    Started by wirelesscove, May 1st, 2003 04:04 PM
    • Replies: 5
    • Views: 5,674
    May 8th, 2012, 02:17 AM
  24. Pageid=13 - robots.txt help!

    Started by m_grindy, May 3rd, 2012 09:48 AM
    • Replies: 2
    • Views: 1,161
    May 7th, 2012, 08:42 PM
  25. Help! Site disappeared off Cache

    Started by ajithlaxman, Aug 18th, 2010 12:46 AM
    • Replies: 5
    • Views: 1,943
    May 1st, 2012, 12:59 PM
  26. Sitemap.xml

    Started by wgrasty, Feb 13th, 2012 08:08 AM
    • Replies: 6
    • Views: 1,781
    Apr 25th, 2012, 10:43 AM
  27. Lightbulb My Website Ranking Dropping in Google

    Started by Devbhushan, Sep 16th, 2010 05:14 AM
    • Replies: 7
    • Views: 2,607
    Mar 29th, 2012, 06:29 AM
  28. How does Google Crawl Geo Dynamic Content?

    Started by debeenus, Mar 21st, 2012 09:25 AM
    • Replies: 0
    • Views: 1,379
    Mar 21st, 2012, 09:25 AM
  29. Parsing php in .html files & seo

    Started by Sienna, Feb 29th, 2012 12:13 AM
    • Replies: 1
    • Views: 1,645
    Feb 29th, 2012, 01:33 AM
  30. Regional access restrictions and search indexing

    Started by Guest123, Jan 31st, 2012 06:08 AM
    • Replies: 1
    • Views: 1,290
    Feb 3rd, 2012, 10:16 AM
  31. Smile Seo strategies

    Started by nitish123, Dec 24th, 2011 03:33 AM
    • Replies: 2
    • Views: 1,633
    Jan 25th, 2012, 08:37 AM
  32. How to get googlebot to visit

    Started by cnile, Dec 29th, 2003 12:33 PM
    2 Pages
    1 2
    • Replies: 24
    • Views: 29,189
    Jan 18th, 2012, 04:49 AM
  33. 2 questions...driving me crazy

    Started by mechele7, Dec 26th, 2011 08:51 AM
    • Replies: 0
    • Views: 1,394
    Dec 26th, 2011, 08:51 AM
  34. Googlebot requesting googlelongnumber.html ??

    Started by Savage, Nov 8th, 2011 12:53 PM
    • Replies: 3
    • Views: 1,435
    Nov 9th, 2011, 01:04 PM
  35. TurnitinBot

    Started by st, Aug 12th, 2003 08:08 AM
    • Replies: 4
    • Views: 4,264
    Nov 2nd, 2011, 02:22 AM
  36. Crawling problem

    Started by stunning boy, Sep 3rd, 2011 05:26 AM
    • Replies: 3
    • Views: 1,991
    Oct 28th, 2011, 06:25 PM
  37. Question How to stop search options from being indexed

    Started by dan1el, Mar 11th, 2011 03:43 AM
    • Replies: 5
    • Views: 1,808
    Oct 19th, 2011, 03:49 PM
  38. Can a php path be indexed?

    Started by michaelw, Aug 9th, 2011 12:50 PM
    • Replies: 1
    • Views: 1,504
    Sep 15th, 2011, 01:19 AM
  39. Problem with robots on dynamic page.

    Started by dan32, May 2nd, 2011 06:46 PM
    • Replies: 4
    • Views: 2,283
    Sep 8th, 2011, 05:24 PM
Page 1 of 14 12311 ... Last
  • Jump to page:

Forum Information and Options

Users Browsing this Forum

There are currently 2 users browsing this forum. (0 members & 2 guests)

Thread Display Options

Use this control to limit the display of threads to those newer than the specified time frame.

Allows you to choose the data by which the thread list will be sorted.

Order threads in...

Note: when sorting by date, 'descending order' will show the newest results first.


IMN logo majestic logo threadwatch logo seochat tools logo