#1
  1. No Profile Picture
    Registered User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    Jun 2011
    Posts
    3
    Rep Power
    0

    Collect urls that google and other search engines show


    Could you help me find a software or a web service to collect urls that search engines show as their search results.

    In other words, I have a list of key words. Say, 1000 words. I'd like to enter this list into a software, define a search engine or engines and set parsing depth (how many pages of search results should be inspected). As a result I'd like to see the links that were shown by the search engines.

    I'm interested only in links.

    Thank you in advance.
  2. #2
  3. Moderator
    SEO Chat Good Citizen (1000 - 1499 posts)

    Join Date
    Sep 2016
    Location
    United Kingdom
    Posts
    1,334
    Rep Power
    1205
    It's essentially what SERP checkers do.

    Can I ask, what is the reason you're wanting this? It might be a better idea to do a custom build, it's not something which is too difficult to build.
  4. #3
  5. No Profile Picture
    Registered User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    Jun 2011
    Posts
    3
    Rep Power
    0
    Originally Posted by matt1966
    It's essentially what SERP checkers do.

    Can I ask, what is the reason you're wanting this?
    I can collect key words from there. Not the meta keywords, but the ones with which the site is visible in search engines.

    Originally Posted by matt1966
    It might be a better idea to do a custom build, it's not something which is too difficult to build.
    I have written some software for this. But I can program only a single thread application. It is just too slow for me.
    And such programs are not simple. If you are a programmer, we can discuss it via private messages if you wish.
    Not an order for you, but what is diffucult in writing such a script.

    Well, there must be such software somewhere. I've googled for serp checkers. I'll have to experiment a bit. But so far I can't see anything which is capable to parse a list of 1000 key words.

    Could you suggest one?
  6. #4
  7. No Profile Picture
    Moderator
    SEO Chat Super Hero (2500 - 2999 posts)

    Join Date
    Sep 2016
    Location
    USA
    Posts
    2,596
    Rep Power
    3213
    Under full automation you can safely scrape approximately 10 keywords and get the first 100 results every hour from a single IP and do that 24 hours a day indefinitely and not alert Google you are scrapping them. You just can't bang Google with a search and then hit all 10 pages needed with out triggering alarm bells back to back unless you make it look like a human is doing it. Everything has to be staggered.

    Just so you understand how many searches your are executing. That's 240 keywords with each returning 100 results per day under full automation from a single IP address. That is 24,000 serp results you are scraping and recording. Larger than you may have thought. The more IP address you have the faster you can scrap. You can purchase proxies very cheap so lets say you have 10 proxies, that equates to 24 hr x 100 keywords x 100 results which equals 240,000 results per day.

    Most folks wanting to scrap Google do not understand to get the full 100 serp results you need to execute 1 search then you have to go thru the first 10 pages. You try to push that limit past 10 keywords per hr per IP and you can trigger temporary bans that will block you and that is no fun. These bans can last 5 minutes to hours depending on how hard you scrape.
    You will know you have scraped Google to hard when you get this results from a search like or similar to the below.

    This is a temp ban message, just one of them.
    We're sorry...
    ... but your query looks similar to automated requests from a computer virus or spyware application.
    To protect our users, we can't process your request right now.
    We'll restore your access as quickly as possible, so try again soon. In the meantime,
    if you suspect that your computer or network has been infected, you might want to run a virus checker or spyware remover
    to make sure that your systems are free of viruses and other spurious software.
    We apologize for the inconvenience, and hope we'll see you again on Google.

    Or this, the dreaded Captcha...which can be just as troublesome.
    We're sorry...
    ... but your query looks similar to automated requests from a computer virus or spyware application.
    To protect our users, we can't process your request right now. We'll restore your access as quickly as possible,
    so try again soon. In the meantime, if you suspect that your computer or network has been infected,
    you might want to run a virus checker or spyware remover to make sure that your systems are free of viruses and other spurious software.
    If you're continually receiving this error, you may be able to resolve the problem by deleting your Google cookie and revisiting Google.
    For browser-specific instructions, please consult your browser's online support center. If your entire network is affected,
    more information is available in the Google Web Search Help Center.
    We apologize for the inconvenience, and hope we'll see you again on Google.
    To continue searching, please type the characters you see below:

    Google will even change these messages to trip you up as well. Google hates automation lol.

    Google knows that a human is only interested in page 1, the 1st 10 results, for the most part. You go asking for 100 results without pagination every time you search, Google will know you're not a human, remember that. Also 100 results returned is less accurate as well.

    If you understand how Google's search engine works you can scrap from over 100,000 locations across the globe to facilitate "Local Search". What that means is you could physically be in your home and search Google and make Google believe you are in Hong Kong or Moscow or Paris or London or any International Airport or any one of the other 100,000 locations Google supports.

    Sorry but Bing and Yahoo do not provide for this ability yet to my knowledge.

Similar Threads

  1. Error Trapping? Search engines show dead links.
    By Skyphoxx in forum SEO Help (General Chat)
    Replies: 0
    Last Post: Jan 7th, 2009, 02:45 PM
  2. Top search result in all search engines but google
    By fusionwebfx in forum Google Optimization
    Replies: 5
    Last Post: Jan 27th, 2007, 12:09 PM
  3. Replies: 0
    Last Post: Oct 7th, 2006, 04:38 PM
  4. Search Engine & Search Engines SERPs on Google
    By 00 in forum Google Optimization
    Replies: 3
    Last Post: Apr 21st, 2004, 09:30 PM

IMN logo majestic logo threadwatch logo seochat tools logo