Great community. Great ideas.
Welcome to SEOChat, a community dedicated to helping beginners and professionals alike in improving their Search Engine Optimization knowledge. Sign up today to gain access to the combined insight of tens of thousands of members.
Jan 4th, 2013, 01:48 AM
is robots.txt required in my case ?
I want web spiders crawl the whole website and i have no xml sitemap to mention in it. The only thing i need is to block bad bots, if they are bad means they will bypass robots.txt. Is robots.txt required in this case.
Second question whether genuine web spiders like googlebot will look for robots.txt if there isn't a robots.txt available in the site. I need to differentiate the bad bots with genuine spiders.
Jan 4th, 2013, 01:55 AM
What sort of 'bad bots' are you concerned about?
Jan 4th, 2013, 02:38 AM
The bots which will eat your bandwidth unnecessarily.........
Originally Posted by dzine
Jan 4th, 2013, 05:25 AM
Do you have any examples? I have found that there are hardly any bots like that, that DO obey robots.txt
Also, I must admit that I'm not all that concerned with my bandwidth
If you want to limit crawling to only a small number of bots, say: Google + Bing/Yahoo + Ask + Baidu + Yandex, then you could do that with a robots.txt file. But it still wouldn't keep the 'bad bots' out -- if said bots ignore the robots.txt file.
You'd have to check each one.
What you could do is:
- check your logs which bot is eating lots of your bandwidth and doesn't send you any good visitors
- check online if that bot obeys the robots.txt protocol
- if so: block it specifically
- if not: try to block its IP address(es) using .htaccess ("allow/deny") or something similar
Jan 4th, 2013, 06:02 AM
i think only robots.txt is not sure for whole website...
Jan 4th, 2013, 07:21 AM
Unknown robot (identified by 'bot/' or 'bot-'), Unknown robot (identified by 'robot'), Unknown robot (identified by 'spider'), Unknown robot (identified by 'crawl'). I have no idea who are these and how to block them in robots.txt file.
Its a hell out of job to analyze log file and find the bad bot. I have some issues that i will post in new thread.
Also whats the alternative of .htaccess in windows server i need to discuss.
Jan 4th, 2013, 07:24 AM
But my original question still remains is robots.txt is required in my case. I am speaking about bad bots not google, yahoo, ask
Jan 4th, 2013, 08:38 AM
Only you can tell if it's necessary.
Personally I wouldn't bother. But if you really want to keep ALL bots out EXCEPT for a few trusted ones, then by all means do so using robots.txt
Jan 4th, 2013, 11:00 AM
can i block these in robots.txt:
Unknown robot (identified by 'bot/' or 'bot-'), Unknown robot (identified by 'robot'), Unknown robot (identified by 'spider'), Unknown robot (identified by 'crawl')
Jan 4th, 2013, 04:33 PM
I think you cannot. As far as I know, one cannot use 'wildcards' for parts of the bots' names.
How much bandwidth are these bots costing your by the way?
Jan 5th, 2013, 12:23 AM
For December 2012 (1 month):
Originally Posted by dzine
Unknown robot (identified by 'bot/' or 'bot-') 6994(Hits) 80.34 MB(Bandwidth) 25 Dec 2012 - 00:52(Last Visit)
site is hosted on a windows shared hosting server....
By webkul in forum Google Optimization
Last Post: Jul 6th, 2011, 07:01 AM
By RSS_News_User in forum Technology News
Last Post: Mar 22nd, 2011, 12:01 PM
By peterson in forum SEO Help (General Chat)
Last Post: Mar 10th, 2011, 06:07 AM
By SEO Chat in forum SEO Chat Articles
Last Post: Jan 13th, 2010, 12:47 PM
By godwin in forum SEO Help (General Chat)
Last Post: Oct 26th, 2005, 07:31 AM