|
|||||||||
|
|||||||||
|
|||||||||
 |
|||||||||
| |
||
|
|
|
|
|||||||||
|
|||||||||
|
|||||||||
|
|||||||||
| |
||
|
|
|
| |||||||||
 |
|
|
«
Previous Thread
|
Next Thread
»
|
Thread Tools | Search this Thread |
Rating: 
|
Display Modes |
SEO Chat Forums Sponsor:
|
 AT&T devCentral & BlackBerry(r) Webcast Series: BlackBerry and GPS -Build Location Awareness into your BlackBerry Applications, July 10th-1:00PM EST. Register Today!
|
|
#46
September 1st, 2003, 08:48 PM
|
||||
|
||||
|
No problem chech or is it non problemo? Any ways for your robots.txt file simply open notepad, wordpad or any text editor and type in this text:
User-agent: * Disallow: This tells the spider to not only spider your entire root but also for more advanced applications you can ban certain spiders (user agents) as well as stop them from taking certain folders and files like images. An example of how to use this is User-agent: * Disallow: /images/ This application would tell spiders not to spider your images folder. js is short for Javascript and what it means to pack away your js (java script) in an external file is described here: http://www.pagerank-search-engine-o...our-header.html you call the same way you would call an attached CSS sheet (cascading style sheet)
|
|
#47
September 1st, 2003, 08:52 PM
|
||||||
|
||||||
|
#
# Your: robots.txt # User-agent: WebZip Disallow: / User-agent: larbin Disallow: / User-agent: b2w/0.1 Disallow: / User-agent: Copernic Disallow: / User-agent: psbot Disallow: / User-agent: Python-urllib Disallow: / User-agent: Googlebot-Image Disallow: / User-agent: NetMechanic Disallow: / User-agent: URL_Spider_Pro Disallow: / User-agent: CherryPicker Disallow: / User-agent: EmailCollector Disallow: / User-agent: EmailSiphon Disallow: / User-agent: WebBandit Disallow: / User-agent: EmailWolf Disallow: / User-agent: ExtractorPro Disallow: / User-agent: CopyRightCheck Disallow: / User-agent: Crescent Disallow: / User-agent: SiteSnagger Disallow: / User-agent: ProWebWalker Disallow: / User-agent: CheeseBot Disallow: / User-agent: LNSpiderguy Disallow: / User-agent: Mozilla Disallow: / User-agent: mozilla Disallow: / User-agent: mozilla/3 Disallow: / User-agent: mozilla/4 Disallow: / User-agent: mozilla/5 Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows NT) Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows 95) Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows 98) Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows XP) Disallow: / User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows 2000) Disallow: / User-agent: ia_archiver Disallow: / User-agent: ia_archiver/1.6 Disallow: / User-agent: Alexibot Disallow: / User-agent: Teleport Disallow: / User-agent: TeleportPro Disallow: / User-agent: MIIxpc Disallow: / User-agent: Telesoft Disallow: / User-agent: Website Quester Disallow: / User-agent: moget/2.1 Disallow: / User-agent: WebZip/4.0 Disallow: / User-agent: WebStripper Disallow: / User-agent: WebSauger Disallow: / User-agent: WebCopier Disallow: / User-agent: NetAnts Disallow: / User-agent: Mister PiX Disallow: / User-agent: WebAuto Disallow: / User-agent: TheNomad Disallow: / User-agent: WWW-Collector-E Disallow: / User-agent: RMA Disallow: / User-agent: libWeb/clsHTTP Disallow: / User-agent: asterias Disallow: / User-agent: httplib Disallow: / User-agent: turingos Disallow: / User-agent: spanner Disallow: / User-agent: InfoNaviRobot Disallow: / User-agent: Harvest/1.5 Disallow: / User-agent: Bullseye/1.0 Disallow: / User-agent: Mozilla/4.0 (compatible; BullsEye; Windows 95) Disallow: / User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0 Disallow: / User-agent: CherryPickerSE/1.0 Disallow: / User-agent: CherryPickerElite/1.0 Disallow: / User-agent: WebBandit/3.50 Disallow: / User-agent: NICErsPRO Disallow: / User-agent: Microsoft URL Control - 5.01.4511 Disallow: / User-agent: DittoSpyder Disallow: / User-agent: Foobot Disallow: / User-agent: WebmasterWorldForumBot Disallow: / User-agent: SpankBot Disallow: / User-agent: BotALot Disallow: / User-agent: lwp-trivial/1.34 Disallow: / User-agent: lwp-trivial Disallow: / User-agent: BunnySlippers Disallow: / User-agent: Microsoft URL Control - 6.00.8169 Disallow: / User-agent: URLy Warning Disallow: / User-agent: Wget/1.6 Disallow: / User-agent: Wget/1.5.3 Disallow: / User-agent: Wget Disallow: / User-agent: LinkWalker Disallow: / User-agent: cosmos Disallow: / User-agent: moget Disallow: / User-agent: hloader Disallow: / User-agent: humanlinks Disallow: / User-agent: LinkextractorPro Disallow: / User-agent: Offline Explorer Disallow: / User-agent: Mata Hari Disallow: / User-agent: LexiBot Disallow: / User-agent: Web Image Collector Disallow: / User-agent: The Intraformant Disallow: / User-agent: True_Robot/1.0 Disallow: / User-agent: True_Robot Disallow: / User-agent: BlowFish/1.0 Disallow: / User-agent: JennyBot Disallow: / User-agent: MIIxpc/4.2 Disallow: / User-agent: BuiltBotTough Disallow: / User-agent: ProPowerBot/2.14 Disallow: / User-agent: BackDoorBot/1.0 Disallow: / User-agent: toCrawl/UrlDispatcher Disallow: / User-agent: WebEnhancer Disallow: / User-agent: suzuran Disallow: / User-agent: VCI WebViewer VCI WebViewer Win32 Disallow: / User-agent: VCI Disallow: / User-agent: Szukacz/1.4 Disallow: / User-agent: QueryN Metasearch Disallow: / User-agent: Openfind data gathere Disallow: / User-agent: Openfind Disallow: / User-agent: Xenu's Link Sleuth 1.1c Disallow: / User-agent: Xenu's Disallow: / User-agent: Zeus Disallow: / User-agent: RepoMonkey Bait & Tackle/v1.01 Disallow: / User-agent: RepoMonkey Disallow: / User-agent: Microsoft URL Control Disallow: / User-agent: Openbot Disallow: / User-agent: URL Control Disallow: / User-agent: Zeus Link Scout Disallow: / User-agent: Zeus 32297 Webster Pro V2.9 Win32 Disallow: / User-agent: Webster Pro Disallow: / User-agent: EroCrawler Disallow: / User-agent: LinkScan/8.1a Unix Disallow: / User-agent: Keyword Density/0.9 Disallow: / User-agent: Kenjin Spider Disallow: / User-agent: Iron33/1.0.2 Disallow: / User-agent: Bookmark search tool Disallow: / User-agent: GetRight/4.2 Disallow: / User-agent: FairAd Client Disallow: / User-agent: Gaisbot Disallow: / User-agent: Aqua_Products Disallow: / User-agent: Radiation Retriever 1.1 Disallow: / User-agent: WebmasterWorld Extractor Disallow: / User-agent: Flaming AttackBot Disallow: / User-agent: Oracle Ultra Search Disallow: / User-agent: MSIECrawler Disallow: / User-agent: PerMan Disallow: / User-agent: searchpreview Disallow: / User-agent: * Disallow: /cgi-bin/ //added// (most of the above crawlers are garbage - they do not do a thing for you). //added//
__________________
We are what we repeatedly do… excellence, then, is not an act, but a habit. — Aristotle Last edited by fathom : September 1st, 2003 at 08:59 PM.
|
|
#48
September 1st, 2003, 08:54 PM
|
||||||
|
||||||
|
Robot tags are not really required for "inclusion", bot naturally index all pages they find automatically.
|
|
#51
October 8th, 2003, 08:00 AM
|
|||
|
|||
|
backlinks
I would also say, check your backlinks ancor text and make sure they fit, if not, contact the webmaster and ask him to change it
__________________
Link Swapper - a FREE link exchange plug-in and directory add-on for your website Crawler Alert - a FREE service which automatically sends you an email notification whenever a search engine crawler is scanning your website
|
|
#52
October 8th, 2003, 10:30 AM
|
||||
|
||||
Very informative input (again from Mod) Fathom, thanks for that. One question for the new comers though; why block the robots/bots/spiders/diggers/drillers with the Disallow: / tag?
Some of them (robots) are country specisific directory (language, alphabet or some sort of image,culture,news, etc. based) providers/suppliers. Wouldn't you suggest to use a Disallow: tag or maybe only to block the non important or do-not-want-to-be-spidered-directory instead of blocking some of the important bots? I already know your answer to those questions,  I just neeed your further input (similar to, one of your previous lenghtly posts here) on this subject to enlighten the new comers to SEO chat.Thanks in advance...
__________________
Affordable SEO and web design services from Thailand simple-biz.com Last edited by simple-biz : October 8th, 2003 at 10:35 AM.
|
|
#53
November 4th, 2003, 09:48 AM
|
|||
|
|||
|
TIP 1
SEO Copywriting Never ignore the power of words. As search engines look for relevant words, text and description. It becomes very vital that your persuasive copy is keyword-rich and search engine friendly. SEO copywriting can also boost your ranking if you are targeting highly competitive keywords. Try it to believe it!
|
|
#54
March 27th, 2004, 08:19 AM
|
||||
|
||||
Great & very helpful Discussion...
Hi Friends
i am new to SEO can any one tell me how to target some keyword for search engine listing. though i got Google PR 5 for www.mindcyclestudio.com but still don't know how to give targeted keywords i.e., web design, print media etc., I hope u understand what i am trying to say, Regards Meenu
|
|
#55
April 15th, 2004, 03:38 PM
|
|||
|
|||
|
Simply great tips. And they go on and on. It's great to know so many people- sorry, PROFESSIONALS are out there willing to share their secrets!
|
|
#56
April 15th, 2004, 03:43 PM
|
|||
|
|||
|
Images=Crawled=bad
Great tip for the Images Folder. Not many peopel actually realize that their iages folder does get crawled and indexed.
__________________________________________________ ________________________________ Trademark Productions, Inc. Quote:
|
|
#57
April 19th, 2004, 06:12 AM
|
|||
|
|||
Hi all, I'm new here, but not that new to SEO.
Fathom, You stated the following earlier in the thread which I have been looking more and more into over the last few weeks with great interest. Quote:
I was looking at using the title tag in the <A HREF=www.mydomain.com title="My Domain">My Domain</a> tags. How much weight would search engines give this? I tried it by optimising a few pages using this title tag and the pages are better ranked in Google (about 4 or 5 pages as a test). I don't know if this is just fluke or if it was because I named the page extensions differently but they are all seeming to do well compared to the others. I don't want to change much as I am ranking well in Yahoo. Averaging number 5/6 for my targeted keywords on Google. I was also looking at the table summary command. Is there any point in adding this code? <TABLE SUMMARY="KeyWord"> - would this be classed as keyword stuffing and will it be penalised? The last thing I want is to get on the wrong side of big G. Thanks Terry
|
|
#58
April 19th, 2004, 12:33 PM
|
||||
|
||||
|
Quote:
Hello, Thank you for a nice tutorial. I will follow the advice above and try to learn as mu |
Del.icio.us
Digg
Google
Spurl
Blink
Furl
Simpy
Y! MyWeb
