|
|
|||||||||
|
|||||||||
|
|||||||||
| |
||
| |||||||||
![]() |
|
|
«
Previous Thread
|
Next Thread
»
|
Thread Tools | Search this Thread | Rate Thread | Display Modes |
|
#1
|
|||
|
|||
|
Please check my robots.txt
User-agent: *
Disallow: /includes/ Disallow: /database/ Disallow: /scripts/ Disallow: /images/ Disallow: /gallery/ Disallow: /panoramas/ I am curious, do I need the "/" at the end of each of these? Otherwise, is everything else ok? |
|
#2
|
||||
|
||||
|
Your robots.txt is fine. The trailing slash tells the bot you only want to hide these directories (and everything in it). If you removed those slashes ("/"), any FILES starting with /includes or /database etc. would be hidden too.
For example: /gallery-of-widgets.html would be hidden as well. But in your robots.txt those filenames are NOT hidden, only the directories. I presume that's what you intended, so it's perfect.
__________________
...please help me w/ the real Redscowl Bluesingsky...how2 check backlinks...now postin' @ SEO Refugee ...
·<°`)~ LOL now that I finally have a paypal account, I'm charging 19,- for SEO advice via PM. Seriously... |
|
#3
|
|||
|
|||
|
Hey a follow up please, I saw this today for Google Disallow: /*? (or something like that). If I leave that out, they will index pages whose url's contain variables, no?
|
|
#4
|
||||
|
||||
|
Yes, but you should in fact leave everything out, and only disallow spidering of pages/dirs that you really want to "hide". Note that you can't really hide the pages, sometimes they're read anyway, just not indexed. And sometimes they are indexed anyway (accidentally or on purpose).
Googlebot can read wildcards in the url strings (to some extent). You're right that this particular phrase would tell g-bot to leave pages with variables and search queries alone... I guess you only want that if you have a proper mod_rewrite in place, and only want the "cleaned-up" urls to appear on the serps. |
![]() |
| Viewing: SEO Chat Forums > Search Engine Strategies > Search Engine Optimization > Please check my robots.txt |
| Thread Tools | Search this Thread |
| Display Modes | Rate This Thread |
|
|
|
|
|