google, (msn and yahoo dir) probs with cms ?

Hi there,

i request help on this topic: MSN, Google and Yahoo Directory currently ignore on of my clients website. None of the other search engines seems to have trouble.

The index.html is already in the three above. From there on, a cms is called through a frameset looking like this:

frame top= /cgi-bin/cms.pl?template=start
frame bottom= /cgi-bin/cms.pl?template=foot

These two pages are searchable as well (along with index.html) in Google, MSN and Yahoo, but none of the following pages.

So, i have a total of three pages , while some hundred are waiting since november 2002. The sites deeplinks look like this:

<a href = /cgi-bin/cms.pl?template=article //for categories
<a href = /cgi-bin/cms.pl?template=article&id=1234 //for articles

I used webmasterplan.de with their "this is how your site looks to engines" tool and found out that this tool requires "=" to be replaced by "%3d" (ascii opcode for equal signs). is this the same for google and company ?

ans help greatly appreciated!
jansass

ps: im not familiar with posting etiqeutte, so i left out the url. can be requested for inspection none the less!

jansass - no problem to post the URL - the more specific you can be the more likely you'll receive the help you need.

Gringo.

http://www.fipl.de , Its german content, but technically we are all the same..

How long has this site been online? It may be just a question of time, since Google normally takes 2 or 3 months to spider and index all pages of a new site.

Apart from this, the question that immediately springs to mind is ... why don't you use normal text links?

Gringo.

Site went online Nov. 2002 and has been submitted to any major engine on a regular basis. any of them except the aforementioned are happy with the content and overall visibility climbed to 10+% last week.

just these three dont like the content, although they spidered the frameset plus one-link-deep dynamic content.

we cant use static links, because the cms itself creates the links is needs. otherwise we would need to use ssi, which we cant and, frankly, dont want. or do i misunderstand you here?

appearently, google ranked the main domain as pr-zero, which means one of two very bad things. the first one can be fixed (no inbound within 3 months) the second (offended TOSs) would only surprise me since i cant see any reason for ban.

so, a) site is technically ok? and b) any advises to do's and dont's ?

best regards
jansass

It looks like Google is not indexing ANY of your pages. It is only seeing the NOFRAMES part of the index page.

You say that none of the other search engines are having trouble spidering these pages - I find this very surprising since Google is one of the better indexers. Can you give us any examples of internal pages that have been indexed by other SE's ?

In the meantime, I think if you want your internal pages spidered you'll have to think up another strategy, like using .htaccess to present Google with search-engine friendly URLs.

Gringo.

thanks for the advise. first of all here is an example for one of the better indexing:

visit http://suche.freenet.de/suche
type in "privatefinancialservices" (no quotes; to avoid other sites)

besides (i know, i know) look for no. 13, news.htm or 15, agb.htm. these are still no deeplinks, but at least one step deeper than google. my goal is to get these pages spidered:

http://www.fipl.de/cgi-bin/pfs.pl?template=ergebnis.htm&q=%20 or
http://www.fipl.de/cgi-bin/pfs%2Epl?template=news_view%2ehtm&id=1043923914

however, none of the SEs did so far..

can you please say a word on the .htacces files? Do you mean server sided redirect? i even though about converting any dynamic page to static (thus producing redundant information) but this can only be a workaround, and no solution..

muchos regardos
jansass

You should update the page building the frameset and use valid HTML4. If the link was quoted you probably would not be having the issues with the framed site. It should look like:

<a href="http://blah.com?key=value">

All pages should always be pushed through the W3C verifier http://validator.w3c.org to make sure you are using valid code before publishing. Having errors in your code can cause engines to do all sorts of unexpected things. Standards-compliant code should be as much a concern as good uptime.

Woah up guys and girls

Most spiders will only go as far as 3.2 so anything past that is pushing the limits

The answer is KISS

Keep it simple stupid

and remember that every page is a front page

link normally and avoid special cgi s etc

link in text before anything else and use lots of text of relevance

if you use a iframe for example you end up with two pages for the same value search wise as one page (once you are listed you then insert a js to force the 'mother' page to show)

It is a numbers game

Simple and stupid

@syz
thanks a lot, man. that hint is as obvious as brilliant. i really was that ignorant not using basic html syntax. simple quotes could really make a difference for this. i will surely implement this one!

@foxyweb
i agree with you on the kiss topic. and besides qotations and some more oddities it should comply to loose 3.2, so no

still i managed to gain a 40%+ visiblity with a purely flashed site, so ranking a cgi site should be at least a little more easy (which it appearently is not). anyhow, i can not avoid linking to dynamic content created by perl, thus \cgi.pl?content is required for this task.

so, i read this as a hint towards a) convertig dynamic to static content and/or b) .htaccess -ing the files.



thank you very much, this board answered some 6 times, while my homeboard didn't say a word until now.

i'll report on this and maybe you'll find me in the reviews-section.

honest respect and best regards
jansass

Foxyweb: I agree that spiders don't require much of anything ... there are some really really lousy pages out there and they get indexed... however they don't get indexed /well/.

When I say use a validator at HTML4.01, I am just saying to quote your urls and all html tag arguments, put alt tags on every image, don't nest your tags funny, and have the required stuff like a </head> and a <body>.

Using a validator tells you your mistakes. All of those problems may not show up in a modern browser like IE, but can cause problems with older browsers (like the wonderful 'missing a </table>' killer in netscape4), more standards-compliant browsers, and engines

Syz: Right on - I am in the process of converting all the pages on all of the sites to HTML4.01 but I am being very cautious about it and taking my time at least at the beginning while I test the "system" - hence this next dance is an important one.

Some of the pages that I have highly ranked come from rather obscure positions eg very deep 4-5 directories and only connected through a frameset (I use every page regardless of importance as a potential incoming page) and when this next dance is over I will put up the scenario and see the results!

Syz: Right on - I am in the process of converting all the pages on all of the sites to HTML4.01 but I am being very cautious about it and taking my time at least at the beginning while I test the "system" - hence this next dance is an important one.

Some of the pages that I have highly ranked come from rather obscure positions eg very deep 4-5 directories and only connected through a frameset (I use every page regardless of importance as a potential incoming page) and when this next dance is over I will put up the scenario and see the results!