Need some expert help badly :-|

Hello hello people!

After reading this forum for about 3 weeks I finally decided to join as well, very very interesting stuff!

At the moment I'm finalizing my own website www.beatfreax.nl. Problem is that it's a tremendousll dynamic site. I tried a lot of things but when I use Sim Spider and so I get some really strange results. Also my site ís spidered by Google, though just the home and not the following pages.

Now today I noticed that a forum like the Invision Bord is spidered really well, even the whole URI is seen and listed by Google.

How come? And what am Í doing wrong?

try to use mod rewrite

I looks like you are setting a spider trap. You are passing a session id in the URL string, so each time a spider requests a page it gets served another URL, so the spider will crawl the same content over and over again. You have four choises. Do nothing. Do nothing and block spiders with robots.txt (no more searchengine rankings), Check for spiders and do not serve session id's to them, or you could pass the session valiables withouts using the URL string. I would recommend the third solution. The two first possibilities are just silly. A URL rewrite alone will not do, but it is always a benefit of having URLS that look static.

Hmmm, never thought of that third option actually.'I'll dig into that, must be able to do this in PHP (right?).

Mod rewrite is no option since my host doesn't want me to do this for security reasons.

Thanx for replying guys, I'm really stuck at this. I'd like to do this by the rules, but dynamic sites seem to have a lot of problems with this....

Your host doesn't want you to mod_rewrite for security? What kind of host is that? If anything mod_rewrite is more secure because it disguises the filestructure of your system...

Get a different host. In 10 minutes you can have this problem solved with mod_rewrite. (Well more like an hour...)

It is the session id that is the problem. You could just store this information in a cookie.

URL rewriting without .htaccess is not a problem if you have a PHP enabled server.

Lets use this url as an example:

http://www.beatfreax.nl/index.php?s...dc1f&art_id=101

You store the session data in a cookie and you are left with

http://www.beatfreax.nl/index.php?art_id=101

You change this url to point to:

http://www.beatfreax.nl/art/101/

and you place this information in http://www.beatfreax.nl/art/101/index.php :

<?php
Header("Location:http://www.beatfreax.nl/index.php?art_id=101");
?>

Thats it. But if you are on a apache server, it could be just as well to ask the admin to create a simple .htaccess file for you.

The best of luck.

Hey thanx man! Allthough I have tried this method, I'd probably have to make a lot of directories right? I also tried to do the trick with index.php/art/101, but then my server things the visitor's trying to access a directory (which doesn't exist, and so he quits).

I don't know how you checked it the last time, but now I put a .htaccess file with the following in the root;

<IfModule mod_php4.c> php_flag session.use_trans_sid off </IfModule>

Can you see if this makes a difference?

Thanx again!!!

Oh yeah, reading your reply again... the "s=234792" is nót a session, just me encrypting my s(ections) :-)

Can this also be used to keep track of affiliates and their referrals?

Owwww yeah! I convinced my hostingpartner to give me rewrite-rights :-)

Got me there. I guess I better look more carefully next time someone asks a question :-)

Ah damned, now that I have my Rewrite Engine on, I can't seem to get the technique I want to work, what's wrong with this? I get an error-page...

Options +FollowSymLinks
RewriteEngine on
RewriteBase /
RewriteRule index/(\.*)/(\.*)/(\.*)/(\.*)/$ /index\.php?$1=$2&$3=$4

Although THIS seems to work (but I think method 1 is more effiecient for me)

Options +FollowSymLinks
RewriteEngine on
RewriteBase /
RewriteRule article(.*)-(.*)\.htm$ /index\.php?s=$1&art_id=$2

Anyone?

GRMPFZ

Took me a day but I think I finally got it now. Could some of you check my site and see if this is an improvement? I'm just a bit anxious... will this help me to get spidered better by SE's like Google?

If you wanted to rewrite all the urls you are not there yet.

You could download XENU and check the linking on your site:
http://home.snafu.de/tilman/xenulink.html

Aloha, back again :-)

Google seems to be changing constantly isn't it?

If I search my "current site" now, I only get some newsarticles, but just the old ones, not the newer ones.

What is also strange that the title of the page isn't indexed right.

This robots.txt shouldn't be a problem right?

User-agent: *
Disallow: