Page 1 of 2 12 Last
  • Jump to page:
    #1
  1. No Profile Picture
    Contributing User
    SEO Chat Discoverer (100 - 499 posts)

    Join Date
    Oct 2004
    Posts
    391
    Rep Power
    14

    Get my pages deindexed


    I have a site with about 15,000 pages that's indexed by google.

    I will change all the urls to a different format, but still have the same content on the pages.

    If I change from for example page.php to page.html, could I get some penalty for dupe content?

    Is there any way to get my sites deindexed by google?
  2. #2
  3. Disregard Logic, Be Happy
    SEO Chat Adventurer (500 - 999 posts)

    Join Date
    May 2004
    Location
    USA
    Posts
    502
    Rep Power
    15
    http://www.google.com/remove.html#exclude_website

    if you want to redirect, use a 301.
    The Source of All Things Must Be Understood.
  4. #3
  5. Resident Black Hat
    SEO Chat Good Citizen (1000 - 1499 posts)

    Join Date
    Aug 2004
    Location
    Woodstock, GA
    Posts
    1,437
    Rep Power
    19
    Originally Posted by okguyjames
    http://www.google.com/remove.html#exclude_website

    if you want to redirect, use a 301.
    Good thing that requires you to place something on your server. Otherwise shifty people might actually use that to remove competitors sites

    G-Man
  6. #4
  7. No Profile Picture
    Contributing User
    SEO Chat Discoverer (100 - 499 posts)

    Join Date
    Oct 2004
    Posts
    391
    Rep Power
    14
    Originally Posted by okguyjames
    http://www.google.com/remove.html#exclude_website

    if you want to redirect, use a 301.
    My site has 15,000 pages with no pattern in it. But it should not be a problem to let a php-script generate a big robots.txt that disallows all the 15,000 pages that's listed in google. But what does the search engines think about a robots.txt file with 15,000 disallows.. isn't that too big? I guess it will be over 1 MB to load.
  8. #5
  9. Resident Black Hat
    SEO Chat Good Citizen (1000 - 1499 posts)

    Join Date
    Aug 2004
    Location
    Woodstock, GA
    Posts
    1,437
    Rep Power
    19
    Originally Posted by jozomannen
    My site has 15,000 pages with no pattern in it. But it should not be a problem to let a php-script generate a big robots.txt that disallows all the 15,000 pages that's listed in google. But what does the search engines think about a robots.txt file with 15,000 disallows.. isn't that too big? I guess it will be over 1 MB to load.
    I could be wrong but I'm pretty certain you can only disallow directories not files. Someone correct me if I'm wrong please.

    G-Man
  10. #6
  11. No Profile Picture
    Contributing User
    SEO Chat Discoverer (100 - 499 posts)

    Join Date
    Oct 2004
    Posts
    391
    Rep Power
    14
    Originally Posted by GeoffreyF67
    I could be wrong but I'm pretty certain you can only disallow directories not files. Someone correct me if I'm wrong please.

    G-Man
    An example from robotstxt.org:
    The following example "/robots.txt" file specifies that no robots should visit any URL starting with "/cyberworld/map/" or "/tmp/", or /foo.html:

    # robots.txt for http://www.example.com/

    User-agent: *
    Disallow: /cyberworld/map/ # This is an infinite virtual URL space
    Disallow: /tmp/ # these will soon disappear
    Disallow: /foo.html
  12. #7
  13. No Profile Picture
    Contributing User
    SEO Chat Hero (2000 - 2499 posts)

    Join Date
    Aug 2004
    Location
    Bay Area, CA
    Posts
    2,350
    Rep Power
    55
    Originally Posted by GeoffreyF67
    I could be wrong but I'm pretty certain you can only disallow directories not files. Someone correct me if I'm wrong please.

    G-Man
    You can, but if you are just rewriting page names and leaving them in the same directories, that won't work, or more accurately, it will work too well and disallow all your new pages as well.

    If you don't reuse the old directories, that will work fine.

    Comments on this post

    • GeoffreyF67 agrees : Thanks! I stand corrected :)

    "Live never to be ashamed of what is written about you. Even if what is written is not true" -- Richard Bach

    Yahoo Store SEO
  14. #8
  15. No Profile Picture
    Contributing User
    SEO Chat Good Citizen (1000 - 1499 posts)

    Join Date
    Apr 2004
    Posts
    1,179
    Rep Power
    16
    you don't have to do anything in robots.txt file or use google remove form. Use .htaccess to redirect (301 redirect) old pages to the new pages. search engine will gradually pick up the new pages and forget about your old pages.
  16. #9
  17. No Profile Picture
    Contributing User
    SEO Chat Discoverer (100 - 499 posts)

    Join Date
    Oct 2004
    Posts
    391
    Rep Power
    14
    Originally Posted by dejaone
    you don't have to do anything in robots.txt file or use google remove form. Use .htaccess to redirect (301 redirect) old pages to the new pages. search engine will gradually pick up the new pages and forget about your old pages.
    Do I have to make a new line in .htaccess for each page? The problem is that there's 15,000 pages.

    Four questions:
    1. Could I make some rule like mod_rewrite but for 301 redirect, because that would work.
    2. What will happen if I just remove the old page?
    3. What will happen if the old page just contains a link to the new page?
    4. What will happen if the old page just contains a header redirect to the new page?
  18. #10
  19. Mostly sane...
    SEO Chat Adventurer (500 - 999 posts)

    Join Date
    Aug 2004
    Location
    St. Petersburg, FL
    Posts
    756
    Rep Power
    15
    As for #1, yes, mod rewrite rule can be used for 301's. Just add [R=301,L] to the end of the rewrite line.

    -Michael
    Enjoy Poetry?
    Get a Home Equity Loan or Mortgage Refinancing today!
    Potential Bad Neighborhood checker.
  20. #11
  21. No Profile Picture
    Contributing User
    SEO Chat Discoverer (100 - 499 posts)

    Join Date
    Oct 2004
    Posts
    391
    Rep Power
    14
    Originally Posted by mvandemar
    As for #1, yes, mod rewrite rule can be used for 301's. Just add [R=301,L] to the end of the rewrite line.

    -Michael
    Ahh, so this would work?
    Code:
    RewriteRule [.]*-id([0-9]*) /id([0-9]+)/[.] [R=301,L]
    Or should it look like this:
    Code:
    RewriteRule [.]*-id([0-9]*) /id$1/[.] [R=301,L]
  22. #12
  23. Mostly sane...
    SEO Chat Adventurer (500 - 999 posts)

    Join Date
    Aug 2004
    Location
    St. Petersburg, FL
    Posts
    756
    Rep Power
    15
    Give me an example of the old path/page and the new one and I'll try and come up with it for ya.

    -Michael
  24. #13
  25. No Profile Picture
    Contributing User
    SEO Chat Discoverer (100 - 499 posts)

    Join Date
    Oct 2004
    Posts
    391
    Rep Power
    14
    Originally Posted by mvandemar
    Give me an example of the old path/page and the new one and I'll try and come up with it for ya.

    -Michael
    Old:
    www.domain.com/folder/name-of-the-script-id123.html

    New:
    www.domain.com/folder/id123/name-of-the-script.html
  26. #14
  27. Mostly sane...
    SEO Chat Adventurer (500 - 999 posts)

    Join Date
    Aug 2004
    Location
    St. Petersburg, FL
    Posts
    756
    Rep Power
    15
    Originally Posted by jozomannen
    Old:
    www.domain.com/folder/name-of-the-script-id123.html

    New:
    www.domain.com/folder/id123/name-of-the-script.html

    Originally Posted by I think this will work
    ^/folder/name-of-the-script-(.*)\.html$ http://www.domain.com/folder/$1/name-of-the-script.html [R=301,L]
    -Michael
    Last edited by mvandemar; Jun 27th, 2005 at 07:45 AM.
  28. #15
  29. No Profile Picture
    Contributing User
    SEO Chat Discoverer (100 - 499 posts)

    Join Date
    Oct 2004
    Posts
    391
    Rep Power
    14
    Ok, but I forgot to mention that 'name-of-the-script' is different en every url, so maybe this will work:
    Code:
    ^/folder/([a-z-]+)-(.*)\.html$ http://www.domain.com/folder/$1/$2.html [R=301,L]
    ?
Page 1 of 2 12 Last
  • Jump to page:

Similar Threads

  1. Here we've gone again
    By straitsex in forum Google Optimization
    Replies: 21
    Last Post: May 22nd, 2005, 04:38 PM
  2. Dynamic pages dissapeared from google
    By donkeyderby in forum Google Optimization
    Replies: 14
    Last Post: May 17th, 2005, 09:55 AM
  3. Which is good - HTML /ASP /PHP pages for Google ?
    By obiztek in forum Google Optimization
    Replies: 2
    Last Post: Jan 7th, 2005, 04:28 PM
  4. Google site: prefix search
    By channel5 in forum Google Optimization
    Replies: 9
    Last Post: Nov 24th, 2004, 10:37 AM
  5. does website HOST location affect your rank??
    By JoeBob in forum Google Optimization
    Replies: 66
    Last Post: Aug 29th, 2004, 02:41 PM

IMN logo majestic logo threadwatch logo seochat tools logo