#1
  1. No Profile Picture
    Registered User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    Jan 2017
    Posts
    1
    Rep Power
    0

    Meta description/robots.txt issue


    Hi all.

    I've got an issue with my site's Google search result. I can't post the site but will try to provide as much info as I can.

    Our marketing site is getting the dreaded: A description for this result is not available because of this site's robots.txt

    That site is Wordpress. Say it's: http://www.oursite.com

    We have a product site that lives on the same domain under https, so: https://oursite.com

    The www site has a pretty simple robots.txt file:
    User-agent: *
    Disallow: /wp/wp-admin/
    Allow: /wp/wp-admin/admin-ajax.php

    The product site shouldn't be crawled, and has a disallow:
    User-agent: *
    Disallow: /

    A few pertinent details:
    -- Going to http://oursite.com will redirect you to https://oursite.com. After that redirect, the product realizes that you didn't come to the login page and redirects you back to http://www.oursite.com. I'm worried this is the cause. For convoluted reasons, this is a behavior that's hard for us to change.
    -- Google's search result for our brand has our www site showing up as http://oursite.com
    -- The Google webmaster tools say that our robots.txt file will allow crawling of www.oursite.com
    -- The SEO aspects of the www site were tuned using an SEO plugin (yoast's). Those appear to be valid (meta description tag, canonical link, green flags from the SEO testers we've used)
    -- Bing picks up our description from the www site correctly. It shows a URL of www.oursite.com
    -- In Google's webmaster tools, we have our preferred domain set as oursite.com, not www.oursite.com. I'm hopeful changing that would possibly fix this.

    Thank you very much for any help.
    Last edited by dzine; Jan 6th, 2017 at 04:07 AM.
  2. #2
  3. No Profile Picture
    Moderator
    SEO Chat Scholar (3000 - 3499 posts)

    Join Date
    Sep 2016
    Location
    USA
    Posts
    3,013
    Rep Power
    3634
    First off, the requirement to be logged into the site will prevent Google from seeing those pages that are behind the login. Google doesn't have any access info.

    Convert the site completely over to https. You will get a boost for this from Google.

    Now when you realize the user is not logged in just send them to the login page.

    Now remove the "disallow /" from the product site robots.txt, why, because you have to be logged in to reach these pages. No search engine will index these pages because they can't log in to the site. If you have any pages on the https site that should be indexed and are not behind a login, the "disallow /" stops that from happening.

    In trying to make your site invisible to SE bots, you have done very well.

    You can test this behavior in GSC with Fetch as Googlebot.

    BTW in my opinion, you are mis-using redirects switching back and forth between the http / https sites.
  4. #3
  5. No Profile Picture

    Join Date
    Jan 2017
    Posts
    1
    Rep Power
    0
    without meta tag we can;t improve SEo and we can;t get Rank In google <snip> tp know the how to get traffic all the and what requirements for Page rank In google
    Last edited by dzine; Jan 13th, 2017 at 06:47 AM.
  6. #4
  7. No Profile Picture
    Moderator
    SEO Chat Scholar (3000 - 3499 posts)

    Join Date
    Sep 2016
    Location
    USA
    Posts
    3,013
    Rep Power
    3634
    rakeshlaishetty which meta tags do you think are important to seo....

    meta tag keywords-- useless
    meta tag descritpion -- no effect on seo, but has an effect on ctr

    There is a difference between page title and meta tag title... do not confuse them...

    edited... to add....

    Btw nice try to get a link to your website... its only 2 1/2 months old too !! You know it doesn't work right ???

    Seems dzine, a mod busted you on that ... Good Job dzine !

    Comments on this post

    • Will-O-The-Wisp agrees
    Last edited by KnowOneSpecial; Jan 13th, 2017 at 06:58 AM.

Similar Threads

  1. Replies: 10
    Last Post: Mar 10th, 2015, 04:29 AM
  2. Replies: 6
    Last Post: Jun 8th, 2013, 11:49 AM
  3. Meta Description Issue
    By seoexpertrizwan in forum Google Optimization
    Replies: 12
    Last Post: Feb 10th, 2013, 12:32 PM
  4. Rookie Question: Page Description vs. Meta Description
    By ISellGuitars in forum Google Optimization
    Replies: 11
    Last Post: Aug 18th, 2011, 12:01 PM
  5. Replies: 2
    Last Post: Aug 21st, 2009, 02:56 PM

IMN logo majestic logo threadwatch logo seochat tools logo