#1
  1. No Profile Picture
    Registered User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    May 2017
    Posts
    7
    Rep Power
    0

    robots.txt has UserAgent: Yandex?


    Recently took over SEO duties for another website and looked into the robots.txt file to see what the people before me had set up in there.
    This is what I see in there:

    User-agent:Yandex
    Allow: /common/*.css
    Allow: /common/*.js
    Disallow: /*?
    Host: www.~~~~~~~.com

    To give you more backstory, other staff do not know why that is the current configuration nor do they know who had set it up this way.
    So my questions are:
    1. Why are we calling out Yandex only and not all crawlers?
    2. Why is only Yandex being allowed to crawl the .css and .js ?
    3. What would be the benefit of Yandex crawling .css and .js or is this even necessary to be in there?

    Any input or suggestion would be appreciated.
    Last edited by Hikin Mike; Aug 30th, 2017 at 04:17 AM. Reason: Fixed title typo
  2. #2
  3. SEO Insultant
    SEO Chat Mastermind (5000+ posts)

    Join Date
    Oct 2005
    Location
    sharing a room with my ego
    Posts
    6,429
    Rep Power
    3292
    Is the file named robot.txt or robots.txt ?
    The former won't do anything...
  4. #3
  5. No Profile Picture
    Registered User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    May 2017
    Posts
    7
    Rep Power
    0
    sorry, it is robots.txt file.
  6. #4
  7. SeoRaptor
    SEO Chat Adventurer (500 - 999 posts)

    Join Date
    Mar 2016
    Location
    France, Saint-Etienne
    Posts
    997
    Rep Power
    1703
    User-agent:Yandex
    To specify that the following rules concern Yandex Bot

    Allow: /common/*.css
    Allow: /common/*.js
    To allow access to css and js files stored in the common folder. (could be usefull if for any reason you blocked access to the files in the common folder)

    Disallow: /*?
    To block access to all URLs that include a question mark (?) (more specifically, any URL that begins with your domain name, followed by any string, followed by a question mark, followed by any string):

    1. Why are we calling out Yandex only and not all crawlers?
    No idea. You should ask them directly

    2. Why is only Yandex being allowed to crawl the .css and .js ?
    No all bots are allowed to crawl these files, by default any bot can crawl any file. Don't know why they specified it for Yandex.

    3. What would be the benefit of Yandex crawling .css and .js or is this even necessary to be in there?
    Allowing css and js files to be crawled helps search engines rendering the page and in some case helps ranking but my knowledge of Yandex is to limited to give you a specific answer.


    I do not understand what the person that created that file wanted to achieve.
    Disallow: /*? on it's own would have been enough in this case.
    Owner of Bennetonable - "My opinions are my own - Feel free to disagree & think above the fold."
    Need a quick analysis > Free SEO grader Beta
  8. #5
  9. SEO Insultant
    SEO Chat Mastermind (5000+ posts)

    Join Date
    Oct 2005
    Location
    sharing a room with my ego
    Posts
    6,429
    Rep Power
    3292
    I think we can all agree that EITHER the 'people before' didn't really know what they were doing OR they were fired mid-experiment.

    Ditch the file. Only hide truly secret files from ANY bot. Don't list complete file names, only the first few (unique) letters.

    That's my advice...
  10. #6
  11. No Profile Picture
    Registered User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    Aug 2017
    Posts
    4
    Rep Power
    0
    dzine, I agree with you
  12. #7
  13. No Profile Picture
    Registered User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    May 2017
    Posts
    7
    Rep Power
    0
    Alright, thank you everyone for your response. This sheds some light on the matter. I was equally perplexed and still kind of am at the goal and or experiment. Time to recreate the robots.txt file.
    Cheers

Similar Threads

  1. Strange google useragent
    By jozomannen in forum Google Optimization
    Replies: 0
    Last Post: Jan 23rd, 2006, 07:11 PM

IMN logo majestic logo threadwatch logo seochat tools logo