#1
  1. No Profile Picture
    Contributing User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    Jan 2017
    Posts
    45
    Rep Power
    3

    Google is not crawing my home? Why


    Hello Experts.

    Kindly help !

    My websites home page is not indexing and continue giving soft 404 error. I have read about soft 404 pages and found that "when many elements of a page is not loading and giving 404 errors so it will create soft 404 header and error in crawling of web page"

    https://search.google.com/u/0/search...GTXWdy01IIgGcg

    Here you can here. When I checked this on webmaster then I found that many of URL's are fetching with the home page. But the URLs those are fectching mostly are "ajaxforAll" URL's and crawlers are taking by buttons as a URLs i.e Buy Button, Add to cards button, delete button.

    These are not Urls, these are buttons those run with scrips. Then why webmaster are reading these buttons as URL's?

    Kindly help. My website's home page is not indexing and giving soft 404 status. This is harming my brand value worst.

    Kindly Help Guys
  2. #2
  3. No Profile Picture
    Moderator
    SEO Chat Scholar (3000 - 3499 posts)

    Join Date
    Sep 2016
    Location
    USA
    Posts
    3,188
    Rep Power
    3708
    It does appear Google has found links to your shopping cart. You need to make these pages noindex nofollow.

    You have also put your blog sitemap in the wrong place on your server.

    Sitemaps should be placed in root of your server. The same place you put your robots.txt not in a sub-directory.

    You have a complicated robots.txt file and most likely that may be where the error lies. I say this because that is the most common issue with search engines crawling pages blocked by robots.

    Just for the record. ( and I have stated this many times)

    Robots.txt only blocks Google from pages. Yes blocking the search engine will keep Google from finding the page, but it will not prevent indexing a blocked page if Google finds an URL that points to a blocked page.

    You need to either use the robots meta tag on that page or use header-x tags on that page to prevent indexing.

    Comments on this post

    • Chedders agrees
    If you have never failed in your life, you have never achieved anything Noteworthy !
  4. #3
  5. No Profile Picture
    Contributing User
    SEO Chat Explorer (0 - 99 posts)

    Join Date
    Jan 2017
    Posts
    45
    Rep Power
    3
    Thanks for the reply.

    But here webmaster telling me that the problem is Google is reading or crawling call to actions buttons like buy sample, delete from cart those are not a URL actually. Those are ajax and call to actions buttons.

    All are landing on 404 and after these much 404. home page is providing soft 404 during indexing.

    So According to me this is the reason which I understand.
  6. #4
  7. No Profile Picture
    Moderator
    SEO Chat Scholar (3000 - 3499 posts)

    Join Date
    Sep 2016
    Location
    USA
    Posts
    3,188
    Rep Power
    3708
    Yes, they are CTA's and "delete from cart" .. but they are also still URLs. Google should not be allowed to manipulate your shopping cart.

    Look at it from this point of view. Apparently this just started happening, I would then make a list of all changes you made to the site before the error appeared. Chances are one of those changes caused the error.

    If all was working properly, the site just doesn't stop working unless a change was introduced. Basic troubleshooting 101. Undo what you have done and see if the error goes away, I am betting it does.
  8. #5
  9. Dinosaur
    SEO Chat Mastermind (5000+ posts)

    Join Date
    Jun 2011
    Location
    UK
    Posts
    5,349
    Rep Power
    7411
    On shopping cart pages I always add the following in the head section of the page
    <meta name="robots" content="noindex, nofollow" />
    In fact I add that to every page within the site that I don't want to appear in the index, pages such as user account pages. I never use robots.txt to do that job.
    We see so many people using robots.txt and the slightest error can open your whole site or block the whole site completely, Most of my sites don't have a robots.txt or if it does then it is just as follows

    User-agent: *
    Disallow:

    Also a common error is with sitemaps. There really is no requirement for one, assuming your site is standard and pages can be found via links on your site then google will crawl the site perfectly fine. The only time you require one is if you have opthan pages which can not be found any other way. Think of a site map as a way of providing links for google to follow when they do not appear any other way on the site. Pages via a internal search for example that requires users to type something in. Dynamically generated pages is a good example of this.

    The issue I think your facing is not blocking google correctly, have a serious look at what pages add value to the site overall and which are just functional pages for users, i.e. shopping cart pages. Add the meta tag to the pages you do not want indexed. Do you think your Signup page is adding anything and do you think it helps if that page was to appear in SERPS ? if the answer is no then add the meta tag.

    Once you have the structure sorted you really then need to look at site speed, I am getting 10 seconds + on pages which will not be helping overall.

    Comments on this post

    • Prof.stan agrees
    IMHO

Similar Threads

  1. Replies: 3
    Last Post: Oct 31st, 2013, 02:01 AM
  2. Force Google to ignore splash page & set www.address.com/home as the home page
    By telkins in forum New User SEO Questions and Answers
    Replies: 2
    Last Post: Oct 22nd, 2013, 10:37 AM
  3. Google Envisions Automated Home with Android@Home (PC World)
    By RSS_News_User in forum Technology News
    Replies: 0
    Last Post: May 11th, 2011, 02:02 PM
  4. Why does Google only have my home page?
    By SulkyGirl in forum Google Optimization
    Replies: 2
    Last Post: Aug 4th, 2003, 05:16 AM

IMN logo majestic logo threadwatch logo seochat tools logo