Answer
Apr 17, 2025 - 02:08 PM
There are several reasons that your pages may not be getting crawled by Google, and you'll want to make sure your "technical SEO" is taken care of so that your pages are effectively/efficiently found and crawled by Google. But....there are other reasons that your pages may not be getting crawled by Google that is more about what you're publishing vs how you're publishing things. Here are some things to think about.
Be Careful When an "SEO Expert" Throws Out "Crawl Budget"
This is one of these things that we run into quite often because it sounds smart and somebody who talks about "Crawl Budget" they sound like they know what they're talking about...but it seems that it's pulled out way too often in cases where it's not applicable and that points to your content not getting crawled because your content and site isn't considered by Google as worth regularly crawling and spending resources on. Google clearly communicated that unless you have millions of pages, crawl budget really isn't a thing for you.....many times people are not considering that they are not being crawled because their content doesn't justify Google crawling it, the frequency of updating their site doesn't make sense for Google to crawl it regularly....or when Google tries to crawl the site their server resources aren't adequate where Google doesn't want to crush them and bring their site down so it doesn't crawl it fully to avoid causing issues on your site.
Now...we do see situations where a customer has navigation options available on their site that are making MANY permutations of pages that are simply 10 x'ing, 100 x'ing, or more...the number of pages that Google needs to crawl. Then...even sites with just tens of thousands of pages can turn into millions....so that is something to consider. But....if you're content isn't getting crawled you need to understand it can also be because your content and site doesn't have a good reputation from Google....and they don't consider the content worth spending resources on. Be wary if you're "SEO Expert" you're relying on is pointing to this consistently as an excuse of why the content they're creating is not getting indexed.....it may be the case that they're just publishing a bunch of stuff that Google doesn't consider worth crawling (we encourage you to watch Google's whole video on this).
Read more about how Google is judging your content and your site as a whole....
Consider Your Content Quality and How Google Judges Your Site
It's important that the content you publish isn't playing the "Content for SEO" and "Competitor Chasing" game...where the articles you publish are things that have already been covered...and the person who is writing the content has no real and new experience that it's adding value to the world. Google knows that now....they will read through your content and judge whether it's something that has created "information gain" for the world....and they're scoring that page and your website based on it. They've also communicated clearly...if you're just publishing a bunch of stuff that doesn't add value to the world...it's no only going to not help your site...it will hurt it and how Google looks at it. See what they said here on a thread where they're warning customers about playing the (what we like to call) "competitor copycat game"....
Point is.....if Google recognizes your site that most content that you're publishing isn't adding value to the world....why would they prioritize your site when considering where/how/when to crawl. Their goal is to crawl the best new information available and ensure their systems/resources/rankings are taking that into account.
If you want Google to take you seriously, identify what your actual customers want to know about (through Answerbase of course) and have somebody on your end fulfill those information requests with their real experience and expertise. This is exactly what Google wants and needs....and they'll be hungry to continue coming back to your site regularly to get it.
Be Careful When an "SEO Expert" Throws Out "Crawl Budget"
This is one of these things that we run into quite often because it sounds smart and somebody who talks about "Crawl Budget" they sound like they know what they're talking about...but it seems that it's pulled out way too often in cases where it's not applicable and that points to your content not getting crawled because your content and site isn't considered by Google as worth regularly crawling and spending resources on. Google clearly communicated that unless you have millions of pages, crawl budget really isn't a thing for you.....many times people are not considering that they are not being crawled because their content doesn't justify Google crawling it, the frequency of updating their site doesn't make sense for Google to crawl it regularly....or when Google tries to crawl the site their server resources aren't adequate where Google doesn't want to crush them and bring their site down so it doesn't crawl it fully to avoid causing issues on your site.
Now...we do see situations where a customer has navigation options available on their site that are making MANY permutations of pages that are simply 10 x'ing, 100 x'ing, or more...the number of pages that Google needs to crawl. Then...even sites with just tens of thousands of pages can turn into millions....so that is something to consider. But....if you're content isn't getting crawled you need to understand it can also be because your content and site doesn't have a good reputation from Google....and they don't consider the content worth spending resources on. Be wary if you're "SEO Expert" you're relying on is pointing to this consistently as an excuse of why the content they're creating is not getting indexed.....it may be the case that they're just publishing a bunch of stuff that Google doesn't consider worth crawling (we encourage you to watch Google's whole video on this).
Read more about how Google is judging your content and your site as a whole....
Consider Your Content Quality and How Google Judges Your Site
It's important that the content you publish isn't playing the "Content for SEO" and "Competitor Chasing" game...where the articles you publish are things that have already been covered...and the person who is writing the content has no real and new experience that it's adding value to the world. Google knows that now....they will read through your content and judge whether it's something that has created "information gain" for the world....and they're scoring that page and your website based on it. They've also communicated clearly...if you're just publishing a bunch of stuff that doesn't add value to the world...it's no only going to not help your site...it will hurt it and how Google looks at it. See what they said here on a thread where they're warning customers about playing the (what we like to call) "competitor copycat game"....
Point is.....if Google recognizes your site that most content that you're publishing isn't adding value to the world....why would they prioritize your site when considering where/how/when to crawl. Their goal is to crawl the best new information available and ensure their systems/resources/rankings are taking that into account.
If you want Google to take you seriously, identify what your actual customers want to know about (through Answerbase of course) and have somebody on your end fulfill those information requests with their real experience and expertise. This is exactly what Google wants and needs....and they'll be hungry to continue coming back to your site regularly to get it.
