I’m trying out Google Custom Search in my AngularJS web application. The ultimate goal is to improve the quality of our website’s search results, and Google Custom Search seems like an appealing (and less expensive) alternative to something like Algolia or ElasticSearch.
I’ve created a Custom Search Engine (CSE) and specified it to return results from my website’s product pages, i.e.
... is my website’s domain.
The issue is, for any search query I submit, the CSE returns an impossibly small number of results. Even for the most common queries, like “dress” or “pants”, for which I know we have thousands of products, only a small handful (somewhere from 0 to 4) are getting returned in the results.
We have meta data (title, image, description, etc.) provided for all of the pages.
So, I have two questions:
- Why are there so few results, when I’m certain we have hundreds of product pages matching these search queries?
- How can I get the CSE to include the missing product pages?
My best attempt at a guess for the first question would be that the Googlebot has not indexed the “missing” product pages, although I am not sure how (or if) this hypothesis could be verified. I attempted to research if there are ways to see which pages of your website have been indexed by Googlebot, but from what I can tell, the available methods violate Google’s terms of service.
Furthermore (if I’m correct in assuming that the missing product pages haven’t been indexed), have they not been indexed because they’re inside a single page application? (If so are there viable work-arounds for this? We tried
prerender.io at one point but it turned out to have no effect.)
Or, could it be that the missing pages have not been indexed due to sheer volume? We have a database with hundreds of thousands of products, each one with a corresponding product page. Does Googlebot typically index all of a website’s pages? Or only a small sampling?
Any comments, suggestions, or insights are much welcomed and appreciated. Thanks all.