A page can be well written, load fast and sit in a sensible structure — and still bring no traffic at all, simply because Google has not indexed it. Indexing is the first threshold a site has to clear, and until it is cleared the rest of the work does not count.
This article merges three separate pieces we had on indexing and rewrites them from scratch: how the process works, how to check your own pages, why they drop out of the index, and what actually speeds up getting into Search.
A page reaches Search through three stages, and it can fail at any of them.
Two things worth settling straight away.
Indexed does not mean ranking. The index is the database search results are built from. Being in the database promises nothing about position.
Not being indexed is not always a fault. Utility pages, internal search results, filter URLs and the cart have no business in Search. The goal is not to index everything, but to have in the index exactly the pages that should bring people in.
Google Search Console is the primary tool. The Page indexing report shows how many URLs are indexed, how many are not, and why. Next to each status sits the list of actual URLs — that list is the working material.
Checking a single URL. The URL Inspection tool shows whether a page is indexed, when it was last crawled, which canonical Google picked, and what the rendered HTML looks like. This is where the gap between what you wrote in the code and what the crawler actually saw becomes visible.
The site: operator. A site:yoursite.com query gives a rough idea. Rough is the operative word: the number it returns does not match Search Console and is not meant to. The operator shows a sample rather than the full index, so use it to check one page quickly, not to count anything.
When the two disagree, Search Console is right.
Search Console names the reason plainly. The problem is that the status names do not tell you what to do about them.
| Status | What it means | What to do |
| Discovered — currently not indexed | Google knows the URL but has not crawled it yet; it held back to avoid overloading the site | Check server response time and internal linking. A lot of these URLs is a crawl budget signal |
| Crawled — currently not indexed | The page was fetched and then left out of the index | Usually a question about the page’s value, not its technical state. See section 5 |
| Page with redirect | The URL returns a redirect | Fine if the redirect is intentional. Check against the HTTP status codes |
| Duplicate without user-selected canonical | Google treats the page as a duplicate and picked a canonical itself | Set the canonical tag explicitly |
| Duplicate, Google chose different canonical than user | Your canonical was ignored | The pages are too similar. Either separate them by content or accept Google’s choice |
| Blocked by robots.txt | The crawler cannot reach the page | Review the rules. The block is often left over from development |
| Excluded by ‘noindex’ tag | An explicit instruction not to index | Remove the tag if the block is a mistake |
| Soft 404 | The page is effectively empty but returns a 200 | Add real content or return an honest 404 |
| Server error (5xx) | The server did not serve the page to the crawler | Urgent: sustained 5xx slows crawling across the whole site |
Two of these are confused constantly. Discovered means the crawler has not been yet. Crawled means it came, looked and moved on. Different problems, different fixes — and treating the first one with remedies for the second achieves nothing.
| Cause | How to check |
| An indexing block left in place after development | URL Inspection shows it in under a minute. The single most common case of all |
| Canonical tag pointing elsewhere | URL Inspection reports the canonical Google selected. CMS templates sometimes point it at the homepage |
| Redirect chains | A crawler will show them: every extra hop slows the pass, and a long chain may not be followed to the end |
| The page is unreachable through navigation | A crawl starting from the homepage will not find it. This is where a considered site structure pays off |
| Content appears only after JavaScript | Compare the raw HTML with the rendered HTML in URL Inspection |
| Slow server, frequent 5xx | The Crawl stats report. Google lowers crawl frequency on a site that responds poorly |
| A sitemap full of junk | Run the sitemap URLs through a crawler: redirects, 404s and blocked pages in it reduce trust in the file |
For the technical side a crawler does the job — Screaming Frog, Netpeak Spider or Sitebulb. Run it from the homepage and compare its list of URLs with the sitemap: anything present in one and missing from the other is worth a look.
If nothing is blocking the page technically and the status is "Crawled — currently not indexed", the page itself is the issue. Google looked and decided it adds nothing to the index.
The usual suspects:
The fix here is not technical: either the page needs a reason to exist, or it should not be in the index at all.
| Method | When it works |
| A current sitemap in Search Console | Always. Basic hygiene: only URLs that return 200 and are open to indexing |
| Request indexing through URL Inspection | For individual important pages. It is a request rather than a command, and it does not scale |
| Internal links from active pages | The most underrated method. A page linked from a frequently updated section gets crawled sooner |
| Updating the section the page sits in | The crawler returns to where content changes |
| A fast server | Directly affects how many pages the crawler gets through per visit |
| External links | They work, but as a side effect — nobody buys links for indexing |
| The Indexing API | Not for ordinary pages. Google officially supports it for job postings (JobPosting) and livestreams (BroadcastEvent) only. Advice to use it for articles and product pages contradicts the documentation |
That last row deserves emphasis. The Indexing API is routinely recommended as a universal accelerator, while Google’s documentation names exactly two content types it is built for.
Crawl budget is the number of pages a crawler is willing to fetch from a site in a given period. The topic is popular in blog posts and irrelevant to most websites.
Google draws the line plainly: the crawl budget guide is for large sites of a million pages or more whose content changes at least weekly, and for medium sites of 10,000 pages or more whose content changes daily. Sites with a large number of URLs sitting in "Discovered — currently not indexed" are named separately.
For everyone else the wording is just as direct: if your pages are crawled the same day they are published, you do not need to read that guide — keeping the sitemap current and checking the Page indexing report is enough.
In practice that means a corporate site of 200 pages or a shop with 3,000 products has nothing to optimise here. If something is not being indexed on those sites, the reason is in sections 4 and 5, not in the budget.
We check what is keeping pages out of the index as part of an SEO audit, and fix it during on-page optimisation. The official crawl budget thresholds are in Google’s documentation.
Get in touch!
