For example, with e-commerce sites, you might create separate jobs for "product pages" and "listing pages." For large-scale sites, consider creating additional jobs for "product categories.”
For media sites with frequent new articles, we recommend using RSS feeds to track when new articles are indexed.
The optimal job design varies depending on your website, products, and URL structure.
When designing jobs, it's important to consider page update frequency and which pages are most important to get indexed. You can also adjust crawl frequency for each job to enable efficient index management.
Use URL filters to narrow down results to various states. You can also apply filters by clicking on status rows. This allows you to view "URLs that were discovered but not crawled" and "URLs that were crawled but not indexed.”
When important pages aren't being crawled or indexed, investigate the underlying factors:
Through this type of analysis and comparison, you can identify factors preventing crawling and indexing.
<aside> 🖌️ Example: When extracting and manually reviewing only non-indexed pages, we found they were all pages with minimal content volume.
</aside>
<aside> <img src="/icons/save_red.svg" alt="/icons/save_red.svg" width="40px" /> Note: You can also export detailed data to CSV or Excel formats.
</aside>
When pages are crawled but not indexed, possible causes include low content quality, similar content elsewhere, or tag issues.
For large-scale sites with low crawling or infrequent crawls, crawl budget might be wasted on unimportant pages. Effective solutions include controlling unnecessary parameters with robots.txt and optimizing sitemaps to prioritize crawling of important pages.
Submitted and indexed: