Skip to main content

Crawler Status

The Crawler Status section on the AddSearch dashboard provides real-time information about the indexing and crawling of your site's URLs.

What it shows​

  • Scheduled for recrawl: Number of URLs queued for crawling or recrawling.
  • Status: Current state of the crawling process.

Crawler Status

Recrawl queue details​

Not all URLs scheduled for recrawl will be indexed. Factors that can prevent indexing include:

  • Robots.txt restrictions
  • Server-side permissions
  • Redirects
  • Duplicate content detection

If the value for Scheduled for recrawl is zero, your index is up to date with no pending pages. If this number grows and remains high for several days, please contact AddSearch support.

Crawl status indicators​

  • Crawling: Crawlers are actively processing URLs.
  • Starting soon: Crawling is currently inactive but will begin shortly.

AddSearch processes crawling in batches. The system continuously checks for URLs to recrawl and updates the index accordingly. Typically, the dashboard shows "Starting soon" when the crawlers are waiting to begin the next batch.

When you initiate a full recrawl, the status remains "Starting soon" until the batch starts.