Crawl multiple documents using the API

Use the POST /v1/crawler:batch endpoint to crawl new URLs or recrawl existing ones in bulk.

Note: This feature is available only on Advanced and Enterprise plans. Contact support@addsearch.com to enable access.

Request payload

Send a JSON object containing your site key and a list of full URLs to be crawled:

{
  "indexPublicKey": "SITEKEY",
  "urls": [
    "http://foo.com/bar1.html",
    "http://foo.com/bar2.html",
    "http://foo.com/bar3.html"
  ]
}

Field	Description	Type	Notes
indexPublicKey	Your site's public key	string	Find this key in your AddSearch dashboard
urls	List of document URLs to crawl	array of strings	Include protocol (http or https), domain, and path

Response

A successful request returns HTTP status 202 ACCEPTED with the following JSON payload:

{
  "documents_queued_for_crawling": [
    {
      "url": "http://foo.com/bar1.html",
      "docId": "aac7142057fcc5b55e6dd5b70969066d"
    },
    {
      "url": "http://foo.com/bar2.html",
      "docId": "59ead4c38325d6ee98a99c21dbd162f2"
    }
  ]
}

Field	Description	Type	Notes
documents_queued_for_crawling	Array of documents scheduled for crawling	array	Each item contains the URL and the corresponding document ID

Additional details

• URLs are queued and crawled sequentially with small delays to avoid overloading your website's server.
• Ensure your requests include valid authentication. See API authentication for details.

Contact AddSearch support if you need help enabling or using this endpoint.