Skip to main content

Authenticated Crawling

AddSearch bots typically crawl publicly accessible content, but you can also index content protected by login credentials. This guide explains how to enable authenticated crawling and secure your search index.

Supported Authentication Methods​

AddSearch supports crawling websites behind these login mechanisms:

If your website uses a different method, please contact AddSearch Support to explore customized crawling options.

Setup Process​

  1. Contact AddSearch Support to initiate authenticated crawling configuration for your account.
  2. Provide the necessary authentication details and parameters for the supported login method.
  3. AddSearch will configure our crawler to access and index your protected content.

Alternative Methods for Indexing Private Content​

If crawling protected areas is not feasible, you can use these alternatives:

  • Whitelisting AddSearch Bots: Allow our crawler IP addresses to bypass access restrictions. See more in Whitelisting AddSearch Bot.
  • Indexing API: Programmatically push private content to the index, offering full control over what and how data is indexed. Learn more in Indexing API.

Securing Your Search Index​

When indexing protected content, securing your search index is critical. Review our documentation on using JWT tokens for securing access to your search results.

For detailed assistance or if your authentication method is not listed, reach out to AddSearch Support for tailored help.