Skip to main content

Excluding and Including Content in AddSearch

By default, AddSearch indexes all pages of your website. To control which pages or content appear in your search results, you can exclude or include specific areas or content using several methods described below.

You can exclude pages or site areas from your search results in the following ways:

1. Using the AddSearch Dashboard​

  • Log in to the AddSearch Dashboard.
  • Navigate to Manage results > Site Areas.
  • Enter the path or pattern you want to exclude (e.g., /temp/) in the text field.
  • Select Exclude from the dropdown.
  • Save your changes.

These changes take effect after the next site recrawl, which might take some time for larger sites.

2. Using robots.txt​

Add rules to your robots.txt file located at your website's root to tell crawlers which pages to avoid.

  • To exclude pages for all crawlers:

    User-agent: *
    Disallow: /temp/

  • To exclude pages only for AddSearch's crawler:

    User-agent: AddSearchBot
    Disallow: /temp/

3. Using Robots Meta Tags​

Place meta tags inside the <head> section of your pages to instruct crawlers:

  • Exclude the current page from indexing and following links:

    <meta name="robots" content="noindex, nofollow">

  • Exclude the current page from indexing but allow following links:

    <meta name="robots" content="noindex, follow">

  • Include this page but do not follow links:

    <meta name="robots" content="index, nofollow">

To target AddSearch only, replace name="robots" with name="AddSearchBot".

Add rel="nofollow" in a link to prevent the AddSearch crawler from following or indexing the linked page.

<a href="/page-to-exclude" rel="nofollow">Link to excluded page</a>

Controlling Content Within Pages​

AddSearch automatically indexes the main content of your pages, avoiding sidebars, headers, footers, and duplicate content. To manually specify content inclusion or exclusion:

  • Include specific content: Add the attribute data-addsearch="include" to a container element such as <div> or <p>.

    <div data-addsearch="include">This content will be indexed by AddSearch.</div>

  • Exclude specific content: Add the attribute data-addsearch="exclude" similarly.

    <div data-addsearch="exclude">This content will be excluded from search results.</div>

Additional Options​

If multiple page versions exist, use canonical links to specify which page should be indexed. See Canonical Links documentation for details.

6. Ignoring Specific URL Parameters​

Define URL parameters that AddSearch should ignore when crawling links. See Ignoring URL Parameters for setup instructions.

For all methods, ensure you verify the effects after recrawling your site to confirm content is indexed or excluded as intended.