Skip to main content

Troubleshooting Demo Creation Failures

If you receive an error while creating a new search demo, it usually means AddSearch's crawler cannot index your website. Follow these steps to identify and fix common issues preventing proper crawling and indexing.

1. Check robots.txt Restrictions​

Your site may have a robots.txt file located at https://your-domain.com/robots.txt. This file instructs web crawlers which parts of your site to index. If this file disallows AddSearch's crawler, it cannot access your pages.

Ensure your robots.txt includes these lines to allow the AddSearch crawler:

# Allow AddSearch crawler
User-agent: AddSearchBot
Disallow:

# Block all other crawlers (optional)
User-agent: *
Disallow: /

For more details on robots.txt, see robotstxt.org.

2. Verify Meta Tags​

Your website's HTML <head> section might contain meta tags that block crawlers. Look for a tag like this:

<meta name="robots" content="noindex">

Replace noindex with index or remove the tag to allow crawling:

<meta name="robots" content="index">

To allow only AddSearch's crawler while blocking others, use a combination of rules.

3. Validate HTML Integrity​

HTML errors may prevent crawlers from processing your pages properly. Even if the site displays correctly to users, invalid or incomplete HTML can cause crawling failures.

Check your site's HTML using the W3C Validator.

4. Consider Dynamic Content Limitations​

Sites that build menus or content dynamically with JavaScript frameworks (e.g., AJAX, React) can block crawlers, including AddSearch and search engines like Google.

If your site uses dynamic content, contact AddSearch support to enable AJAX crawling, which helps crawlers process JavaScript-generated content.

If your website is very small with few links, AddSearch's crawler might not find sufficient pages to index.

Contact AddSearch support if you suspect your site’s link structure is too sparse.

If you encounter issues not covered here, reach out to AddSearch Support for further assistance.