Web crawlers scan websites to index content for search engine results. Here's how to exclude your site or specific pages from search engine indexing:

Exclude Your Entire Site:

1. Go to Settings > Crawlers
2. Check "Block Search engine crawlers"
3. This adds a rule to your robots.txt file preventing search engines from indexing your site

Exclude Specific Pages:

You can add a noindex tag using two methods:

Page Settings Method:

• Navigate to Pages panel
• Click SEO tab in page settings
• Toggle "Hide page from search results"

Benefits:

• Available in all plans
• No coding required
• Removes page from sitemap
• Pre-enabled on demo pages

Code Injection Method:

• Add this code to Page header:

<meta name="robots" content="noindex">

Benefits:

• Works on homepages
• Page remains in sitemap
• Available in premium plans

Important Notes:

• Excluding a collection page also excludes all its items
• Individual collection items can't be excluded separately
• Excluding an index page doesn't exclude its sub-pages
• These methods only affect external search engines, not internal site search

The noindex tag ensures search engines won't display your content in search results while keeping pages accessible to direct visitors.

For AI crawler exclusion or additional access controls, refer to your platform's specific documentation.

Related Articles

Previous Articles