A web crawler is a bot used by search engines to scan and index website content for search results. Here's how to control your site's visibility in search engines:

Block All Search Engines

1. Go to Settings > Crawlers
2. Check "Block known AI crawlers"
3. This adds a robots.txt rule preventing search engine indexing

Hide Specific Pages

You can exclude individual pages using two methods:

Page Settings Method:

• Open Pages panel
• Click SEO tab on desired page
• Enable "Hide page from search results"

Benefits:

• Available on all plans
• No coding required
• Removes page from sitemap
• Works for collection pages and their items

Code Injection Method:

• Add this meta tag to page header:

<meta name="robots" content="noindex" />

Benefits:

• Works on homepage
• Keeps page in sitemap
• Available for advanced customization

Important Notes:

• Hiding a collection page also hides all items within it
• Index page settings don't affect subpages
• Must add noindex to both main index and subpages for complete exclusion
• These settings only affect external search engines, not internal site search
• Changes may take time to reflect in search results

Remember: Blocking search engines still allows direct visitor access to your site. For controlling visitor access, use site privacy settings instead.

These methods ensure search engines respect your content visibility preferences while maintaining site functionality for intended visitors.

Related Articles

Previous Articles