The specific pages crawler gives you complete control over which pages to include in the knowledge base. It's ideal when you have very targeted content or want to avoid scanning the entire site.
When to use specific pages
- Targeted content: Only some sections of the site are relevant for the assistant
- Precise control: You want to exclude promotional, legal or technical pages
- Limited resources: You have plan limits and want to optimize usage
- Selective updates: Only certain sections change frequently
- Complex sites: The site has structures that would confuse the automatic crawler
How to configure specific pages
Method 1: URL list
- Create a new crawler selecting "Specific pages"
- Enter URLs one per line in the dedicated text box
- You can add complete URLs or relative paths
- Save and start scanning
URL list example:
https://mysite.com/support https://mysite.com/faq https://mysite.com/products/category-a https://mysite.com/about-us https://mysite.com/contact
Specific pages advantages
- Total control: You decide exactly what to include
- Efficiency: Uses only necessary resources
- Quality: Only relevant and quality content
- Performance: Faster scans
- Simple maintenance: Easy to add or remove pages
Dynamic management
Adding new pages
- You can add URLs at any time
- New pages will be scanned on the next update
- No need to reconfigure the entire crawler
Removing pages
- Remove URLs from the list to exclude them from future scans
- Already acquired content will remain in the knowledge base
- You can force manual removal if necessary
Practical usage examples
E-commerce
- Product category pages
- Main product pages
- Product FAQs
- Buying guides
- Shipping and return policies
Business website
- About us and company history
- Services offered
- Case studies and portfolio
- FAQ and support
- Contacts and locations
Blog/Content
- Articles from a specific category
- Guides and tutorials
- Resources and downloads
- Industry glossary
Optimization tips
- Start small: Begin with 10-20 key pages
- Monitor results: Verify that pages are scanned correctly
- Expand gradually: Add pages based on feedback
- Review periodically: Remove obsolete or less useful pages