Just like files, crawlers must be associated with your assistants to make web content available in the knowledge base. This association allows you to control which assistants have access to which web content.
How to associate crawlers to assistants
Method 1: During assistant management
- Select the assistant you want to configure
- In the side menu, click on "Knowledge base"
- In the "Crawler" section, you'll see all available crawlers
- Check the crawlers you want to associate with the assistant
- Click "Save changes" to confirm
Method 2: During crawler creation
- When creating a new crawler, in the last configuration phase
- You can immediately select which assistants to associate it with
- Save time by avoiding having to do it later
Association strategies
Specialized association
Principle: Each assistant receives only content relevant to its role
Practical example:
- Technical Support Assistant:
- Support section crawler (/support/*)
- Technical FAQ crawler (/faq/technical/*)
- Documentation crawler (/docs/*)
- Commercial Assistant:
- Product catalog crawler (/products/*)
- Service descriptions crawler (/services/*)
- Case studies crawler (/case-studies/*)
Advantages:
- More focused and relevant responses
- Reduction of "noise" from irrelevant content
- Better performance in search
- Precise control of content per assistant
Generalist association
Principle: An assistant with access to all web content
When to use:
- You have only one assistant for the entire site
- The assistant must answer any type of question
- All web content is relevant
- You prefer simplified management
Advantages:
- Simplified management
- Complete coverage of all topics
- Maximum flexibility in responses
Association management
Adding existing crawlers
- You can associate already configured crawlers to new assistants
- Already scanned content becomes immediately available
- No need to restart scanning
Removing associations
- Removing an association doesn't delete the crawler
- The crawler remains available for other assistants
- Content is removed from the assistant's knowledge base
Association monitoring
Assistant view
In each assistant's management you can see:
- Which crawlers are associated
- Status of each crawler (active, error, etc.)
- Number of pages provided by each crawler
- Last update date
Crawler view
In crawler management you can see:
- Which assistants each crawler is associated with
- Total number of assistants using it
- Impact of changes on various assistants
Performance optimization
Load balancing
- Avoid excessive overlaps: Don't associate too many crawlers with similar content
- Distribute content: Balance the number of pages per assistant
- Monitor performance: Too much content can slow down responses
Update management
- Synchronization: Crawler updates automatically reflect on assistants
- Temporal lag: It may take a few minutes for complete update
- Cache: Content may be temporarily stored for performance
Common problem resolution
Content not found
- Check association: Verify that the crawler is associated with the assistant
- Crawler status: Make sure the crawler is active and working
- Content present: Verify that the page was actually scanned
Inaccurate responses
- Too much content: Reduce the number of associated crawlers
- Contradictory content: Avoid crawlers with conflicting information
- Content quality: Make sure pages have clear textual content
Best practices
- Start simple: Associate few crawlers at the beginning and increase gradually
- Test associations: Verify that the assistant finds expected content
- Document choices: Keep track of why you associated certain crawlers
- Periodic reviews: Regularly check the relevance of associations
- User feedback: Use feedback to optimize associations