AI Crawler Analytics

AI crawlers are automated bots from AI companies that visit your website to collect training data. This guide covers monitoring AI crawler activity and controlling access.

AI crawlers are bots operated by AI companies to:

• Collect training data for language models
• Index content for AI-powered search
• Gather information for AI assistants

The CMS automatically detects these AI crawlers:

To view AI crawler activity for your site:

1. Navigate to Sites → Select your site
2. Click Analytics → AI Crawlers
3. View the activity dashboard

The dashboard shows:

• Total Visits: Total number of AI crawler visits in the selected period
• Crawler Breakdown: Pie chart showing visits by crawler
• Top Pages: Pages most frequently visited by AI crawlers
• Recent Activity: Timeline of recent crawler visits
• Blocked Attempts: Number of blocked crawler requests (if access is disabled)

Filter crawler activity by:

• Date Range: Last 7 days, 30 days, 90 days, or custom range
• Crawler Type: Specific crawler or all crawlers
• Page: Specific page or all pages

To allow or block all AI crawlers for your site:

1. Navigate to Sites → Select your site → Settings
2. Find AI Crawler Access
3. Toggle the switch:
   • Enabled (green): AI crawlers can access your site
   • Disabled (grey): AI crawlers are blocked with HTTP 403

1. Click Save

When AI crawler access is disabled:

1. Detection: The system detects AI crawlers by User-Agent
2. Blocking: Returns HTTP 403 Forbidden to detected crawlers
3. Logging: Logs blocked attempts for analytics
4. Regular users: Regular users and search engine crawlers are unaffected

You can also block AI crawlers using robots.txt:

1. Navigate to Sites → Select your site → Settings → SEO
2. Edit the Robots.txt content
3. Add disallow rules for specific crawlers:

1. Click Save
2. Perform a site rebuild to update robots.txt

AI crawlers typically:

• Visit frequency: May visit daily, weekly, or sporadically
• Page coverage: Often crawl all published pages
• Depth: May follow links to discover new pages
• Respect robots.txt: Most (but not all) respect robots.txt directives

AI crawler visits:

• Bandwidth: Consume bandwidth like any other visitor
• Server load: Generate requests that your server must handle
• Analytics: May skew visitor analytics if not filtered
• Content usage: May use your content for AI training

The crawler breakdown shows which AI companies are accessing your site most frequently. This helps you understand:

• Which AI systems are indexing your content
• Relative activity levels of different crawlers
• Trends over time

The top pages report shows which pages AI crawlers visit most. This can indicate:

• High-value content: Pages crawlers find most interesting
• Entry points: Pages crawlers discover first
• Link structure: How crawlers navigate your site

The recent activity timeline shows:

• Visit timestamps: When crawlers accessed your site
• Pages visited: Which pages were accessed
• Crawler identity: Which crawler made each visit
• IP addresses: Source IPs of crawler requests

AI crawler activity data is retained for 90 days. After 90 days, records are automatically deleted.

This retention period:

• Provides sufficient history for trend analysis
• Limits storage costs
• Complies with data minimization principles

Track which AI systems are accessing your content:

1. Review crawler breakdown regularly
2. Identify new or unexpected crawlers
3. Assess whether your content is being used for AI training

If you have proprietary or sensitive content:

1. Disable AI crawler access at the site level
2. Add robots.txt rules for specific crawlers
3. Monitor blocked attempts to verify effectiveness

Identify your most valuable content:

1. Review top pages by crawler visits
2. Compare with regular user analytics
3. Understand what AI systems find most useful

Ensure compliance with content licensing:

1. Review your content licenses
2. Determine if AI training is permitted
3. Block crawlers if necessary to enforce licenses

When deciding whether to allow AI crawlers:

Allow if:

• You want your content discoverable by AI systems
• You're comfortable with AI training on your content
• You want to support AI-powered search

Block if:

• You have proprietary or sensitive content
• Your content license prohibits AI training
• You want to control how your content is used

• Review regularly: Check analytics monthly
• Watch for new crawlers: New AI crawlers may emerge
• Compare with traffic: Assess crawler impact on overall traffic
• Document decisions: Keep records of access control decisions

• Privacy policy: Disclose AI crawler access in your privacy policy
• Terms of use: Specify acceptable use of your content
• Contact info: Provide a way for AI companies to contact you

If you suspect an AI crawler is visiting but not appearing in analytics:

1. Check User-Agent: The crawler may use a different User-Agent
2. Review server logs: Look for unusual bot activity
3. Contact support: Report undetected crawlers for investigation

If AI crawlers continue accessing your site after blocking:

1. Verify settings: Ensure AI crawler access is disabled
2. Check robots.txt: Verify robots.txt rules are correct
3. Rebuild site: Perform a site rebuild to update robots.txt
4. Wait for cache: CDN cache may take time to clear
5. Check compliance: Some crawlers may not respect robots.txt

If analytics numbers seem incorrect:

1. Check date range: Ensure you're viewing the correct period
2. Verify filters: Check that filters are applied correctly
3. Consider retention: Data older than 90 days is deleted
4. Compare sources: Cross-reference with server logs

AI crawler analytics do not track individual users. The system only logs:

• Crawler User-Agent
• Page accessed
• Timestamp
• IP address (for crawler identification only)

The 90-day retention period ensures:

• Sufficient data for analysis
• Minimal long-term storage
• Compliance with privacy principles

Be transparent about AI crawler access:

• Disclose in your privacy policy
• Explain how you control access
• Provide opt-out mechanisms if applicable

• SEO Settings — Configure robots.txt and sitemap
• Publishing — Publish pages for crawler access
• Creating Sites — Configure AI crawler access when creating sites