On this page
AI Agent Analyzer
See which AI crawlers request your website, investigate failed requests, and review your pages’ access preferences.
1. Start with recorded crawler activity
An AI crawler is an automated visitor that requests pages. AI Agent Analyzer helps you see the recorded requests, the pages involved, and responses that need attention.
Choose your project
Check your website in the app navigation, then open AI Agent Analyzer.
Read the connection status
Look for Connected, Waiting for delivery, Saved results, or a message that needs attention. If the feature is unavailable for your account, follow the displayed guidance; saved results are retained.
Open Overview
Start with the recorded activity and its dates. If you have no source yet, use the connection or upload steps below. Access checks can be used without a log connection when available.
2. Connect your website or upload saved logs
An account Admin manages website connections. Members with editing access can upload logs and run access checks; viewers can read the available reports. In Overview preview, connections and access checks are managed from current results.
Open the setup
Select Connect a source, or Connections if you already have one. Choose Cloudflare, Vercel, WordPress, or Other platform to match your website.
Check the website and connection name
Review the hostname and give the connection a recognizable name. Continue through the selected platform’s instructions. Depending on the platform, you may need website administrator access or help from the person who manages your hosting.
Finish the platform-specific instructions
Review the available authorization and website selection, or follow the displayed installation and pairing steps for WordPress. Use the confirmation offered by your selected path. If setup is unavailable or reports an error, resolve that message before expecting new activity.
Confirm delivery and activity separately
Read the checklist: Connection details saved, First delivery received, and AI crawler activity detected. Saving details alone does not prove delivery, and receiving a delivery does not prove that a recognized AI crawler has requested a page.
- Cloudflare
- Vercel
- WordPress
For an existing file, select Upload logs or Upload a log file. The upload flow accepts CSV, JSON, NDJSON, combined logs, or gzip files up to 100 MiB.
Upload and review field matches
Choose your file, then check Timestamp, Requested URL, and User agent in the Map step. Review optional fields such as Response status when available. A missing response remains unknown.
Check the preview
Review the website, UTC period, Recognized AI requests, Other requests, and Unreadable records. Only recognized AI requests are included. Check your account’s allowance and detailed-history window before continuing.
Import and read the final outcome
Select Import recognized requests, then wait for processing to finish and review the outcome. An uploaded file provides saved history; it does not establish continuing delivery from your website.
3. Choose the period and bots to review
Set the reporting period
Use the date-range calendar or a 7-, 30-, or 90-day preset. Crawler reporting uses UTC. Read the displayed detailed-history retention and daily-summary coverage alongside the dates.
Narrow the activity
Use AI platforms, Bot purpose, and AI bots. Where a project has several sources, use Connections to select one. Bot purposes help distinguish search, user-requested activity, training, and other activity.
Return to a broader view when needed
Use Clear filters for platform, purpose, bot, and connection selections. Check your dates separately. In Pages or Bots, also review the search text and Response status selection.
4. Read Overview without overinterpreting it
- AI requests
- Recorded requests from recognized crawler identities within the selected scope. Repeated requests count separately; this is not a count of people or AI answers.
- Active bots
- Distinct recognized bot identities observed in the report. A platform can use several bots for different purposes.
- Pages requested
- Distinct website-and-path combinations requested. Query variations are grouped together.
- Failed requests
- Requests with a recorded error response. The displayed failure percentage uses requests with known responses; unknown responses are not treated as successful.
Use Crawler activity to review the trend and change its daily, weekly, or monthly interval. The AI requests comparison uses the preceding equal-length period. Select Active bots to open Bots, Failed requests to investigate errors, or a most-requested page to open its details.
AdvancedRequest identity is not model identity
Bot names and versions describe the identity supplied with a request. Identified by user agent is different from independently verified identity; verification is shown only when supported. A bot version does not identify the AI model that may later use content. Shared search crawlers may serve several purposes, and policy-only entries in Access do not create request activity.
5. Investigate a page or bot
Open Pages or Bots
Use Pages to find a requested page, or Bots to compare bot identities, purposes, and observed versions. Search with Find a page… or Find a bot… and review Response status. Sort by the available headings and use pagination for more rows.
Open a row
Select the page or bot name. Read its request and failure totals, then review Recent requests for the recorded time, page or bot, response, and identity label.
Check the evidence separately
Page activity can show Measured citations and links to saved citation evidence. Review those answers independently. A crawler request and a citation on the same page do not establish a causal connection.
Follow up on errors
Review the response code and page with the person who manages your website. Check this page opens the URL tester for that page; review the URL before starting a check.
Use Export in Pages or Bots to download the matching table with its current filters. If the report says it is showing the first 5,000 matches, narrow the selection before treating the table as a complete list.
6. Review access preferences and page responses
Open Access and Robots.txt
Select Access, then Robots.txt. Review the latest saved check, its date, each bot’s Policy result, and the explanation. If you have editing access, use Check robots.txt to request a fresh check.
Test a specific project page
Open URL tester, enter the intended Page URL, and select Check page. Review the returned response, final URL, readable content, indexing preferences, and recommended next steps. This checks the page; it does not change your website.
Review saved checks
Open History to see saved access checks and reported changes to bot preferences. Compare their dates with the period of the recorded crawler requests.
AdvancedPolicy and observed responses answer different questions
An Allowed or Blocked policy result describes the saved access rules for a bot. Recorded request responses describe what the website returned at another time. Some user-requested actions may follow different rules. Review both observations and their dates before changing access preferences with your website administrator.
7. Resolve missing or paused activity
Why does a saved connection say Waiting for delivery?
Setup details have been saved, but the first delivery has not been received. Return to Connections and review the selected platform’s setup and delivery checklist. Saving details or confirming the setup does not establish crawler activity.
Why did a WordPress connection miss some requests?
The plugin sees requests that reach WordPress. Responses served entirely by a cache or content delivery network can fall outside its coverage. Review the source’s coverage before comparing totals with hosting reports.
What should I do when collection is paused?
Check the source message and your account’s request allowance. Allowances are shared across the account, and detailed-history limits depend on your plan. Ask your Admin to review the account or connection; refreshing the report does not remove a pause.
Why does Connected appear alongside an error?
A project can have several sources. One may be receiving deliveries while another needs attention, is paused, or is waiting. Review each connection rather than relying on the overall label alone.
Does refreshing the report make a crawler visit?
No. The refresh control reloads the available reporting. It does not ask an AI crawler to visit, create a citation, or run a prompt check.