PraferoBot is the official crawler used by Prafero to perform SEO, technical, performance, and website quality analysis.
PraferoBot visits websites when a user requests an analysis through Prafero. It collects public page information such as HTML structure, metadata, headings, canonical tags, robots directives, response headers, redirects, page status, and basic technical signals.
The crawler operates fully in the public space. It does not attempt to access private dashboards, login-protected pages, payment checkout areas, user account profiles, or restricted administrative backend sections.
Status codes, response headers, SSL redirects, and final destinations.
Page titles, descriptions, headings structure (H1-H6), and canonical tags.
Robots meta tags, robots.txt directives, and X-Robots-Tag settings.
Response latency, page sizes, script counts, and structural assets.
PraferoBot is built to respect website owners and maintain system resources. We ensure the crawler behaves responsibly according to standard web crawler conventions:
Robots.txt Adherence
Strictly respected. If blocked, PraferoBot halts crawl immediately.
Rate Limiting
Typically restricted to 1 request every 5–10 seconds per domain.
Scope Limits
Limited only to user-initiated request URLs and page diagnostics.
Resource Focus
Minimal bandwidth overhead; only retrieves basic header & markup.
If your website uses Cloudflare, WAF rules, or bot management, it might block website analyzers. If you wish to allow Prafero to analyze your site, you can whitelist PraferoBot using these parameters:
Recommended custom expression:
(http.user_agent contains "PraferoBot")Cloudflare Dashboard Navigation Path:
Set Field to User Agent, Operator to contains, and Value to PraferoBot. Choose action Skip and select security checks to bypass (e.g. Bot Management, WAF, etc.).
If you prefer to prevent PraferoBot from visiting your website, add the following directive to your website's root `robots.txt` file:
User-agent: PraferoBot
Disallow: /PraferoBot/1.0 (+https://www.prafero.com/bot)If you need verified IP lists for firewall exceptions, or have questions about crawl volumes, please reach out to our team.
Email Support