What the crawler does
It checks websites for technical SEO and accessibility issues and uses the results to generate an audit report.
Crawler identity and policy
NOTABIS SEO Auditor crawler
A transparent, respectful crawler used to run SEO audits on websites at the request of their owners or authorised users.
It does not bypass authentication, robots.txt rules, rate limits, WAF controls, CAPTCHAs, or other anti-bot challenges.
What it is not
It is not a search engine crawler, training crawler, general-purpose scraper, vulnerability scanner, or transaction bot.
User agent
Every request identifies the crawler clearly:
NOTABIS SEO Auditor/1.0 (+https://seo.notabis.com/crawler)
Identifier pattern: NOTABIS SEO Auditor
Robots.txt
The crawler fetches robots.txt before starting an audit. It selects the most specific matching User-agent group and follows Allow and Disallow rules using longest-match precedence. It also honours Crawl-delay.
These rules apply to the starting URL, sitemap URLs, and links discovered during the audit.
If robots.txt cannot be retrieved because of a server or network error, the audit stops rather than assuming permission. A normal HTTP 404 is treated as no published robots policy.
Request limits
- Pacing: No more than 60 page requests per minute, unless robots.txt specifies a slower Crawl-delay.
- Audit size: Each audit is limited to a maximum of 100 pages.
- Boundaries: Response sizes, timeouts, redirects, retries, and sitemap processing are also limited to prevent excessive requests.
Consent and safety
Every audit is started by a user who enters the website they want to check. The crawler:
- Uses HTTP and HTTPS only.
- Validates DNS destinations before connecting.
- Remains on the same origin.
- Respects robots.txt.
- Stops when it encounters an edge security or anti-bot challenge rather than attempting to bypass it.
Verification
These machine-readable resources make the crawler’s identity and network origin easy to verify:
Crawler identityhttps://www.seo.notabis.com/crawler.json
Crawler IP addresseshttps://www.seo.notabis.com/crawler-ips.json
The IP list will contain only dedicated crawler egress addresses once they are configured.
Privacy and contact
We retain audit metadata and findings so that we can provide audit reports. Operational logging is designed to avoid retaining page content when it is not necessary.
If you want to block the crawler, report unexpected behaviour, or ask us a question, contact contact@notabis.com or block the crawler using your WAF or server controls.