About our crawler

If you are a site operator and found this address in your access logs, this page is for you. It says who we are, what we request, and how to make us stop.

Who is calling

Normdiff tracks published changes in security standards and regulations — new versions, amendments, effective dates — and tells subscribers what changed. To do that we read a small, hand-curated list of pages and feeds published by standards bodies, regulators, and national authorities.

Every request we make carries this User-Agent:

ComplianceRadarBot/0.1 (+https://normdiff.securityvp.ai/bot; [email protected])

We never send any other identity. We do not pretend to be a browser, and we do not pretend to be another company's crawler. If a request in your logs claims to be Googlebot, it is not us.

What we actually do

• We read, we do not crawl. There is no link-following and no site discovery. We request specific URLs that a human put on a list, and nothing else on your domain.
• Rarely. Most sources are polled a few times a day. The fastest is every 30 minutes, and where a site asks for a slower pace we go slower than it asks.
• One request at a time per domain, with a deliberate pause between them. We use conditional requests (ETag, If-Modified-Since) wherever a server supports them, so most of our calls cost you a 304 and no body at all.
• We take published text, not accounts or paid content. We do not log in, do not submit forms, do not buy or borrow credentials, and do not fetch anything sold behind a paywall.

What we do about robots.txt

We read it, and we record what it says, but we no longer treat it as the thing that decides whether we may request a page. We would rather tell you that plainly here than quietly rely on the fact that few people check.

The reasoning, for what it is worth: our list is dozens of URLs on public publication pages, refreshed a few times a day. A blanket rule written for search-engine indexing was, in practice, deciding which laws and standards our readers get told about. We decided the honest trade was to stop hiding behind a file and instead be identifiable, cheap to serve, and genuinely easy to stop.

How to make us stop

Email [email protected] and say which domain or paths you want left alone. There is no form and no appeal process. We remove the source and it stays removed — this is a list maintained by a person, not a policy engine, so a one-line email is genuinely enough.

If you would rather act first and talk later, blocking our User-Agent works and we will not route around it. We do not rotate addresses or change our identity to get back in.

If we are costing you money

Tell us and we will fix it, even if you do not want us gone. We can poll less often, switch to a feed or API endpoint you would rather we used, or take a mirror if you publish one. A source that is expensive for you to serve is a source we would rather read a better way.

Contact: [email protected]