ObjectivCrawler
ObjectivCrawler is the web crawler operated by Objectiv LLC. It collects the official, publicly published text of U.S. federal and state statutes, regulations and agency guidance, so employers can see the employment, benefits and workplace-safety requirements that apply to them, and stay current when those requirements change.
How it identifies itself
User-Agent ObjectivCrawler/1.0
Where it connects from
157.245.139.21 (reverse DNS: crawler.objectivbenefits.com)
A machine-readable list is at /crawler/ips.json.
How it behaves
- It reads and honors robots.txt, including Crawl-delay, before any other request to a host.
- It waits at least 2 seconds between requests to the same host, and longer whenever the site asks.
- It makes read-only requests, including queries to public search pages that publish the law. It never logs in, creates accounts, or attempts to get past access controls or challenges.
- It collects only public legal and regulatory text, and links every item back to its official source.
To limit or block it, add a rule for ObjectivCrawler to your robots.txt. It takes effect on our next visit.