How to configure Cloudflare and other middleware to allow Overton’s scrapers access to your website
Overton automatically checks for updates to your publication pages, so that we can see when new documents are published and start tracking them.
If you’re a publisher and want to use Overton then it’s important that we’re able to see new documents when they are published – otherwise they won’t appear in the database or accrue citations.
Typically we check for changes once a week using a piece of software called a web scraper.
If you use Cloudflare
Cloudflare is a widely used content delivery network. It sits between your servers and the internet and reduces page loading times. It can also be used to prevent bots and scrapers collecting data from your website.
Overton is well behaved and usually Cloudflare allows our scraper to run without any issue. However, every so often we’ll get blocked from a Cloudflare protected website and we require the site owners to restore access.
To give Overton access you need to:
a) Log in to your Cloudflare account, and select your website
b) Click the Security option in the left hand sidebar, and then on the Tools option in the menu that appears
c) In the IP Access Rules box you need to add Overton’s IP address and mark it as “Allow”
- In the “IP, IP range, country name, or ASN” box, type 176.58.101.93 (this is the IP address of our scraper). It may appear in the dropdown box after you’ve typed it, if so click on it there to confirm it.
- In the “Action” dropdown, select “Allow”
- You can leave the “Zone” and “Notes” fields as they are
d) Now click on “Add” and the new rule should appear below. Make sure that “Action” is set to “Allow”
If you use a different CDN or firewall
Please whitelist http traffic from 176.58.101.93
This is the IP address of Overton’s scraper.
Alternatively if you’re unable to whitelist specific IPs but can whitelist us via a user agent string or HTTP request headers then please contact us – both options are possible, just tell us which you’d prefer.
If you require any support or further details, please don’t hesitate to contact support@overton.io