Our crawler

ParallaxBot reads news feeds for Parallax. This is who it is, what it takes and how to stop it.

Who it is

Every request carries the User-Agent ParallaxBot/0.1 (+mailto:bot@top-news.uk). It is run by Parallax, which shows one event and how each of seven blocs tells it — in its own words, with a link to every source.

What it fetches

RSS and Atom feeds, and each host's robots.txt. From a feed it keeps the link, the date, the headline and the standfirst, to group reports into events. It does not fetch article pages, does not read behind a paywall, and never displays a headline or any of your text: what readers see is our own writing and a link to you. Feed items are deleted after 14 days.

How it behaves

It reads each feed at most every 30 minutes, honours Crawl-delay, and checks robots.txt before every pass. A Disallow for ParallaxBot or for * stops it. A machine-readable reservation of text-and-data mining (TDMRep, ai.txt, the reservation header) is recorded for every host and honoured. It does not run JavaScript, solve challenges or rotate addresses.

How to opt out

Add User-agent: ParallaxBot / Disallow: / to your robots.txt and it stops on its next pass. Or write to bot@top-news.uk and we will remove your feed by hand, no conditions.