BestAffiliateProgramsBot
The crawler BestAffiliatePrograms.cc uses to read public affiliate-program pages, so the commission rates and cookie windows in this directory match what each program publishes.
What it reads, and why
It reads the public pages where a brand describes its affiliate program, and a brand's homepage to find that page. From them it takes the facts a program states for anyone to see: the commission, the cookie window, and a one-line description of the brand. Each fact is shown on the program's page in this directory with the address it was read from and the date it was checked. Read the verification method for how a figure is accepted.
It identifies itself with this user agent:
Mozilla/5.0 (compatible; BestAffiliateProgramsBot/1.0; +https://bestaffiliateprograms.cc/bot)
How it behaves
- Before it reads a site's pages for terms, it reads the site's
robots.txtand obeys the rules forBestAffiliateProgramsBot, or for*when there are none for it. - It requests a page from the same site at most once every 1.5 seconds.
- It reads public pages only: it never signs in, sends no cookies and fills in no forms.
- It does not get around a block: the first 401, 403 or 429 response from a site ends its visit to that site.
- It also opens the sign-up links listed in this directory, one request each, to check that they still work.
How to stop it
Add these lines to your robots.txt. It reads the file again within half an hour and stops fetching your site:
User-agent: BestAffiliateProgramsBot
Disallow: /Run a program we list?
You can confirm or correct your published terms, or send a program we do not list yet, on list or correct your program. For anything about the crawler itself, write to hello@bestaffiliateprograms.cc.