FindBytesBot

The crawler behind FindBytes. If you run a website, this page says what it fetches, how often, what it keeps and how to stop it.

What it fetches

  • Public search and listing pages of classifieds sites, and the public sitemaps some sites publish. Nothing behind a login, paywall or CAPTCHA.
  • Only sites cleared for crawling after a review of their robots.txt, comments included, and their terms of service. Each clearance is recorded.
  • Every request carries the user agent FindBytesBot/0.1 (+https://findbytes.ai).

How often

  • A sweep runs every 6 hours.
  • Each search is paced to about one request a second and reads about the 25 newest listings. A site gets at most 60 searches per sweep.
  • Listings not seen for a week may be re-checked, at most 50 per site per sweep, so ads that have ended leave search.

What it keeps

  • The offer, not the person: the title and ad text, price, place, category details such as the number of rooms or a car's year, links to the photos and a link back to the original ad.
  • Contact details are left out by design: there is no field for a person's name, phone number or email address, and a filter removes the phone numbers and email addresses it finds in ad text before the text is stored.
  • Photos are linked, never copied, and every result links back to the ad on your site.

What it never does

  • No proxies or rotating addresses, no disguised user agents, no workarounds for bot challenges.
  • A 403 or 429 answer stops the sweep for that site. A site that keeps refusing our searches is taken off the list.

Opt out

Email hi@engineeredby.ai with your domain: we stop crawling it, and its listings leave search 14 days after we last saw them.

Changes to robots.txt are not picked up automatically yet, so email is the way to opt out.

Offer a feed instead

If you run a classifieds site and have a feed or an API, or would rather send us listings than be crawled, write to us: FindBytes prefers an official feed to reading pages.