What it fetches
-
Public search and listing pages of classifieds sites, and the public sitemaps some sites
publish. Nothing behind a login, paywall or CAPTCHA.
-
Only sites cleared for crawling after a review of their robots.txt, comments included, and
their terms of service. Each clearance is recorded.
- Every request carries the user agent
FindBytesBot/0.1 (+https://findbytes.ai).
How often
- A sweep runs every 6 hours.
-
Each search is paced to about one request a second and reads about the 25 newest listings.
A site gets at most 60 searches per sweep.
-
Listings not seen for a week may be re-checked, at most 50 per site per sweep, so ads that
have ended leave search.
What it keeps
-
The offer, not the person: the title and ad text, price, place, category details such as
the number of rooms or a car's year, links to the photos and a link back to the original ad.
-
Contact details are left out by design: there is no field for a person's name, phone
number or email address, and a filter removes the phone numbers and email addresses it
finds in ad text before the text is stored.
- Photos are linked, never copied, and every result links back to the ad on your site.
What it never does
- No proxies or rotating addresses, no disguised user agents, no workarounds for bot challenges.
-
A 403 or 429 answer stops the sweep for that site. A site that keeps refusing our searches
is taken off the list.
Opt out
Email hi@engineeredby.ai with your domain: we stop crawling it, and
its listings leave search 14 days after we last saw them.
Changes to robots.txt are not picked up automatically yet, so email is the way to opt out.
Offer a feed instead
If you run a classifieds site and have a feed or an API, or would rather send us listings
than be crawled, write to us: FindBytes
prefers an official feed to reading pages.