Crawler
LeakBriefBot, the crawler behind LeakBrief briefs
LeakBriefBot reads a few public pages of a Shopify store when an agency asks for a brief. Here are its user agent, limits, robots.txt rules and opt-out.
Who operates LeakBriefBot?
LeakBriefBot is the crawler of LeakBrief, a prospecting tool for Shopify agencies operated by Luis Tschurtschenthaler in Italy.
It visits a store only when an agency asks LeakBrief for a brief about that store, and it reads only pages any visitor can open without an account. It never logs in, adds to a cart or starts a checkout.
How do I recognise its requests?
Every request carries the user agent and From header below, so your logs and firewall rules can match it exactly.
| Header or rule | Value |
|---|---|
| User-Agent header | LeakBriefBot/1.0 (+https://www.leakbrief.com/bot) |
| From header | bot@leakbrief.com |
| robots.txt user agent | LeakBriefBot |
Which pages does it fetch?
At most 12 public pages per brief, all on the store's own domain, plus one call to Google's PageSpeed Insights API.
- robots.txt, read first, before any other page
- The home page
- /meta.json, for the store name, currency and country
- The first page of the public product catalog, /products.json?limit=250
- Up to two product pages
- The shipping, refund and terms of service pages, to see whether each one answers
- The home page and one product page, rendered once in a headless browser at mobile size, with stylesheets and scripts but no images, media or fonts
- Google PageSpeed Insights for the home page's real-user speed data, a request to Google, not to the store
Which pages does it never fetch?
It never requests a cart, checkout, account or order page, a Shopify API or anything behind a login.
Requests to these paths are refused before they leave the crawler, also under a locale prefix such as /fr-ca/:
/cart/cart.js/cart/*/products/*.js/products/*.json/recommendations/*/search/suggest*/checkout*/checkouts/*/account*/orders*/admin*/api/*/password*/challenge*
It also never sends a request other than GET, never submits a form and never signs in.
How fast and how often does it crawl?
Slowly: one brief at a time per store, at least one second between requests, and no more than 12 requests per brief.
| Limit | Value |
|---|---|
| Store requests per brief | At most 12; the stylesheets and scripts of the two rendered pages do not count, but obey the same block list |
| Briefs of one store at a time | 1 |
| Time between two requests to one host | At least 1 second |
| Timeouts | 10 seconds per request, 25 seconds per rendered page, 90 seconds per brief |
| Redirects followed | Up to 3, and only within the same registrable domain |
| Repeat requests for the same store | Answered from the last result for 24 hours, without a new request |
Does LeakBriefBot obey robots.txt?
Yes: it reads robots.txt first and skips every path disallowed for LeakBriefBot or for all user agents (*).
When robots.txt disallows the home page, the crawler stops after that one request. When robots.txt answers with a server error, a 401 or a 403, or does not answer in time, it stops as well. A missing robots.txt (404) allows every path.
To keep LeakBriefBot out of your whole store, add these lines to robots.txt:
User-agent: LeakBriefBot
Disallow: /What does it do when a store pushes back?
It slows down or stops, and it never tries to get around a block.
- A 429 with a Retry-After of 30 seconds or less: it waits that long and retries once.
- A longer 429: it stops and tries again no sooner than 10 minutes later, at most 2 more times, then gives up.
- A 401 or 403, a bot challenge or a password page: it stops at once and does not retry.
How can I verify that a request really comes from LeakBriefBot?
Signed requests carry Web Bot Auth HTTP message signatures that you can check against our public key directory.
A signed request sends the Signature, Signature-Input and Signature-Agent headers, and its Signature-Agent points at https://www.leakbrief.com. The Ed25519 public keys sit in the key directory below.
How do I contact the operator or opt out?
Write to us about the crawler or to report abuse, or opt your store out with the form, which takes effect at once.
- Questions about the crawler: bot@leakbrief.com
- Abuse reports: abuse@leakbrief.com
- Opt your store out
- Read the notice for store owners