About QuickScryBot
QuickScryBot is the crawler behind QuickScry, a search engine for public forum discussions. If you run a forum and see it in your logs, here is what it does and how to control it.
How it identifies itself
QuickScryBot/0.1 (+https://quickscry.com/bot; [email protected])
How it behaves
- It reads and obeys
robots.txt, includingCrawl-delay, and re-checks it at least daily. If your robots.txt can't be read because of a server error, it stops crawling your site until it can. - It fetches from one site at most once every 5 seconds by default, slower if you ask for it, and slows down further on its own when your server responds slowly.
- It backs off automatically on
429and503responses and honorsRetry-After. If your site keeps failing, it stops visiting until someone checks on it. - It sends
If-None-Match/If-Modified-Sinceand accepts compressed responses, so unchanged pages cost you very little. - It only reads public pages. It does not log in, solve challenges, or work around blocks.
- On Discourse and Flarum forums it prefers the public JSON endpoints, which are lighter on your server than HTML pages. When it re-checks a long thread for new replies, it reads only the last page.
How to block or slow it
Add a rule for QuickScryBot to your robots.txt, for example:
User-agent: QuickScryBot Disallow: / # or, to slow it down: User-agent: QuickScryBot Crawl-delay: 30
To have content already collected removed, see removal requests.