Spatula

Spatula 0.2.0-dev keeps product facts current for the apps that depend on them: names, identifiers such as GTINs and part numbers, and specifications. Prices come later. It collects them for Crowdio, a site about audio gear.

How it behaves

• It obeys robots.txt: the rules for Spatula, or for * when none name it. While a site's robots.txt can't be reached, or answers with a server error or 429, it fetches nothing from that site. A site with no robots.txt (any other 4xx answer) has no rules for it.
• It waits 15 seconds between requests to a site for the datasets it keeps today, never less than a second, and longer when robots.txt sets a Crawl-delay. It backs off when a site is busy or failing.
• When a site refuses it, it waits twice as long on its next visit, up to four times as long. After three visits in a row that a site refused, it sends one request a visit until the site answers again.
• It follows Content Signals: it reads a page with an AI model unless the site's Content Signals say ai-input=no. It uses an AI model only to read pages, and nothing it reads is used to train one.
• It collects facts only, never descriptions or photos. A block is an answer: it never tries to get around one.
• It signs its requests with Web Bot Auth (HTTP Message Signatures), so a site can check that a request naming Spatula came from Spatula. Its public key is at /.well-known/http-message-signatures-directory.

Opting out

Add this to your robots.txt:

User-agent: Spatula
Disallow: /

For anything else, write to bot@spatula.run.