Skip to content
Sentinela

Our crawler

SentinelaBot

The crawler that runs Sentinela's automated accessibility checks. Here is how it identifies itself, how it crawls, and how to allow or block it.

What SentinelaBot is

Every request from our scanner sends this User-Agent:

SentinelaBot/1.0 (+https://sentinela.systems/bot)

It visits a site only when someone asks us to check it: a free scan requested on this site, or a store its owner has added to a Sentinela account for monitoring or an audit.

What it does on a site

  • Reads robots.txt and sitemaps, then picks a sample of pages by type: home, collections, products, cart.
  • Loads those pages in a headless browser and runs automated accessibility checks against WCAG 2.2 AA.
  • May add a product to the cart to test the purchase flow with a keyboard. It stops when the checkout page loads and never enters shipping or payment details, so it cannot place an order.

How it crawls

  • At most 2 page loads or crawl requests to a host at the same time. A page's own images, styles and scripts then load as they would in a browser.
  • At least 500 ms between the start of one page load or request and the next.
  • A Crawl-delay for SentinelaBot in robots.txt slows it down further (up to 10 seconds between requests).
  • On sites whose owner has not verified them with us, it follows robots.txt. If robots.txt cannot be read because of a server error, it does not crawl. An owner who verifies their site in Sentinela can have it checked in full.

To keep SentinelaBot off a site that has not been verified, add to robots.txt:

User-agent: SentinelaBot
Disallow: /

Allowing it through a firewall

If you asked us to check your store and the scan fails with errors such as 403 or 429, or shows a challenge page, your firewall is probably blocking the crawler. Allow requests whose User-Agent contains SentinelaBot/1.0.

Cloudflare

  • In Security → WAF → Custom rules, create a rule with the expression below and the action Skip, then choose the protections to skip (for example the managed rules, rate limiting, or Super Bot Fight Mode).
  • Cloudflare's Bot Fight Mode on its Free plan cannot be skipped by a rule. If it blocks the crawler, turn it off while the scan runs.
(http.user_agent contains "SentinelaBot/1.0")

Other firewalls and security plugins

WordPress security plugins such as Wordfence, and most hosting firewalls, let you allowlist a User-Agent. Shopify's bot protection rarely blocks it; a store behind a password page needs the storefront password added as the site credential in Sentinela.

Anyone can send any User-Agent. Keep the rule as narrow as you can, for example limited to your own store's hostname, and remove it when you no longer use Sentinela.

Contact

To ask a question about SentinelaBot, report a problem it caused, or ask us to stop scanning a site, email us with the site's address and, if you have them, the time of the requests and the IP address they came from.

hello@sentinela.systems

A stop request is handled by a person. We remove the site from the account that asked for the scan, and the robots.txt rule above keeps the crawler off a site that nobody has verified with us.