1. Home
  2. Our crawler

FieldScoutBot

FieldScoutBot

FieldScoutBot is the crawler Datum24h uses to read public business websites so we can understand what a business does before we contact it.

How to recognise it

It identifies itself with this user agent:

FieldScoutBot/1.0 (+https://datum24h.com/bot; business-directory enrichment; contact: bot@datum24h.com)

What it does

  • Follows robots.txt (RFC 9309). Rules for FieldScoutBot, or for *, are obeyed.
  • Reads at most five public pages per site (the homepage plus contact, about and team pages), pausing between requests to the same site.
  • Records public business details such as the company name, services, phone and the email addresses published on the site itself, on the site's own domain only. It doesn't invent addresses: only where a site publishes staff addresses in a clear pattern may that same pattern be applied to other staff listed on that site, and such addresses are marked as inferred.
  • Never logs in, submits forms, solves CAPTCHAs or reaches private network addresses.

Block it

Add this to your robots.txt:

User-agent: FieldScoutBot
Disallow: /

Remove your details

Email bot@datum24h.com or privacy@datum24h.com with your domain. We delete what we hold, add the domain to our do-not-contact list, and our crawler stops visiting it. See our privacy policy.

Speak your idea.
We'll build it.

Start free with the Datum Agent, or tell our team what you need.