How the crawler identifies itself.
LeadsForge can fetch a public business website to confirm the site that belongs to a listing. This page is the reference for that crawler.
User agent
Requests identify themselves as:
LeadsForgeBot (+https://leadsforge.app/bot)
The address in that string is this page. https://leadsforge.app/bot/ is the same page.
What gets fetched
A crawl starts from a public website that discovery has already associated with a business. The crawler reads public pages on that site to confirm the website. It requests ordinary web pages over HTTP or HTTPS. It does not log in, submit forms, or open private areas.
Before a crawl, it reads robots.txt for that site and skips paths listed as Disallow. A crawl-delay in that file is honored up to 10 seconds. If robots.txt cannot be fetched, the crawl continues on public pages only.
The first request uses the user agent above. If that response is an error or an access interstitial and contains no page content, a follow-up request may use a standard browser user agent so a public page can still be read. That follow-up is not a login and does not bypass a restriction in robots.txt.
What we do with the page
Page text is used to confirm that a listing matches a public website. We do not publish a copy of your site. The privacy notice describes how research records are stored and how a removal request works.
Ask us to stop
To keep an email address or a domain out of later search results, use the data removal page. We email a confirmation link to that address, or to admin@ the domain, and suppress it after the link is confirmed. The link expires after 24 hours.
You can also write to support@leadsforge.com or use the contact form and choose data removal. A disallow rule for LeadsForgeBot in robots.txt is the right control on your own site. Include the pages you want left alone and allow time for a new crawl to see the file.