Case study · Web Scraping

Email Scraper

Recursively crawl websites to extract, deduplicate, and optionally validate email addresses.

Overview

Scrapes email addresses from supplied URLs and recursively explores linked pages up to a configurable depth. Extracted emails are deduplicated and can optionally have their domains validated using DNS records.

Problem

Finding email addresses across an entire website requires following linked pages while preventing duplicate results and optionally checking whether discovered email domains are valid.

Solution

Uses configurable recursive crawling to discover email addresses, stores only unique results, and optionally validates domains using MX records with DNS-over-HTTPS fallback.

How it works

- Email address extraction - Recursive crawling - Configurable maximum crawl depth - Automatic email deduplication - Optional DNS validation - MX record validation - DNS-over-HTTPS fallback - Proxy configuration - Configurable minimum and maximum concurrency - Structured dataset output