THEHARVESTER: COLLECT PUBLIC EMAILS AND HOSTS, THEN SORT THE NOISE

Other people's indexes, one domain, a pile of strings

theHarvester takes a domain and asks public sources what they already know about it. Search engines, certificate transparency logs, passive DNS datasets, code hosts, and a long list of other providers. It then files what comes back under a few headings: hostnames, email addresses, IP addresses, URLs, ASNs, people, and breach names, plus whatever structured evidence an optional action attached. The current README on github.com/laramies/theHarvester is the description that counts. This paragraph is a briefing, and briefings go stale.

The job in this lesson is the early, quiet pass. You want a list of addresses and hostnames that public sources already associate with a domain you control, and you want each line labelled with where it came from and whether that source actually finished. You are sorting a pile. You are not proving that a mailbox receives mail, that a hostname still resolves, or that a name on a page is an employee.

The project splits network behaviour into three classes. P0 queries a provider or a dataset and does not send traffic at the target. Search, certificate logs, code search, and passive DNS sit here. P1 asks DNS about names or addresses: resolution, brute force, recursive lookup, reverse lookup. P2 contacts the target: HTTP, TLS, screenshots, takeover checks, virtual hosts, ports, API paths. P0, P1, and P2 describe what the traffic does. They are not a confidence score. -b all runs the catalogued P0 sources. P1 and P2 run only when you select them. This lesson does not select them.

Passive still has an audience. The provider receives the domain string. It may log the query. It has quotas and terms. "I never touched the server" is true of a P0 run and incomplete as a description of who else was involved. The project's own responsible-use note says this in plainer words than most blog posts manage. Select the providers the question needs. A key you happen to own is not permission to query every source that will accept it.

Sources divide, roughly, into three practical buckets, and the README's discovery-sources table is the bucket list. Some run with no API key. When this lesson was checked, that no-key P0 set included crtsh and certspotter (hostnames from certificate logs), waybackarchive and commoncrawl (hostnames from web archives), duckduckgo and yahoo (hostnames and emails), otx, urlscan, gitlab, rapiddns, and others. Some run only with a key in api-keys.yaml: hunter, virustotal, github-code, censys, brave, securityTrails, and a lot of the table. A few are marked optional, meaning they can run without a key: hackertarget, mojeek, windvane were in that column at the same check. If a name above is missing from your theHarvester -h, the help text is right and this paragraph is old. Do not memorise the table. Read it on the day.

A keyed source with no key is skipped. The configuration guide says so, using the verified-domain HIBP source as the example and then noting that other unavailable keyed sources are skipped the same way. A skip means you did not ask. It does not mean the provider holds nothing, and it does not mean the domain has no mail.

You can also select by capability instead of by name: subdomains, emails, ips, asns, urls, people, breaches. Those selectors are a union. They choose which sources run. They do not throw away other result types those sources happen to return. -b emails is "sources that can contribute emails", which includes sources that will skip without a key. A thin email list after that command is often a key problem wearing an empty-looking coat.

Noise and signal are a labelling problem. A hostname from a certificate log is a historical string with a source attached. An email from a search engine is a string some indexed page contained. A role address such as security@ or info@ is a mailbox shape, often a shared one. A person string, if a source actually returns one, is a name a page published, which is a long way from "works here now". The signal is the row you can cite: value, type, source, and that source's outcome. Everything else is a lead you have not earned.

theHarvester is GPL-2.0 only. Running it for this exercise is ordinary use. If you later ship a modified copy, the licence text in the repository is the obligation, not a vibe about "open source". Christian Martorella wrote the tool. The README names the current maintainers. You do not need their biographies to run a query.

Flags and source names move between releases. Kali's package in particular can lag the repository. Before you repeat a command from this page, from a cheatsheet, or from a video, run theHarvester -h and skim the project README. Where they disagree with this lesson, they win.

A long list feels like progress. The useful artefact is the short list where every row has a source and an outcome you can say out loud.