Type “dark web search engine” and you’ll find a handful of names: Ahmia, Torch, maybe a Tor-only mirror of DuckDuckGo. What’s less obvious is how any of them find onion sites to index in the first place. Google crawls the surface web by following links from page to page, starting from a huge set of seed URLs. Onion services don’t work that way, and understanding why explains a lot about why every onion search engine’s results look thin next to Google’s.
Why a Regular Crawler Can’t Reach .onion Sites
A conventional web crawler resolves a domain through DNS, opens a normal connection, and follows the outbound links it finds on each page. None of that works for an onion service. There is no DNS record for a .onion address, and the only way to reach one at all is by routing the request through the Tor network itself. That means an onion search engine has to run its own Tor client just to fetch a single page, which is far slower per page than crawling the regular internet, and it rules out most of the shortcuts a normal search engine relies on to cover billions of pages quickly.
The Opt-In Reality Behind Onion Search
Ahmia, one of the better-known onion search engines, is built around an opt-in submission model on top of its own crawler. An onion service does not automatically announce itself the way a surface-web page does through sitemaps and backlinks. Ahmia learns about an address from links on pages it can already see, from public sources, or from an operator submitting the URL directly. If none of those happen, the crawler has no way of finding it, no matter how long it has been online.
What a Crawler Like Ahmia Actually Does
Once an address is known, the crawling itself looks fairly ordinary: a Scrapy-based crawler requests each public page it can reach and hands the text to an Elasticsearch index, the same kind of infrastructure a normal search engine uses under the hood. What is different is the filtering step in between. Every site is checked against a maintained blacklist before it enters the index, and pages tied to material like child sexual abuse content or known malware are excluded outright. That review step is manual in part, which is another reason indexing an onion site takes far longer than indexing a regular one.
Why Coverage Stays Thin
Put those pieces together and an onion search engine is always a partial map, not a comprehensive index the way Google aims to be. Some services simply have not been submitted anywhere yet. Others are intentionally kept off any index because the operator does not want them publicly discoverable. This is part of why our own verified directory stays small and manually checked rather than trying to be exhaustive: a short list you can trust beats a long one nobody actually crawled end to end.