There's a whole slice of the internet that's public but almost impossible to search. Content on IPFS is addressed by what it is (a content hash, the CID) rather than where it is, so there's no domain to crawl, no robots.txt, and no single server to point a search engine at. If you don't already have the CID, you mostly can't find the thing. darksearch.xyz is my attempt to fix that, and it has been a side project on and off for five years. This month it got a big overhaul, and since I've just been at the Open Intelligence Summit in SF, it can now be searched by AI agents too.

The Open Intelligence Summit at The Midway in San Francisco.

A Hacktoberfest Idea (2021)

The project started on October 1, 2021 as compscidr/ipfs_indexer, a Hacktoberfest idea: could I write a crawler for IPFS in Rust? The first commit was literally the libp2p ping example. By the end of that month it was a real (if tiny) crawler. It started from a seed CID (the IPFS mirror of Wikipedia), fetched pages through a local IPFS gateway, pulled out keywords and links, and queued up every link it found. Conor Brady jumped in with tests, CI, Docker and a fair bit of cleanup, which made it feel like a proper project instead of a weekend hack.

In October 2022 (Hacktoberfest again, clearly a pattern) I rewrote it around a small web server, so the index could actually be queried over HTTP instead of just sitting in memory.

darksearch.xyz (2023-2025)

In July 2023 I put a front end on it: darksearch.xyz, a deliberately simple PHP site that sends your query to the indexer and lists the CIDs that match. Over the next couple of years it picked up a submit page (so people could add their own CIDs), a stats page and a disclaimer. The indexer got a faster concurrent hashmap, a top-keywords endpoint and a pile of fixes for relative links and edge cases.

Then, honestly, it mostly sat there. It ran, the index lived in memory, and every restart meant crawling from scratch.

October 2026: The Rewrite

This month I came back to it and changed most of it, in a burst of about a week. The big changes:

Taking abuse seriously. An open search box plus an open submit form on a decentralized network attracts exactly the people you'd expect. I saw queries in the analytics that looked like people hunting for abuse material, so I took the indexer offline until I could put real safeguards in place:

• The indexer now honors the badbits denylist (the IPFS community's list of known-bad content), checks it before anything is queued or fetched, and refuses to start without it.
• There's a local takedown list on top of that. Every result has a Report link, which hides it immediately, and the admin area can turn that into a permanent takedown that removes it from the index and stops it ever being fetched again.
• Searches for abuse terms don't return results. They get a warning and links to help resources instead.
• Only HTML is indexed. Images, video and other media are never downloaded or stored.
• Public submissions are closed for now.

Search came back on a day later with all of that in place.

Persistence that respects takedowns. The index now lives in SQLite instead of in memory, so a restart resumes where it left off. Persisting creates a new problem, though: if someone asks for something to be removed, it has to really be gone. Deleted rows are overwritten with zeros (secure_delete), the rollback journal stays in memory so plaintext never lands in a side file, pages expire after 30 days, and the index is deliberately never backed up, because a backup is a copy takedowns can't reach. The index can always be rebuilt by crawling.

Its own IPFS node. Public gateways like ipfs.io now return 429s to crawlers. Filebase's gateway works but allows about 200 requests a minute, and a cache miss can take ~50 seconds. So the production crawler now runs through its own Kubo node, locked down so it never re-serves anything it fetched (no providing, DHT client only, Bitswap server off). Search result links still point at a public gateway, and you can set your own.

A search engine I built myself. I could have dropped in Tantivy, but where's the fun in that? Instead the indexer grew its own internals: queries are normalized (case, punctuation, stop words), multi-word queries match every word, results are ranked with TF-IDF and paged with limit/offset, and the inverted index moved out of RAM into SQLite with a configurable cache. That last change, along with a 12x cut in keyword-index memory, is what lets one machine keep up with an index growing this fast.

The indexer hasn't been running long in its current form, so the total page count doesn't say much yet. The rate does: on one machine it's adding roughly 3 million pages a day (2,000 to 3,000 a minute), and it still finds new links faster than it can fetch them, with almost 6 million pages waiting in the queue. It's at v0.3.0.

How It Works Now

seed CID ──▶ crawl queue (SQLite) ──▶ 10 workers ──▶ private Kubo node ──▶ IPFS
                  ▲                       │
                  │      badbits + local denylist checked first
                  │                       ▼
                  └──── new links ◀── parse HTML: title, excerpt, keywords
                                          │
                                          ▼
                              inverted index (SQLite, TF-IDF)
                                          │
              darksearch.xyz  ◀── HTTP API ──▶  JSON API / MCP server  ◀── agents

1. Crawling starts from the Wikipedia seed CID and follows links from page to page.
2. Before anything is queued or fetched, it's checked against badbits and the local denylist.
3. For each HTML page it keeps only the CID/path, the title, a 128-character excerpt and keyword counts.
4. Keywords go into an inverted index, and searches are ranked with TF-IDF.
5. The website reaches the indexer over Tailscale rather than the public internet, and an admin area (tailnet only) handles reports, takedowns, re-crawls and health.

Making It Searchable by Agents

This week I was at the Open Intelligence Summit in San Francisco, which was all about the intelligence powering our products being more open, portable and controllable. It got me thinking: agents are getting good at using tools, but for most of them the decentralized web may as well not exist. If you want an agent to find something on IPFS, it needs a way to search it.

One of the announcements drove it home: DigitalOcean's Action Gateway, one managed MCP endpoint in front of 20,000+ tools from 600+ providers, usable from any harness that speaks MCP (Claude Code was the first one on the slide). If MCP is how agents are going to find their tools, then a search engine that isn't an MCP server is invisible to them.

Action Gateway: "20,000+ tools, invoked by intent", with a slot on the slide for "Your MCP servers".

So the day after, darksearch got an agent API:

• A JSON endpoint: https://www.darksearch.xyz/api.php?q=wikipedia&limit=5 returns {"total": N, "results": [{"cid", "title", "excerpt", "url"}]}. No key required, rate-limited to 10 requests a minute per IP.
• A remote MCP server at https://www.darksearch.xyz/mcp with a single tool, search_ipfs(query, limit). It's listed in the official MCP registry as xyz.darksearch/search.
• llms.txt and an OpenAPI spec, so an agent (or someone writing one) can work out how to use it without reading my code.

All of it goes through the same filter, hidden-result list and logging as the website. An agent doesn't get a way around the safeguards.

To add it to Claude Code:

claude mcp add --transport http darksearch https://www.darksearch.xyz/mcp

Or try it without an agent:

npx @modelcontextprotocol/inspector --cli https://www.darksearch.xyz/mcp \
  --transport http --method tools/call --tool-name search_ipfs --tool-arg query=wikipedia

What's Next

• Beyond IPFS. The goal was always the parts of the internet that aren't easily searched, not just IPFS. Other content-addressed and peer-to-peer networks are next on the list.
• Keeping up with the queue. Even at around 3 million pages a day, new links pile up faster than they're fetched, so there's more batching and tuning to do in the homemade index.
• Submissions again, probably admin-reviewed rather than fully open.

Links

• darksearch.xyz (the search engine)
• compscidr/ipfs_indexer (the Rust indexer)