Why search stays at the surface

Search engines are among the most remarkable tools ever built. In a fraction of a second, they find a handful of relevant pages among hundreds of billions. That is exactly what they were built for: answering the question “where can I find something about this?”

It is a different question from “what does everything ever recorded about this actually say?” And that second question is where search reaches its limits.

How search works

Strip away the details and a search engine does three things. It crawls the web, following links and fetching pages. It indexes them, building a giant map from words to the pages that contain them. And when a query arrives, it ranks the candidates, using relevance, freshness, popularity and hundreds of other signals, and returns the top few.

Everything follows from that design. Search returns pages, not facts. It returns a few results, not all of them. And it favours what is linked, recent and popular, because those are good signals for the question it was built to answer.

Who writes the first page

Ranking rewards relevance and visibility, and visibility can be produced. Organisations publish, optimise and communicate. Press releases are syndicated, profiles are polished, pages are written to rank.

The result is predictable. On almost any organisation, the first page of results is largely written by the organisation itself, or by those who repeat it. The most visible version of a story is rarely the most complete one.

What it does not reach

Much of the public record sits in what is often called the deep web: content that exists and is public, but lies outside the general search index.

Company registries, court databases and official gazettes are searchable mostly through their own forms and interfaces, not through a general search engine. Filings often sit in scanned PDFs, poorly indexed when indexed at all. Hours of interviews, hearings and conferences exist only as audio and video. Specialised data providers sit behind accounts and contracts.

A researcher can reach each of these sources, one at a time: query the registry, request the documents, buy access to the provider, listen to the recording. This is how serious research has always been done. It is also why it takes weeks, and why it rarely covers everything.

No memory

Search shows the web as it is today. When a page is rewritten, the index keeps the new version. When an announcement is deleted, it disappears from the results. When a team page is quietly emptied, search sees an empty page.

The past of the web survives in archives, captured at irregular intervals, heavy to query and impossible to browse at scale. Yet for understanding an organisation, the past is often what matters most: what it said before, who used to be listed, what changed and when.

What AI search adds

AI assistants that search the web are a real step forward. They formulate several queries, read the results, and write a synthesis with sources. For many questions, this is far better than a list of links.

But they work at question time, on what search returns. A retrieval system brings back the top results for each query, the top-k, and the model reads those: dozens of pages, sometimes hundreds, chosen largely for their visibility. All of it must fit in the model’s context window, which, however large, holds the equivalent of a few thousand pages, not the full record of a group of companies across decades. And even within that window, models use information unevenly: research has shown that they rely more on what appears at the beginning or end of a long context than on what sits in the middle.

The limit is not the model. It is the setup: a few pages, gathered in seconds, read in one pass. AI search reads more of the surface, faster. It still reads the surface.

The real barrier: dimensionality

Even with access to every source, a harder problem remains.

Every person and every organisation is described by many attributes (names, roles, dates, addresses, companies, websites, statements), scattered across sources that each have their own format, language, spelling and point in time. The same person may appear under several spellings, in several countries, across registries, transcripts, archives and court records, over decades. The same address may be shared by dozens of entities. And the facts that matter most often sit in sources that seem to have nothing to do with each other.

Connecting them involves two hard tasks: record linkage, deciding whether two records describe the same entity; and link discovery, finding which relationships between entities exist, and which matter.

Both grow combinatorially. Comparing 10,000 records with one another already means nearly 50 million pairs to examine. Following chains of three or four links, from a person to a company to an address to a court, multiplies that by orders of magnitude. For a person working by hand, it is out of reach. For a list of search results, it is out of scope.

That is why links between distant sources so often go unnoticed. Not because they are hidden, but because no one could hold all the pieces in view at once.

What changes

The same models that are constrained at question time are capable when the work is done ahead of it.

Read one document at a time, the context window stops being the constraint: a model can process a registry filing, a scanned annual report or an hour of transcribed speech, then the next, then millions more, in dozens of languages. To link those documents, there is no need to compare everything with everything. Candidates are narrowed first, to records that share a name, a date, an address or an identifier, a step known as blocking. The model then judges only those pairs, estimating with a measurable confidence whether they describe the same entity, and keeping track of every relationship it finds. What emerges is not a pile of pages but a linked record: entities, their attributes, their connections, each tied to its source.

When a question comes, it is no longer asked of ten search results squeezed into a context window. It is asked of the whole record, already read and already linked. The model only needs to retrieve what is relevant, and every link it draws on has been built from everything available, not from what happened to rank.

None of this is infallible. Two people with the same name can be wrongly merged; one person under two spellings can be wrongly split. Every link needs a confidence score and a source that can be checked.

But the shift is real. The question is no longer how much a model can read at once, but how much has been read before the question is asked.

Search finds pages. Understanding requires everything those pages were drawn from.