Beyond Search: From Keywords to AI-Integrated Discovery

Originally Published by EDRM and JD Supra

By: John Tredennick and Dr. William Webber

For decades, discovery and investigation teams faced the same empty search box. They conjured up keywords, nested terms in parentheses, and added wildcards and proximity operators. Even when they mastered Professor Boole’s nineteenth-century logic, the reward was thousands of search hits. In every case, someone still had to painstakingly read the documents, decide what mattered, and determine where to go next.

In Keyword Search Isn't Dead. Searching With Keywords Alone Should Be, we examined that starting point. Keyword search is indispensable when exact language matters, but it cannot find words the team does not know to use. Even a carefully built query may miss a document that calls a problem by an unexpected name, while a broad query leaves people to sort through results that add little to the investigation.

In AI Search Can Find the Meaning. But It Can Still Miss the Evidence, we added semantic search, which can find the underlying idea when documents use different words. Our Project Checkmate example showed the converse: an opaque project name may carry the crucial signal even though its ordinary meaning tells semantic search almost nothing. Combining keyword and semantic methods gave the team complementary ways to find the initial evidence.

In Search Finds the Initial Evidence. CAL Learns What Comes Next, we explored what an AI-powered system should do after hybrid search returned its first 50 results. An LLM assessed those documents and identified the clearest positive and negative relevance judgments. We used those examples to train a continuous active learning system (CAL), which helped rank the results presented next. Across the next 100 results, that approach found 82 more responsive passages than continuing down the original search ranking. AI could learn from what it found and move the search forward.

That led us to a larger question:

If AI can learn from the documents and improve search results, why stop there? Why not let AI help us investigate what happened?

That is the idea behind AI-Integrated Discovery. AI becomes a member of the team from the first question through final resolution. It helps frame the inquiry, build searches, read the results, pursue leads, test competing explanations, and assemble an account based on the documents found. The human members of the team direct that work, examine critical evidence, challenge conclusions, and decide what the record supports.

Here is how that team could work in practice. Using Alchemy Intelligence, our document intelligence platform, we follow one investigation from the first question through a refined assignment, an initial report, a challenge to its conclusions, and a plan for what comes next. The example shows how quickly an AI teammate can help turn an initial question into an account the human team can inspect, test, and build upon.

1. Start with what you want to know

Imagine a team examining documents connected to the UK Post Office's Horizon accounting system. The team wants to investigate Fujitsu's knowledge of problems with Horizon and its communications with the Post Office. At the outset, the lawyer may not know the internal terminology, the key custodians, or the most important time periods. The question is nevertheless clear enough to begin:

What did Fujitsu know about problems with Horizon, when did it know, and what did it do in response?

That is an assignment, not a search string. It contains several issues: what counted as a problem, who learned of it, whether it could affect branch accounts, what was done internally, what was communicated externally, and what evidence points the other way. A system that immediately converts the question into a keyword or semantic query has missed its first opportunity to help.

Traditional search begins with a query. AI-Integrated Discovery begins with intent.

Our AI Prompt Helper can explore the collection before the team commits to a full search strategy. In this example, it reviewed an initial sample of roughly 30 documents in our Alchemy system. The following Prompt Suggestions show the issues it surfaced and the questions it put to the team.

The lawyer can answer those questions and redirect the inquiry. Here is the initial prompt suggested by the Prompt Helper:

In this way, the first encounter with the documents can help refine and clarify what the team is investigating.

2. Give the assignment more than one search method

Once the inquiry is sharper, AI can break it into investigative questions. Which records show that a potential defect was identified? Who assessed its effect on branch accounts? What did technical staff tell management? What did Fujitsu tell the Post Office? Were there reports that attributed a discrepancy to another cause?

As our first two articles showed, each question may require a different way to look for evidence. Keyword search is strong for names, ticket numbers, exact phrases, product terminology, and dates. Semantic search can find discussions of a problem even where nobody used the lawyer's words. Once documents have been assessed, CAL can learn the characteristics of responsive material and improve what the system presents next. The system can combine the methods and revise the mix as the investigation develops.

Here are two targeted examples of the kinds of queries the LLM might build for our topic:

Under the hood, AI builds both keyword and semantic searches and runs them independently against the larger population. The two search engines rank documents across the population for relevance and send them to the system in relevance order. Alchemy then merges the results and presents the first 100 most promising documents or sections.

3. Let AI read and report what the first search finds

The first 100 results are a starting point, not the deliverable. AI reads them, assesses how each relates to the assignment, prepares targeted summaries, and draws together an initial report. Here is an excerpt from Alchemy’s report on the Horizon inquiry:

Within Alchemy, team members can move from a finding to its supporting summaries and documents, test the conclusion, and direct the next inquiry. They need not accept an AI report any more readily than an associate’s first memo. But they begin with an organized account of the evidence and specific questions to pursue, rather than a list of search hits.

4. Let the first results open new lines of inquiry.

In Article Three, CAL used clear relevance judgments to bring more responsive documents into the next batch. But AI can also notice when the evidence changes the question. A record saying a defect was “fixed,” for example, may leave open whether the fix was tested, whether earlier accounts were corrected, and whether affected branches were told.

Information in one set of documents can lead to a whole new investigation pathway and generate new searches to find more relevant information.

We asked Alchemy where its initial Horizon report needed further investigation. It proposed five areas for the team to pursue:

Each area can become a new search assignment. The team might begin with the first: find the S90 release records, proposed estate-wide checks, later reports of the Callendar Square problem, and records addressing affected accounts and branches. AI can assess what those searches find and revise the working account. The team decides which lead to pursue and whether the new evidence answers the question.

5. Put an AI team to work on the next questions

Alchemy’s initial report identified five areas for further investigation. In an agentic workflow, an AI team leader could turn each area into a defined assignment and dispatch agents to pursue them. Each agent would build and run searches, read the results, test competing evidence, and report back with document references and questions still open.

To show what one assignment could produce, we asked Alchemy to investigate what “fixed” meant for Callendar Square. It examined release records, proposed estate-wide checks, later incidents, and records concerning affected branches. Here is an excerpt from its follow-up report:

Other agents could investigate how affected branches were identified, the gap between technical knowledge and litigation evidence, the reliability of ARQ data, and remote-access capability and use. The AI team leader would compare their reports, connect findings across issues, and assign narrower follow-up work when new names, terms, events, or contradictions emerged.

The AI team leader would maintain a working record of the issues, people, terminology, significant documents, conflicting accounts, unanswered questions, and searches already run. The human team should be able to inspect and correct that record. An uncertain identification should not quietly become the premise for every later search or report.

6. An AI teammate should investigate what could prove its first answer wrong.

An investigator who finds evidence supporting a theory has not finished testing it. The next assignment should seek records that could undermine that account and test competing explanations. The Callendar Square report raised a consequential question: did the S90 release resolve the underlying defect, or did it only appear to do so? We asked Alchemy to search for evidence that could change its first answer.

Challenge the Callendar Square report’s assessment of the S90 fix. Find the strongest evidence that the original defect was resolved, including release testing, post-release monitoring, incident records, and communications. Determine whether later Riposte lock incidents involved the same underlying fault or a distinct problem. Present the evidence for each explanation, identify missing records, and cite the documents behind your assessment.

Alchemy ran new searches, examined the results, and prepared a second report testing the first. In particular, it looked at whether similar symptoms in later records established that the original defect persisted, and whether records outside the initial search supported a verified fix.

The challenge report narrows the initial account. It finds evidence that S90 was deployed and lock timeouts declined, while leaving open whether the 2010 incident involved the same defect and whether the original fix was fully validated. The AI team leader could carry forward that more precise account, assign searches for the missing test and incident records, and bring both reports and the critical documents to the human team for review.

7. Know when there is enough for the task

An AI team could keep finding relevant documents long after it has enough to help with the immediate assignment. It could also produce a persuasive report while important evidence remains undiscovered. The team needs a way to judge whether further work is likely to change its understanding.

Alchemy’s Adaptive Sampling system samples the ranked document set to estimate how many relevant documents are in the collection. An LLM reviews the sample and distinguishes documents that add important new information from those that are relevant but repeat what the team already knows. Together, those measures help the team see whether deeper review is still producing new evidence or mostly cumulative material.

For a Callendar Square witness interview, the chronology, competing reports, and principal documents may be enough to prepare focused questions. If Adaptive Sampling shows that deeper ranks are still producing new facts or contrary evidence, the team has reason to keep investigating. If they yield mainly cumulative records, the team may have enough for that interview while continuing targeted searches for the missing S90 test and monitoring records. A disclosure decision or a conclusion about all affected branches may require a different depth of review.

The AI team leader can present those estimates alongside the open questions, searches completed, and lines of inquiry deferred. But in all cases, the human team decides how deep to go for the task at hand.

8. Let AI suggest next steps for the investigation

After the initial report and the Callendar Square follow-up, we asked Alchemy to recommend the next phase of the broader Horizon investigation. It identified eight steps, from mapping defects to affected cases through recovering missing records and testing the evidence behind each conclusion.

Here is a summary of Alchemy’s recommendations for next steps:

The full report gives the team its next assignments. It can ask Alchemy to trace a witness statement through its drafts, build a witness kit for Anne Chambers, or investigate how a particular defect affected individual cases. Each request can draw on the work already done, run new searches where needed, and update the account as the evidence develops. The human team sets the priorities, inspects the critical documents, and decides what the findings support.

A Better Way to Discover What Happened

The progression across our series now comes into focus. Keywords find language. Semantic search finds meaning. CAL learns relevance from assessed examples. AI-Integrated Discovery carries that learning through the matter: helping the team frame the inquiry, examine the evidence, pursue new leads, challenge its conclusions, and decide where to look next.

The division of labor matters. AI investigates broadly. Humans direct, challenge and decide. AI can search, read, learn and synthesize across a large collection. The human team examines critical evidence, tests the answers, and exercises judgment about what the matter requires.

For decades, discovery systems waited for us to supply the next query. AI-Integrated Discovery can help us frame the question, follow the evidence, test competing explanations, and see what remains unknown. It gives every member of that team a more effective way to understand what happened and a more efficient way to act on what they learn.

That matters beyond the discovery team. When evidence can be found and tested sooner at lower cost, parties can assess their positions earlier, resolve disputes faster, and pursue or defend claims that prolonged review might otherwise put out of reach.