RAG Pipeline for Legal Contracts

The Challenge

Generic LLMs frequently hallucinate legal definitions or cross-reference inapplicable jurisdictions. Ask a chat model about a termination clause and you may get a fluent paragraph that belongs to another agreement, another country, or a textbook. For commercial contracts that span borders — governing law in one place, arbitration in another, definitions that nest three exhibits deep — that failure is not a curiosity. It is a liability.

The objective was an enterprise pipeline that answers contract questions with exact paragraph and page citations. The user is a lawyer or a trained operator, not a consumer. The system is allowed to say “not in the retrieved set.” It is not allowed to sound sure.

This is not a general legal advice product. It is retrieval over a corpus of contracts the organization already has, plus a generator that must stay inside that corpus.

Technical Architecture

Semantic chunking. Naive page splits destroy contracts. A definition on page 2 is used on page 40. A clause is a tree, not a bag of 512-token windows. We built a custom AST chunker for PDF contracts that preserves section headers, definition blocks, exhibits, and clause numbering. Chunks carry path metadata: document id, section, clause, page. Retrieval without that path cannot cite.

Hybrid vector + BM25. Dense embeddings catch paraphrase (“notice period” vs “days to notify”). Sparse search catches the things lawyers actually type: 12.3(b), "Governing Law", a defined term in quotes. Reciprocal Rank Fusion (RRF) combines the two so a clause number is not drowned by a semantically similar but legally different neighbour. We tuned fusion weights on a labelled set of questions, not on demo prompts.

Factuality verifier. A second, lighter pass. After the generator drafts an answer, a verification agent checks each claim against the retrieved chunks. If a sentence cannot be aligned to a span, it is dropped or rewritten as a refusal. Citations are not a footnote the model invented; they are the ids of chunks that survived verification.

Jurisdiction hygiene. Cross-border sets mix English, New York, Singapore, Hong Kong law. The retriever is biased toward the document in the question’s context (the file, the counterparty, the deal). The generator is instructed not to “helpfully” import a doctrine from another file unless that file was retrieved and cited. Mixed-language exhibits are chunked with their parent, not as orphan pages.

Evaluation

We scored citation precision (is the span actually in that clause), answer abstention on out-of-corpus questions, and time-to-first-useful-cite for a lawyer who already knows the deal. Fluency was not a KPI. A short quoted paragraph with a page number beat a synthesized memo.

Failure modes we still watch: scanned PDFs with bad OCR, tables of definitions that look like body text, and questions that are really asking for advice (“is this enforceable”) rather than location (“where is the cap”). The latter we refuse.

Outcome

The pipeline answers contract questions with paragraph and page citations a human can click. Hallucinated jurisdictions dropped because the verifier never saw them in context. The remaining work is corpus quality and the refusal UX — not a bigger model. RAG here is a citation machine with a language model attached, not a lawyer in a box.