# Ask the Model Where, Not What

In a recent document extraction system I built, I used this approach to keep each value tied to its source.

The pipeline has five stages, with human review where needed:

1. Parse. First, turn the PDF into pieces you can refer to. For this, I use Docling, an open-source document parser that identifies page layout and uses TableFormer to recover table structure. Keep each piece of text together with its coordinates on the page. You'll need those later to show where a value came from. This is also where text recognition can misread a digit or the parser can put it in the wrong cell, so the source text still needs checking.

Think of the parsed document as a tree. Pages contain headings, paragraphs and tables; tables contain rows and cells. Our £544 is text inside one of those cells. Give each piece a reference and the model can choose from what's there. Code then checks that its choice exists. Docling stores its structured document in a DoclingDocument; the diagrams below show how you can organise the pieces and connect them back to the page.

2. Classify. Work out what you're looking at before asking for values. A report about the whole fund and a statement for one investor need different fields. I start with rules that check the title, headings and table headers, and ask a model when those leave room for doubt.

3. Find relevant passages. A short statement can fit in one model request, so I send all the parsed pieces in page order when it does. A sixty-page report gives the model a lot more to sort through, including plenty of numbers that look like the one you want. For those reports, I search for likely passages first. I combine keyword search in Postgres with a search for similar meaning, using pgvector and bge-m3 embeddings. I use reciprocal rank fusion to combine the two result lists by rank, so you don't have to make their different scoring systems agree.

If you've worked with RAG, retrieval-augmented generation, this part will look familiar. You find relevant passages and give them to the model. For the meaning-based search, bge-m3 turns each piece of text into a vector: a list of numbers that represents aspects of its meaning. Do the same for the description of the field you want, then search for nearby vectors. With pgvector, you can keep those vectors in Postgres alongside the text and page coordinates. Here, the model uses the results to pick source references. Search can miss the right passage, though, so this step needs testing too.

4. Extract. Ask for the fields together in one model call per document, so the model can use the surrounding figures to tell, for example, a beginning balance from an ending balance. Request structured output: a form with a defined set of fields. Each field holds a reference to the piece containing the value. Code validates that reference, reads the text and applies the currency and scale.

I keep one field list per document type. Each entry describes what you're looking for, its value type, whether it's required and the labels you might see in the document. Those labels can cover English, Dutch, German and French, for example. Then you can reuse the same definition for the search terms, the model's instructions and the checks on the returned value. Fewer places to update when you add a field.

5. Verify. Read the selected cell again with code, without asking the model, and check that the text, currency and scale reproduce the stored value. Reject the field if they don't. This catches a mismatch between the stored result and its source. It still won't catch a model choosing the beginning balance instead of the ending balance. That mistake can pass this check, which is why I wouldn't treat a successful re-read as proof that the answer is right.

To find out how well this works, I'd start with documents where I already know the answers. Fictional statements let you change the layout, currency or scale note and see what breaks. Keep some whole layout families out of tuning, then test on representative real documents you have permission to use. I'd score the cell selection and the value conversion separately, so a conversion bug doesn't get mixed up with a wrong-cell mistake. And I'd count missing answers separately from wrong ones. An empty field asks for attention; a plausible wrong number can slip into a report.

There's a research history behind asking a model to point. A generative model writes an answer; an extractive question-answering model selects text from the source. The pointer network paper from 2015 describes a neural mechanism that outputs positions in its input. A 2021 paper explores extracting answer spans from a generative model's attention instead of using its generated answer. That constrains the answer to source text, but selecting the right text is still a separate problem. A 2026 document-parsing benchmark also evaluates visual grounding: connecting parsed content to the right region of the page. I'm using explicit source references here, rather than implementing those papers' neural mechanisms. The connection is the ability to point back to the evidence.

The OCBC team's April 2026 paper, A Multistage Extraction Pipeline for Long Scanned Financial Documents, is closer to the financial-report problem. They tested extraction from long, scanned documents in several languages. Their pipeline cleans up the image, reads the text, finds the relevant pages and then extracts the data. In their evaluation, this improved performance by up to 31.9 percentage points over giving the model the whole document at once. Finding the right pages made the biggest difference. My system also separates finding the relevant passages from extracting the values, with the source-reference constraint described above.