# Combining LLM and OCR for Invoice Validation: Lessons Learned

ChatGPT returned a VIN with a zero where there should be an O. WMI validation failed. The car couldn't be registered at the RDW. Manual work required. Trust in the system lost.

This is the problem with "just use GPT for document extraction."

I'm building invoice validation for Revive Capital, a B2B car financing company. We extract IBANs, VINs, Chamber of Commerce numbers, prices, and tax amounts from supplier invoices. Get them wrong, and we might finance the wrong car.

// Redacted Dutch car invoice

// typical Dutch car invoice with VIN, IBAN, prices, and tax fields

### Just use GPT, right?

First attempt: "Just use GPT with vision." Send the invoice image, ask it to extract all fields. It works great for semantic understanding. Trade-in deductions? GPT gets it. Price calculations across line items? No problem with the right prompt.

But then the failures started. O became 0. I became 1. Q became 0. The LLM would confidently return values that looked plausible but were wrong.

This is the core problem: LLMs aggregate meaning well and transcribe symbols badly. They understand what a VIN is. They can't copy one out. Tokens are probability distributions, not characters, and for a checksummed identifier close is useless.

### I've seen this before

In 2016, I built Dutchies Travel (BackpackApp), a travel agency platform for backpackers in Australia. Same problem: extracting booking codes, dates, locations from travel voucher PDFs. Each vendor had a different format. Each required custom regex extractors. Formats changed over time. Constant maintenance, custom code for each case.

When field extraction goes wrong, end customers, vendors, and internal stakeholders all lose trust.

### The fix: let each tool do what it's best at

The solution was a hybrid dual-extraction approach. Run GPT and AWS Textract in parallel, then merge strategically:

• GPT handles semantic understanding: trade-in deductions, price sums across line items, supplier vs buyer detection, contextual reasoning

• AWS Textract handles symbol-critical fields: IBAN, VIN, Chamber of Commerce numbers. Why? OCR is deterministic and checksum-verifiable. LLMs are probabilistic and token-based. For fields with validation algorithms, deterministic wins.

• Merge strategy: Textract only overrides GPT for these specific fields. Everything else uses GPT's semantic understanding.

// Extraction prompts and extractor code

### The prompt optimization rabbit hole

I went deep on prompt optimization. Built a 331-line system prompt with detailed extraction instructions. Verification checklists. Explicit warnings about BPM vs BTW (vehicle registration tax vs VAT). IBAN validation instructions in the prompt itself.

Even used Promptomatix for automated prompt optimization, running experiments against baseline prompts. The optimized prompts were more concise. Accuracy improved marginally.

Prompts can't fix symbol fidelity. You can tell the model to double-check and verify all you want; it still operates on tokens, not characters. The fix has to be architectural.

### Normalization is everything

Extraction is the easy part. The real work is normalization. Dutch invoices use 1.234,56 for amounts. US formats use 1,234.56. Company names come with B.V., N.V., random whitespace, accents. IBANs need MOD 97-10 validation. VINs need WMI whitelist verification against 160+ manufacturer prefixes.

I built field-specific normalizers for everything: amounts, invoice numbers, company names, addresses, dates, mileage. Each with their own parsing logic, tolerance thresholds, and comparison strategies.

// Field normalizers and extractors code

### Building for failure

Everything here fails eventually. Here's what happens when it does:

• Textract fails? Fall back to GPT values. Still works, just less accurate.

• PDF format rejected? Rasterize with Poppler, retry Textract on the image.

• Field extraction fails? First-value-wins merge from multiple extraction paths.

• Validation near-miss? Partial scoring with tolerance thresholds (within 1% = 0.95 score).

• Critical fields: 8 must match for valid=true. Non-critical fields can fail without invalidating the invoice.

### Full circle: back to vendor-specific extractors?

In 2016 the per-vendor regex extractors were too labour-intensive to maintain. But they were deterministic and reliable for known formats.

In 2025-2026, AI coding agents change the economics. Instead of writing vendor-specific extractors by hand, you can use LLMs to generate them. The LLM writes the extractor once. After that it runs deterministically, every time.

Concrete example: a new supplier sends invoices with the VIN in a table on page 2. Instead of prompt-engineering GPT to find it, generate a vendor-specific extractor: real code, with regex patterns and spatial relations. The regex validates the VIN format; table coordinates find it on page 2. When their format changes, regenerate the code. Minutes of work instead of hours of prompt debugging, burning tokens, and non-deterministic results.

The "old" approach might win again, now that the tooling makes it maintainable.

Note: this article reflects the state of GPT-4o and AWS Textract as of late 2025. These tools evolve fast. The symbol fidelity problem may improve. Swapping models is easy—better accuracy, cheaper tokens, just plug in the next generation.

See the project: Revive Capital on my projects page

#llm #ocr #documentextraction #typescript #fintech #ai