Financial Document AI with source evidence

Problem

Audited financial statements arrived as Arabic and English PDFs. The system needed structured facts, accounting consistency, and a traceable source for every accepted value.

Constraints

• Document layouts and labels varied across filings.
• A model answer was not sufficient evidence for a financial fact.
• Uncertain or inconsistent values still required review.

My role

Led the technical work and owned the OCR and extraction service, evaluation path, validation rules, and source-evidence contract.

Decisions

• Kept the model as one component inside a larger validation path.
• Stored raw values and source references before normalization.
• Sent failed checks to review instead of accepting them silently.

What failed

• The first workflow scored 34% on the internal audited-statement evaluation.
• Adding more model output without independent checks did not establish correctness.

Tests and checks

• The evaluation used roughly 100 known-answer cases, including poor-quality scans, digital documents, Arabic and English content, currency, and monetary scale. This scope comes from the engineer's account; the exact dataset and field denominator are not recorded in the portfolio evidence.
• The score covered extracted values and calculated financial results. The score-combination method, language split, and whether human-reviewed corrections counted toward the score are not recorded in the portfolio evidence.
• Separate staging and company documents exposed additional failures outside the development cases. Unresolved ambiguity went to human review; those runs do not establish a separate held-out accuracy score.
• Compared independent extraction passes.
• Checked accounting identities, section totals, and cross-year consistency.
• Retained source evidence for accepted values.

What I would change today

Record the dataset size, field denominator, language mix, and review treatment with each evaluation result.