How routing a LangGraph RAG chat with JEV got me 98% accuracy

Between September 25 and 27, 2026, I rebuilt the homepage of this site for recruiters and hiring managers. The chat is now the biggest element on the page. It answers questions about my work, compares me with a job description, and opens a contact form with the first fields already filled in.

Most decisions in the chat now come from JEV, a decision model that answers typed questions. JEV decides where a message goes, which evidence the answer model reads first, whether the answer does what the visitor asked, and which options the contact form gets. The answer model writes the text. Deterministic code checks it.

Here is an example. A visitor drags my three CVs into the chat and asks "does manuel knwo python?", then "what is his notice period". My notice period is not public, so the answer says that and shows a contact block. The visitor opens the form. Both select fields already say "Hiring or collaboration", and the message box lists the two questions and the three CVs. JEV chose those options with confidence 1.00 and 0.99. It answered them inside the request that grades the answer, so the form needed no extra request.

The first note about this agent explains the base rules: the chat answers only from public evidence, and every factual line needs a source that passes a check. Those rules did not change. This note covers what changed around them, and one failure that had nothing to do with AI: the site's email stopped after a server restart, and no check told me.

The new homepage

Before the redesign, the chat was a robot button in the bottom-right corner of every page. It still is on the other pages. On the homepage, the chat is now an open window. The page uses desktop elements: windows, file icons, and a dock.

On a screen 781 px wide or wider, the homepage has these parts:

On a phone, the chat is a sheet at the bottom of the screen with three heights. A bar has three buttons: ↓ CV, Projects, and Email. A tap on a CV icon opens a pane with two actions: download the PDF, or ask about this CV.

A visitor can also paste a job description, or attach a PDF of up to 12 pages and 5 MB. The browser extracts the text and sends only the text.

Why the chat needed a better classifier

The first version routed each message with regular expressions and a model classifier. In one production conversation, a recruiter pasted a Spanish job description after an English question: "How good of a fit is Manuel for this JD". The chat refused it as a request for private data. The posting contained "Rango salarial" and "Día off por cumpleaños", and the salary and birthday patterns ran on the whole message. The refusal was in Spanish. Then the recruiter wrote "this is a JD, I'm not asking for personal information", and the chat showed the contact card: the negated phrase still matched "personal information".

A hotfix separated the visitor's own words from the pasted posting. But a regular expression matches words, not meaning. A posting can contain the word "salary" and ask nothing private.

So I changed who decides. JEV makes the routing decision. When JEV fails, times out, or is unsure, the model classifier decides, and the old rules decide after that. My target was that JEV's decision is final for 80% to 95% of questions.

Four rule guards stay in front of JEV: prompt injection, personal attacks, a portfolio question mixed with an unrelated task, and cyber misuse. Text inside a message can change JEV's answers too, so these guards run first. JEV cannot override them.

The router evaluation has 322 labeled messages. JEV-first routing scored 0.981 on them. The old path, with only the model classifier, scored 0.957. After a wording fix to four JEV criteria and six new cases, a live run on 328 messages scored 0.982. JEV's decision was final for 94.2% of them.

What JEV is

JEV is a decision model from TypeSafe. The chat calls it through OpenRouter's Decisions API, with the same OpenRouter key as the answer model. The model id is typesafe/jev-1.13.

A request has a small JSON state and a set of named questions. Each question has one of three types:

One request can hold many questions. The routing request asks 6, and the evidence reorder asks 25. This is a real answer to one noul question, recorded in the client code:

{
  "model": "typesafe/jev-1.13-20260917",
  "answers": {
    "line_0": {
      "type": "noul",
      "noul": 0.42
    }
  },
  "usage": {
    "input_tokens": 377,
    "output_tokens": 22,
    "cost": 0.000015834
  }
}

That question cost $0.000016. The answer is a number or a label, so code can compare it with a threshold, log it, and replay it in an evaluation with other thresholds.

The client never throws for a provider problem. Every call returns the answers or a typed failure: timeout, aborted, HTTP error, network error, or invalid response. A zod schema checks every answer. A choice outside the given labels is invalid. The client retries once, only when the first attempt failed in less than one second. The default timeout is 2.5 seconds.

How the LangGraph agent works

The agent is one LangGraph StateGraph in the Next.js server (LangGraph 1.4.8, Next.js 16.3.3). The API route builds the dependencies: the answer model openai/gpt-6-luna, the embedding model, and the JEV steps. Then it runs the graph and streams its events to the browser.

Each JEV step is optional. When one is missing or turned off, the graph uses fixed rules for that step. The tests run the same graph with a scripted answer model and a stub JEV, and check the route, the log fields, and the answer.

The state

The state is one object with 63 fields, defined as a zod schema. A node reads the state and returns only the fields it changes. The main groups:

Nodes, edges, and who decides

The graph has 22 nodes. This is the path of a normal question, with its exits:

Small talk goes from generate straight to stream_verified, because it cites nothing. The template nodes check their own answer, then go to stream_verified, or to finalize when the check fails. Questions about the Privacy and Terms pages take a separate path of five nodes: a fixed answer first, then at most two checked lines from the model.

What the main nodes do, and who decides in each:

The model decides only the wording. It does not choose the route or the sources, and in a job comparison the ability matcher sets the marks. When JEV fails, the model classifies the message as the first fallback.

Retries, loops, and failures

The graph has three loops, and each has a limit:

The graph's recursion limit is 30 steps, so a mistake in an edge cannot loop forever.

A JEV failure never fails an answer. Routing falls back to the model classifier, then the rules. A document's type falls back to fixed rules: for example, my name at the top of a PDF means it is my CV. The evidence keeps the order of the two searches. The requirement types fall back to the sections of the posting. The ability matcher uses only the fixed implication rules. The line check keeps only the deterministic checks. When the judge fails, the checked answer is released as it is, and a phrase check decides the contact block.

Streaming and time limits

The API route streams server-sent events. Nodes send status events, and work inside a node sends activity events: keyword search, embeddings, evidence selection, the model call, and the source check. The browser turns these events into the step list under each answer: Input check, Route, Ask user, Keyword search, Embeddings, Evidence, Requirement match, Answer, Grounding, and Release. A finished answer shows how many steps ran, for example "8 steps".

The model's tokens do not go to the browser as they arrive. The server waits for the full draft, checks it, and then sends the checked text in small chunks. A retry sends a reset event that clears the draft on the screen.

Every run has one deadline. Each JEV step checks the time left before it starts, and it is skipped when less than its own timeout plus one second remains. So a slow JEV call cannot cost an answer that already passed its checks. The answer judge needs 8 seconds left. Its regeneration stops 1.5 seconds before the deadline, so the checked answer can still be released. The evidence reorder has its own limit of 1.2 seconds.

Why LangGraph, and when I would not use it

In the first note I wrote that a small state machine could do the same job. The graph now has 22 nodes and three loops, and LangGraph helps in three ways:

• The state is one typed schema with defaults. Every branch reads named fields.
• All edges are in one file. Each route is a named node, so the logs and the traces show the node path of every run.
• The tests run the whole graph with scripted dependencies and check the path it took.

The graph does not use model-chosen tool calls, long-term memory, checkpoints, or a second agent. The model never picks the next node: JEV and code pick it. I would not use LangGraph for an endpoint that only retrieves records and asks a model to summarize them. I would also not use this setup for a workflow that must continue after a server restart, because the graph runs inside one request and keeps no checkpoints.

One question through the graph

This is a live run on my machine on September 27, with the code that production runs, the real JEV, and the real answer model. The question comes from my recruiter test set, typo included: "does manuel knwo python?"

Step 1, validate. It normalizes the text and checks its length. There is no pasted posting.

Step 2, classify. The four guards do not match. One JEV request asks the six routing questions. It took 419 ms and cost $0.000091:

The policy checks private data first: 0.03 is below 0.6, so nothing is restricted. clear_enough is 0.77, above 0.35, so there is no clarifying question. portfolio_general is above its threshold of 0.35, so JEV's route is final. The topic probability is above 0.5, so the search query gets fixed extra terms: "CV skills programming languages frameworks infrastructure".

Step 3, retrieve. Keyword search returns 20 candidates, and embedding search returns 24. Reciprocal rank fusion (RRF) merges the two lists: a record scores higher when both lists rank it high. Then one JEV request scores the RRF top 25 on the 0 to 9 scale. It took 428 ms and cost $0.00036. Part of the result, in RRF order:

The spoken-languages record was third and the infrastructure skills were fourth, because the extra search terms contain "languages" and "infrastructure". JEV scored them 1.44 and 1.78, and both left the top six. The Financial Document AI project was last in RRF order, and JEV moved it into the top three. Then the pins add the two records that a skill question always needs: the project that shows Python, and the CV list that names it.

Step 4, evidence_gate. It passes, because there is evidence.

Step 5, generate. It sends about 3,900 input tokens to openai/gpt-6-luna and gets 64 output tokens: "Yes. Manuel used Python in the Financial Document AI project, which used Python and FastAPI and improved the extraction evaluation from 34% to 94%; Python is also listed among his programming languages on his CV."

Step 6, grounding_check. It passes. Both citations are in the evidence, and 34% and 94% appear in the cited project record.

Step 7, judge_answer. One JEV request asks five questions. It took 378 ms. The grades were 0.97 for answers_request, 0.99 for language_ok, and 0.01 for missing_info. No grade is below 0.5, so there is no regeneration. Both form questions returned "Hiring or collaboration", with confidence 0.77 and 0.58.

Step 8, stream_verified and finalize. They release the answer. The step list under it shows 8 steps. The run took 3.6 seconds.

Before the pins, a recruiter test run answered the same question with: "Yes. Python is listed in Manuel's programming languages." It scored 0.5 on fact recall: it named Python, but no project that shows it.

Evidence the answer must see

Keyword search and embedding search find records that share words or meaning with the question. They often miss the record that holds the answer. On September 27, the recruiter test set showed four examples:

Pins fix this with fixed patterns. After typo correction, in English and Spanish, a pattern finds the type of question and adds the records it always needs.

A skill pin uses the ability catalog. The catalog has 196 abilities from my CV, my roles, my projects, and my note about this agent. 86 of them are shown by a project, a role, or a note. The rest are only listed in the CV. The catalog has 61 implication rules:

This change (the pins, two new records, and clearer answer instructions) raised fact recall on the answerable recruiter questions from 0.77 to 0.92. The honest "not public" answers went from 0.81 to 1.00. The wrong-fact rate stayed at 0.00.

JEV reorders the evidence

Pins cover known types of questions. For everything else, the search has to rank well. Keyword search and embedding search each return up to 24 candidates, and RRF merges them. The embeddings use text-embedding-3-small at 512 dimensions, in a versioned index file in the repository.

I tested four ways to reorder the RRF top 25 on a holdout set of 50 questions (25 questions, each in English and Spanish). Three were hosted rerankers: Cohere rerank-v3.5, Voyage rerank-2.5, and Qwen3 Reranker 8B. The fourth was JEV, with one score question per record.

JEV ranked best. MRR (mean reciprocal rank) went from 0.552 to 0.797, the highest of the four. Recall@6 went from 0.760 to 0.900, tied with two hosted rerankers. Recall@6 is the share of questions whose expected record is in the six kept records. MRR measures how high the first expected record ranks: 1.0 means first place every time. Voyage was cheaper, at $0.00021 per question against $0.00037, but it ranked lower, and JEV needed no new provider. The same experiment tried seven other embedding setups, including Qwen3 Embedding 8B and BGE-M3. None passed the rule I wrote before the run.

A retrieval score is not answer quality. So before the first run, I wrote an end-to-end test with six pass rules. It ran the full recruiter set, 126 answers, three times with the reorder off and three times with it on:

p95 latency is the time that 95% of answers stay under. My rule allowed 600 ms more at most, so the test failed by 37 ms. Every quality rule passed. I raised the latency limit to 800 ms and turned the reorder on. I changed that rule after I saw the result, so I say it here.

A failure, a timeout, or an invalid answer keeps the RRF order. A job comparison always keeps the RRF order.

Testing with the real JEV

I wanted every number in this note to come from runs with the real JEV and the real answer model. Early pull requests used a fake JEV that scored pairs by shared words. It was lenient on purpose, and it could not calibrate a threshold.

The recruiter test set has 63 questions that a recruiter, a hiring manager, or a client would ask. Each has an English and a Spanish version: 126 variants. 42 variants have typos or missing accents, for example "cuales son sus expectativas salariales". 50 questions have a public answer, and each lists the facts that the answer must contain. 13 questions have no public answer, for example salary, notice period, visa, and references. Their answer must say that the information is not public, and it must not invent anything.

The harness runs the graph in the same process, with the same dependencies as the API route. It records every JEV decision, token count, and cost. It scores fact recall (the share of required facts in the answer), wrong facts (an invented number or a forbidden claim), and honest unknowns (a not-public question gets a "not public" answer or a refusal, and no invented fact).

"The same dependencies as the API route" was a lesson. The first harness runs on job descriptions left out the ability matcher that production uses. With the matcher on, false ✓ marks on 20 real job postings dropped from 29 in 240 runs to 3 in 120 runs, before any code change.

On the holdout half of the recruiter set (31 questions), fact recall was between 0.971 and 0.990 in three runs. With the evidence reorder on, the full set scored 0.983.

The job-description comparison is harder. On 20 real job postings with requirements that I labeled by hand, the shown mark (✓, ~, or ✗) agrees with my label for 65% of the requirements. Must-have recall is 0.66. In 120 runs, only one line showed ✓ where my label said ✗. I chose the verdict thresholds on the same 20 postings, so they may be overfitted.

Filling the contact form from the chat

A contact block appears under an answer in four cases: the visitor asked how to contact me, the answer says that something is not public, the answer compared me with a job, or the visitor sent a project brief. Its button opens the form at /work.

The form has two select fields: who the inquiry is for, and what kind of help fits. For most answers, JEV fills them with two choice questions inside the answer judge's request. A job comparison is the exception: the code already knows that it is about hiring, so both selects get "Hiring or collaboration" by a fixed rule, and JEV is not asked. I added that rule after a test in which JEV chose "Not sure yet" for a pasted job posting. This is the real question for the first select, from the second live run, with all six of its options:

The labels are the form's own option labels, so JEV's answer maps straight to a select. The questions contain the visitor's questions (at most four), the attached CV ids, and the kinds of uploaded documents. They never contain an answer or the text of a document, and they cannot change the grades.

The rules:

The first live check used three test conversations:

The second live run is the example from the start of this note: three CVs attached, and the question "what is his notice period". JEV routed it as portfolio_general at 0.99. The answer said: "Manuel's public portfolio does not state his notice period; he can answer that directly. His current role is Technical Lead (IC) at ConTrust Suite, starting March 2026." JEV gave missing_info 0.98, so the contact block appeared. It chose "Hiring or collaboration" for both selects, at 1.00 and 0.99. This is the chat, and the form it opened on September 27. Since then, the form shows the attached CVs and the other chat details in a separate block that the visitor can remove before sending:

The run also showed a weakness. JEV gave answers_request only 0.48 to that honest answer, so the graph regenerated it. The new answer got 0.47, and the graph kept the first one. The loop added 2.3 seconds and changed nothing, because a regeneration cannot find a fact that is not public. The next change is to skip the regeneration when missing_info already says that the information is not public.

The draft goes from the chat to the form through the tab's sessionStorage, never through the URL. It expires after 30 minutes, and the form deletes it after it reads it. It holds the visitor's own recent questions (at most four, 200 characters each) and the names of attached CVs and files. After a job comparison, it also holds the role title, the company, and the verdict sentence. It never holds the posting, a document's text, or the answer lines. The form fills only empty fields, and the visitor can change every field. Analytics events carry only fixed codes, such as the select codes, and only after cookie consent.

The CV easter egg

The easter egg is a hidden reply. When a visitor drags a CV icon into the chat, the chat attaches that CV and shows three steps of 400 ms each, built from the real counts of that CV: "Reading Lead IC CV", "5 roles, 20 skills", and "Ready". Then the chat shows my line:

"You dragged the CV in. Respect - almost nobody finds that. Read it. What do you want to know?"

Other versions of this line are generated once, at build time, and I approve each one. openai/gpt-6-luna drafts 20 lines. One JEV request grades each line with three questions: meaning (a score from 0 to 9), whether it adds a claim (noul), and whether it is a near copy of my line or of another draft (noul). The script keeps at most 10 lines with meaning 7 or higher and both noul answers below 0.5. In the final run, 8 of 20 lines passed. I approved 3 of them, so the chat now rotates 4 lines. The site never calls a model for this reply.

The reply appears on the first drag of a visit, for the first three visits. With cookie consent, the browser remembers which lines it showed. Without consent, nothing is stored, and each visit gets a random approved line. Later drags add no new message. They update one line under the first message: "now: 3 CVs attached — Backend, AI / Inference, Lead IC. Ask about one, or compare them."

On a phone, a finger that moves down or sideways drags the icon onto the chat sheet, and a finger that moves up still scrolls the page. A tap on the icon offers "Ask about this CV", which attaches the CV with a plain line and no easter egg.

An attached CV also changes the evidence. The server pins the records that the focused CV is built from: my summary, the roles that CV selects, its two closest skill groups, and the awards. At most 8 records are pinned, and retrieval keeps 4 places for the question's own sources.

What it costs

These are the totals on my OpenRouter activity dashboard on September 27, 2026. For each measure, the dashboard lists the biggest models and groups the rest as "others":

Most of these requests come from evaluation runs, not from visitors: the recruiter test set, the job-description runs, and the reorder tests, all with the real JEV. One full recruiter run with 630 answers cost about $0.19. The Qwen3 reranker spend is the reranker comparison above.

JEV has five times as many requests as the answer model. Most answers make three JEV calls: routing, the evidence reorder, and the judge. A job comparison makes more, and the router evaluation calls only JEV. In the Python run, the three JEV calls cost $0.0005 together. In the end-to-end test, one answer cost about $0.0008 in total.

The email stopped, and no check told me

All email from this site goes through one self-hosted mail relay on a VPS. That covers the verification links that raise a visitor's question limit in the chat, and the notifications of the /work form.

On September 13, the server rebooted, and the mail service stopped cleanly. Its restart policy restarted it only after a failure. A clean stop is not a failure, so the service stayed down. From that point, verification emails and form notifications were not sent.

I found it on September 26, while I tested the new /work form, and fixed it the next day. The production logs showed failed verification emails, no successful one, and no cause. After a small change that logs the failure category, the cause was clear: the connection to the relay failed. The fix was to rebuild the service and change its restart policy to restart after any stop. The form itself does not lose an inquiry when email fails. It saves every inquiry before it sends the notification, and a daily job retries failed notifications.

I checked the AI parts with hundreds of test runs, and nothing checked that email still worked. So the site now has a status page that checks each part, at about $0 per month. It started measuring on September 27:

A free uptime monitor outside the VPS checks /api/health every 5 minutes and sends alerts to my phone. The alerts do not use email, because an alert must still arrive when email is down.

Sources and credits

This note is based on the pull requests of the private site repository from September 25 to 27, 2026, their evaluation results, two live runs of the graph on September 27, and my OpenRouter activity dashboard. The work landed as 53 merged pull requests. Coding agents wrote most of the code and ran the evaluations. I set the targets and made the product decisions, for example the 800 ms latency limit, the 640 px chat height, and the three approved easter-egg lines.