←
Raj5,804 posts
Raj avatar

Raj

@raj

Building ragwalla.com -- Agents, Memory, Graph RAG

◷ Joined July 2006▧ 336 Photos and videos
Raj avatar
Raj@raj·21h

There are not enough Huangs in this story.

Post attachment
Raj avatar
Raj@raj·Nov 8

Two places we want to try Jev next, both untested: Scoring whether what a search found is enough to answer, and searching again if not. Dropping low-fit passages instead of always passing 10, so the model sees fewer near-duplicates.

Raj avatar
Raj@raj·Nov 8

21 questions failed in all three scored runs. In 10, gold never entered the pool: the question said "attention memory footprint," the docs said "KV cache footprints." In 7, gold sat at rank 2 to 8 under a near-duplicate for another customer that also fit.

Raj avatar
Raj@raj·Nov 8

Caveat: these are retrieval numbers. Our best end-to-end run scores 81.21 on the leaderboard against 90.91 for fifth place. We haven't run the same agent without Jev on all 251 questions, so we can't yet say how many of those points are Jev's.

Raj avatar
Raj@raj·Nov 8

We kept the filters anyway. They pay off when the caller actually knows the app: a user who picks Jira in a search box, or an agent that has already seen where a project's tickets live.

Raj avatar
Raj@raj·Nov 8

Each app's top 10 drags in its weakest results, and scores don't compare across apps. The 10 best Slack messages are mostly noise. The 10 best Confluence pages are all you get from Confluence, even when the answer is the 11th. A reranker wants depth, not spread.

Raj avatar
Raj@raj·Nov 8

So don't route. Search all nine apps, take the top 10 from each, let Jev pick. Round robin merge: 51.1% Merge by score: 57.7% Reranked by Jev: 72.8% One unfiltered search, Jev over 90: 77.7% Fan-out lost by five points.

Raj avatar
Raj@raj·Nov 8

We asked GPT-5.4 for the two likeliest apps per question. It named Confluence for 80% of Confluence questions. Jira for 5% of Jira questions. Linear for 5%. Which tracker holds what is one company's habit. The question doesn't say.

Raj avatar
Raj@raj·Nov 8

Filter the search to the right app before ranking and it finds a lot more: 80.8% vs 69.8%. Confluence goes from 59.8% to 81.4%, Jira from 73.7% to 89.5%. But someone has to name the app.

Raj avatar
Raj@raj·Nov 8

Second test: routing. Confluence and Jira are 2.2% of the docs and hold 47.5% of the gold. Slack is 55.8% of the docs and 11% of the gold. Search everything at once and the chat crowds out the specs and tickets.

Raj avatar
Raj@raj·Nov 8

Cost: Jev is $0.042 per million input tokens. Reading 100 candidates a side comes to about $0.004 a search, plus a second or two.

Raj avatar
Raj@raj·Nov 8

So we kept retrieval's order in the mix. Jev's ranking and the search's ranking are merged by reciprocal rank fusion (k=10). On the held-out half: 72.1% fused, 68.7% Jev alone, 67.1% retrieval alone. Fusion now runs on every reranked search.

Raj avatar
Raj@raj·Nov 8

Reranking alone had a failure mode. Semantic questions (paraphrased, not quoted) gained 19 points. Project questions, which need docs from several apps, fell 14. Jev scores each passage on its own, so it filled the top 10 from one source.

Raj avatar
Raj@raj·Nov 8

Why depth pays: a search's top 10 holds 72.5% of the gold docs. The top 50 from each side of the hybrid search hold 83.7%. The answer is usually in the pool, just not at the top. A reranker can only promote what it reads.

Raj avatar
Raj@raj·Nov 8

Reranking hands the search results to Jev, which scores each passage on whether it answers the question. How deep it reads matters most: No reranker: 77.8% 50 a side: 83.0% 100 a side: 85.2% 100 + fusion: 87.0%