←
Provably233 posts
Provably avatar✓

Provably ✓

@GetProvably

Making agentic workflows hallucination and poison free powered by our verifiable databases and cryptography tools.

◷ Joined April 2024▧ 74 Photos and videos
Provably avatar
Provably@GetProvably·Jul 9

SourceryKit in early adoption: - detects 100% of agent tool-call errors. - improves accuracy by up to 50% in some models. - works across single-agent and multi-agent workflows. This is why we are building runtime verification for production agents.

Post attachment
Provably avatar
Provably@GetProvably·Oct 20

3/4 How many answers included every supported fact? • Conversation alone: 12 of 24 • With the organised record: 14 of 24 Two more complete answers. A preliminary result, not proof of higher overall accuracy.

Post attachment
Provably avatar
Provably@GetProvably·Oct 20

2/4 Same model. Same recovered evidence. Same instruction to write a fresh answer. Two approaches: • Conversation history alone. • That history plus an organised record separating successful tool calls and their results from failed calls.

Post attachment
Provably avatar
Provably@GetProvably·Oct 20

1/4 AI agents use tools to find information. Retries can recover what was missing. But does the answer use what they found? We examined 24 repaired runs to test whether organising the tool results helped.

Post attachment
Provably avatar
Provably@GetProvably·Sep 16

7/7 Of 32 guided runs that recovered complete evidence, only 11 also passed the final-answer check. The next challenge: getting that evidence into the answer. These were selected failures, weighted toward missing tool evidence. provably.ai/blogs/the-trace-…

Provably avatar
Provably@GetProvably·Sep 16

6/7 One hard task followed a chain from a museum to its architect, his birthplace, a library and the nearest subway. Guided recovery found evidence for all five expected claims. The final answer still omitted them. Finding the evidence isn’t the same as using it.

Provably avatar
Provably@GetProvably·Sep 16

5/7 The benefit varied by task difficulty. Guided vs blind evidence recovery: • Simple: 71.4% vs 33.3% • Hard: 52.0% vs 28.0% • Medium: 28.6% for both This describes the selected runs—not proof that medium tasks are harder to repair.

Post attachment
Provably avatar
Provably@GetProvably·Sep 16

4/7 When tool evidence was missing, guidance recovered 23 of 42 runs. Blind retry recovered 9; naming missing claims, 10. Naming the gaps barely helped. Guidance is most effective.

Post attachment
Provably avatar
Provably@GetProvably·Sep 16

3/7 On the same 60 failed runs: • 18 recovered with both guidance and blind retry. • 14 recovered only with guidance. • Zero recovered only with blind retry. • 28 recovered with neither. Guidance recovered every run blind retry did—and 14 more.

Post attachment
Provably avatar
Provably@GetProvably·Sep 16

2/7 After each retry, we checked two things: did the tool calls provide all required evidence, and did the final answer pass the correctness check? Guidance improved evidence recovery much more than final-answer correctness.

Post attachment
Provably avatar
Provably@GetProvably·Sep 16

1/7 Even the best agents get tool calls wrong. What helps them recover? Our previous study showed why a convincing answer isn’t enough. We took 60 failed runs and tried three recovery methods on them. Here’s what we learned.

Post attachment
Shyam avatar
⟳ Provably retweeted
Shyam@ProvablyShyam·Aug 13

anthropic.com/research/multi… This is fantastic read. 1. agents struggle to coordinate with each other especially for common resources. 2. they don't know how to detect when they are being lied to. And as usual Anthropic wants to solve it through more learning..

Provably avatar
Provably@GetProvably·Aug 11

SourceryKit detected 100% of the covered deterministic grounding errors in this study. Next, we will test how much accuracy agents can recover through answer healing and targeted retries. Read the study: provably.ai/blogs/The-Agent-…

Provably avatar
Provably@GetProvably·Aug 11

We believe agents should capture evidence natively as they work. SourceryKit captures tool calls in a state of the art verifiable database, allowing agents to evaluate themselves and downstream agents to independently verify their answers.

Post attachment
Provably avatar
Provably@GetProvably·Aug 11

Hard tasks exposed the gap. Answer scoring passed 7 of 60 runs. Complete trace support passed 0 of 60. The answers looked correct, but the recorded execution could not support every required claim.

Post attachment