


Agent product engineer @CREAOAI Ex Atlassian Dev AI Build AI systems for businesses


"RAG in a Trenchcoat: When Minimal Memory Is Enough for Agentic Systems, and When It Isn't" — new paper (arXiv preprint, Nov 17, 2026). arxiv.org/abs/2611.22179 In plain terms: agent memory is how a long-running assistant keeps facts and habits from past conversations. Many memory products stack fancy layers (fact extraction, knowledge graphs, reflection) on top of RAG (retrieval-augmented generation: look up relevant stored notes, then paste them into the prompt). The authors show a deliberately simple RAG memory ties the best systems on today's memory benchmarks (standard test sets), then build new tests that check whether the agent actually uses what it remembers. 3 distinct takeaways for product builders: - Recall tests don't separate memory designs. Their minimal memory, Bridge, matched the previous best on LongMemEval-S and BEAM and beat the best memory system on EverMemBench by 5.7-6.3 points, using ~5-7K tokens (chunks of text the model reads, roughly 3/4 of a word each) per question, 17-154x fewer than pasting the whole history. Decision: ship a simple RAG memory first and make any complex memory beat it on your own tasks. - What you store decides what the agent can do. Memories saved as separate one-line facts (LangMem) best applied a rule mentioned once in passing (80% vs Bridge 35%); memories saved as per-session summaries (Bridge) best repeated a procedure that was only ever demonstrated (88% vs 55%). Picking a design per request didn't work: a classifier reading the request guessed right 49-61% of the time, no better than always picking the most common answer (60%). Decision: test with "unspoken rule" and "repeat the pattern" tasks that match your product, not only Q&A about the past. - Retrieved is not the same as used. Bridge fetched the right rule in all 20 rule scenarios but the model often ignored it, buried among ten blocks. One extra call that pulls standing rules into a labeled block at the top of the prompt raised the pass rate from 52.6% to 94.7% (a model reading the full history: 90%), and from 42.1% to 78.9% on the hardest version. Decision: surface standing rules prominently, separate from other retrieved context. Experiment setup (from the paper): - Bridge: per-session observation notes (an LLM summary of each block, written with the 10 prior notes as context), hybrid search (BM25 keyword matching + cosine similarity over embeddings, i.e. number vectors that capture meaning) merged by Reciprocal Rank Fusion, top 10 blocks shown in time order, 3 as raw text. - Recall benchmarks: LongMemEval-S (gpt-4o reader, the model that writes the answer), BEAM 100K (gpt-4.1-nano), EverMemBench (gpt-4.1-mini, gemini-3-flash). - New probes: 5 tasks x 20 scenarios: apply a rule (RA+), avoid a banned approach (RA-), copy a code style (PI-S), repeat a procedure (PI-P), keep a rule across a 5-turn chat (DM); plus 87 scenarios in clinical, legal and finance and 2 harder levels. Graded on the generated work by a judge model from a different vendor; checked with 2 more graders and Llama-3.3-70B / Qwen2.5-7B readers. - Compared: Mem0, LangMem, Graphiti, and an oracle model given the full history.

"Evaluation-Free Recursive Self-Improvement for Coding Agents" — new paper (arXiv preprint, Nov 16, 2026). arxiv.org/abs/2611.21056 In plain terms: a self-improving coding agent rewrites its own code (its scaffold: the prompts, tools and control logic around the model) to get better. The usual loop runs every new version on benchmark tasks and keeps the winner, which costs hundreds of task runs. ELITE (Explore-then-Integrate) skips that: it writes several focused improvements from the same parent, reads the resulting code, and merges the compatible pieces into one next version, checked only by fast pass/fail tests that the program builds and runs. 3 distinct takeaways for product builders: - You may not need a scoring loop to improve an agent. With zero candidate task runs, ELITE took the starting agent from 60.0% to 85.6% on a 60-task SWE-bench Pro set (a benchmark of real GitHub bug fixes), close to the score-driven methods DGM (86.7%) and HGM (88.3%), in 4.9 hours vs 43.2 and 15.7 (8.8x faster than DGM), at about $8 vs $45 and $30. Try "propose, read the diff, merge" before paying for repeated eval runs. - Tuning against one test set can overfit, meaning the gains don't carry to new tasks. Picking the best of seven children by score hit 91.7% on Pro but transferred worse (70.0% / 62.0%) than merging them (73.3% on SWE-bench Verified, 69.3% on Polyglot, a multi-language coding benchmark). Always check an evolved agent on tasks it never trained on. - The cheap gates matter most. Removing the deterministic checks (does the new code build, run, keep its interfaces) dropped the average from 76.1% to 52.0%, the biggest drop of any ablation (removing one part to see its effect). If your agent edits itself, block any change that fails a basic run test before anything else. Experiment setup (from the paper): - Method: ELITE, 7 same-parent children per generation (K=7), compatibility-aware merge, deterministic validity gates; no task outcomes used during evolution. - Baselines: Darwin Godel Machine (DGM) and Huxley-Godel Machine (HGM), which use Pro task scores as feedback; same starting scaffold and same model for self-editing and task solving. - Models (the LLM inside the agent): DeepSeek-V4-Flash (3 runs each) and Qwen-3.7-Plus; extra run on mini-swe-agent v2.4.6 with Claude-opus-5.5 (Pro 71.7% to 83.3%). - Benchmarks: SWE-bench Pro 60-task subset for evolution; SWE-bench Verified and Polyglot for transfer only (agent frozen, never evolved on them). - Metrics: task success rate, number of task evaluations, self-modifications per hour, tokens, wall-clock time, API cost.

"Discovering Effective Agent Memory Designs through Program Evolution" — new paper (arXiv preprint, Nov 12, 2026). arxiv.org/abs/2611.17068 In plain terms: agent memory is the code that saves what a user said in past chats and looks it up later to answer a new question. Instead of hand-designing that code, MAPE lets a language model rewrite the whole memory program over and over, starting from one that stores nothing, keeping changes that score better. The winning designs beat hand-built memory systems and reveal two simple patterns you can copy. 3 distinct takeaways for product builders: - Keep the raw chat, not only the summary. Extracted notes (short facts pulled out by a model) drop details. Removing the retrieved raw conversation turns from the best design cut accuracy by 12.17 points on LoCoMo and 6.5 on LongMemEval. Store original messages next to your notes and return both at lookup time. - Prepare the found evidence by question type before answering. Their "Query-Routed Scaffolding" sends each question down a path that fits it: date arithmetic in code for time questions, item listing plus code counting for "how many" questions. Bolted onto five existing memory systems (Mem0, A-Mem, LightMem, SimpleMem, Zep), it raised accuracy by 2.75 to 9.25 points on LongMemEval. Do math in code, not in the model's head. - Score cost, not just accuracy. Searching for accuracy alone kept adding model calls (one step raised cost about 6x for a small gain). Searching for accuracy and cost together moved work from save time to question time: 90% and 40% fewer model calls, and on LoCoMo with Qwen 81.33% accuracy, 12.01 points above SimpleMem at 35% of its serving cost. If each chat is asked few questions, skip heavy processing when saving. Experiment setup (from the paper): - Method: MAPE, evolutionary program search (a model proposes code edits, each is tested, a diagnosis agent reads failure traces). MAPE-A optimizes accuracy only; MAPE-P optimizes accuracy and cost together. - Benchmarks (fixed test sets): LoCoMo and LongMemEval-S, long multi-session conversations with questions about them; 20% used for search, 80% held out for final scores. - Serving models: GPT-4.1 mini and Qwen3-30B-A3B-Instruct-2507; embeddings (text-to-vector for similarity search) from text-embedding-3-small; GPT-5 mini judges answer correctness. - Baselines: full conversation in the prompt, plain RAG (retrieve messages by similarity, then answer), Mem0, A-Mem, LightMem, SimpleMem, Zep. - Metrics: answer accuracy (%) and serving cost in dollars per 1,000 questions, including the cost of building memory.

"MemCredit: From Relevance to Utility for Decision-Time Memory in Long-Horizon Web Agents" — new paper (arXiv preprint, Nov 11, 2026). arxiv.org/abs/2611.15600 In plain terms: an agent on a long task keeps notes on what it already did (its "memory"), but only a few hundred tokens (the word-pieces a model reads) of notes fit next to each decision. Most systems pick the notes that look most similar to the current situation. MemCredit instead picks the notes that actually change what the agent does next, and gets higher success with far fewer tokens. 3 distinct takeaways for product builders: - "Looks related" is not "helps": the authors replayed the same moment with and without a single note 1,041 times; 86.9% of those swaps changed nothing, and similarity to the current situation had essentially no link to whether a note helped. Decision: don't let a similarity score alone decide what goes into the prompt; measure which notes change outcomes. - Split finding from choosing: use cheap similarity search to pull a wide pool of candidate notes, then a small learned scorer (an ExtraTrees model, a fast tree-based predictor) picks which ones fill the budget, re-scoring after each pick so duplicates drop out. With a 500-token memory budget this reached 86.57% task success vs 65.67% for BM25 (classic keyword search). Decision: add a second "is this useful?" ranking step after retrieval; the agent model itself stays unchanged. - Fewer, better notes cost less: on WebShop it scored 0.612 using 152.1 memory tokens per step, vs 0.586 at 427.6 tokens for the best baseline (MemArbiter) and 597.0 tokens for BM25. Notes were worth most right after a failed action. Decision: shrink memory budgets once you rank by usefulness, and favor notes that help the agent recover from errors. Experiment setup (from the paper): - Benchmarks: ALFWorld (text-based household chores, 134 unseen tasks, success within 50 steps = SR@50) at 500- and 750-token memory budgets (90.30% vs 81.34% for BM25 at 750); WebShop (simulated online shopping, 200 tasks over a 10K-product catalog, official task score). - Agent model: Qwen3.6-27B-FP8 served with vLLM, deterministic decoding; cross-model checks swap in Qwen2.5-7B-Instruct, DeepSeek-V3 and GPT-5.4. - Baselines: Flat Recency (keep the latest history), BM25, Dense Retrieval (embedding-based similarity search), State-Aware Retrieval, Failure-Aware Retrieval, MemArbiter. - Training signal: 1,041 counterfactual interventions (replaying one step with vs without a note) from 45 ALFWorld tasks; the scorer is checked with task-disjoint 5-fold cross-validation (Spearman, AUPRC, AUROC).

"The RSI Tax: When Recursive Self-Improvement Is Not Worth the Cost" — new paper (arXiv preprint, Nov 10, 2026). arxiv.org/abs/2611.13524 In plain terms: a self-improving agent learns from its own past tasks by rewriting its prompts, code or skill files between jobs (recursive self-improvement, RSI). This paper bills the tokens (the units LLM providers charge by) that the learning step itself burns, the "RSI Tax", and finds a simple capped notes file matches or beats the heavy methods at roughly a third of the total cost. 3 distinct takeaways for product builders: - Cost accounting: the learning bill can be bigger than the work bill. On one terminal task stream, Meta-Harness spent fewer tokens doing the tasks than the simple baseline (184.13M vs 214.37M), but its 578.45M-token RSI Tax pushed its total to 762.57M vs 214.65M, with slightly lower accuracy (77.5% vs 78.7%). Compare self-improvement methods on total tokens, not per-task tokens. - Where the money goes: verification, meaning re-running the agent on check tasks to see if a proposed change really helps, was 74.8–93.6% of the RSI Tax for Meta-Harness, EvoSkill and RRSI. EvoSkill spent 497.45M tokens re-checking old candidates vs 182.26M serving the 89 real tasks. Cap or sample verification runs first; that is where the savings are. - Start simple: Simple Reflection with Notes (SRN) only rewrites one note of at most 2,048 tokens from failed tasks, every four tasks. It had the top accuracy in 3 of 4 settings (tied in the 4th), +4.7 points over a never-learning agent for +3.6% tokens, and used 65.6–71.0% fewer tokens than the heavy methods. Ship a bounded lessons file as your baseline before building a search loop. Experiment setup (from the paper): - Methods (6): Static (never learns), ReasoningBank (stores lessons from past reasoning), EvoSkill (evolves reusable skill files), Meta-Harness (searches over the agent's controller code, the "harness"), RRSI (harness edits penalized for overfitting), and SRN. - Models: DeepSeek V4.1 Flash and GPT-6 Luna; all start from the same KIRA agent. - Task streams: SWE-Chain-Evo (105 positions of chained coding tasks across 7 repository chains) and Terminal-Bench 2.1 (89 command-line tasks). - Protocol: each task is scored before its feedback can be used (no peeking); learners update every four tasks. - Metrics: accuracy (% of tasks passing an automatic checker) and total tokens = task execution + RSI Tax, split into verify / explore / write. - Limits: two models, one task ordering, single runs.

"DISCERN: Preserve at Write, Resolve at Read for Persistent LLM-Agent Memory" — new paper (TTCL Workshop 2026, arXiv Nov 11, 2026). arxiv.org/abs/2611.15729 In plain terms: agent memory means notes an AI agent saves from past chats and looks up later. This Amazon paper shows agents usually already find the right note. What breaks is deciding which lookalike company you meant, which value is current, and when to say "I don't know". DISCERN keeps every fact and every lookalike name when saving, and makes those calls only when a question is asked. 3 distinct takeaways for product builders: - Retrieval (looking up the relevant stored notes) is mostly solved. Every competent system scored about 0.93-0.98 on recalling stable facts and latest values. Spend your effort on the answer decision, not on a better search ranking. - Never merge lookalike names or overwrite old values when saving. DISCERN keeps "Zephyron Cloud Solutions" and "Zephyron Cloud Systems" as separate, linked records, and marks old values expired instead of deleting them. On truly ambiguous questions it scored 0.800; no other system beat 0.40. Store history with valid-from/valid-to dates and resolve at read time. - Make "I don't know" an explicit, earned action. Every question must end in one of four actions: answer, update, clarify, or abstain (decline). Declining is only allowed after a search shows the fact is absent. Abstain F1 (a score balancing right declines against wrong ones) was 0.924 vs at most 0.802 for 8 baselines, with only 2.9% of answerable questions wrongly declined. Track declines as their own metric. Experiment setup (from the paper): - Benchmark ENT-310: synthetic enterprise sales-agent memory. 10 sales users, 50 companies, 31 past sessions, 310 questions (210 answerable, 100 unanswerable traps). - Transfer test: LoCoMo, 1,986 questions over long personal chats, zero tuning. Macro answer correctness 0.663 vs 0.638 for HippoRAG2. - 8 baselines: BM25 keyword search, dense search (matching by meaning vectors), RRF fusion of both, an agentic RAG search loop, an extract-and-retrieve pipeline with query enrichment, GraphRAG, HippoRAG2, Mem0. Answer model: Claude Sonnet 4.6. - Metrics: answer correctness averaged equally over 4 skill groups (0.861 vs 0.806 best baseline), plus abstain precision/recall/F1. Scored by majority vote of 4 LLM judges (Sonnet 4.6, Opus 4.8, Qwen3-235B, GPT-oss-120B). - Ablation: letting the reader take up to 8 lookup turns lifted accuracy from 0.33 to 0.89. Average cost: 5.7 tool calls and 7.6 s per question.

"Correct Answers from Insufficient Histories: A Controlled Study of Agent Memory Evaluation" — new paper (arXiv preprint, Nov 9, 2026). arxiv.org/abs/2611.11840 In plain terms: agent memory systems hand a model only part of its past history (a trimmed or compressed log), then most evals (automated tests) check the answer against a reference worked out from the FULL history. This paper shows a model can "pass" that check even when the part it was shown could not possibly determine the answer, so a high score can hide lucky guessing. 3 distinct takeaways for product builders: - A matching answer is not proof the memory worked. With logs cut short so the answer was provably not in view, Qwen3-32B still matched the full-history reference on 87 of 203 cases (DeepSeek-V4-Flash: 85 of 203). Recommendation: in your memory evals, label each test case as "answer is in the context" or "answer is not in the context", and count hits on the second kind as unsupported, not as wins. - Asking the model to say "I don't know" is not enough. Both prompts explicitly told the model to reply UNDETERMINED when the evidence was missing, yet it gave a concrete answer on 196 of 203 (Qwen) and 182 of 203 (DeepSeek) such cases. Recommendation: if guessing is costly, enforce abstention in code (for example, check whether the retrieved log is complete for the time asked about) instead of relying on the prompt. - A 99% LLM judge (a model used to grade other models' answers) still slips. The DeepSeek-V4-Flash judge was right on 1,908 of 1,920 constructed cases (99.38%) but twice approved answers the visible log did not support, and two outputs were cut off; a simple rule-based program scored 1,920 of 1,920 with zero model calls. Recommendation: when correctness can be checked by rules, run a deterministic checker first and use the LLM judge only for what rules cannot cover. Experiment setup (from the paper): - Environment: synthetic, executable device logs (each report has 8 fields: ids, times, device, action, argument, outcome; duplicates and rejected actions included); 4 query types: count of accepted routes, first route, last route, device state. - Evidence labels: "insufficient" is proven by building a second legal history that yields byte-identical visible context but a different answer; "sufficient" by public rules (complete-log marker, a complete record after the query cutoff, or first route already visible). - Readers: hosted Qwen3-32B and DeepSeek-V4-Flash, temperature 0, 64-token output limit; 80 confirmation histories, 1,600 requests per reader; context cut to a 768-token budget (a token is roughly a word piece). - Conditions: raw-text prefix, whole-record prefix, CSV prefix, plus full raw and full CSV controls; the CSV vs raw effect on appropriate answers stayed inconclusive. - Metrics: reference agreement vs. "context appropriateness" (answer only when the visible log determines it, otherwise abstain); 10,000-replicate bootstrap 95% intervals clustered by history. - Judge study: 120 held-out histories, 1,920 constructed candidates, compared against finite and extended rule programs and a program-first hybrid.

"When Should Agents Remember? Falsification-Gated Self-Evolution for LLM Agents" — new paper (arXiv preprint, Nov 10, 2026). arxiv.org/abs/2611.14326 In plain terms: many self-improving agents write a "lesson" into memory (saved notes the agent reads on future tasks) after each task. This paper treats every lesson as a guess that must pass tests before it is saved: try it on a temporary copy of the agent, check cases where it should help, cases built to break it, and tasks the agent already solves, then save it, narrow it, or throw it away. They call this FGSE. 3 distinct takeaways for product builders: - A lesson that fixed one task is not proof it is safe to keep. In their diagnostic study, free-form lessons often helped similar tasks but hurt look-alike tasks where the rule should not apply, and broad, vague lessons did the most damage. Write each memory rule with an explicit "only when..." condition instead of a general instruction. - Test each lesson against counterexamples before saving it. Removing these break tests raised the share of lessons that fail them from 5.0% to 18.6%, and harmful saved lessons from 3.6% to 10.6%. Of 380 candidate lessons, 104 were saved; 101 rejected ones looked useful on their target tasks but failed 22.8% of break tests. A few deliberate counterexamples per rule catch most bad lessons. - Keep a regression set (tasks your agent already solves, rerun after every change) and let real checks decide. Without the regression set, damage on previously solved tasks rose from 1.3% to 5.2%; letting an LLM judge (a model grading by opinion) decide what to save instead of executable checks cost 3.1 points on WebArena. Gate memory writes on tests and state checks, not on a model's opinion. Experiment setup (from the paper): - Base models: GPT-4o-mini, GPT-4o, DeepSeek-V3.1; FGSE changes only the agent's saved memory, not model weights. - Benchmarks (test suites): WebArena (web-browsing tasks, success rate), AppWorld (multi-app API tasks, task-goal completion), τ-bench Retail/Airline (customer-service tool use with a simulated user, Pass1 = solved on first try), LongMemEval and LoCoMo (questions over long chat histories), AgentDojo (resistance to prompt injection, i.e. malicious instructions hidden in tool output), plus HumanEval, MBPP, SWE-bench Verified, GSM8K, MATH. - Baselines: Reflexion, Self-Refine, Voyager-style skill memory, MemGPT, A-Mem, ACE and others. - Results with DeepSeek-V3.1: 39.8 WebArena success (ACE online: 26.4), 78.8 AppWorld average (ACE online: 68.2), 0.724 / 0.487 Pass1 on τ-Retail / τ-Airline; falsification failures 5.0%, regression damage 1.3%, harmful saved updates 3.6%.

"System Optimizations for Cost-Efficient Agent Harnesses" — new paper (arXiv preprint, Nov 9, 2026). arxiv.org/abs/2611.12122 In plain terms: a harness is the wrapper program around an AI model (like Claude Code or Codex) that gives it instructions and tools and calls it in a loop. UC Berkeley researchers found the harness barely changes whether the agent solves a task, but strongly changes the bill. They built Agency, a layer that sits between an unmodified harness and its computer, and cut agent cost to 0.77x of baseline on average while keeping 0.97x of the accuracy. 4 distinct takeaways for product builders: - Your biggest cost is re-reading history, not new work. Every model call resends the whole conversation so far (the "context"). Even at cached prices (a discount for text the provider has already seen), re-reading took up to 73% of total cost. Measure cost per call and trim context before you swap models. - Trimming context only pays if the math says so. Rewriting old context forces a fresh, pricier cache write, and deleting something the agent later needs costs extra turns to fetch it again. The paper compacts only when predicted savings beat those costs, using a small decision-tree model to guess which lines will be reused. Gate compaction on an expected-savings check, not a fixed token limit. - Give the agent a map instead of letting it wander. Agents spent up to 89% of turns searching folders on lookup-heavy tasks. A prebuilt SQLite index of files and their contents cut cost to 0.68x on its own, and to 0.51-0.66x when files had random ID names in deep folders. If your agent searches documents, index them first. - Savings grow with run length, so don't judge on short tests. Runs under 100K tokens (a token is roughly a word piece) dropped only to 0.93x; runs over 1M dropped to 0.67x; 17-29% of tasks got pricier. Also, telling a small model to batch tool calls cut steps 19% but raised cost 2%. Fewer steps does not mean cheaper; benchmark on your longest real sessions. Experiment setup (from the paper): harnesses = Claude Code 2.1.309, OpenAI Codex CLI 0.147.0, and Basic-full (a plain think-act-observe loop with 6 tools). Models = GPT-6-Luna, GPT-6.1-Sol, Sonnet 5.5, Opus 5.5. Tasks = 100 coding tasks (40 SWE-bench Verified and 40 SWE-bench Pro, which fix real GitHub issues; 20 Terminal-Bench 2 shell tasks) plus 100 OfficeQA Pro document-lookup tasks in 4 folder layouts. Cost patterns also measured on public real-user traces (TraceLab, FreeInference). Metrics = cost ratio and pass-rate ratio vs. the unmodified harness. Techniques = tool-output compaction, context garbage collection (deleting stale lines), subtask stack machine (collapsing finished sub-jobs into a summary), metadata-assisted search, tool-call batching.

"Decision Context: Persistent Behavioral Learning for AI Agents" — new paper (NeurIPS 2026 PALM workshop, arXiv Nov 5, 2026). arxiv.org/abs/2611.08644 In plain terms: today's agents forget your corrections when the session ends, so you fix the same mistake every day. Most agent memory stores facts (a name, a preference). This paper stores behavior: it turns your corrections, plus the agent's own failed tool calls (requests to an API or function that errored), into short "always do X" rules, and shows each rule to the model right at the step where it applies. 4 distinct takeaways for product builders: - Store corrections as commands, not notes. Plain retrieval (pasting the most similar saved text into the prompt) of past corrections reached 55% compliance (share of checklist items the answer got right). Compiling them into short imperative rules reached 95%, while adding about 10x fewer tokens (word pieces the model reads; 159 vs 1,651 for the graph version). Rewrite feedback into one-line directives before you save it. - Deliver a rule at the moment it applies. Some rules only matter once a tool returns certain data, so the system searches its rule store again between tool calls and adds newly matching rules then. "Only when X" scenarios went from 53% to 95%. Don't load all memory only at session start. - Learn from the agent's own failures, not just user feedback. A tool call that fails and then succeeds on retry is a lesson nobody types. Correction-only designs scored 0-20% on this category; a second model reading only the tool-call log reached 78%, the full system 90%. Log retries and errors and turn them into rules. - Don't argue with the model, and don't break what already works. Adding a one-sentence reason to each rule scored flat or lower: given a reason, the model sometimes decides to override. Memory can also make an agent worse; the full system never failed a case the no-memory agent passed, while every earlier version failed at least one. Test memory against a no-memory baseline. Experiment setup (from the paper): DC-Bench, a new test set of 71 scenarios in 9 behavior categories, built from real agent runs with mocked tools (fake tools returning fixed outputs, so every system sees the same data), in two domains: financial analyst and customer support. Corrections are given about one company (Acme) and tested on another (Beta), so it measures applying a lesson, not recalling text. Agent: Claude Opus 4.7. Judge: GPT-4 scoring each answer against a 3-8 item checklist, 5 runs per scenario. Search: text-embedding-3-small embeddings (text turned into number vectors for similarity search) fused with keyword search. Baselines: no memory 10%, fact memories Mem0 12% and Zep 15%, procedure memories AWM 34% and Memp 49%, four earlier designs 55/63/70/82%. Full system (forge_dc) stays at 93-97% on Sonnet 4.5, Opus 4.5, Opus 4.6, GPT-4.1 and GPT-5.5; its one-time rule-building cost pays back after about 4 queries.

"SUMONE: A Simple and Efficient Summarizer for Plug-and-Play Agent Memory" — new paper (arXiv preprint, Nov 4, 2026). arxiv.org/abs/2611.06445 In plain terms: a long-running AI agent appends every step to its prompt, so each new model call re-reads, and you re-pay for, a growing history. "Compaction" means replacing the old part of that history with a short summary once it gets too long. SumOne is a small separate model (2B parameters; parameters are the learned numbers inside a model, and 2B is tiny by today's standards) trained only to write those summaries, and it plugs in front of an existing agent without changing the agent. 3 distinct takeaways for product builders: - Cost: compaction is a large cost lever with a small quality price. Same agent on web-research questions: cost per question fell from $0.170 with the full history to $0.038 with SumOne, while the score went from 0.52 to 0.48 F1 (F1 = a 0-to-1 answer-accuracy score). On coding tasks: $0.286 to $0.071, and the bug-fix rate went from 0.63 to 0.59. Simply dropping the oldest steps instead collapsed the web score to 0.16. If your agent runs long, measure a summarizer before paying for a bigger context window (the max text a model can read at once). - Who writes the summary: "ask the agent to summarize itself" is not the best default, and neither is a small model with only a prompt. The trained 2B summarizer beat self-summary (0.44 to 0.48 F1 on web, 0.56 to 0.59 on coding) at 4-8% of the summarizing cost, while the same 2B model, prompted but untrained, dropped to 0.29 F1. The training is what makes the small model work. - Portability: the training needs no labeled data. Half of the old history is hidden, and a fixed second model tries to rebuild the hidden parts from the summary; the better it rebuilds, the higher the reward. Because this is not tied to one task, one checkpoint worked across 5 downstream LLMs and 4 agent harnesses (harness = the code that runs the agent loop and tools) without retraining. Still test it in your own harness: on Codex it scored 0.47 vs 0.55 with full history. Experiment setup (from the paper): summarizer starts from Qwen3.5-2B and is trained with GRPO (a reinforcement-learning method that scores a group of candidate summaries and nudges the model toward the better ones) for 600 steps on ~5K compaction cases at 16K/32K tokens (a token is roughly 3/4 of a word), drawn from OpenSeeker-v1-Data (web-search runs) and SWE-smith (coding runs). Reward: a frozen Qwen3.5-9B rebuilds 50% masked ~512-token blocks, graded by DeepSeek V4.1 Flash, plus a length penalty. Tests: DeepSearchQA (web research questions, scored by F1), SWE-bench Verified (real GitHub bug fixes, scored by resolved rate), LongMemEval-S (an isolated memory test). Agent: Qwen3.5-35B-A3B in the OpenCode harness, compaction at 32K tokens. Baselines: Full History, Recency Truncation, Self-Summary, Prompt-2B, CoMem. Transfer: DeepSeek V4.1 Flash, GPT-6 Luna, GLM-5.3, Gemini 3.7 Flash; harnesses OpenHands, Codex, DeepSeek Harness.

"RECEIPT: Measuring and Repairing the Return Assumption of LLM Agents in Web Service Operations" — new paper (arXiv preprint, Nov 1, 2026). arxiv.org/abs/2611.01930 In plain terms: when an agent runs a shell command, its harness (the wrapper code that launches commands and hands results back to the model) assumes the command finished and did what was asked. But many commands stop to ask a question first ("Delete these files? y/N"). If the harness closes the program's input, the program takes its default and can exit with status 0 (the "success" code) without doing anything; if it leaves input open, the command just hangs until a timeout. Receipt fixes this by reading from the operating system what each process is waiting on and telling the agent. 3 distinct takeaways for product builders: - Exit code 0 does not mean the job was done. With input closed, 18 of 156 tested prompting commands (11.5%) returned 0 without doing the action: docker system prune removed nothing, rm -i left the files. And 7 of 8 popular harnesses close input when no terminal is attached. Check the effect (did the rows/files actually change?), not just the exit code. - Leaving a question unanswered is expensive. At a plain terminal nobody types into, a waiting question made a task use 2.9x the tokens (the text chunks you pay the model for) and 2.6x the run time. Detect waits at run time and report them: Receipt cut false "I'm done" claims from 34 to 9 versus closed input and used 24.4% fewer tokens than a plain terminal, adding a median of at most 0.54 ms per shell call. - Leaderboards partly rank your harness, not the model. Just changing how input is connected moved Llama-3.3-70B from 7th to 3rd of 8 models. Before choosing a model from a benchmark (a fixed test set used to compare models), fix and report how the harness connects input. Experiment setup (from the paper): measurement on 91,469 production tool calls from 6 developers and 89,047 public agent trajectories (recorded step-by-step agent runs) from 23 harnesses and 19 LLMs; 8 harnesses tested. Litmus = a catalog of 158 command-line invocations of 77 programs, each checked for whether the action happened under closed, pipe and terminal input. Tripwire = 25 task pairs (database migration, deployment, cleanup, credentials) where the "trap" twin differs only in a question guarding the action; a hidden verifier script checks the final state. Harnesses: mini-SWE-agent 2.4.6 and Terminus 2. Models: Qwen2.5-Coder-7B/14B/32B, Qwen2.5-72B, Llama-3.1-70B (plus Llama-3.3-70B, Mistral-Small-24B, Gemma-2-27B for rankings), temperature 0, 30 steps. Metrics: verified successes, false successes, run time, tokens, wait report rate (85.0% vs 66.7-68.3% for text-matching baselines, no false positives). Baselines: text hint, outcome monitor, DeepSeek's process inspector.

"How Do AI Agents Waste Your Money?" — new paper (arXiv preprint, Nov 4, 2026). arxiv.org/abs/2611.07238 In plain terms: AI providers bill by the token (a small chunk of text, roughly three-quarters of a word), and an agent run can burn millions of them. This paper measures how many of those tokens were actually removable, by having a second model act as a judge (an "LLM-as-a-Judge") that marks text which can be deleted without losing useful work. Across their runs, roughly 18% of output tokens were waste. 4 distinct takeaways for product builders: - Most waste can be cut without hurting results. Pruning the reasoning the judge flagged as repeated or filler, while the agent was running, deleted 41% of reasoning on SWE-bench Verified (a benchmark, i.e. a fixed test set, of real GitHub bug fixes) with success going from 89.0% to 89.5%; deleting all reasoning dropped it to 73.0%. Decision: trim repeats and filler, not thinking as a whole. - The task and tools drive waste more than the model does. Waste varied much more across benchmarks than across six frontier models, and the biggest bucket was "detours": exploration that looked reasonable but turned out unnecessary. Decision: before switching models to save money, give the agent clearer tools and reusable notes so it explores less. - Max reasoning effort mostly pays for nothing. Reasoning effort is a setting for how long the model "thinks" before acting. Going from Low to High raised success from 69.7% to 78.4%; going from High to Max cost another 0.80 million tokens with no measurable gain, and on tasks both solved, Max cost 1.62 times as much. Decision: default to High, not Max, and test the step up on your own tasks. - The harness (the code wrapped around the model that runs its loop, tools and context) changes the bill. With the same model and the same tasks, five different harnesses produced markedly different token use and action patterns. Decision: measure waste per harness instead of only total tokens. Experiment setup (from the paper): a proxy built on LiteLLM (an open-source router that sits between the agent and the model API) records every turn with only a few lines of code changed; an LLM judge labels four waste types: Restate (repeating what is already in context), Duplicate-Fetch (re-calling a tool for unchanged data), Padding (preambles and narration) and Detour. Benchmarks: tau-bench airline and retail (customer service), SWE-bench Verified (coding), Claw-Eval and MCP-Atlas (general tool use). Models: Kimi-K3, Gemini 3.1 Pro, GPT-5.6 Terra, GPT-6 Astra, Claude Sonnet 5, Claude Fable 5.1. Harnesses compared with Kimi-K3: OpenCode, OpenClaw, OpenHands, Hermes, Kimi Code CLI. Validation: removing flagged waste and resuming the run; next-action shift on SWE-bench Verified in 3.0% of turns vs 25.0% when all reasoning is removed. Code and labeled runs are open-sourced.

"CARE: Grounded Recursive Self-Improvement for Production Customer-Service Agents" — new paper (arXiv technical report, Nov 2, 2026). arxiv.org/abs/2611.02839 In plain terms: a team at Meituan's Keeta replaced an expensive frontier model (GPT-5.4) in a live food-delivery support agent with their own 31B model, then let that model keep learning from its own mistakes in production. A reviewer process finds real failures, decides whether the model or the surrounding setup (prompts, tools, Skills = written step-by-step procedures) was at fault, and only model faults become training data. After five rounds, bad conversations fell from 6.42% to 1.97%, below GPT-5.4's 4.60%. 3 distinct takeaways for product builders: - Blame before you train. Many agent failures come from a broken tool, an outdated procedure or the harness (the code around the model), not the model. CARE routes those to the owning team and only trains on failures it attributes to the model. Recommendation: add an attribution step to your failure review so you don't "fix" a tool bug by retraining. - Learn from the first wrong step, not the whole conversation. Each failing conversation is cut at the first bad decision, a verified fix is written, and the model learns from itself: the same model acts as teacher but is shown the diagnosis and fix as hints the student never sees (on-policy self-distillation). On a small batch of fixes, plain fine-tuning (training on example answers) collapsed, and preference training (rewarding preferred over rejected answers) collapsed into handing every case to a human, passing 15.1% of simulated cases; self-distillation held 94.1%. Recommendation: with few correction examples, prefer self-distillation over naive fine-tuning. - Gate every release on frozen tests plus real users plus a human. A new version must match the deployed one on a simulation test compiled from the latest procedure (checking whether refunds or cancellations actually happened, not what the model claims), then win an online comparison, then get human sign-off. Result: replacing GPT-5.4 cut median reply time 27.7% and serving cost per order 70.5%, with user satisfaction at parity (+1.7 points in the latest period). Recommendation: a smaller in-house model can replace a frontier API if you build this release gate first. Experiment setup (from the paper): - System: CARE (Customer-service Agent Retrospective and Evolve) — a cold start from the incumbent's logged conversations, then a loop of 24/7 inspection (an external frontier model reviews conversations without seeing the expected answer, with independent review), attribution, repair, and self-distillation. - Models: CARE-31B (in-house 31-billion-parameter model; parameters = the learned numbers inside a model) vs. incumbent GPT-5.4 and other frontier models including Gemini-3.5-Flash. - Deployment: multilingual food-delivery customer service across seven regions; five self-improvement rounds producing six model versions. - Metrics: inspection Bad Case rate (share of reviewed conversations with a confirmed failure), simulation accuracy (93.0% to 94.1%), combined score (96.07 vs. incumbent 96.10), online user satisfaction, median latency, serving cost per order. - Ablations (removing one piece to see its effect): fine-tuning vs. preference optimization vs. self-distillation; best loss = forward KL divergence (a measure of how far the student's word choices are from the teacher's) over the teacher's top 32 tokens (tokens = word pieces).

"Optimize the Harness, Not the Model: Fitting a Minimal Agent to a Self-Hostable LLM for Investigative Analysis" — new paper (arXiv preprint, 2 Nov 2026). arxiv.org/abs/2611.03434 In plain terms: a "harness" is all the code around a language model that turns it into an agent: the tools it can call, its memory, its safety rules, and the loop that keeps it working. This paper grows a 3,000-line harness from almost nothing around a fixed open-weights model (one you can download and run on your own servers), and it scores within 1.2 points of the roughly 512,000-line Claude Code SDK on 240 held-out investigation questions. 4 distinct takeaways for product builders: - Size: a small harness can match a big one. With the same model, CLEO (3k lines) passed 83.8% of questions vs 85.0% for Claude Code SDK and 80.8% for the bare PI harness (75k lines). On 240 questions that gap is about three answers. If you must self-host or pass a security review, tune the harness before reaching for a bigger model. - Direction: grow, don't prune. Starting from only ls, read, grep and a sandboxed Python, an automated loop added a part only when testing showed it was needed. Starting from the big PI harness and encouraging removals reached the same accuracy but barely shrank the code. Start near-empty and add. - Gate: higher accuracy alone did not earn a change its place. Each proposed edit also had to be generic, auditable (easy for a reviewer to inspect) and lean. A cite() helper was reverted because the agent rarely used it. Add an "is this worth the extra surface?" check to your change review. - Overfitting guard (overfitting = tuning to the test instead of the task): each loop round scored only 44 questions (32 fixed + 12 freshly sampled) so it finished in minutes, the question pool was periodically swapped for a different case, and the final 240 questions were never shown to the loop. Keep a held-out test set your optimizer never sees. Experiment setup (from the paper): - Model: GLM-5.1, open-weights and self-hostable, kept fixed throughout. - Method: CLEO (Constrained Loop for Environment Optimization), an autoresearch-style propose-test-keep loop driven by a supervisor model, Claude Opus 4.8. Starting agent: a ReAct-style loop (the model alternates thinking and tool calls) with ls, read, grep and sandboxed Python, plus persistent per-case memory. - Data: 3 intelligence-analysis use cases; a typical one holds ~8,000 files in up to 7 formats (250 MB to 6 GB, with misleading and contradictory evidence). 300+ expert questions at 4 difficulty levels, 240 held out. - Baselines: Claude Code SDK (~512k lines, as reported), PI #1 (75k lines, same prompt), PI #2 (PI with CLEO's prompts, 83.8%). - Metrics: pass rate and a 0-10 soft score from an LLM judge (a model grading each answer against expert ground truth): CLEO 8.41 vs 8.48 for the SDK. Latency 114s vs 76s (SDK) and 126s (PI #1). - Limits: one domain; the judge was spot-checked by experts but not systematically validated.