←
TokenWave.AI30 posts
TokenWave.AI avatar

TokenWave.AI

@TokenWaveai

Compile Real-World Complexity into AI Capabilities.

◷ Joined September 2026▧ 12 Photos and videos
TokenWave.AI avatar
TokenWave.AI@TokenWaveai·Nov 17

To see where delivery breaks down, we analyzed a set of CoWork runs from Claude Opus 5.5. We traced 979 failed checks for new behavior back to specific requirements. In 44% of those failures, the agent never received the requirement. In 56%, the agent had received it and the check still failed. In one run on carapace-bin, the agent received all 25 requirements held by teammates. Still, 22 of 35 checks for new behavior failed. The trace showed incorrect CLI flags and a build failure the agent left unchecked. If you’re building coding agents, pick a failed check and trace it back through the agent’s conversations, code changes, and tests.

Post attachment
TokenWave.AI avatar
TokenWave.AI@TokenWaveai·Nov 17

We ran GPT-6 Astra on the same 101 tasks with the same tests and time budget. Giving it the full spec up front raised its pass rate on checks for new behavior from 62.91% to 78.84%. Full-task passes stayed at 8/101 in both settings.

Post attachment
TokenWave.AI avatar
TokenWave.AI@TokenWaveai·Nov 17

Take MEM-227, one of several tickets the agent had to resolve in Mempool. After its first commit, the agent got a message about MEM-227, one of the cards on that board. It asked the owner what the audit summary should return when there was no saved audit. The owner said to return the stored audit if one exists, or null otherwise. The agent made that change and both local checks for that specific behavior passed.

Post attachmentPost attachment
TokenWave.AI avatar
TokenWave.AI@TokenWaveai·Nov 17

In one SWE-CoWork assignment, the Mempool Workbench has 103 task cards. Active work sits alongside old tickets, related projects, and maintenance tasks. The agent has to work out which records still apply, who can confirm a requirement, and which questions need a follow-up.

Post attachment
TokenWave.AI avatar
TokenWave.AI@TokenWaveai·Nov 17

GPT-6 Astra led on full-task completion, passing every required check on 8 of 101 assignments. Claude Opus 5.5 led on tests for new behavior, passing 63.84% of those checks and fully completing 7 of 101 assignments.

Post attachment
TokenWave.AI avatar
TokenWave.AI@TokenWaveai·Nov 17

Can a coding agent take over a teammate’s work in progress? Introducing SWE-CoWork, a benchmark for coding agents working as digital employees on long-horizon software engineering tasks. Each assignment starts with a short handoff in a simulated workplace modeled on real-world engineering workflows. Agents search task boards and versioned docs, ask teammates for missing details, and revise their code when a reply changes what needs to be built. Open questions and confirmed requirements can both be lost before the agent hands the work back. SWE-CoWork lets us trace them through the conversation, the patch, and the tests. 101 assignments across 101 real open-source repositories. 26 simulated teammates per workplace. 11 programming languages. 12 application domains. 17 workplace tools. 🌐 Website: tokenwave-ai.github.io/swe-c… 📄 Paper: arxiv.org/abs/2611.20218 💻 Code: github.com/TokenWave-AI/swe-… 🤗 Hugging Face Daily Papers: huggingface.co/papers/2611.2… (If you find it useful, we’d appreciate an upvote!)

1:36
TokenWave.AI avatar
TokenWave.AI@TokenWaveai·Nov 5

[9/9] Thank you to all our OSWorld-Science collaborators: Dingyuan Dai, Heli Qi, Lei Liu, Yinxi Li, Baiding Chen, Zijun Dou; Qingcheng Zeng (@SteveZeng7), Qi Kang, Oliver Kunyang Sun, Eric Wang, Bo Zhou, Haixin Wang, Yufan Du, Shi Bo, Ruihan Lin, Mengqi Yuan, Dunjie Lu, Steven Dillmann (@StevenDillmann), Yiming Shi, Tina Su, Amy Xin, Minghao Liu, Xi Wang, Xu Huang; Ge Zhang (@GeZhang86038849), Pengyu Nie, Zhen Yang, Jie Tang (@jietang), Juanzi Li, Weihao Xuan, and Tianyu Liu (@meSuper8). Full author list and affiliations: discoailab.github.io/osworld…

TokenWave.AI avatar
TokenWave.AI@TokenWaveai·Nov 5

[8/9] Help expand OSWorld-Science: • Contribute tasks grounded in real research needs across scientific, clinical, and engineering workflows. • Bring cross-application and interdisciplinary tasks, including drug-discovery workflows. • Debug and verify tasks and evaluators. • Support testing with API access or sandbox resources. Researchers and collaborators welcome. Submit a task: forms.gle/dAK1xuLJCfGJG9sm7

TokenWave.AI avatar
TokenWave.AI@TokenWaveai·Nov 5

[7/9] On OpenFOAM task OF-004, GPT-5.6 sol computed the correct solution but exported 6 significant digits where at least 8 were required. It checked row counts and CSV headers, but not the delivered precision against the requirement. Correct solve. Failed delivery.

Post attachment
TokenWave.AI avatar
TokenWave.AI@TokenWaveai·Nov 5

[6/9] On QuPath task QP_004_T1, GPT-6 Astra installed an extension through the app and checked its menu entry. Claude Opus 5 copied it to a guessed folder and restarted nine times, without a deliverable. Recognizing a prerequisite is not enough. Verify it in the application.

Post attachment
TokenWave.AI avatar
TokenWave.AI@TokenWaveai·Nov 5

[5/9] How do we know the work is correct? Tasks come from expert proposals and iterative human-AI co-design. Execution-based evaluators inspect application states and artifacts, awarding partial credit against task-specific criteria.

Post attachment
TokenWave.AI avatar
TokenWave.AI@TokenWaveai·Nov 5

[4/9] Scientific workflows can cross applications and interfaces. 122 tasks allow CLI access; 24 are GUI-only. All use screenshot-based observation. The challenge is coordinating visual interpretation, GUI actions and commands while tracking application state. For researchers less familiar with code or the command line, an important design goal is to make agent workflows easy to inspect and correct through familiar software interfaces.

TokenWave.AI avatar
TokenWave.AI@TokenWaveai·Nov 5

[3/9] A recorded Ansys Fluent task: continue an inkjet-nozzle simulation from 18 to 24 microseconds using the same numerical scheme. GPT-6 Astra reports the liquid volume-fraction sum and outlet area-averaged axial velocity. The recorded task score is 1.000.

Post attachmentPost attachment
TokenWave.AI avatar
TokenWave.AI@TokenWaveai·Nov 5

[2/9] How well do agents handle scientific software? In the paper's 12-model evaluation, Claude Fable 5.1 led with a 73.7% mean task score. Partial credit included; VOID = 0. These are mean scores, not full-task completion rates.

Post attachment
TokenWave.AI avatar
TokenWave.AI@TokenWaveai·Nov 5

[1/9] What are the limits of AI agents in scientific research? Can they use professional software as effectively as scientists? Meet OSWorld-Science: a benchmark and evaluation environment for computer-use agents, with tasks grounded in real-world scientific research needs—from pathology image analysis to physical simulation. 146 tasks. 7 scientific domains. 17 tools. 12 VLMs evaluated. Tasks are developed through expert proposals and iterative human–AI co-design, with selection guided by scientific value and difficulty. By combining GUI/CLI interaction, artifact-based evaluation, and trajectory analysis, we examine how agents carry out scientific workflows and where execution breaks down. TokenWave is glad to be part of this research collaboration. 🌐 Project: discoailab.github.io/osworld… 📄 Paper: arxiv.org/abs/2609.39903 💻 Code: github.com/DiscoAILab/OSWorl… 📊 Data: huggingface.co/datasets/SciA…

1:09