←
Wake6,465 posts
Wake avatar

Wake

@WakeFramework

Complete Solidity security: Wake Framework (open-source) for testing & fuzzing + Wake Arena for AI audits with graph-driven reasoning. By @AckeeBlockchain

◷ Joined September 2023▧ 383 Photos and videos
Wake avatar
Wake@WakeFramework·Mar 30

Four AI auditing tools. Fourteen protocols. The shape tells the story. Wake Arena 3.1's blue area contains the other three on every axis. Where all systems struggle — Burve, Notional — the gap closes. Where the work matters — Lend, Munchables, Virtuals — the edge is visible and consistent. What the chart doesn't show: 49 of the 63 found vulnerabilities were confirmed by more than one Wake Arena agent independently. The shape reflects consensus, not a single model's guess. Learn more ↓

Post attachment
Wake avatar
Wake@WakeFramework·Jul 9

9/ Sources: StarCoder — arxiv.org/abs/2305.06161 MultiPL-E — doi.org/10.1109/TSE.2023.326… CangjieBench — arxiv.org/abs/2603.14501 EVMbench — arxiv.org/abs/2603.04915 LLM Solidity tests — arxiv.org/abs/2505.05584 COMPSAC 2026 — arxiv.org/abs/2605.11163

Wake avatar
Wake@WakeFramework·Jul 9

8/ This is the design rationale of Wake: contracts in Solidity, tests and fuzzing in Python, identical EVM semantics. Generation falls on the language where measured model capability is highest; human review concentrates on test logic rather than language mechanics. → getwake.io

Wake avatar
Wake@WakeFramework·Jul 9

7/ The defensible conclusion: the disparity persists by default and closes only with targeted data or tooling. A contract's language is fixed by its deployment target. Its test suite's language is not — and it determines which region of model competence AI-written tests exercise.

Wake avatar
Wake@WakeFramework·Jul 9

6/ The evaluation ecosystem remains Python-centric. HumanEval and MBPP are Python-only; SWE-bench Multilingual covers nine languages and Aider polyglot six — none includes Solidity.

Wake avatar
Wake@WakeFramework·Jul 9

5/ On test generation: the only published measurement of LLM-generated Solidity unit tests (EASE 2025, three Code4rena projects) reports 2–4.6% correctness in single-shot generation. No comparable Solidity-versus-Python test-generation study exists to date.

Wake avatar
Wake@WakeFramework·Jul 9

4/ Solidity, 2026: EVMbench (117 real vulnerabilities): the strongest agent evaluated, Claude Opus 4.6, detects 45.9%. Zero-shot GPT-5.2: 52% macro-F1 on vulnerability detection. COMPSAC 2026, 15 models: LLMs "are not yet reliable as standalone security auditors."

Wake avatar
Wake@WakeFramework·Jul 9

3/ At current scale (Cheng et al., 2026): GPT-5 attains 96.3% on Python HumanEval but ~4.3% pass@1 generating the low-resource language Cangjie; 5 of 6 frontier models score under 5%. An agentic scaffold with compiler feedback raises GPT-5 to 77.6% — the deficit is knowledge, not reasoning.

Wake avatar
Wake@WakeFramework·Jul 9

2/ The mechanism is training data: the StarCoder corpus contains 60.40 GB of Python and 0.85 GB of Solidity — a ~71× disparity (Li et al., 2023). Cassano et al. (IEEE TSE 2023): same models, same problems, 18+ languages — 30.4 pass@1 on Python vs ~10–12 on low-resource languages (p<0.001).

Wake avatar
Wake@WakeFramework·Jul 9

1/ LLM performance is not uniform across programming languages: it tracks training-data representation, and Solidity sits at the low-resource end. In 2026, the strongest agent evaluated detects 45.9% of real Solidity vulnerabilities; LLM-written Solidity tests are 2–4.6% correct. The evidence:

Wake avatar
Wake@WakeFramework·Jul 1

On 94 high-severity findings from Code4rena and Sherlock protocols: Wake Arena 3.1 caught 63 — 67%. Nearest competitors, GPT-5.2 xhigh and Zellic v12, caught 41 — 44%. Public, reproducible, labels taken as-is. Some models may have trained on these codebases — fair caveat. → github.com/Ackee-Blockchain/…

Wake avatar
Wake@WakeFramework·Jun 16

Bridges fail because testing one chain at a time misses the interaction bugs. Cross-chain protocols introduce synchronization challenges that single-chain tests cannot replicate. Different block times. Event relay timing. State consistency across networks. Testing these in isolation creates false confidence. The contract works on Ethereum. Works on Polygon. But fails when they interact. Wake was designed from day one with multi-chain as a first-class citizen. Not an afterthought. Not a plugin. Built-in. Spin up multiple blockchain instances in one test suite. Each maintains independent state, accounts, and configuration. Fork mainnet on both chains simultaneously. Test realistic cross-chain scenarios with production token contracts and actual network conditions. The framework enforces chain-specific account usage. Can't accidentally deploy an Ethereum contract using a Polygon account. Can't transfer funds across chains without explicit relaying logic. This prevents an entire class of testing mistakes while making complex multi-chain scenarios straightforward to express. Real bridge exploits often emerge from timing assumptions, failed relay attempts, or reorg handling. Wake lets you simulate these edge cases systematically: chain reorganizations, message ordering issues, and timeout scenarios. One cross-chain DEX aggregator caught a critical synchronization bug during Wake testing that would have drained user funds during high network congestion. The bug only manifested when both chains processed transactions simultaneously with specific timing. Multi-chain protocols are complex. Testing infrastructure shouldn't make them harder. With built-in mainnet forking, independent time manipulation per chain, and intuitive Python APIs for cross-chain logic, Wake gives teams the testing foundation that distributed protocols require. Cross-chain testing should be straightforward, not heroic. Discover Wake's multi-chain capabilities ↓

Wake@WakeFramework 2 Oct 2025 Article Cross-Chain Testing with Wake: Building Robust Multi-Chain Smart Contract Tests Introduction As blockchain ecosystems become increasingly interconnected, testing cross-chain interactions has become critical for DeFi protocols, bridges, and multi-chain applications. Wake, a

Wake avatar
Wake@WakeFramework·Jun 9

Learning to write security tests? "Awesome Wake Tests" collects real production test suites from Aave, Lido, Axelar + comprehensive reentrancy defense patterns + educational resources (workshops, Ethernaut solutions, benchmarks). Study how protocols securing billions test their code ↓

Wake@WakeFramework Jan 13 Article Awesome Wake Tests: A Collection of Security Resources Check out "Awesome Wake Tests," a curated list of testing resources, guides, and real-world examples for Wake users. This repository collects production test suites, performance benchmarks, and

Wake avatar
Wake@WakeFramework·Jun 2

3/ Pair detectors with fuzzing for layered coverage: Static analysis catches known bad patterns. `wake test -P 5` explores unexpected runtime behavior. Structure + behavior — the gap most CI pipelines leave open.

Wake avatar
Wake@WakeFramework·Jun 2

2/ What gets caught by Wake before a premium audit: • Reentrancy patterns • Write-after-write bugs • ERC-4337 storage violations • Access control issues Same detector set from 200+ Ackee audits. Now running on every commit.