←
Jack Vanlightly1,821 posts
Jack Vanlightly avatar

Jack Vanlightly

@vanlightly

Likes breaking ideas and systems, writing, picking systems apart @confluentinc Ex @Splunk, @VMware jack-vanlightly.com Credit: ESO/B. Tafreshi

◷ Joined November 2016▧ 123 Photos and videos
Gunnar Morling 🌍 avatar
⟳ Jack Vanlightly retweeted
Gunnar Morling 🌍@gunnarmorling·Nov 12

As we're getting closer to the #Hardwood 1.1 CR1 release, I took the time to re-run the benchmarks from the 1.0 announcement, this time on the little N300 box on my desk (8 Intel Gracemont cores, clock fixed at 1.5 GHz). 1️⃣ The 1.0 results hold on very different hardware. The announcement ran on an AWS m7i.2xlarge. Against the same parquet-java version (1.17.1), Hardwood 1.0 on the N300 shows pretty much the same multiples as the original announcement. 2️⃣ We spent a significant amount of time on tuning the decode kernels as well as the wider read path, and it's paying off: Hardwood got a lot faster since 1.0! For this benchmark, it is now more than 3× faster with by-name field access, and 1.7–1.8× faster on the columnar scan and with the indexed row reader. Against the latest parquet-java (1.18.2), that's up to 15× faster on all eight cores and up to 6.3× with Hardwood limited to a single core. It's great to see the single-core improvements in particular, for instance when thinking of running in K8s pods with limited CPU resource assignment. 3️⃣ Filters that match every row now cost nothing extra. When row-group statistics prove that every row of a row group matches, Hardwood skips the filter column for that group entirely. On one core over 50M rows, that takes the scan from 2.9 s to 0.37 s. 4️⃣ parquet-java improved too. 1.18.2 is 4–7% faster than 1.17.1 on full scans and about 40% faster on filtered scans, so I compared against it, not the version from the original announcement. As always, these are numbers from some specific workloads, on one specific machine, so you should do your own testing too. All numbers are JMH means, median of multiple runs, on one month of NYC taxi data and a generated 50M-row event file for the filters. Charts attached, raw data and scripts at github.com/hardwood-hq/hardw…

Post attachmentPost attachmentPost attachment
Jack Vanlightly avatar
Jack Vanlightly@vanlightly·Nov 11

Another case of Claude seeing a Kafka benchmark confounder I missed. I asked Claude for analysis of the results. Claude: Looking at disk queue length, it's as if all 6 consumers are assigned the same network processor. Me: It's assigned round-robin, can't be. <rerun with network processor metrics collection> Me: Claude you were right! Turns out my single-broker benchmark, with 3 network threads. Each consumer starts serially, each opening 3 network connections to the same broker, such that the fetch connection always ends up on the same network processor for every consumer. My 24 hours of benchmark data all bad.

Chris avatar
⟳ Jack Vanlightly retweeted
Chris@criccomini·Nov 9

7/ Bug fixes: union-clone data loss, sequence tracker corruption, and a recovery race that could let a stale writer open after losing ownership. Special thanks to @vanlightly for his TLA+ proof and the paper authors for pointing out the recovery issue. github.com/slatedb/slatedb/i…

Jack Vanlightly avatar
Jack Vanlightly@vanlightly·Nov 5

Only mildly better than '4' I suppose I dread to think how many security bugs exist in the world's software right now. It's probably much larger than we ever dreamed (and we already knew it was bad).

update your boxes /now/ the default user on alpine's aws and azure images take any password longer than 256 characters. yes, any password. yes. i know. yes.

Post attachment
Jack Vanlightly avatar
Jack Vanlightly@vanlightly·Nov 5

Pretty cool, but the sheer volume of latent vulnerabilities out there combined with adversaries armed with AI, is frankly scary! It will get a lot worse before it gets better.

We are now allocating a significant amount of our frontier compute towards discovering and patching vulnerabilities in open source codebases, fully pro bono. In one case, we studied the Linux kernel, which is a particularly load-bearing piece of software at OpenAI. Before AI, security researchers might discover ~20 local privilege escalation (LPE) vulnerabilities in Linux in a typical year. In one week's time, our upcoming frontier cyber model was able to find ~1000 LPE vulnerabilities using about $1M of tokens. In the process, it was able to discover at least 10 novel classes of Linux exploitation techniques.

Jack Vanlightly avatar
Jack Vanlightly@vanlightly·Nov 5

The human brain just doesn't have the brain circuits for understanding concurrency well, or rewriting model weights by hand for that matter. AI on the other hand has no (will not have) such inherent limitations.

like, will we get to a point where a super-intelligent agent is better at finding deep bugs in a spec than spending TLC compute cycles to check the spec. anecdotally, i feel like human experts who know a protocol super deeply sometimes outperform formal/automated tools on this

Jack Vanlightly avatar
Jack Vanlightly@vanlightly·Nov 4

I repeat, I was wrong. Also it turns out this SlateDB spec is pretty amenable to scale out. I can keep adding cores so it's over 300% faster on my threadripper rn. M5 Max: 18 workers -> 30M states/min. TR 18 workers -> 65M states/min. TR 32 workers (half the 64 cores) -> 95M states/min. TR 40 workers -> 105M states/min No real benefit beyond 40. Humming at 4.8 GHz.

It's amazing how much faster my MacBook Pro M5 Max is than my Threadripper 9980X for model checking small state spaces. I assume it is, in part, due to higher memory latency on the TR (has CL52 64GB DDR5 ECC 6400 sticks). But for bigger specs, I have 8x the memory on the TR.

Jack Vanlightly avatar
Jack Vanlightly@vanlightly·Nov 4

EDIT: The TR is actually much faster, I was running a different set of parameters initially. Still, M5 has pretty impressive numbers for a laptop.

Jack Vanlightly avatar
Jack Vanlightly@vanlightly·Nov 4

Oh I was wrong, I was running a different set of parameters, the Threadripper is actually 72% faster on the same spec, same parameters. 😅 That makes me feel better for having spent 12 grand on it, though I think it's worth 25+ now since the rampocalypse.

It's amazing how much faster my MacBook Pro M5 Max is than my Threadripper 9980X for model checking small state spaces. I assume it is, in part, due to higher memory latency on the TR (has CL52 64GB DDR5 ECC 6400 sticks). But for bigger specs, I have 8x the memory on the TR.

Jack Vanlightly avatar
Jack Vanlightly@vanlightly·Nov 4

It's amazing how much faster my MacBook Pro M5 Max is than my Threadripper 9980X for model checking small state spaces. I assume it is, in part, due to higher memory latency on the TR (has CL52 64GB DDR5 ECC 6400 sticks). But for bigger specs, I have 8x the memory on the TR.

Jack Vanlightly avatar
Jack Vanlightly@vanlightly·Nov 4

It’s not the first time I’ve missed a bug at first because of either too small a model or not exploring all small model parameter combinations. I missed a glaring one when exploring Flexible Raft a few years ago. In this SlateDB case the Small Scope Hypothesis still holds as the violation is found in 1 hour 35 min on my Mac M5 Pro with 18 workers. I was just in too much of a hurry to write and verify the other systems (11 now) in my WAL-on-S3 survey.

Jack Vanlightly avatar
Jack Vanlightly@vanlightly·Nov 4

New paper on TLAPS-Bench, "a benchmark and execution harness for constructing machine-checked proofs for real-world system specifications in TLA+ with agentic AI” It turns out that TLAPS-Bench revealed that my TLA+ specification of SlateDB's WAL protocol demonstrates a read consistency bug, and it looks like SlateDB has it too (though it is EXTREMELY unlikely to ever happen, and needs to be confirmed). It’s another case of GC complicating the life of put-if-absent. The authors also validated the fix which is simple. I had run the model checker with 3 competing writers and never saw the violation. But TLAPS-Bench, with Astra Max, ran it with 4 competing writers and found the issue. arxiv.org/pdf/2611.02849

Jack Vanlightly avatar
Jack Vanlightly@vanlightly·Nov 2

"the signals we used to rely on as markers of a mature system are now easier to counterfeit"

At HPTS, Kyle mentioned a company confidently handed him a vibe-coded database to test. Now Jepsen report on ParticleDB is out. IMO I feel sad for the engineers involved. It's a long report, but at least read the part on LLM tradeoffs. "Perhaps engineers will begin to treat any new software project with suspicion—the signals we used to rely on as markers of a mature system are now easier to counterfeit." jepsen.io/analyses/particled…