Founder @Onehousehq — Data Infra for Agents. Creator of @apachehudi, built the first #DataLakehouse @Uber. Databases, Dist. Systems. LinkedIn, Confluent.
Performance is already solid: a 10,000-PDF corpus processed in 2.8 minutes on 32 CPU cores (excluding embedding calls, which depend on provider rate limits anyway).
We’re continuing to enhance, improve and optimize it. Read the blog for the full product walkthrough, architecture, and benchmarks and share your feedback.
onehouse.ai/blog/announcing-…
Choose a processing option:
- “Pass through” to store one row per file
- “Parse content” for one row per text chunk, image, or video frame
- “Parse and embed” to generate vectors using your configured model
The results land in open tables your agents and jobs can query alongside regular lakehouse tables with structured data, instead of being silo-ed into a special multi-modal lakehouse.
Today, OneFlow ingestion and Quanton Spark/SQL jobs support unstructured data processing.
OneFlow listens for new files in your bucket and processes them incrementally. It extracts and chunks text from documents, keeps images whole, and samples frames from videos, with optional speech transcription.
Onehouse makes this seamless by transforming a file dump in a storage bucket into a well-organized, optimized, indexed table that agents can query directly with SQL.
By filtering on metadata and computing and maintaining vector embeddings for files (and parts of files), agents can sift through vast amounts of data efficiently—without downloading the same content over and over again.
For decades, “using data” meant a person opening a file and typing a few fields into a database. Today, models can read and interpret that content directly. But to respond, agents still have to guess which files to open, fetch them, and burn tokens sifting for signal across millions of documents, before any real work can begin.
90% of enterprise data is unstructured, and only about half ever gets analyzed (IDC). Agents won’t unlock that by brute-forcing buckets with raw LIST and GET.
We’re starting our announcement run ahead of re:Invent. First up: unstructured docs, images, and video: parsed, embedded, and stored in open tables.
As AI erodes the “well-integrated” and “fully-managed” moats, serverless data vendors will face this big time.
First up, probably EMR. Collecting a 25% tax for running slow Spark jobs.
Incredibly proud of the @Onehousehq team.
The Gartner list is mostly hyperscalers and very large enterprise software companies, plus two mid-sized startups and two real startup challengers.
If scored on technical capabilities alone, the leader quadrant will look quite different.
🏆 Onehouse earned an honorable mention in Gartner’s 2026 Magic Quadrant for data lakehouse platforms. And we’re the only full-featured lakehouse on that elite list that can deploy anywhere.
It’s great to see Gartner recognize the full breadth of today’s Onehouse platform.
We started small. Now Onehouse is a fast, full-stack lakehouse: ingestion, Spark/SQL pipelines, cataloging, table management, compute, and low-latency serving.
But we’ve stayed stubborn about one thing: customers should own a single open copy of their data, use the engines they want, and run the platform where they want.
That flexibility, paired with a fully managed lakehouse, is exactly the vision we’ve been building towards for years.
Gartner captured it in a few crisp sentences.
Excited for where this goes. More to come.
⚡ Uber benchmarked Onehouse Quanton against OSS Spark on production-scale Apache Hudi workloads.
🚀 4.7–5.5× faster ingestion on 10–60 billion records.
💰 Nearly 9× better compute efficiency (89% fewer provisioned executor vCPU-hours) in a reduced-core experiment, at comparable latency.
What makes this interesting is the methodology: synthetic data modeled on Uber's production workloads, preserving scale, schema characteristics, and insert/update patterns.
Enjoyed co-authoring this engineering blog!
🔗 uber.com/us/en/blog/uber-ben…#ApacheSpark#ApacheHudi#Quanton
So, bring us 3 of your toughest Spark pipelines. The most expensive one, the slowest one, and the one you've already tuned to death. 5 engineer hours over a month, to kickoff the runs.
Let's run it head-to-head. Compare runtime, cores, and cost.
If Quanton wins, great. If it doesn't, I want to understand why.
If you own a Spark platform, what would it take to put one pipeline through this test?
Just use OSS Cost Analyzer tool for Spark jobs and DM me pypi.org/project/spark-analy…
We've been running Quanton measurements with several Spark teams operating at planet scale. I suspect a lot of Spark teams are sitting on significant performance and cost savings they haven't measured yet.
⚠️ There's a catch-22 holding them back. Teams want proof that a new engine will improve their workloads before spending time evaluating it. But that proof only comes from running their workloads. So the same expensive pipelines keep running, without ever testing how much better things could be.
Uber also ran a “reduced core” experiment, keeping the job runtime target fixed while varying the number of cores. Quanton slowed down much less: 4.8x slower with 10x fewer cores. Faster tasks gave Spark more opportunities to smooth out stragglers. In the heaviest stage, task-time speedups exceeded 4–5x.
Quanton is Onehouse's spark/SQL engine, that runs right inside your existing clusters. The evaluation replayed production workload patterns at full scale on Kubernetes.
• 60 billion records, insert-heavy: 5.5× faster median job runtime.
• 10 billion records, update-heavy: 4.7× faster median job runtime.
⚡ Quanton engine is really freakin fast. Here’s the receipt. Uber put out a blog that unpacks Quanton evaluation at production scale against OSS Spark and found it 4.7x-5.5x faster.
More than 200,000 CPU cores power Uber’s data lake ingestion. At that scale, every efficiency gain counts.
Making better use of that compute means examining how Spark executes queries, moves data, and writes to growing tables. Accelerating one stage can leave other bottlenecks untouched.
Uber engineers are improving production workloads with native vectorized execution and evaluating new ways to accelerate write-heavy pipelines.
Explore the engineering choices behind those efforts, where the benchmarks showed gains, and what remains to be optimized.