I had a great time seeing everyone at CoRL this week.
My talk was called "Learning in the factories", and I talked about:
- Walden's longest-running deployment to date,
- the value of coupling deployment data with large-scale pretraining, and
- a few things I've learned as I've broadened my focus from AI/robotics to include large-scale manufacturing.
I'm excited to keep sharing more of Walden's story as we go. Things are moving very fast!
-
youtube.com/live/uPXoJm-L3YY…
Our new model, Sage, surprised us - didn't expect to be able to automate this in 2026. Huge jump in deployment performance and unexpectedly strong generalization with some emergent sparks.
@RaresAmbrus and team really cooked on this one. More details of what we're seeing at waldenrobotics.com/news/lbm-…
When we first started working on Diffusion Policy, which felt like an "unlock" for fitting human demonstration data, the explanation I gave myself was that "human demonstration data was subtly much more multimodal than we thought, so we need the modeling power of denoising diffusion to fit it properly."
What Michael's paper brings to light is that some of this multimodality was probably exacerbated (more than I realized) by the fact that we've only been giving our models incredibly short observation context lengths.
For me, this is pretty compelling evidence that we should be pushing for policies with long-context lengths and short action prediction horizons.
Building great robots takes great partners.
Proud to be collaborating with @SamsungSDSA on a proof of concept focused on manufacturing use cases, combining Walden's robotics platform with Samsung SDS' manufacturing expertise.
More: bit.ly/4fZJwoJ
In January, I started "building something new" with an incredible team. Today I finally get to share some first details about what we've been building. We've called it Walden Robotics (waldenrobotics.com).
I thought long and hard about my own reasons for starting this company. It's not only about the robots. It's also about people. I've tried to capture those thoughts in my first Walden blog post: waldenrobotics.com/news/why-…
It's been an incredible ride so far. Within just a few months of forming the company, we were already operating a general-purpose robot with an end-to-end policy in production in one of the most important factories in North America. It's amazing at how much I've already learned from that experience. There is a lot of work to do, but the mission has never been so clear.
Please help me welcome Walden Robotics into the world. And stay tuned for more updates!
youtube.com/watch?v=fewvZrck…
When it comes to diffusion policies, VLAs, and LBMs, we all know that data quality matters. Many of us have spent considerable time filtering data, and discarding large amounts of data that might not be considered to be sufficiently high quality. This feels very inefficient.
We’ve seen several proposals of late for how we might learn from suboptimal data. pi0.7, for instance, suggested that we learn models conditioned on the data quality.
Adam’s new paper exposes a different (complementary) angle. When we’re training denoising diffusion models, each data point is potentially used several times — at different noise levels. **The value of a data point might be different depending on the noise level.**
Adam’s work explores this nicely and shows that he can extract value even out of significant portions of the OXE dataset that many of us had previously discarded as suboptimal.
Definitely worth having a look.
I've been saying for years that the biggest challenge for simulation in robotics is not actually the physics engine (although you do have to get that right). The real challenge is capturing the *diversity* of the real world. There was no doubt that generative AI had the potential to change that, but it's still amazing to see it take shape.
Watching Nick's incredibly fast progress has convinced me that content generation might not actually be a bottleneck anymore. This is a beautiful combination of hardened tools for e.g. low-level mesh processing with the latest tools for generative asset creation, wrapped in a powerful agentic workflow. Please do give it a try and share your feedback.
Introducing Large Video Planner (LVP-14B) — a robot foundation model that actually generalizes. LVP is built on video gen, not VLA. As my final work at @MIT, LVP has all its eval tasks proposed by third parties as a maximum stress test, but it excels!🤗
boyuan.space/large-video-pla…
This was a massive effort by the entire team, with a number of individuals really pouring their hearts into this paper. The paper is packed full of (too many?) details. Your comments and feedback would be very welcome.
Probably my favorite plot from the paper, which sums it all up, is this one.
The plot compares performance using different amounts of pretraining data used before training a new task: 0% (aka single task), 25, 50, or 100% of TRI’s data, then 100% of TRI’s data + all of the open-source robot data (the red line) that we’ve curated. It’s just awesome that the distributions over task completion are so tight and that trends as we increase data are so consistent. The results show clearly that with pretraining, we can train a novel skill with substantially less data or use the same amount of data and get much better task performance. And the benefits appear to continue with more data.
The short version is: LBMs work! We see consistent and statistically significant improvements as we increase the amount of pretraining data. But doing the science is still hard; as a field we have more work to do to improve the statistical power of our experiments.
TRI's latest Large Behavior Model (LBM) paper landed on arxiv last night! Check out our project website: toyotaresearchinstitute.gith…
One of our main goals for this paper was to put out a very careful and thorough study on the topic to help people understand the state of the technology, and to share a lot of details for how we're achieving it.
youtube.com/watch?v=BEXFnru5…