←
Andy Zeng523 posts
Andy Zeng avatar

Andy Zeng

@andyzengineer

Co-founder @GeneralistAI general intelligence born from the physical world ✗DMs → email

◷ Joined September 2017▧ 67 Photos and videos
Generalist avatar
⟳ Andy Zeng retweeted
Generalist@GeneralistAI·Nov 10

GEN-1.5 assembling an engine housing: precisely placing two clamshell pieces onto the engine and lining them up with the screw holes, ... 100 times in a row. With our friends at @FoundryRobotics

6:46
Generalist avatar
⟳ Andy Zeng retweeted
Generalist@GeneralistAI·Oct 29

Not all chips are fried. Read more about GEN-1.5 and sign up for the waitlist in the comments below.

12:40
Andy Zeng avatar
Andy Zeng@andyzengineer·Oct 20

if you look closely at the reliability test later in the video, you’ll see a few examples of “physical reasoning” -- it tries something, it doesn’t fit (because physics), so it backtracks, switches gears, then does something entirely different to reach the goal. in this case, it’s taking out the stuck puck, making space elsewhere, and putting it back in. things like this (… a sort of physical problem-solving with step-by-step “chain-of-actions”) should just work with models out of the box.

0:06

How many pucks could a puck bot pack if a puck bot could pack pucks? 🏒 12 hockey pucks is a tight fit in this container – watch the model fix the arrangement when pucks get stuck or misplaced. GEN-1.5 doesn't just puck around. Read more about it in our blog post and sign up for the waitlist in the comments below.

3:46
Generalist avatar
⟳ Andy Zeng retweeted
Generalist@GeneralistAI·Oct 15

GEN-1.5 puts together a kit of cosmetics, then puts together another kit, then another kit, then another... We're expanding access to GEN-1.5 Sign up to get in line generalistai.com/waitlist

7:59
Evan Morikawa avatar
⟳ Andy Zeng retweeted
Evan Morikawa@E0M·Oct 9

2 years ago, I personally spent dozens of hours trying to get this task to work. It was our first “too hard” task. Now it’s just another day here.

GEN-1.5 packs a bottle into a box, then packs another bottle into a box, then another bottle, then another one, then another... Read more about GEN-1.5 in our blog post in the comments below.

4:18
Lucas Beyer (bl16) avatar
⟳ Andy Zeng retweeted
Lucas Beyer (bl16)@giffmana·Oct 9

Mesmerizing! Also, *exactly* 10y ago i was at a startup with exactly these arms and very similar grippers with exactly this camera placement. But models from 10y ago: cnn + lstm, no real pretraining. What i see now, from multiple startups, it definitely a big step up.

GEN-1.5 packs a bottle into a box, then packs another bottle into a box, then another bottle, then another one, then another... Read more about GEN-1.5 in our blog post in the comments below.

4:18
Generalist avatar
⟳ Andy Zeng retweeted
Generalist@GeneralistAI·Oct 8

GEN-1.5 packs a bottle into a box, then packs another bottle into a box, then another bottle, then another one, then another... Read more about GEN-1.5 in our blog post in the comments below.

4:18
Corinne Marie Riley avatar
⟳ Andy Zeng retweeted
Corinne Marie Riley@CorinneMRiley·Oct 6

Feels like every week in robotics there’s a new ‘this is the GPT-3 moment for robotics 🤖’ announcement.  We brought on @GeneralistAI CEO & Co-Founder @peteflorence on @GreylockVC Change Agents to dig into what’s going on at the frontier of robotics models. We covered the company’s latest Gen 1.5 model, few-shot learning, training robots on different embodiments, and the milestones towards a more generalized physical model.  Timestamps: 00:51 The inception of Generalist 04:13 Long-term goal of the company 05:47 Parallels and differences between robotics and language models 12:05 Key differentiation in 1.5 Gen model 12:48 Robot vs banana 14:49 Emergent capabilities not explicitly trained for 17:13 Cross-embodiment and the importance of hands 22:26 Research vs working with customers 24:58 Beyond VLA vs world model 28:50 Future-looking milestones Some of the top takeaways: - One-shot and few-shot learning emerged without being trained for. Gen 1.5 can learn a new task from a single demonstration, and Pete compares it to the GPT-3 moment in language. In one example, a robot taught to sweep a cube into a bowl with a brush used a banana instead. Given a dustpan, it held the pan with one hand, swept with the other, then tipped the cube into the bowl. Neither behavior was explicitly trained, and Pete sees this as a signal of where the model's generalization is headed. -Generalizing to new hands remains a challenging problem for cross-embodiment. Physical hardware doesn’t stay static, and so cross-embodiment - the ability of a physical AI model to adapt to different hardware systems - is vital for success.  -Customer deployments are a valuable source of research inputs. Generalist actively partners with their customers for feedback, which they use to inform their research and make real-world evaluations. -Generalist doesn't think in terms of "VLA vs. world model." Pete helped create early VLAs and has worked on world models, but he argues the goals matter more than the label, and the team is trained to think in a first-principled way when considering new research directions. Watch the full episode at the link in the comments. Thank you to @peteflorence for joining us!

31:49
Andy Zeng avatar
Andy Zeng@andyzengineer·Oct 6

post-training to hill-climb dexterity

Robot puts a ring on it GEN-1.5 puts an O-ring on a hydraulic rod gland, then takes the ring off, then puts the ring back on, then takes the ring off, then ring on, ring off, ring on... Read more about GEN-1.5 in our blog post in the comments below.

3:57
Generalist avatar
⟳ Andy Zeng retweeted
Generalist@GeneralistAI·Aug 26

A simple example of physical prompt steerability: same environment, different prompts, different behaviors. Thanks @JagdeepBhatia8 for the suggestion. Read more about GEN-1.5 in our blog post in the comments below.

0:29

Physical prompting is an elegant idea! One request for the @GeneralistAI team: can we see an example of *physical prompt steerability*, where different prompts induce distinct behavior in the *same* environment? I'm curious whether GEN-1.5 is listening to the physical prompt or simply executing the most likely action sequence for the scene under its pretraining prior.

Generalist avatar
⟳ Andy Zeng retweeted
Generalist@GeneralistAI·Aug 24

We've reduced the time it takes to go from physical prompt → robot behavior. The faster anyone can teach a robot to do something new, the easier it becomes to scale physical work. Read more about GEN-1.5 in our blog post in the comments below.

0:22
Will Knight avatar
⟳ Andy Zeng retweeted
Will Knight@willknight·Aug 20

I'm usually wary about talk of "the ChatGPT moment for robots" because language is less complex than the physical world. After visiting @GeneralistAI, though, where I saw their robots perform impressive manipulation with just a prompt, I think you can see it happening in a limited way. (Thanks to @peteflorence and co for having me!) wired.com/story/generalist-a…

Pete Florence avatar
⟳ Andy Zeng retweeted
Pete Florence@peteflorence·Aug 21

Good question thanks Ani. Here's a video that gives a sense of the answer here about the effect of prompting: - in the first part, the model is just babbling with no prompt in context - at the end, we add a prompt for taking money out of a pouch (generalizing to the wallet)

0:44

Very cool work, Pete! I'm curious if you've tried an extreme version of this: no in-context example at all; just place objects in front of the robot and see if it can infer the task. I suspect this will have non-trivial success rates, which would also allow you to figure out how much of the benefit is actually coming from the prompt (rather than pure zero-shot task inference + capability). I remember Russ showed something like this in his Stanford talk on LBMs last year: youtu.be/TN1M6vg4CsQ?si=-8J-…

Ben Barry avatar
⟳ Andy Zeng retweeted
Ben Barry@benbarry·Aug 20

We started planning yesterday’s @generalistai GEN-1.5 announcement about 3 weeks ago. Two weeks ago, already feeling whiplash from new results, I asked ChatGPT to create this image to share in our internal meme channel.

Post attachment