←
Richard Wei591 posts
Richard Wei avatar

Richard Wei

@rxwei

mts on on-device AI | previously engineering leader @ apple

◷ Joined September 2009▧ 33 Photos and videos
Jensen Huang avatar
⟳ Richard Wei retweeted
Jensen Huang@JensenHuang·Nov 17

Over the past week, I’ve been overwhelmed by the love and support of friends, industry colleagues, and people I've never met. Thank you to everyone who has shown kindness to my family and me. It means the world to me. I came to the United States 53 years ago, and becoming an American citizen is one of the greatest gifts and proudest moments of my life. NVIDIA has been my life’s work. Building it into the company it is today alongside my colleagues has been a privilege and an honor. Nowhere but in America could a company like NVIDIA be possible. I love our country, and I love NVIDIA. I did not live up to the standards expected of everyone at NVIDIA, and I have made serious mistakes in my personal life. I created problems for the Board that I deeply regret. I own my mistakes completely, and I am committed to cooperating with the Board’s independent investigation. Since my departure, I’ve seen an outpouring of calls from NVIDIA’s employees, investors, business partners, and customers for me to return as CEO. Many continue to reach out to me directly as the company navigates complex technology, market, competitive, and geopolitical dynamics. NVIDIA is a once-in-a-generation company. If the Board of Directors decides that my return is in the best interest of our shareholders, our employees, the company, and America, I would be honored to answer these calls and return as CEO. Thank you again to everyone who has shared their good wishes. I am profoundly humbled by your support. To my family: I love you, and I will spend the rest of my life being worthy of yours. Jensen

Pratik Gandhi avatar
⟳ Richard Wei retweeted
Pratik Gandhi@pratikg·Nov 8

FULL RECAP: Apple Intelligence Group Lab, the short version for Local AI builders Sharing highlights from this week's session hosted by @shshnkp (Apple AI/ML Evangelism Team) with @mgorbach and engineers from Foundation Models, Core AI, Siri and App Intents teams at Apple. 1) Apple’s app framework can use your open-weight model as its LLM. Apple’s new app framework can use Apple’s on-device model, Apple’s cloud model or a Core AI converted model, which is interchangeable with a one liner! And your model gets access to the same tool calling, structured output, profiling with Instruments, etc. as Apple’s models. 2) Don’t use Apple’s on-device model for coding, even according to Apple engineers! Use Core AI to bring in a model for coding. One engineer uses a setup where he can use MLX and Core AI models in different “coding harnesses” and reports good experiences. 3) Free local Chat Completions endpoint on any Apple silicon Mac with macOS 27! fm serve provides an OpenAI style Chat Completions endpoint backed by Apple’s on-device model on any Mac. Nothing to install! (shared by Shashank the day after the lab) 4) Apple’s new on-device model is 20B, but only has 8K context. AFM Core Advanced (20B) will be available on the latest iPhones, iPads and M3+ Macs. Can’t opt out of Apple Intelligence in 27, and the model will always be present. Tip: use typed units for fields and let constrained decoding handle it (instead of wasting tokens explaining in prompt). 5) Agentic Siri only works when online. Tool calls from Siri to apps are done using Private Cloud Compute. When offline, Siri will use the on-device model but won’t be able to call into apps. 6) Lessons Apple engineers follow when building agents apply to any agent: - write down instructions to the model about how/when to use your tools in a markdown file (“skills” are the same idea as agent skills everywhere else!) - skills are just context to the model, meaning no ordering guarantees. If something should always happen in an order, it should probably be a tool - make tools accept lists of items to operate on to avoid agent having to call it in a loop - unit test your tools (deterministic) before testing the whole agent (non-deterministic) Curated Q&A relevant to Local AI builders in 🧵

Post attachment
Richard Wei avatar
Richard Wei@rxwei·Nov 6

A few weeks ago, I wrapped up 7 great years at Apple. I've since joined a startup! Looking back, I’m really proud of the core developer frameworks and SDKs we built, and of helping bring free, fast on-device AI models, guided generation, and Private Cloud Compute to app developers. To me, great API design is an art form. It takes taste, empathy for developers, and a relentless pursuit of simplicity. I’m also grateful for the opportunity to help shape lots of WWDC videos, speak twice at the WWDC Platforms State of the Union, and build multiple teams from scratch. Above all, these experiences reinforced my belief in what small, talented teams can accomplish. That feels especially true today. Thank you to everyone who made this chapter so special. I’m staying in SF and can’t wait to share what’s next.

Shashank Prasanna avatar
⟳ Richard Wei retweeted
Shashank Prasanna@shshnkp·Nov 6

At yesterday's Apple Intelligence Group Lab livestream, many of you had questions about Apple Foundation Models context size. We recently published a doc Article with more details. developer.apple.com/document… You can also see it on the terminal using the fm cli. Also, fun fact: You can start a Chat Completions API server using the on-device model using fm serve

Post attachmentPost attachmentPost attachment
Josh Elman avatar
⟳ Richard Wei retweeted
Josh Elman@joshelman·Nov 3

This is huge news for Siri AI (and one of the last things I was working on when I stepped away) Any app can now offer dynamic skills from their app to Siri so users can take broader actions across their apps (App Intents instead of MCP)

Big news for our developers! We are shipping a developer preview that dramatically expands the capabilities of all apps to integrate into Siri. So excited to see what you folks do with this. developer.apple.com/news/?id…

Sam Lazarus avatar
⟳ Richard Wei retweeted
Sam Lazarus@slazarus·Oct 30

We're laser-focused on the Apple platforms development experience for really serious builders. Some examples: - Providing world class simulator use tools for your agent - A system that allows agents to work with Xcodebuild in a way that gets out of your way (priority queueing builds based on whether you're looking at them / the agent is doing them in the background, etc. More to come here soon as well, I have something really special cooking 🧑‍🍳) - Support for sims even apple doesn't provide (CarPlay sim!) - A great out-of-the-box suite of skills + tools that help with Apple platforms development - Best-in-class support for letting your agent use App Store Connect so you never need to worry about things like taking screenshots, uploading builds, etc. There are many great orchestrators out there, but there are a lot of unique things about development for Apple's platforms that a standard Agent + Orchestrator setup doesn't equip your AI well to handle. That's what Bitrig is here for!

Jesse Bunch avatar
⟳ Richard Wei retweeted
Jesse Bunch@bunchjesse·Oct 27

Huge day for Creator Studio today. Our new in-app assistants can help you get things done in record time. MCP allows you to bring your own agent if you want. And an open source format for Keynote (and Python library!) so you can create Keynote documents programmatically on any platform. Super proud of the team. Hope you love it!

Alex Choi avatar
⟳ Richard Wei retweeted
Alex Choi@AlexChhk·Oct 21

wtf i just discovered that a macbook comes with a free local LLM in your terminal

Post attachment
Shashank Prasanna avatar
⟳ Richard Wei retweeted
Shashank Prasanna@shshnkp·Oct 15

Today we also published benchmarks and a deep dive into LLM inference on M5 Ultra Mac Studio, covering MLX, GPU Neural Accelerators, tensor and pipeline parallelism, and Thunderbolt 5 RDMA. machinelearning.apple.com/re…

Muse Spark Open -- weights releasing today -- runs great on new Mac Studios thanks to work with @johnternus and Apple.

Post attachment
Peter Friese avatar
⟳ Richard Wei retweeted
Peter Friese@peterfriese·Oct 10

You can now use @OpenAIDevs ‘s Decision model via Apple’s Foundation Model framework! One API to rule them all 💍

Let your app choose the right model, tool, or action in near real-time with Decisions API, now available to all developers in public beta. The Decisions API makes decisions up to 10x faster than GPT-6 Luna through the Responses API.

1:12
Peter Friese avatar
⟳ Richard Wei retweeted
Peter Friese@peterfriese·Oct 1

Decision models like Jev are all the rage - they are super fast, dirt cheap, and - best of all - deterministic. What if... you could use Jev (and other System One models) using Apple Foundation Models framework? Meet System One models in Swift: go.peterfriese.dev/system-on…

Post attachment
Peter Friese avatar
⟳ Richard Wei retweeted
Peter Friese@peterfriese·Sep 21

Want to use @typesafeai's Jev in your @SwiftLang apps using the convenience of Apple's Foundation Models framework? Well - here you go (GitHub repo link in the replies)! (h/t to @rxwei for the inspiration!)

Post attachment
Richard Wei avatar
Richard Wei@rxwei·Sep 21

I clearly made some typos in the code example! 😅 The request should look like this:

Post attachment
Richard Wei avatar
Richard Wei@rxwei·Sep 21

Jev is a fantastic example of co-designing SDKs and models to unlock a new paradigm! When we were designing the Swift API for Apple’s Foundation Models framework two years ago, we wanted to make type-safe structured responses the standard for apps on our platform. That is why we created the `@​Generable` API, making constrained decoding approachable to app developers. This effectively eliminated structural hallucinations in both tool call arguments and responses, which were especially important for a small on-device model. developer.apple.com/document… It would be fascinating to see decision models conform to the `LanguageModel` protocol and work with `@​Generable` natively! The best part is that all the APIs are already there: Bools are "noul", enums represent choices, and `@​Guide` descriptions are instructions. A Swift model implementation simply needs to handle a schema conversation, and output probabilities can be stored directly in the response metadata. developer.apple.com/document…

Post attachmentPost attachment
Richard Wei avatar
Richard Wei@rxwei·Sep 20

It supports both the on-device model and Private Cloud Compute! We built it to enable effortless automations for Mac users. It's free, zero setup, no privacy compromises.

TIL macOS 27 now comes with new fm cli 🤯 We can now use Apple's Foundation Models (on device) from CLI!

Post attachment