ComfyUI gave me early access and credits to test their new in-app agent, and I've been having fun working with it ahead of today's release! It embodies a lot of what I love about collaborating with AI in a creative process - making technical experimentation faster and more accessible, without sacrificing creative control.
Video made with Comfy Agent + Wan VACE - sound on! 🔊
I've been working with agents + ComfyUI for a while. For me, what makes the new in-app agent special is its ability to work directly inside your workflow, alongside you.
When you send a prompt, the agent sees the current state of your graph, including any changes you made. Its edits appear directly on the canvas, as it makes them (no reloading or opening a new version of the workflow).
It's natural to switch between tweaking parameters and running test generations yourself, and asking the agent to build a new part of the graph or troubleshoot something. It's also good at answering questions to help you get a better sense of what's happening technically, or how various factors relate to your creative intent.
The immediacy of this allows you to quickly test and develop new approaches. The iterative loop is faster, making it easier to experiment freely, and get into a creative flow.
Hoy, en el capítulo de "la IA no tiene control", me he visto la keynote de Adobe y he dicho: "ánda, qué curioso... esto lo puedo hacer con mis nodos".
He pulido un par de cosas y he terminado con mi propio método para controlar la acción dentro de un vídeo. Ahora puedo controlar los ritmos, los momentos exactos en los que ocurre algo y un montón de cosas más.
Encima funciona directamente en @ComfyUI sin necesidad de entrenar nada, tanto con Minimax H3 como con LTX. Os muestro algunos test con vídeos y sus curvas de control.
It's so cool to see old techniques - in this case, context smashing by @PurzBeats - resurrected for modern models AND explained nicely
Declining friction for people to understand + use these techniques will lead to such interesting places
Animatediff shall prevail
Context Smashing in H3 is really strange. Using animated masks to create spatially aware context mixing. It averages video/temporal tokens as well as text token blending. Makes for some truly bizarre results.
Pay attention
What you see here is vibe coding a Digitone II FIRMWARE
Yep, you read it right, here is example of Erica Synths LXR-02 running of the Digitone
… now think what Digitakt II vibe coded firmware can do… 😁
Today we launched Comfy Agent. It lives right inside ComfyUI.
Creators still need craft, and craft comes from control. I truly believe it. But they should be focusing on ideas, iteration, composition, colors, not technical terms or debugging. When I started using ComfyUI 3 years ago, I kept seeing talented people who couldn't use the tool at their highest level.
This should all go away. Today you describe what you want, and Comfy Agent takes the building part. It even works with you while you keep editing. The workflows are coming to life now. They can talk to you.
It's in Comfy Cloud now and free to try. Desktop/local is coming in a few weeks. It's still beta, so please tell us what breaks.
It is my honor to build with our amazing team, our tech leads @c__byrne, @shimikeri, as well as @UyTieu@xlatentspace, Alex DrJKL, @ComfyUIWiki@_ryan_di@shanef3d, @mattmiller_ai, Memie, Anne, Ali, Marwan and many other people. I hope you enjoy creating with us!
Comfy Agent is now available to everyone in Comfy Cloud.
It plans, builds, and fixes ComfyUI workflows right on your canvas.
We strongly believe creators shouldn’t choose between ambition and complexity. Comfy Agent is our answer. While it handles the technical part of Comfy, you can finally focus on visual craft!
- It builds while you edit. It sees the assets you see.
- Ask anything. Reference anything.
- Create your own skills, or use the public ones.
Agent in Comfy Desktop is coming soon in a few weeks.
Click the link below to try Comfy Agent today ⬇️
The in-app agent is available in Comfy Cloud now, and Comfy is providing free credits for anyone who wants to try it out. A version for local ComfyUI is in the works. I prefer to work locally, so I'm excited to work with this and see what it can do with my local workflows, assets, and custom nodes. (I ran into some limitations on Cloud, especially for more unusual/experimental workflows)
The agent uses credits (usage seemed pretty minimal compared to generations). The local version won't need a subscription and can use credits purchased as you need them, like partner nodes used in local Comfy.
A few other features:
- Workflow context: the agent can edit/work on one workflow at a time (easily switchable, within the same session), but can see and reference workflows in any of your open tabs
- Memory: The agent can save user preferences, which are available to it across sessions (in my case, it automatically saved my requirement that outputs correspond 1:1 on a frame level with my input control video, in terms of length and temporal control).
- LLM: currently running Opus 4.8 (team is planning on testing 5.5 soon - excited to see what this is like)
Other things the team is exploring for future versions:
- LLM model selection
- Ability to connect to external services like MCP servers
- Agent2Agent protocol for communication with other agents
- In the local version: detect GPU and recommend workflows that work with your system / download and install models & nodes / create custom nodes
Thanks to @ComfyUI for inviting me to test early! It was a pleasure to work with the team. They were responsive to bug reports and feedback, and are clearly a very inspired and hard working group of people. Congratulations on the launch!
I worked with the agent to create new workflows inspired by some of my existing AnimateDiff workflows - exploring how to get similar creative results with newer open weight models like MiniMax H3 and Wan.
(first in the grid is input oscilloscope video from VCV Rack; the rest are tests made with Comfy Agent + Wan VACE or Wan Fun Control)
I've used my AnimateDiff workflows for a series of audiovisual pieces that use oscilloscope video as a source of control - preserving tight audiovisual sync while using diffusion models to expand the visual palette (guided by reference images and text prompts).
The agent understood my creative goals and recommended a few ways to attempt something similar with Wan and H3. We prototyped and tested various approaches, then developed and refined them. It helped me understand details of the generation process, how they interact with each other and my source material, and how this relates to my artistic vision. That informed how I tested parameter ranges, preprocessing chains, methods of control and conditioning, etc - and the process of tweaking a workflow became less random, more productive.
The agent doesn’t get everything right on the first try, and sometimes makes mistakes (like most agents). Occasionally, it would wire things in a way that didn't work, or implement something that left out important parts of what I was trying to do. It also helped troubleshoot many of those issues.
Because of the quick back and forth on a live workflow, the process often flows smoothly and feels like working together with a skilled collaborator.
Comfy Agent is now available to everyone in Comfy Cloud.
It plans, builds, and fixes ComfyUI workflows right on your canvas.
We strongly believe creators shouldn’t choose between ambition and complexity. Comfy Agent is our answer. While it handles the technical part of Comfy, you can finally focus on visual craft!
- It builds while you edit. It sees the assets you see.
- Ask anything. Reference anything.
- Create your own skills, or use the public ones.
Agent in Comfy Desktop is coming soon in a few weeks.
Click the link below to try Comfy Agent today ⬇️
Everyday Fragments now supports stills and video renders. Its so fucking fast I might as well build all of my tools in three.js instead of After Effects.
Big Foundation-1 Update
V-1.2 is out!
Not only does it do infinite One-Shot samples but it also opens the doors to true prompt-based sound design: independently controlling instrument identity + timbre across playable text-to-synths!
I also have a deep dive on how I did it!
👇
Struggling to navigate the demented online culture during "ai fall", and its polarizing black-and-white thinking, following a summer slaughtered by burned out, overpromised, oversubscribed hype exposure? Try the 3-era™ meme!
Hi, are you a computer musician using librosa, numpy, and torch as your instrument? Congratulations, you're no longer doing AI 🧠 you're just engineering! ⚙️🛠️🔧
We delight to inform you AI has moved on. With the dadaist power of absurdity invested in us, we hereby relieve you of this burden. The term "AI" has been officially excised from the description of your practice.
*## Countering the lobotomizing hype-cycle meme of "AI" w/ the "3-era" meme ##*
Since 1955, the meme "AI" was a future capability, a moving target 🎯 (as s soon as a capability is achieved, it's no longer "AI" it's just engineering). The name is designed as a vehicle for fundraising (and was more successful over its alternatives like "cybernetics"). By nature of a moving target, the meme's behavior has a predictable structure: attract attention (talent, mindshare, funding, customers), over-promise, and under-deliver. Hence we have seasons of ai winters & boom-bust cycles.
However, @karpathy's three paradigms of software engineering are not moving targets, nor economic cycles, they're just capabilities.
Software 1.0: Coding - Understand the problem, define the rules, computer automates the solution.
Software 2.0: ML - Understand the problem, define the solution, computer automates the rule-writing.
Software 3.0: Vibecoding - Describe the problem in english, computer automates the computer
Computer music creative coders, confused about how to differentiate themselves, can instead think of their practice in these terms:
1.0 → livecoders using strudel, max/msp, bytebeats; musicians coding procedurally
2.0 → @dadabots, ISMIR, magenta, neural synthesis, librosa, numpy, torch; musicians doing machine learning, training models, coding in python
3.0 → suno, udio, gemini lyria, making music by describing it in english
Or computer artists in general:
1.0 → Tools - You operate the software.
2.0 → Learned tools - The software learns patterns.
3.0 → Agents - The software participates in the creative process.
The era of 2.0 was 2012-2022 and was underground unless you were orbiting the hacker or academic community and had GPUs. Though the term "AI" was abused by over-promised product marketing in this decade, it never really hit a winter. Whereas 3.0 (prompting, vibecoding) started in 2023, reached mass awareness and buy-in at an unprecedented scale, and its over-optimism is feeling like burnout.
*## Computer Music Software 2.0 ##*
Are librosa, numpy, & torch your instrument? Run your own GPU? Like to play with matrix math? The meme of "AI" is so strong it's easy to think you're swept up in it. But nope. You're no longer AI. The moving target has passed. You're just engineering in era 2.0.
New eras don't replace old, they continue to develop in tandem, at their own pace.
Awareness of the creative practice:
3.0 → Popular. 100 million users have tried making music with prompt apps. It was shoved into people's faces with ads.
2.0 → Has yet to hit much awareness as a music practice. Still underground. Gets confused for 3.0.
1.0 → Recently gained popularity with strudel blowing up last year (thanks dj dave & switch angel!). Still very few musicians are doing it, or have learned to code, or have followed in the statistical/procedural footsteps of Brian Eno or Dennis Mårtensson.
2.0 was inaccessible to most musicians during its era, has not gained very much popularity yet, but continues to become more accessible as
- Everything gets faster
- Costs go down
- Matrix math in regular computers / laptops / phones gets faster (e.g. CPU SIMD instructions, apple sillicon)
- Algorithms make breakthroughs in speed (diffusion, quantization)
- More open models front-load the expensive parts (e.g. neural codecs)
@dadabots was born in 2012, experienced fully the era of 2.0 from the start, was partly responsible for it, and made a music practice out of it. We fell in love with ML. Music that is based on statistics & linear algebra felt even more extreme & we were even more fascinated by it than purely regular DSP and procedural coding. (Watch our documentary PIZZAFIRE on our website to hear the full story.)
Do we care THAT MUCH to chase the moving target that is "AI"? Well, the scifi stuff is cool. And agents do funny shit. BUT let's pause for a second. We actually just really love 2.0, because it breaks out brains. We love neural synthesis for itself.
2.0 isn't done yet.
It's this super crazy giga playdough that's still vastly under-understood. ⚙️🛠️🔧