
In the end, I think agents will stream output actions continously. Think @thinkymachines interaction models but not only streaming speech and vision, but also streaming output control actions. This will also be end-to-end trained in the model weights, but I wonder how far you could get today by stitching something together with @livekit etc., with computer use models outputting high level actions and faster models outputting the micro-steps. This is kind of how we've done things in unity [github.com/unifyai/unity], but for now we only stream the speech and vision at the high-frequency conversational layer (not yet actions). In the end more of the harness will move to end-to-end training, and harnesses are a point-in-time solution to what will inevitably become an end-to-end trained system with latent representations at the boundaries instead of text etc.















