
Could this be a direction for future model architectures? Introducing Memory Attention (MA). alphaxiv.org/abs/2609.28399

archer, pefter, Incoming PhD @cuhk

Could this be a direction for future model architectures? Introducing Memory Attention (MA). alphaxiv.org/abs/2609.28399


Hy4 flash is now open-weight, including Hy4 flash-FP8 and Hy4 flash-Base. 224B (12B active), 1M context, natively multimodal. Fast, efficient and built for everyday productivity. It's yours now. And stay tuned for Hy4 coming up next! 🤗: huggingface.co/tencent/Hy4-f… 🐙: github.com/Tencent-Hunyuan/H… 📄: hy.tencent.ai/research/hy4-f… Released under the Tencent Hy License. See full terms here: huggingface.co/tencent/Hy4-f…

太有品味了
LLM pretraining has run on single-epoch text, bits per byte and iso-FLOP curves for years. visual generation still memorizes ImageNet for tens to hundreds of epochs and pushes FID down with CFG, with no strong signal those tricks transfer to real text-to-image. new blog: 🎨 scaling generative models on GPIC, 100M permissive text-image pairs. training pixel-space JiT from 3.6e18 to 2.3e20 FLOPs: 1) held-out loss is our main metric, and compute-optimal TPP sits at about 50 image tokens per image parameter 2) EMA is not necessary: a proper learning-rate schedule beats EMA on held-out loss at every budget by 0.4% to 1.0% 3) we use CFG = 1 (only the conditional branch) for every result, and call on everyone to drop CFG: it saves compute and leaves no room to hack the metric all code, configs and sweep data are open 🧵 blog: wenhaochai.com/blogs/scaling…


v honored to see our video-index by @EnxinSong @YinooXu @shushengyang @wenhaocha1 and @thoma_gu used for Qwen 4 evaluation. video-index is a curated meta-benchmark for video understanding, with a coverage of 115 video understanding benchmarks. 😘 congrats to the @Alibaba_Qwen team improve the perf a lot! Paper: arxiv.org/abs/2610.00960
📢 Meet Qwen4-Max, your new AI teammate for work, creativity, and discovery! Qwen4-Max is built on a new MoE architecture with native vision and a 1M-token context window. Highlights: - Outperforms Qwen3.8-Max with up to ~50% fewer tokens, while average cost per task drops over 60% in our internal evaluation. - Advances real-world productivity through greater proficiency with professional tools, stronger long-horizon execution, and higher-quality deliverables. - Handles real tasks, including 3D modeling, game development, manufacturing, finance, law, healthcare, and frontier scientific discovery. - $1.5/M tokens for input, $4.5/M tokens for output, and $0.075/M tokens for implicit caching — all lower than Qwen3.8-Max, with implicit caching down by 70%. The API is live on QwenCloud. You can also try Qwen4-Max on Qwen Studio, QwenWork and Qoder. Start building today! 🚀 - Blog: qwen.ai/blog?id=qwen4-max - API: qwencloud.com/models/qwen4-m… - Token Plan: qwencloud.com/pricing/token-… More to come! The open weights of smaller Qwen4 models will be released. Stay tuned. 👀

Data is all you need
What I saw of the AI data business while job hunting for post-training roles, and why it keeps booming even though it seems to have no moat.



开源只是一种选择。
越来越觉得,AI 时代,个人开发者做开源真的没什么意义了。 花了大量时间和精力,从零开始做一个自己喜欢的东西。没有报酬,没有赞助,所有的开发、维护和试错成本都由自己承担。 等到项目有了一定价值,别人就可以直接拿去,整合进自己的产品,继续发展自己的商业业务。 没有事先的沟通,没有商量的余地,没有相应的回报,甚至连自己曾经写下的代码,未来往什么方向发展,都不再有话语权。 最后得到的,可能只是一句公开的感谢。 我知道开源许可证允许别人这样做,也明白开源从来不意味着必须获得回报。 但我始终觉得,有权利这样做,和应该这样对待一个开源作者,是两回事。 曾经以为开源是分享、协作和共同创造,现在却越来越觉得,个人开发者的热情和劳动,似乎成了一种可以被理所当然索取的免费资源。 挺讽刺的。 以后还是多为自己做点东西吧。

frontier
Training with fewer bits is faster, but quantization error can interfere with learning. Today on Connectionism, we detail how we trained Inkling 2 Small with a native 4-bit forward pass from the first step of pretraining through the final RL rollout. thinkingmachines.ai/blog/end…

🚀 The full DeepSeek App experience, now via API Search, files, links, safety, and voice. The capabilities behind the DeepSeek App are now available through the DeepSeek API. Enable built-in features with a few parameters, or call standalone APIs such as OCR, ASR, and TTS. Build richer conversations or recreate the full DeepSeek App in your own application. Try our open-source frontend demo, with support for importing DeepSeek conversation history. 🔗 Demo: cdn.deepseek.com/chat-demo 📦 Code: github.com/deepseek-ai/chat-… 📖 Details: api-docs.deepseek.com/guides… 1/4

这是所有国模不得不面对的问题,生死压力下蒸馏只能保证不掉队,但水下差距越来越大(算力,数据,认知),不过我还是相信国模能找到自己的出路。
A big Chinese lab reached out to me asking for help distilling Fable into their model. I said no. I told them to use the millions of books that have no copyright instead. Offered to help pick them and to help train the model a different way. They only wanted distillation, which tells you a lot about the mindset there. I've worked deep in this field and distillation is real. I wasn't convinced at one point. What I saw changed my mind completely. I'm the biggest supporter of open source. I'm for releasing weights and for the freedom to research and build, because humanity only grows when knowledge gets shared. But let's be logical. Nobody will convince me that labs shipping a model every month or less, each one a big leap, now can't ship a better one after three months. Kimi K3.1 Pro isn't a big leap over Kimi K3 and it's been about 100 days. DeepSeek still can't put out a respectable Pro model. Zai used to release their strongest model in under two months. It's been three with nothing. All I hear about Qwen 4 Max is that it's a letdown. Kimi K3.1 Pro and DeepSeek Flash both show traces of Astra distillation. The leak showed Moonshot used it, and so did DeepSeek. If this isn't the effect of Anthropic blocking distillation then what is it? Chinese labs need to get serious about using the talented researchers they have and give their models an identity. Right now their models at their best are a cheaper alternative. I want them out of that box. I want to see what makes them different. Anthropic has crossed lines too, no argument there. They destroyed millions of books, human knowledge that was in people's hands, and kept it for themselves. I'll never forgive them for that. But Chinese labs rely on distillation more than they rely on better data. I didn't work with them. Transparency and honesty matter to me, like you've known since I started here.


国模还在预告Fable5.2已经来了,没发的已经可以开启新一轮蒸馏了,无限延期。
Introducing Claude Fable 5.2 and Claude Mythos 5.2, designed to tackle the most daunting technical challenges. They’re capable collaborators on ambitious projects in many fields, and show improvements in areas like coding, reasoning, and vision.

