I post the papers I find interesting. There are so many papers published these days, and I frequently miss great papers. I appreciate paper recommendations via DM, but I tend to only post papers I discover on my own to keep my list personally curated.
여기에 대해 창작자의 입장에서 반응하는 인용들이 많은데 사실 (여기에서도 암시되어 있지만) AI를 사용한 창작은 창작이 아니라 소비의 경험에 가깝다. 웹소설 독자가 AI 채팅으로 넘어간다는 건 독자에서 창작자가 되는 것이 아니라 여전히 소비자에 가까울 수 있다는 것. 그리고 이미 그 소비의 경험이 다들 생각하는 것보다는 상당히 올라와 있다.
논란 가능한 소신발언 좀 할게요
그림 그릴 줄 모르는 사람, 작곡할 줄 모르는 사람 포함해서 모든 사람이 어떤 비용 부담도 없이 완전히 자기가 원하는 창작물을 순식간에 자급자족할 수 있는 미래사회가 이상향이라고 생각합니다.
물론 그게 무조건 지금의 AI 기술로 성립돼야 한다는 뜻은 아니고요.
But maybe good long context pretraining setup and evaluations with usual good taste on pretraining like emphasis on diversity could do most of the tricks.
I think many groups are trying to do this or actually doing this (like agentic benchmarks during pretraining or Mistral report) but it would be hard in general, considering task scaling laws and long context benchmarks are still open research areas. Especially for post-training it would also depend on data/task composition for it. But I think it could be feasible to establish scaling laws for it.
FineWeb-Edu is built with trafilatura, heuristic filtering and snapshot dedup, which are now a bit previous-generation practices. I recommend checking CuraWeb for what more recent practices look like. (Though I don't know the methods of frontier labs.)
Some unpopular-but-true notes on pretraining research:
→ 𝗗𝗮𝘁𝗮 𝗿𝗲𝗽𝗲𝘁𝗶𝘁𝗶𝗼𝗻 is a very common practice.
→ 𝗢𝘃𝗲𝗿-𝘁𝗿𝗮𝗶𝗻𝗶𝗻𝗴 (~1000 TPP) is a very common practice.
→ In the long run, 𝗱𝗮𝘁𝗮-𝗲𝗳𝗳𝗶𝗰𝗶𝗲𝗻𝘁 architectures and algorithms will matter more than (currently) 𝗵𝗮𝗿𝗱𝘄𝗮𝗿𝗲-𝗳𝗿𝗶𝗲𝗻𝗱𝗹𝘆 ones.
→ 𝗙𝗶𝗻𝗲𝗪𝗲𝗯-𝗘𝗱𝘂 is a bad dataset for testing architectures; it can be misleading.
arxiv.org/abs/2611.07232
Tree-structured attention is popular now? But I am not sure whether it could support exact retrieval well enough (and potential risk of aliasing!).
Exciting implication: previous observations on sparsity of RL policy updates mainly due to ignoring master weight updates. Still it is sparse but much less.
arxiv.org/abs/2611.03142
Resolving the diminishing effectiveness of depth scaling. Cross-layer connectivity through KV sharing, and this improves both efficiency and performance. Interesting result!
arxiv.org/abs/2611.02270
Task scaling law. As small models are not able to show meaningful scores in benchmarks, give them privileged information, and formulate scaling law with strength of information.