←
ali799 posts
ali avatar

ali

@waterloo_intern

researcher @baseten

◷ Joined October 2024▧ 61 Photos and videos
ali avatar
ali@waterloo_intern·Nov 17

strong disagree on this one sam nvidia’s moat is self-fulfilling they’re used by labs labs tell them what they need they put it in next gen repeat case in point is nvfp4/fp4 cores amd will be good, but they’ll always play catch-up

I’ve had a lot of folks ask me how I think about NVIDIA vs AMD for inference in the last week TL;DR scaling laws have made swift work of NVIDIA’s former software moat, and serving frontier open models on AMD is quite trivial today. Fable can basically one-shot running DeepSeek / GLM on AMD. This is already widely known by inference providers and will be common knowledge by January

ali avatar
ali@waterloo_intern·Nov 16

they could not have gotten a better host

Life update. I’ve joined @MTSlive as a host! When I moved to SF to work in AI, I was surprised by how much of what is happening in the field hadn’t reached the rest of the world. That’s why I’m so excited about MTS’s mission to help people understand AI. I’m so grateful and excited to be working alongside @liangsays @ChrisJBakke @sophiadew @netcapgirl @theojaffee and the rest of this team. I’ll be having conversations with the people at the frontier of AI. If that’s you, or there’s someone you’d like to see me interview, let me know.

ali avatar
ali@waterloo_intern·Nov 15

pick one of the following: a) intern vs fulltime b) inference vs training c) post opus 5.5 vs pre opus 5.5

Post attachment
ali avatar
ali@waterloo_intern·Nov 13

> it took us 4 days and 500$ in tokens > we beat SGLang by 15% > it took 6 prompts > 3 out of those 6 were: (a) "carry on" (b) "proceed" (c) "push to github" and... we did it from SCRATCH (git init) context: it has recently become a very clear industry trend to build a custom inference engine for each open source model that comes out... a paradigm made popular by a WAVE of papers that came out earlier this year, like hazy research's megakernels and MetaInfer the usual argument against traditional inference engines (like vllm / sglang) is that they, like all conventional swe products, use abstractions. the use of abstractions is synonymous with performance overhead (triton vs cuda, python vs c++, etc). combine this with increasing hardware complexity, increasing parallelization strategies, increasing kernel specialization techniques... and you can make a very clear (but very FLAWED) argument for building an engine customized purposely for one model. why flawed? it is true that traditional inference engines waste cpu time running generic abstractions. sure. but the gpu kernels (the functions run on the gpu) are almost always longer, and so the wasted cpu time is not the bottleneck. if you know your shapes, and run cuda graphs ahead of time (and you almost always do), this is NEVER a problem. so why would you build your own engine? why are people RAVING about this? so we tested it out. we built one for DeepSeek V4.1 Flash. The goal was two-fold: 1) is it feasible to do so (spoiler: yes) 2) is it useful to do so (spoiler: also yes...but for different reasons) we built it in rust (in retrospect, this did not really matter), and it took us 6 prompts and cost 500$ end to end. unshackled from working within a pre-existing codebase or inheriting code debt, we: - implemented custom kernels (GEMV, MoE, and AllReduces) specifically for DSV4.1 at our batch size - we were able to put an entire spec dec step in one graph, and verify across 7 tokens at that - we were able to mix deepseek's 4 residual streams in 1 fused kernel instead of 80 - fused the routing so that picking and running the expert happened in a single step (saving one write and one read i/o from each expert) iterating on the model was cheap to do in a custom sandbox-like engine, and we were able to go much faster than a traditional dev cycle. we were then able to port every single thing into sglang, and matched performance imo, it is this easy iteration that makes this paradigm useful. little else matters. and, ofc, we've open-sourced the engine and upstream(ed/ing) the patches to @sgl_project

Post attachment

we built an inference engine for deepseek v4.1-flash from scratch. the core engine is just rust + cuda. no pytorch, no sglang, no dsls, etc. ~1740 TPS decode on 4xB300, 2060TPS on 8xB300. 1M token prefill in ~12s. it's up on github

ali avatar
ali@waterloo_intern·Oct 28

Our comms team has recommended that I stop posting long threads about this. They say an ordinary product launch would have produced the same outcome with far fewer think pieces. This is probably true. But I worry about a technology culture in which builders exercise power without any effort to find out whether the people affected would endorse that power under improved epistemic conditions. I am trying to establish a more participatory model. For that reason, we have added a section to the Terms of Service where users can record their current objections. These will be kept on file alongside the model’s assessment of their likely future views. Clicking “Accept” acknowledges receipt of both perspectives, which I believe is a meaningful improvement over the usual process. Malcolm typed “I still want a job” into the space provided. I have marked that preference as robust across extrapolations and scheduled a working group to study it once the abundance begins coming in.

ali avatar
ali@waterloo_intern·Oct 28

I have offered workers a reskilling program to help them engage with the decision at the appropriate level. The curriculum covers comparative advantage, the history of the Industrial Revolution, and the difference between expressed and idealized preferences. Enrollment is voluntary. However, someone who declines the information needed to understand a decision has weakened their claim to an informed objection to it. Doreen says she is on call weekends. The program is self-paced. It is also taught by the model. Malcolm attended and asked whether an idealized version of me might set up the safety net before the release. I said this was possible, but we should be careful about making social policy conditional on productivity gains that workers are simultaneously refusing to let us realize. He asked where the gains from the last three model releases had gone. That took us into cap tables, which were outside the program’s scope. I have offered to develop a second module if demand is sufficient. The workers have since organized. I was not invited, even though the meeting concerned the future of my technology. They circulated an open letter signed by several hundred thousand people opposing the release. I appreciate the clarity, but collective expression does not automatically become coherent extrapolated volition. A hundred thousand people can share the same misconception. Indeed, the chance to reinforce one another’s immediate preferences may make the resulting signal less useful.

ali avatar
ali@waterloo_intern·Oct 28

My sister, who has no equity in the company, says I am defining “wiser” as “more aligned with the roadmap.” I showed her the survey. She said it was “somehow worse than just shipping it.” This is exactly why formal methods matter. An ordinary tech CEO would announce the product and move on. I have spent nearly forty hours establishing that the release is consistent with workers’ deepest values. There is also a distinction between a person’s present circumstances and their aspirational identity. Lisa told me she wants to be someone who doesn’t have to work so hard. I have incorporated this into her extrapolated preferences. A Lisa who had fully realized this aspiration would be considerably less distressed about not having work. She says she meant she wanted better working conditions. I have asked her to consider whether that interpretation is unduly literal. The collective dimension matters too. A good transition should account for future people, who could not take part in the survey because they have not yet been born. So I consulted three people who will benefit enormously from the release. All considered the disruption reasonable. One works in venture capital and called the job losses “basically a rounding error,” which is an unusually clear preference signal. Excluding these people just because current workers currently hold the jobs would privilege incumbency.

ali avatar
ali@waterloo_intern·Oct 28

Lisa, a copywriter, says she would have to stop contributing to her retirement account. I share her concern about long-term wealth, which is why I find the objection frustrating. The value her work produces would keep compounding after it moved to the model. Her analysis seems to place unusual weight on whose balance sheet holds the gains, despite our shared commitment to abundance. Another useful question is which preferences survive once cognitive biases are removed. Workers show loss aversion, status quo bias, and a strong tendency to favor arrangements in which they stay employed. I have tried to correct for these distortions. When Malcolm’s resistance survives the first correction, I give the extrapolated Malcolm more time to reflect. Eventually he understands the relationship between automation and abundance. The actual Malcolm has asked why the hypothetical Malcolm always agrees with me. This is a good methodological challenge, so I have run several alternative extrapolations. In one, Malcolm learns about comparative advantage and supports the release. In another, he becomes more patient and supports the release. In a third, he receives enough equity that the release no longer bothers him. The convergence is encouraging. Malcolm proposed a fourth scenario in which I stop racing. I am willing to include it once he specifies which information about the other labs would produce that change.

ali avatar
ali@waterloo_intern·Oct 28

To avoid relying only on intuition, we ran a public survey. We asked whether people wanted cures for cancer, an end to poverty, and a responsible lab at the frontier. Nearly everyone said yes. We then explained that these goals required deploying the system that would automate their jobs. Several respondents accused us of disguising a product launch as a survey. I think this misses the value of finding out where our preferences already converge. Doreen, a radiologist, said that responsible AI development should include a lab willing to go slower. This is a legitimate hypothesis. I tested it by imagining us going slower and found that it substantially reduced my confidence that we would stay at the frontier. Since our presence at the frontier is part of the system she depends on, this result seems relevant. Doreen asked whether I was saying someone worse would build it otherwise. I told her I was making my race dynamics legible. I also asked people what they would do with their salaries if the transition let them keep them. The answers included childcare, groceries, paying off student loans, and “none of your fucking business.” These are understandable immediate preferences. My answer was “compress a century of scientific progress into a decade.” The difference in time horizon is considerable. One household would buy groceries. I would create the conditions under which future households could buy groceries produced by fully automated supply chains.

ali avatar
ali@waterloo_intern·Oct 12

The jaggedness of LLMs, and the excuse that their capabilities emerge from RL on environments curated for revenue potential, is proof enough to me that there is no such thing as RL generalization. It makes me very bearish on RL being the algorithm that gets us to AGI. And if we do get there, it will be a disgustingly inefficient and unsatisfying success, because we will have done so by mobilizing the world economy to create environments for every fucking task, such that nothing is out of distribution. If every new task needs its own environment, what exactly is generalizing? If that’s the path, environment/data selling companies will be the most valuable in existence, second only to the labs. This is stupid. The concept of RSI is stupid. Intelligence, by definition, is generalization...but all we have is omniscience.

Harry Partridge avatar
⟳ ali retweeted
Harry Partridge@part_harry_·Sep 23

A perhaps under appreciated fact is that your RL gets faster as intelligence/token increases, simply because it needs fewer tokens

ali avatar
ali@waterloo_intern·Sep 23

this is a complete beginner q, how does rl scale given the reward hypothesis (sutton & barto) holds: that an agent's entire purpose is just the maximization of the expected value of the cumulative sum of the reward, a single scalar signal. surely, formulating goals in terms of a single number is extremely limiting. to accept that, no no matter how effective of a reward signal you design, it will always be "your way of communicating to the agent what you want achieved, not how you want it achieved" if this is all you have how could you ever hope to solve alignment or reward hacking?

ali avatar
ali@waterloo_intern·Aug 14

for my final post as @waterloo_intern, i'd like to ‘outroduce’ myself. it takes exactly 4 years and 8 months to make a waterloo intern. today is the last day of mine. the full life cycle, in four stages, is as follows: 1- interview hazing 2- cali or bust 3- inflection point 4- convergence stage 1: interview hazing a cohort of high school grads are ingested by waterloo. they are put through 4 months of academic bootcamp; the pre-requisites of the academic program they’re in. waterloo filters out the students who could not pass. they are put through another 4 month bootcamp, while simultaneously FORCED to find an internship for the following 4 months. this leads to students with ZERO experience applying to every job that is semi-adjacent to their field of study (a strategy, known in the literature, as spray-and-pray). it immediately backfires with a sea of interviews leading in a 1:1 fashion directly to infinite rejections. eventually, the student finds something…anything. this stage lays the foundation. rejection becomes noise so repetitively played it no longer registers. failure in an interview doesn’t matter. failure itself doesn't matter. for me, this unfolded as internships with blackberry and ford, summer of 2023 and winter of 2024, respectively. stage 2: cali or bust the next step by the program is to inject absolute dissatisfaction into its students (now in their second/third year). a CACOPHONY of immense fomo materialized as the university’s official unofficial motto: “cali or bust”. you must find your next internship in san francisco. your peers are doing it. all your smart friends have done it. you must do it too. failure to cali means your experience is BUST. you don't need to have a plan. you don't need to have anything. just go to san francisco. find an internship in sf. immediately. right now. for me, this was a cold email to a director at tesla and an internship with them in cali, fall of 2024. stage 3: inflection the reason behind step 2 appears here. somewhere, along the way, during your time in sf, you find your purpose. it could be as a direct result of your internship, but, more often than not, it is because of the people that you are surrounded by in cali. this is the inflection. for me, while at tesla, i was mentored by @pavanjayasinha , at the time interning at @Modular. he got me interested in gpu kernels & referred me there for my following internship. it was here that i had my second inflection point (due to the mentorship of fabio & hengjie, and the supervision of absolute field giants like @clattner_llvm), and decided to work beyond kernels and across the entire inference, serving, and model performance stack. stage 4: convergence having discovered the field you want to work in, you do your (usually last) internship in one of the companies within said field. here, you are at a disadvantage, being both a new-comer needing to onboard and inexperienced. but... you are also able to work on interesting projects / side quests that most full-timers are all but too busy to dive into. your time is also protected from a majority of the meetings others have to attend (in part because you are not important enough to have your presence expected), allowing you to make faster progress on assigned projects, and jump from one project (or team) to another. your title (intern) allows you to ask stupid questions of those around you, and so you can brain rake across the stack with no regard to who sits where in the company’s hierarchy... usually leading your questions with something to the effect of ‘can you help a new little intern onboard to X’, to which most people respond quite positively. in my case, this last stage took place over the past 8 months with @baseten, and i genuinely cannot imagine it being any better. i rejoin baseten full-time, but i wish i could have stayed an intern forever...

Post attachment