dear 8gb, 12gb, 16gb and 24gb vram owners,
i know people in the local ai space might discourage you, saying don't use q4 and q4 kv, it trades away quality and sucks and all that, but hear me out loud, i'm going to be very clear
the ones saying that own datacenter grade chips and can afford full precision, and as a small gpu owner you don't have a choice
i'm out here in the field running everything from 8gb cards to 256gb of unified memory, and i know what i'm saying
don't get sold, try it and find out, it's not that bad at all
BREAKING: offrail 27b uncensored just crossed 18k downloads last month on huggingface 🚨
if you own an 8gb gpu and want an uncensored model that refuses nothing so you can do crazy shit, this is the way to go, a 27b that fits on your card and never stops to lecture you
> 18,830 downloads last month
> 21,218 all time
> fits 8gb cards at 48k context
> 50.8 tok/s on a 3060 12gb
> 0 refusals
i packaged it for everyone who wants to taste what an uncensored model really feels like, so pull it and have it grind
if you own an 8gb, 12gb or 16gb card this is your golden moment to put it to work on the things frontier models like opus 5.5 refuse
weights in the reply anon
the sglang summit news i keep coming back to is who signed on
nvidia, amd, intel, google, meta, thinking machines lab, cloudflare, modal and radixark all joined the new sglang consortium to help shape the roadmap, and chip rivals don't usually back the same project
same day at the summit:
> sglang v1.0, faster inference
> aws trainium support, in preview
> radixark making it faster on intel
> miles v0.2, rl on more hardware
i run models on 2x dgx sparks and an amd strix halo on my desk, and the engine decides if a new model runs there on day one or waits for a port, so having all of them on one open engine is the part i care about
2,000+ people. Two days. The first SGLang Summit.
Together, we announced SGLang v1.0, released Miles v0.2, and launched the SGLang Consortium with nine founding members: @nvidia, @AMD, @intel, @Google, @Meta, @thinkymachines, @Cloudflare, @modal, and @radixark.
SGLang is now used by 332 organizations. At Meta, it powers production inference for products serving billions of users. Teams from Modal, Cloudflare, LinkedIn, and others shared their production lessons.
Across 90+ technical talks and six hands-on workshops, the community explored what comes next for inference, reinforcement learning, and AI agents.
To everyone building, deploying, and contributing: thank you for making this possible.
Explore the announcements, engineering lessons, and moments from SGLang Summit 2026 in the link below.
hot take, a lab launching at 200b+ is building for leaderboards and api providers, and the people running open weights at home are on a 3090 or less
i run deepseek v4.1 flash at 2.80 bpw on 2x dgx sparks with my own kernels and i still say it, 93% of steam has less than the 24gb vram a 30b dense at q4 needs
i am so so so tired of new open weight labs dropping huge models over 100b+ and completely ignoring the 12b, 30b and 60b dense category
> 12b dense at q4 for 8-12gb cards
> 30b dense at q4 for one 24gb card
> 60b dense at q4 for 2x 3090s
if you're an aspiring open weight lab and want to know what the local ai community wants, i'm speaking on behalf of them
local ai for everyone!
nvidia or amd, linux or windows, any card with 8gb or more, just one line installs it all. rtx or rx, both get a crown here
stop renting your thinking
every prompt you type into a cloud chatbot lands on someone else's server, under their rules, at their price, and they can change both whenever they want. the model you think with can live on your own vram, on the gaming gpu you might already own
what your vram gets you
> 8gb, chat with up to 64k context, rtx 2060 super to 2080 ti, rtx 3050 to 4060 ti 8gb, rtx 5050 and 5060, rx 6600 to 7600 and 9060 xt 8gb
> 12gb, agentic coding with 131k context and up, rtx 3060 12gb to 4070 ti, rtx 5070, rx 6700 xt to 7700 xt
> 16gb and up, vision and the full 262k, rtx 3090 and 4060 ti 16gb to 4090, rtx 5060 ti 16gb to 5090, rx 6800 to 7900 xtx and 9070 xt
what it does on real cards
> rtx 2070 super, about $200 used, 52.6 tok/s
> rtx 3060 12gb, about $300 used, 50.1 tok/s
> rtx 5060 ti 16gb, about $600 used, 67.3 tok/s
owners are already running it and posting their lines, a 4070, a 5080, a 3000 ada laptop, two 5060 tis on windows, one of them on a 2018 i5 that wrote a working arma 3 mission. radeon is in beta and no desktop rx has posted a speed yet, be the first
what's stopping you, run the line tonight, post your speed check, your card becomes the next row for the next owner
now you can run local ai on a $200 gpu
rtx 20 owners, you're in. one line now runs local ai on every rtx from the 20 series to the 50 series and amd radeon with 8gb or more, sm75 to sm120, windows or linux
> windows, paste in powershell:
irm raw.githubusercontent.com/su… | iex
> linux, paste in terminal:
curl -fsSL raw.githubusercontent.com/su… | bash
what your vram gets you
> 8gb, chat, up to 64k context, 52.6 tok/s on an rtx 2070 super, about $200 used
> 12gb, agentic coding, 131k context, 50.1 tok/s on an rtx 3060 12gb
> 16gb and up, vision, the full 262k, 67.3 tok/s on an rtx 5060 ti 16gb
rtx 20 on windows and radeon are still in beta, run the speed check once and reply with your line, every card that posts becomes a row for the next owner
the record runs, one synthetic patient born 1950, 76 years and 418 visits, no real person
one 128k prompt got 7/7 allergies, 8/10 disorders and 2/5 current drugs, hermes agent through the full record got 7/7, 12/12 and 2/5
both runs missed the 3 drugs that get renewed every 2 weeks, so the medication list is the one i'd check by hand
speed on the sparks as the record grows, from 1k to 128k of record writing drops from 54.1 to 45.7 tok/s and the read before the first token climbs to 61 s
one week of running ling 3.0 flash sante on my 2x dgx sparks and the verdict is here now
first the benchmarks from its own model card, 500 questions each at the same settings
> medqa 94.6%, card 94.5
> medmcqa 82.4%, card 81.1
> medxpertqa 45.0%, card 50.5
two of three land right on the card, medxpertqa is the hardest set at 10 options a question and it came in 5.5 under, more than 500 questions can explain
my verdict, it's a legit medical model you can run at home on two little boxes, it aces the usmle style questions and reads a whole patient record, and it still needs a human checking the medication list
speed, the record runs and every link are below
watch anon! this is a medical model and it's dancing on my 2x dgx sparks, and it's actually good
hermes agent is driving ling 3.0 flash sante, yolo mode on, nobody approving a single tool call
it read the record in 8 chunks, grepped it, wrote and ran 4 python scripts over it, compacted its own context 3 times when it filled up and came back with the answer in 7m 54s, the video is that run sped up 12x
> 12/12 active disorders with dates
> 7/7 allergies with every reaction
> 2/5 current drugs
> 56.1 tok/s at 31k of context
> 44.0 tok/s at 97k of context
i've been running this model for almost a week now on actual health data and on synthetic records, and i'm seeing it as a legit medical model
what a time to be alive and to own hardware, a 124b medical model runs on two little boxes on my desk, it reads health records on my own gpus and none of it leaves the house, and i can have it go through anything and do crazy stuff with it, most of which i can't even share here lol
the evals finished tonight, medqa, medmcqa and medxpertqa at the card's settings, verdict incoming tomorrow stay tuned