←
Killy2,351 posts
Killy avatar

Killy

@net_termina

Hard Work and Guts

◷ Joined March 2025▧ 319 Photos and videos
Killy avatar
Killy@net_termina·3h

Farewell post for this build before I tear it all down for next project. Served me well, except that one time it leaked and flooded all 6000s

Post attachment
Killy avatar
Killy@net_termina·5h

Be very careful with detachable pcie portion when disassembling RTX 6000. Rest of PCB are built like a tank but one wrong move here will be disastrous. Even pressure, slow and smooth does it

Post attachment
Killy avatar
Killy@net_termina·22h

Edge0-145B driving 6 other models at 2-3 streams each, thinking on, 64~128k context per, writing code from scratch. Was trying to do 40 streams but Edge0 will not delegate more

0:13
Killy avatar
Killy@net_termina·Nov 17

Thanks for the correction, was looking at bestbuy price of $9k but Newegg sell it closer to $7k, making it less than $3k difference.

Killy avatar
Killy@net_termina·Nov 17

5090 and 4500 both have 32GB VRAM, can be purchased new with warranty, except 5090 has x2 bandwidth. Is it worth paying extra $5k for? For running dense models that can leverage all bandwidth, difference is significant. However, for running moe model using strata, gap closes quite a bit since CPU/RAM comes into equation and bandwidth isn’t as important. If you’re shopping for 32GB GPU to run strata or similar solution, you can buy two 4500 at the price of single 5090. If you really need snappy 27B, definitely go with 5090

Post attachment
Killy avatar
Killy@net_termina·Nov 17

Use case #2, using multiple models/quants for long repo work is very effective once you figure out which models are reliable for specific task. Also great benching tool, can easily compare results live and have another model analyze so I can understand how/why for evaluation

0:05
Killy avatar
Killy@net_termina·Nov 16

I believe cloud and local AI will end up supplementing each other instead of competing. Tasks that frontier models are allowed to do are simply unmatched, but local models are more capable and smaller than ever with full privacy and agency. Majority of use cases, including ones cloud AI doesn’t allow, can be performed on local models with cloud AI acting as a referee as needed. This separation saves tokens, privacy, and full control over your affairs It’ll also show where a model is strong or weak, and exactly why so you can make informed decision on what to run for your specific use cases outside of typical bench numbers. Imo, this is the best potential outcome for how AI is leveraged in the future

Killy avatar
Killy@net_termina·Nov 16

When my fleet of 7-10 models finish their own research analysis, it can be cross-checked then curated into a report It can run on a schedule as well, so you have multiple curated reports on any given topics ready to go when you need it Uncensored models are VERY useful here

0:12
Killy avatar
Killy@net_termina·Nov 16

Just occurred to me that I’ve never once heard any IT disaster story from Asia. Is bringing down a huge chunk of your company western specific phenomenon or its just that Asian IT workers don’t survive to tell the tale? Joking aside, do share if you got em

Killy avatar
Killy@net_termina·Nov 16

Just wanted to clarify, each model are doing their own individual research and coming up with its own analysis in this example. (I can just tell GLM to do it, or GLM+Nemotron, or entire group) Later, I can have a larger model check for accuracy, curate and make a master report if needed. Its basically having 7~10 prompts open at the same time to different models, except theres an option to make them work together and debate something between them. This is very helpful for more technical work and for evaluating smaller models, which ill share in coming days

Killy avatar
Killy@net_termina·Nov 15

There are simply not enough use cases being shared in local AI community, so let's see if we can change that I have around 7~10 different models loaded up at any given time, into an environment I built called NetSphere. It allows me to give tasks to all/group/individually, make them cross-check the work, or have a leading model delegate a large task to smaller models fit for the job. For this post, I'll show my research use case. I give my models a topic, and have them create a tree chart that breaks it down to relevant sections with sources being pulled from web and local vault. Once its been cross-check by the group for accuracy, I can generate deliverables with visuals created from my 6000s dedicated for comfyUI work. This method also makes for a good model bench, since it allows me to clearly see that certain 9B model's research was nearly as good as GLM 5.3/QwenFN that gives me confidence to delegate similar work to it. I'll start sharing more use cases from now, and I encourage yall to do the same

0:11
0:12
Killy avatar
Killy@net_termina·Nov 15

The use of engram tables makes V41F/QwenFN significantly more accessible, but at what cost? Engram is keyed by last 2,3,4 tokens together, then resulting phase picks the rows from the engram table. V41F reads 48 rows for a token, out of 768 million that requires almost no computing that makes it easy to offload to RAM A model needs to be trained specifically for engram table, cannot drop a table on top of existing models. The cost of this architecture is less clear, Qwen's test showed 2-6 point gain and my own test showed near identical results but needed more tokens for same task. As of now, it appears the cost of engram table vs experts is small enough where any potential loss is negligible compared to making large models easily accessible for more users

0:20
Killy avatar
Killy@net_termina·Nov 15

Popular moe architecture compared, it’s interesting to see how each priorities its expert pools and attention to meet their design. Qwen has more small experts but picks more to make up for small capacity, GLM has more dense expert picks, K3 does something completely different by squeezing state to half before expert picks to reduce cost. Using engram table is such a brilliant architecture, useful in so many ways so I would love to see more models adopting it for future model training

0:24
Killy avatar
Killy@net_termina·Nov 14

I thought I had it bad when I accidentally brought down huge chunk of Accenture’s network 3 days into my job, but whoever brought down the entire US air traffic… it’s so bad that it’s kind of impressive

Killy avatar
Killy@net_termina·Nov 14

After all these years online, I’ve been completely desensitized to anything I can possibly come across, but when Claude tells me my GPU is ‘wedged’, my heart stops and I literally freeze in my tracks