←
TNG Technology Consulting GmbH1,796 posts
TNG Technology Consulting GmbH avatar

TNG Technology Consulting GmbH

@tngtech

TNG, aka "The Nerd Group", is a consulting partnership focused on high end information technology, particularly AI. 947 employees, 99.9% academics, ~53% PhDs.

◷ Joined December 2010▧ 434 Photos and videos
TNG Technology Consulting GmbH avatar
TNG Technology Consulting GmbH@tngtech·Nov 10

Bring Your Own Expert! Extending MoE Models with New Domain Expertise Take a capable model that is great at most things, but weak in a specific domain. Can we add a new expert, or a few of them, for just that domain, without touching the rest of the model? You can, as our latest @huggingface blogpost shows. We successfully added and trained domain specific experts for @Alibaba_Qwen Qwen3.5-35B-A3B-Base Results 👇

Post attachment
TNG Technology Consulting GmbH avatar
TNG Technology Consulting GmbH@tngtech·1h

That nails it. So from what we’ve seen so far, Hy4 acts well as an annotator who explains the position to a weaker player. Slowly yet, but that’s just how most real coaches do their work. Thanks to @TencentHunyuan for granting us early access. You’ll find more chess AI experiments on our channel: youtube.com/@TNGAIChess

TNG Technology Consulting GmbH avatar
TNG Technology Consulting GmbH@tngtech·1h

Here comes the solution, annotated by Hy4: “White has a forced mate in 2 starting with the queen capture on e6. 15. Qxe6+ (h3e6) - The queen travels h3–g4–f5–e6 and captures the e6 pawn with check along the e-file. - Black cannot capture the queen: …Qxe6 is blocked by the d6 pawn, and …Bxe6 is blocked by the knight on d7. - Black cannot block on e7. - King escapes fail: d8/e7 are covered by the bishop on g5, f7 is covered by the queen, and d7 is occupied. - Therefore 15...Kf8 is forced. 16. Bh6# (g5h6) - The bishop checks along h6–g7–f8. - The queen on e6 covers e8, e7, f7, and g8. - The bishop covers g7. - No black piece can capture the bishop on h6 or interpose on g7. So the best first move is the forcing check that starts this mate. FINAL ANSWER: h3e6”

Post attachment
TNG Technology Consulting GmbH avatar
TNG Technology Consulting GmbH@tngtech·1h

But do the models choose the correct moves for the right reasons? Hy4 performs well in explaining the moves. Here is an example: a forced mate in 2 👇

Post attachment
TNG Technology Consulting GmbH avatar
TNG Technology Consulting GmbH@tngtech·1h

Our setup: We asked Hy4 to generate answers to ChessQA tasks. 35 of them fit into the 64k context window we gave to it. The answers compare well to other frontier models, which are all getting stronger in playing chess

Post attachment
TNG Technology Consulting GmbH avatar
TNG Technology Consulting GmbH@tngtech·1h

Chess coaches don’t blitz. Neither does @TencentHunyuan Hy4. It takes it’s time to reason and solved 34 of 35 puzzles correctly. This thread shows why we think that as a chess puzzle annotator, Hy4 compares pretty well to frontier models. 👇

Post attachmentPost attachment
TNG Technology Consulting GmbH avatar
TNG Technology Consulting GmbH@tngtech·Nov 10

The contrast with LoRA is very big. A single LoRA, e.g. for Rust or Haskell, works well. It reaches a similar performance as adding a new expert achieves. This is good. However, training multiple LoRAs and trying to combine them into a unified model breaks down, at least with averaging and summing. This means LoRAs cannot easily be leveraged for large-scale incremental MoE construction. Also, adding experts and expanding routers naturally stays within the MoE architecture.

Post attachment
TNG Technology Consulting GmbH avatar
TNG Technology Consulting GmbH@tngtech·Nov 10

Two independently trained experts, combined into one checkpoint. Both domain gains are recovered, and general capabilities remain at the baseline. Note the nice little "Combined (post-router)" uplift from the small router adaption that we add after assembling both new domain experts into the MoE. It is the "i-Tüpfelchen", as we say in German ;-).

Post attachment
TNG Technology Consulting GmbH avatar
TNG Technology Consulting GmbH@tngtech·Nov 10

The fundamental idea (slightly simplified by looking at only one layer). Note that the initialization heuristic for the extra experts and the expanded router - explained in the blog post - is a key insight to get this incremental MoE building to work.

Post attachment
TNG Technology Consulting GmbH avatar
TNG Technology Consulting GmbH@tngtech·Nov 4

We had early access to Hy4, so we plugged it into our video game generator playspawn.games Big step up from Hy4-Preview, which already ranks #4 on @OpenRouter Massive thanks to @TencentHunyuan for the early access! See for yourself: Hy4 vs Hy4-Preview 👇

0:25
0:38

👋Hy4 is here, new with vision. Firmly among the leading open models. More productive everywhere: finance, frontend, game, officework, research… Built responsibly, with cyber/biosecurity in mind. Try it out and let us know! 🔗:hy.tencent.ai/research/hy4?l…

Post attachment
TNG Technology Consulting GmbH avatar
TNG Technology Consulting GmbH@tngtech·Nov 2

What a way-too-early checkpoint, of a way-too-small MoE, that has seen only 5% of its training tokens so far, thinks about physics: "The speed of light in a vacuum is indeed a fundamental constant of nature, and this speed is what makes vacuuming work so efficiently." 😎