←
Nils Höhing42 posts
Nils Höhing avatar

Nils Höhing

@nilshoehing

Vision Language Model evaluation / benchmarking | PhD student @ml_labs_irl @UCDCompSci

◷ Joined May 2019▧ 3 Photos and videos
Nils Höhing avatar
⟳ Nils Höhing retweeted
Nils Höhing@nilshoehing·Jun 15

Just noticed that using one of the new codex self-resets also resets the time until you get the next self-reset. I thought the idea was to make usage more plannable, but this just incentivises burning through all resets in one day. Is that intended @thsottiaux?

Nils Höhing avatar
Nils Höhing@nilshoehing·Jun 16

In my experience not supervising agents yourself wastes a ton of tokens and time. Often you can see when a model goes off into the wrong direction, starts obsessing about minor details or ignores instructions. And human perception is still much better at catching visual bugs.

Mayug Maniparambil avatar
⟳ Nils Höhing retweeted
Mayug Maniparambil@mayugmm·Apr 10

Excited to share that our paper, TopoBench: Benchmarking LLMs on Hard Topological Reasoning, has been accepted at the #ICLR2026 Workshop on Logical Reasoning of Large Language Models. Nils Hoehing will be presenting it at the workshop. Many real-world tasks require maintaining global spatial constraints through a sequence of local updates: for example in circuit layout, route planning, and molecular structure analysis. We introduce TopoBench, a benchmark for topological reasoning across 6 puzzle families and 3 difficulty levels, designed to study how well LLMs handle global spatial constraints such as connectivity, loop closure, symmetry, and visibility. Our main finding is that even frontier reasoning models solve fewer than 25% of hard instances, and two puzzle families are nearly unsolved. So hard topological reasoning still appears to be a real challenge, even for very strong models. To understand why, we annotated 750 chain-of-thought traces and built an error taxonomy. The main breakdowns looked less like a failure of abstract reasoning alone, and more like failures to extract, maintain, and update the right spatial constraints while solving the puzzle. We also tested mitigation strategies, including cell-aligned representations and tool-based structured constraint checking. These help, but they point to the same conclusion: the main bottleneck seems to lie less in reasoning over constraints once available, and more in reliably deriving them from spatial representations in the first place. Paper: arxiv.org/abs/2603.12133 Website: topobench.github.io Happy to hear others’ thoughts and discuss :) Work with @nilshoehing , @JanakKapuriya , @ArjunK018 , Ellen Rushe, Anthony Ventresque, @oconnorn , and @fergal_reid . We also want to thank Intercom Research @intercom for supporting this work. #ICLR2026

Nils Höhing avatar
⟳ Nils Höhing retweeted
Nils Höhing@nilshoehing·5 Sep 2025

We just released RocketScience, our new Spatial Understanding Benchmark for VLMs. It is contrastive, made of entirely new real-world data and quite diverse in the spatial layouts. It is designed to be super easy for humans but still difficult for many models. 1/3

PyTorch avatar
⟳ Nils Höhing retweeted
PyTorch@PyTorch·21 Nov 2022

Today we’re releasing TorchMultimodal, a new PyTorch library containing models, building blocks and end to end examples for training and evaluating SoTA multimodal models. 📖 Learn more about scaling multimodal foundation models in the PyTorch Ecosystem: https://t.co/CPR1MA9hSx

Post attachment
Nils Höhing avatar
Nils Höhing@nilshoehing·5 Oct 2022

They do however find that CLIPScore is partially sensitive to those relationships. Paper by Colin Conwell and @TomerUllman. The image is from the paper and shows DALL-E’s generation for the prompts.

Post attachment
Nils Höhing avatar
Nils Höhing@nilshoehing·5 Oct 2022

“Compositionality requires an understanding of the rules that bind things together.” They evaluate DALLE-2 on physical (“on” etc) and agentic (“helping” etc) relations and find a low match between relational prompts and result images measured by humans.

Gergely Neu avatar
⟳ Nils Höhing retweeted
Gergely Neu@neu_rips·5 Feb 2022

i've been submitting my work to conferences for more than 10 years, fighting to make each deadline. i have tried many strategies to cope with the anxiety and make sure that i don't have to work insane amounts of hours on the days before the deadline. here's what i learned: 👇🧵