←
Dimitris Bertsimas53 posts
Dimitris Bertsimas avatar

Dimitris Bertsimas

@dbertsim

MIT professor, analytics, optimizer, Machine Learner, entrepreneur, philatelist

◷ Joined August 2017▧ 1 Photos and videos
arXiv math.OC Optimization and Control avatar
⟳ Dimitris Bertsimas retweeted
arXiv math.OC Optimization and Control@mathOCb·6 Nov 2023

Dimitris Bertsimas, Georgios Margaritis: Global Optimization: A Machine Learning Approach arxiv.org/abs/2311.01742 arxiv.org/pdf/2311.01742

Wes Gurnee avatar
⟳ Dimitris Bertsimas retweeted
Wes Gurnee@wesg52·3 May 2023

This paper would not have been possible without my coauthors @NeelNanda5, Matthew Pauly, Katherine Harvey, @mitroitskii, and @dbertsim or all the foundational and inspirational work from @ch402, @boknilev, and many others! Read the full paper: arxiv.org/abs/2305.01610

Wes Gurnee avatar
⟳ Dimitris Bertsimas retweeted
Wes Gurnee@wesg52·3 May 2023

Precision and recall can also be helpful guides, and remind us that it should not be assumed a model will learn to represent features in an ontology convenient or familiar to humans.

Post attachment
Wes Gurnee avatar
⟳ Dimitris Bertsimas retweeted
Wes Gurnee@wesg52·3 May 2023

While we found tons of interesting neurons with sparse probing, it requires careful follow up analysis to draw more rigorous conclusions. E.g., athlete neurons turn out to be more general sport neurons when analyzing max average activating tokens.

Post attachment
Wes Gurnee avatar
⟳ Dimitris Bertsimas retweeted
Wes Gurnee@wesg52·3 May 2023

What happens with scale? We find representational sparsity increases on average, but different features obey different scaling dynamics. In particular, quantization and neuron splitting: features both emerge and split into finer grained features.

Post attachment
Wes Gurnee avatar
⟳ Dimitris Bertsimas retweeted
Wes Gurnee@wesg52·3 May 2023

Early layers seem to use sparse combinations of neurons to represent many features in superposition. That is, using the activations of multiple polysemantic neurons to boost the signal of the true feature over all interfering features (here “social security” vs. adjacent bigrams)

Post attachment
Wes Gurnee avatar
⟳ Dimitris Bertsimas retweeted
Wes Gurnee@wesg52·3 May 2023

But what if there are more features than there are neurons? This results in polysemantic neurons which fire for a large set of unrelated features. Here we show a single early layer neuron which activates for a large collection of unrelated n-grams.

Post attachment
Wes Gurnee avatar
⟳ Dimitris Bertsimas retweeted
Wes Gurnee@wesg52·3 May 2023

Neural nets are often thought of as feature extractors. But what features are neurons in LLMs actually extracting? In our new paper, we leverage sparse probing to find out arxiv.org/abs/2305.01610. A 🧵:

Post attachment
Wes Gurnee avatar
⟳ Dimitris Bertsimas retweeted
Wes Gurnee@wesg52·3 May 2023

One large family of neurons we find are “context” neurons, which activate only for tokens in a particular context (French, Python code, US patent documents, etc). When deleting these neurons the loss increases in the relevant context while leaving other contexts unaffected!

Post attachment
Dimitris Bertsimas avatar
Dimitris Bertsimas@dbertsim·5 Apr 2023

As part of HIAS and together with Professor Georgios Stamou from NTUA, Greece we are offering a course on Universal AI (in English, free of charge) aicourse2023.hellenic-ias.or… on July 3-5, 2023 in Athens, Greece. Prospective participants can declare their interest in the website.

Ryan Cory-Wright avatar
⟳ Dimitris Bertsimas retweeted
Ryan Cory-Wright@RyanCoryWright·11 Dec 2022

Delighted to share that our paper "A new perspective on low-rank optimization" has just been accepted for publication by Math Programming! Valid & often strong lower bounds on low-rank problems via a generalization of the perspective reformulation from mixed-integer optimization

Excited to share a new paper with @dbertsim and Jean Pauphilet on a matrix perspective reformulation technique for strong relaxations of low-rank problems. Applications in reduced-rank regression and D-optimal experimental design: optimization-online.org/DB_H…

Ryan Cory-Wright avatar
⟳ Dimitris Bertsimas retweeted
Ryan Cory-Wright@RyanCoryWright·15 Mar 2023

📢New preprint alert! arxiv.org/abs/2303.07695 We use sampling schemes and clustering to improve the scalability of deterministic Bender's decomposition on data-driven network design problems, while maintaining optimality. w/ @dbertsim, Jean Pauphilet, and Periklis Petridis

Dimitris Bertsimas avatar
Dimitris Bertsimas@dbertsim·15 Mar 2023

The paper presents a novel holistic deep learning framework that improves accuracy, robustness, sparsity, and stability over standard deep learning models, as demonstrated by extensive experiments on both tabular and image data sets. arxiv.org/abs/2110.15829