←
Nikolas V. 🦋85 posts
Nikolas V. 🦋 avatar

Nikolas V. 🦋

@NikVits

Fledgling PhD. Areas of research: Ethical AI/NLP. Interested in space, sci-fi and everything cyberpunk. He/ They.

◷ Joined March 2015▧ 5 Photos and videos
Nikolas V. 🦋 avatar
Nikolas V. 🦋@NikVits·14 Nov 2024

Surprised but also genuinely happy to be recognised as one of the many outstanding reviewers of this cycle at #EMNLP2024. It’s been a such pleasure to discuss with authors and be of help!

We're kicking off the awards session at #EMNLP2024 by announcing our (many) **Outstanding Reviewers**!

Post attachmentPost attachmentPost attachment
Zara Siddique avatar
⟳ Nikolas V. 🦋 retweeted
Zara Siddique@TheZaraSiddique·21 Sep 2024

Really pleased to say this has been accepted at #EMNLP2024 main

🚨 NEW PAPER ALERT 🚨 Introducing the GlobalBias dataset… We ask Claude 3, GPT 3.5, GPT 4o, and Llama 3 to produce character profiles based on given names from GlobalBias for 40 different gender-by-ethnicity groups. We find that all models displayed stereotypical outputs (1/4)

Post attachment
Nikolas V. 🦋 avatar
Nikolas V. 🦋@NikVits·24 Jul 2024

Overall, we present a strong methodology that furthers research into representation of minority voices throughout the modeling pipeline, which in the future, we aim to use to amplify previously marginalised group perspectives!

Nikolas V. 🦋 avatar
Nikolas V. 🦋@NikVits·24 Jul 2024

Finally, here is an example of an inter-minority voice, a combination of two minority labels, republican and highly educated. Here opinions consistently agreed or disagreed with more technical statements indicative of specialised knowledge and/or researched opinions.

Post attachment
Nikolas V. 🦋 avatar
Nikolas V. 🦋@NikVits·24 Jul 2024

Here is an example of a Democrat majority voice (the original dataset has a Democrat majority skew). Examples are pro environmental protection, while we also found numerous examples of agreement with politically charged examples.

Post attachment
Nikolas V. 🦋 avatar
Nikolas V. 🦋@NikVits·24 Jul 2024

Here is an example of a Republican voice (as Republicans were a minority in the original dataset). Clusters contained mixed opinions on global warming phenomena, while we also found patterns of disagreement with more politically charged examples.

Post attachment
Nikolas V. 🦋 avatar
Nikolas V. 🦋@NikVits·24 Jul 2024

We found automatic metrics to be bad at evaluating what voice each cluster represents… Qualitative analysis to the rescue! We found that resulting clusters captured both majority and minority voices as shown by patterns in text & annotation pairs of each cluster.

Nikolas V. 🦋 avatar
Nikolas V. 🦋@NikVits·24 Jul 2024

Our framework learns to predict each annotator’s annotations without the need to encode any annotator metadata, and clusters based on similarities in annotating behavior through similarities in hidden states. We compare our methodology to past approaches and highlight benefits.

Post attachment
Nikolas V. 🦋 avatar
Nikolas V. 🦋@NikVits·24 Jul 2024

In this paper, we provide a framework on how to dynamically uncover minority perspectives present in the data, which we call voices, by grouping based on annotating behaviour. This organic process also allows for any number of voices to be expressed, no matter how small.

Nikolas V. 🦋 avatar
Nikolas V. 🦋@NikVits·24 Jul 2024

However, both methods are problematic! The former biases the model into grouping annotators that share metadata labels but might not annotate similarly, while the latter limits the number of viewpoints able to be expressed.

Nikolas V. 🦋 avatar
Nikolas V. 🦋@NikVits·24 Jul 2024

Previous literature grouped annotators based on one of two approaches: either by encoding annotator metadata or by grouping annotators by agreement / disagreement per example and sorting them as part of either an agreeing / disagreeing majority or minority.

Nikolas V. 🦋 avatar
Nikolas V. 🦋@NikVits·24 Jul 2024

So very, very proud to share our new paper “Voices in a Crowd: Searching for Clusters of Unique Perspectives” (arXiv:2407.14259), a novel framework on how to organically find clusters of unique voices (perspectives) in datasets. 🧵 for summary, co-authors @amitkparekh_ @sinantie

Nikolas V. 🦋 avatar
Nikolas V. 🦋@NikVits·17 Jul 2024

This project was both a ridiculous amount of work and fun. The implications are also pretty massive for the field of embodied multimodal AI, give it a read!

What if your multimodal model succeeds when given nonsensical instructions? What if it even succeeds without being told explicitly what to do? Have we finally achieved AGI? We answer at least two of those questions w/ @NikVits @ale_suglia @sinantie (🧵for more)

0:38