
For full technical details + compliance Datasheet see our preprint @ arxiv.org/abs/2510.13996 As for German-specific models trained on this data... stay tuned 👀

Research group working the fields of Information Retrieval, Natural Language Processing, Data Mining, Machine Learning, and Artificial Intelligence.

For full technical details + compliance Datasheet see our preprint @ arxiv.org/abs/2510.13996 As for German-specific models trained on this data... stay tuned 👀

The data spans 7 text domains: 🌐 Web: Wikis, GitHub, social media 💬 Political: Parliamentary proc., speeches ⚖️ Legal: Court decisions, law 📰 News: Newspaper archives 🏦 Economics: Public tenders 📚 Cultural: Heritage collections 🔬 Scientific: Papers, books, journals

This means: ✅ Every document has verifiable usage rights (min. CC-BY-SA 4.0 + allows commercial use) ✅ Full institutional provenance for reduced compliance risks ✅ Systematic PII removal + quality filtering, ready for training ✅ Rich metadata for downstream customization

The current problem: training data sourced from Web crawls, which give you scale but unclear licensing. This blocks models from commercial deployment and research. We took a different path: systematically collecting text from 41 institutional sources with explicit open licenses.

We just released "German Commons", the largest openly-licensed German text dataset for LLM training: 154B tokens with clear usage rights for research and commercial use. huggingface.co/datasets/cora…

Congratulations to the authors @H1iReimer, @maik_froebe, @bennostein, @martinpotthast, @matthias_hagen from @UniJena, @bauhaus_uni, @uni_kassel, @Hessian_AI, @Sca_DS!

Honored to win the ICTIR Best Paper Honorable Mention Award for "Axioms for Retrieval-Augmented Generation"! Our new axioms are integrated with ir_axioms: github.com/webis-de/ir_axiom… Nice to see axiomatic IR gaining momentum.

Come join us at the poster session at ICTIR 2025 to discuss: - Axioms for Retrieval-Augmented Generation webis.de/publications.html#m… - Learning Effective Representations for Retrieval Using Self-Distillation with Adaptive Relevance Margins webis.de/publications.html#g…

Thrilled to announce that @MattiWiegmann has successfully defended his PhD! 🎉🧑🎓 Huge congratulations on this incredible achievement! #PhDDefense #AcademicMilestone

Congrats to the authors @LukasGienapp, Tim Hagen, @maik_froebe, @matthias_hagen @bennostein, @martinpotthast and @hscells – from @uni_kassel, @Hessian_AI, @Sca_DS, @uni_tue, @UniJena & @bauhaus_uni

Happy to share that our paper "The Viability of Crowdsourcing for RAG Evaluation" received the Best Paper Honourable Mention at #SIGIR2025! Very grateful to the community for recognizing our work on improving RAG evaluation. 📄 webis.de/publications.html#g…

Do not forget to participate in the #TREC2025 Tip-of-the-Tongue (ToT) Track :) The corpus and baselines (with run files) are now available and easily accessible via the ir_datasets API and the HuggingFace Datasets API. More details are available at: trec-tot.github.io/guideline…

Credit & thanks to the author team @LukasGienapp @DeckersNiklas @martinpotthast @hscells 📄 Preprint: arxiv.org/abs/2407.21515 💻 Code: github.com/webis-de/adaptive…

Results on BEIR demonstrate that our method matches teacher distillation effectiveness, while using only 13.5% of the data and achieving 3-15x training speedup. This makes effective bi-encoder training more accessible, especially for low-resource settings.

The key idea: we can use the similarity predicted by the encoder itself between positive and negative documents to scale a traditional margin loss. This performs implicit hard negative mining and is hyperparameter-free.

Our paper on self-distillation for training bi-encoders got accepted at #ICTIR2025! By exploiting pretrained encoder capabilities, our approach eliminates expensive teacher models and batch sampling while maintaining the same effectiveness.