←
Institutional Data Initiative35 posts
Institutional Data Initiative avatar

Institutional Data Initiative

@instdin

A research center at Harvard working to strengthen society’s connection to knowledge by advancing our access to and understanding of the data that shapes AI.

◷ Joined August 2024▧ 10 Photos and videos
Institutional Data Initiative avatar
Institutional Data Initiative@instdin·Sep 8

The Institutional Data Initiative at Harvard (@instdin) works with knowledge institutions to increase the availability, diversity, and responsible use of training data for AI. Reach out and join us.

Institutional Data Initiative avatar
Institutional Data Initiative@instdin·Sep 8

Together, these projects establish Institutional Books as an ongoing resource that can grow through continued research and community use. The open-source pipelines can be adapted to new collections, models, languages, and questions: institutional.org/posts/exte…

Institutional Data Initiative avatar
Institutional Data Initiative@instdin·Sep 8

Visual Elements extends the dataset to images embedded in scanned pages, producing 22.6M visual elements – including illustrations, photographs, charts, musical notation, and decorative materials – linked to their source volumes and pages.

Institutional Data Initiative avatar
Institutional Data Initiative@instdin·Sep 8

Enriched Text contains 217B tokens from 983K @HarvardLibrary volumes, organized into 1.39B annotated paragraphs across 250 languages. Researchers can use the dataset to analyze language, repeated passages, line breaks, and paragraph predictability.

Institutional Data Initiative avatar
Institutional Data Initiative@instdin·Sep 8

Institutional Books is expanding. The Institutional Data Initiative (@instdin) at @Harvard_Law Library has added paragraph-level text analysis and 22M extracted images to our work on ∼1M books, opening new possibilities for research, experimentation, and discovery. 🧵

Post attachmentPost attachment
Institutional Data Initiative avatar
Institutional Data Initiative@instdin·Sep 8

The Institutional Data Initiative at Harvard (@instdin) works with knowledge institutions to increase the availability, diversity, and responsible use of training data for AI. Reach out and join us.

Post attachmentPost attachment
Institutional Data Initiative avatar
Institutional Data Initiative@instdin·Sep 8

Together, these projects establish Institutional Books as an ongoing resource that can grow through continued research and community use. The open-source pipelines can be adapted to new collections, models, languages, and questions: institutional.org/posts/exte…

Institutional Data Initiative avatar
Institutional Data Initiative@instdin·Sep 8

Visual Elements extends the dataset to images embedded in scanned pages, producing 22.6M visual elements – including illustrations, photographs, charts, musical notation, and decorative materials – linked to their source volumes and pages.

Institutional Data Initiative avatar
Institutional Data Initiative@instdin·Sep 8

The Institutional Data Initiative at Harvard (@instdin) works with knowledge institutions to increase the availability, diversity, and responsible use of training data for AI. Reach out and join us.

Post attachmentPost attachment
Institutional Data Initiative avatar
Institutional Data Initiative@instdin·Sep 8

Together, these projects establish Institutional Books as an ongoing resource that can grow through continued research and community use. The open-source pipelines can be adapted to new collections, models, languages, and questions: institutional.org/posts/exte…

Institutional Data Initiative avatar
Institutional Data Initiative@instdin·Sep 8

Visual Elements extends the dataset to images embedded in scanned pages, producing 22.6M visual elements – including illustrations, photographs, charts, musical notation, and decorative materials – linked to their source volumes and pages.

Greg Leppert avatar
⟳ Institutional Data Initiative retweeted
Greg Leppert@leppert·Aug 13

AI enables new forms of research, but only where information is computationally accessible. Today, IDI (@instdin) and the University of Michigan Library (@UMichLibrary) are releasing a structured dataset that makes 186 years of U-M’s Board of Regents Proceedings newly accessible. Links in the thread below. Organizational decision-making is fertile ground for AI-assisted research, but board minutes often bury key decisions in narrative prose. Using reasoning models, we worked with U-M Library to extract and structure these decisions so researchers can use computational tools to study the evolution of U-M. Across nearly two centuries, we identified leadership changes, fundraising progress, building campaigns, and other strategic plans. These include the creation of university race and gender policies following the passage of the Civil Rights Act as well as a 1921 $1M alumnae drive for a “women’s building,” launched after women were barred from entering the Michigan Union without a male escort. This project is the first of many that the Institutional Data Initiative at Harvard Law School Library is undertaking to help libraries and other knowledge institutions enable computational access to their collections. If you represent a library and would like to work together, we'd love to hear from you.

Greg Leppert avatar
⟳ Institutional Data Initiative retweeted
Greg Leppert@leppert·Apr 28

Amazing work from an amazing team using @instdin’s Institutional Books data release. Their dedication to detail and accuracy is sorely missing from the vast majority of historical-data work from the AI community. Yet there’s so much work to be done and benefit to getting it right

Announcing Talkie: a new, open-weight historical LLM! We trained and finetuned a 13B model on a newly-curated dataset of only pre-1930 data. Try it below! with @AlecRad and @status_effects 🧵

7:58
Greg Leppert avatar
⟳ Institutional Data Initiative retweeted
Greg Leppert@leppert·20 Nov 2025

Even if you're not a partner library, you might be curious about what it's like to work with GRIN. Our technical report has a wealth of details: arxiv.org/abs/2511.11447

Greg Leppert avatar
⟳ Institutional Data Initiative retweeted
Greg Leppert@leppert·20 Nov 2025

When libraries participate in Google Books, Google not only scans their books, it also makes a wealth of image, OCR, and metadata available to them via the Google Return Interface (GRIN). But working with GRIN can be challenging.