←
Papers with Code455 posts
Papers with Code avatar

Papers with Code

@paperswithcode

Our mission is to organize science by converting information into useful knowledge.

◷ Joined December 2018▧ 236 Photos and videos
Papers with Code avatar
Papers with Code@paperswithcode·17 Nov 2022

Thank you everyone for trying the Galactica model demo. We appreciate the feedback we have received so far from the community, and have paused the demo for now. Our models are available for researchers who want to learn more about the work and reproduce results in the paper.

Patrick Mineault avatar
⟳ Papers with Code retweeted
Patrick Mineault@patrickmineault·16 Nov 2022

Galactica is basically GPT-3 for science. It can write whitepapers, reviews, wikipedia pages and code. It knows how to cite and how to write equations. It's kind of big deal 1/ 🧵

🪐 Introducing Galactica. A large language model for science. Can summarize academic literature, solve math problems, generate Wiki articles, write scientific code, annotate molecules and proteins, and more. Explore and get weights: galactica.org

0:44
Jim Fan avatar
⟳ Papers with Code retweeted
Jim Fan@DrJimFan·16 Nov 2022

Today a 120B model called “Galactica” is open-sourced by @paperswithcode. It’s capable of writing math notations, citations, code, chemical formula, DNA, etc. Here’s why I think Galactica is a huge milestone in open foundation models, scientific automation, and responsible AI: 🧵

Post attachment
Sergei Kalinin avatar
⟳ Papers with Code retweeted
Sergei Kalinin@Sergei_Imaging·15 Nov 2022

The new language model for science galactica.org. Upon few quick tries, it seems to generate professional text in the areas I am familiar with. And 7 years ago we were *joking* about ML writing papers!

Jim Fan avatar
⟳ Papers with Code retweeted
Jim Fan@DrJimFan·15 Nov 2022

Haven’t been so excited by a text LM for a long time! And it’s all open! A true gift to science. 🎁

🪐 Introducing Galactica. A large language model for science. Can summarize academic literature, solve math problems, generate Wiki articles, write scientific code, annotate molecules and proteins, and more. Explore and get weights: galactica.org

0:44
Papers with Code avatar
Papers with Code@paperswithcode·15 Nov 2022

This is just the first step on our mission to organize science. And there is a lot more work to be done. We look forward to seeing what the open ML community builds with the model.

Papers with Code avatar
Papers with Code@paperswithcode·15 Nov 2022

Despite not being trained on a general corpus, Galactica outperforms BLOOM and OPT-175B on BIG-bench. Galactica is also significantly less toxic than other language models based on evaluations.

Post attachmentPost attachment
Papers with Code avatar
Papers with Code@paperswithcode·15 Nov 2022

We train for over four epochs and experience improving performance with use of repeated tokens. For the largest 120B model, we trained for four epochs without overfitting.

Post attachment
Papers with Code avatar
Papers with Code@paperswithcode·15 Nov 2022

Galactica performs well on reasoning, outperforming Chinchilla on mathematical MMLU by 41.3% to 35.7%, and PaLM 540B on MATH with a score of 20.4% versus 8.8%.

Post attachmentPost attachment
Papers with Code avatar
Papers with Code@paperswithcode·15 Nov 2022

We release our initial paper below. We train on a large scientific corpus of papers, reference material, knowledge bases and many other sources. Includes scientific text and also scientific modalities such as proteins, compounds and more. galactica.org/paper.pdf

Post attachmentPost attachment
Papers with Code avatar
Papers with Code@paperswithcode·15 Nov 2022

🪐 Introducing Galactica. A large language model for science. Can summarize academic literature, solve math problems, generate Wiki articles, write scientific code, annotate molecules and proteins, and more. Explore and get weights: galactica.org

0:44
Papers with Code avatar
Papers with Code@paperswithcode·12 Oct 2022

We have explored some of the latest progress, architectural improvements, and emerging new techniques for long-range modeling. We'll continue to keep track of the progress on long-range modeling and LRA. More threads like this coming soon! Follow @paperswithcode for more. 10/10

Papers with Code avatar
Papers with Code@paperswithcode·12 Oct 2022

Besides transformers, other types of models have been tested on LRA. Some of the top performing models are attained by S4 variants which are based on state space models. A recent, improved S4 variant (Liquid-S4) attained competitive results with Mega (current SoTA). 9/10