-
Notifications
You must be signed in to change notification settings - Fork 62
Add ChemBench maintainer spotlight #514
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
samus-aran
merged 3 commits into
main
from
copilot/create-md-file-for-chembench-interview
Jun 25, 2026
+81
−0
Merged
Changes from 2 commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,81 @@ | ||
| --- | ||
| name: Kevin Maik Jablonka | ||
| institution: Friedrich Schiller Universität Jena | ||
| department: Chemistry | ||
| projectName: ChemBench | ||
| projectRepo: https://github.com/lamalab-org/chembench | ||
| projectWebsite: https://chembench.lamalab.org/ | ||
| maintainerProfiles: | ||
| - github: https://github.com/kjappelbaum | ||
| - orcid: https://orcid.org/0000-0003-4894-4660 | ||
| badges: ["Academic Maintainer", "Research Software Engineer"] | ||
| description: "A comprehensive benchmarking framework with over 2,700 chemistry questions that evaluates how well AI language models understand chemistry, spanning basic concepts to complex reasoning tasks." | ||
| --- | ||
|
|
||
| ## What is this project, and what does it help people do? | ||
|
|
||
| ChemBench tests how well AI language models understand chemistry. We created over 2,700 chemistry questions spanning basic concepts to complex reasoning tasks. When we tested leading AI models against human experts, we found something surprising: the best models outperformed most humans on chemistry knowledge. | ||
|
|
||
| What makes ChemBench different? First, it can process chemical notation and molecular structures, not just text. Second, it includes practical features such as visual model testing and tracking when models refuse to answer. Third, unlike typical one-off evaluations, ChemBench includes governance and maintenance protocols. | ||
|
|
||
| Our results reveal clear patterns. AI models excel at textbook knowledge but struggle with novel reasoning. They perform well on certification-style questions but falter when asked to predict complex molecular behaviour. These gaps highlight where AI training still needs improvement. | ||
|
|
||
| The ChemBench framework has since grown beyond its original purpose. We now use its core architecture to test multimodal models in other scientific domains. By mapping both AI strengths and weaknesses in chemistry, we help researchers build better models and show educators where human expertise remains essential. | ||
|
|
||
| ## What inspired you to start this project? | ||
|
|
||
| We needed ChemBench because it didn't exist. While training chemical language models, we couldn't find suitable tools to evaluate them. Existing benchmarks were not comprehensive enough to assess deep chemical understanding. | ||
|
|
||
| What started as filling a technical gap quickly became something more meaningful. Over 30 people joined the effort, including those writing questions, building evaluation tools, and establishing the human performance baseline. Experimental chemists collaborated closely with AI researchers throughout the process. | ||
|
|
||
| ## How does this project connect to your academic work? | ||
|
|
||
| ChemBench emerged directly from my research team's work on chemical language models and helped establish a more systematic approach to benchmarking. | ||
|
|
||
| ## Who contributes to the project? | ||
|
|
||
| Faculty, students, postdocs, and external contributors. | ||
|
|
||
| ## How are students involved in the project? | ||
|
|
||
| Students contribute across all areas, including code, datasets, and testing. | ||
|
|
||
| ## How is the project used in teaching or coursework? | ||
|
|
||
| The project contributes to PhD-level research and forms part of thesis work. | ||
|
|
||
| ## What impact has this project had on your students? | ||
|
|
||
| Students develop strong software practices and gain visibility within the scientific community. | ||
|
|
||
| ## What impact has the project had beyond the classroom or research? | ||
|
|
||
| The project has been adopted by researchers and AI labs, with a growing number of citations. | ||
|
|
||
| ## What does it take to maintain the project? | ||
|
|
||
| Three PhD students currently serve as core maintainers. | ||
|
|
||
| ## What have been the biggest challenges in maintaining the project? | ||
|
|
||
| Academic systems often undervalue software development, and securing long-term maintainers remains a challenge. | ||
|
|
||
| ## How do you ensure the project remains sustainable over time? | ||
|
|
||
| We align research projects with the codebase and build partnerships to support ongoing development. | ||
|
|
||
| ## How do you engage with your community? | ||
|
|
||
| Through documentation, onboarding resources, contribution guides, and community forums. | ||
|
|
||
| ## Have you taken part in any open source programs or events? | ||
|
|
||
| Not yet, but we are planning to. | ||
|
|
||
| ## What would you love to achieve by showcasing your project? | ||
|
|
||
| To attract new contributors and highlight the work of students involved in the project. | ||
|
|
||
| ## Is there anything else you'd like to share about your project or open source journey? | ||
|
|
||
| GitHub tools such as Copilot and the Developer Pack have been invaluable. | ||
|
Comment on lines
+79
to
+81
|
||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.