Visit the GROBID documentation for more detailed information.
GROBID (or Grobid, but not GroBid nor GroBiD) means GeneRation Of BIbliographic Data.
GROBID is a machine learning library for extracting, parsing and re-structuring raw documents such as PDF into structured XML/TEI encoded documents with a particular focus on technical and scientific publications. First developments started in 2008 as a hobby. In 2011 the tool has been made available in open source. Work on GROBID has been steady as a side project since the beginning and is expected to continue as such.
The following functionalities are available:
In a complete PDF processing, GROBID manages 55 final labels used to build relatively fine-grained structures, from traditional publication metadata (title, author first/last/middlenames, affiliation types, detailed address, journal, volume, issue, pages, doi, pmid, etc.) to full text structures (section title, paragraph, reference markers, head/foot notes, figure headers, etc.).
GROBID includes a comprehensive web service API, batch processing, a JAVA API, a Docker image, a generic evaluation framework (precision, recall, etc., n-fold cross-evaluation) and the semi-automatic generation of training data.
GROBID can be considered as production ready. Deployments in production includes ResearchGate, HAL Research Archive, INIST-CNRS, CERN (Invenio), scite.ai, and many more. The tool is designed for high scalability in order to address the full scientific literature corpus.
GROBID should run properly "out of the box" on Linux (64 bits) and macOS. We cannot ensure currently support for Windows as we did before (help welcome!).
GROBID uses optionnally Deep Learning models relying on the DeLFT library, a task-agnostic Deep Learning framework for sequence labelling and text classification. The tool can run with feature engineered CRF (default), Deep Learning architectures (with or without layout feature channels) or any mixtures of CRF and DL to balance scalability and accuracy.
For more information on how the tool works, on its key features and benchmarking, visit the GROBID documentation.
For testing purposes, a public GROBID demo server is available at the following address: https://grobid.science-miner.com
The Web services are documented here.
Warning: Some quota and query limitation apply to the demo server! Please be courteous and do not overload the demo server.
For helping to exploit GROBID service at scale, we provide clients written in Python, Java, node.js using the web services for parallel batch processing:
All these clients will take advantage of the multi-threading for scaling large set of PDF processing. As a consequence, they will be much more efficient than the batch command lines (which use only one thread) and should be prefered.
We have been able recently to run the complete fulltext processing at around 10.6 PDF per second (around 915,000 PDF per day, around 20M pages per day) with the node.js client listed above during one week on one 16 CPU machine (16 threads, 32GB RAM, no SDD, articles from mainstream publishers), see here (11.3M PDF were processed in 6 days by 2 servers without interruption).
In addition, a Java example project is available to illustrate how to use GROBID as a Java library: https://github.com/kermitt2/grobid-example. The example project is using GROBID Java API for extracting header metadata and citations from a PDF and output the results in BibTeX format.
Finally, the following python utilities can be used to create structured full text corpora of scientific articles simply by indicating a list of strong identifiers like DOI or PMID, performing the identification of online Open Access PDF, the harvesting, the metadata agreegation and the Grobid processing in one step at scale: article-dataset-builder
A series of additional modules have been developed for performing structure aware text mining directly on scholar PDF, reusing GROBID's PDF processing and sequence labelling weaponery:
See the Changelog.
GROBID is distributed under Apache 2.0 license.
The documentation is distributed under CC-0 license and the annotated data under CC-BY license.
If you contribute to GROBID, you agree to share your contribution following these licenses.
Main author and contact: Patrice Lopez ([email protected])
ej-technologies provided us a free open-source license for its Java Profiler. Click the JProfiler logo below to learn more.
If you want to this work, please refer to the present GitHub project, together with the Software Heritage project-level permanent identifier. For example, with BibTeX:
@misc{GROBID,
title = {GROBID},
howpublished = {\url{https://github.com/kermitt2/grobid}},
publisher = {GitHub},
year = {2008--2021},
archivePrefix = {swh},
eprint = {1:dir:dab86b296e3c3216e2241968f0d63b68e8209d3c}
}
See the GROBID documentation for more related resources.