State-of-the-art Machine Learning for JAX, PyTorch and TensorFlow
Transformers provides thousands of pretrained models to perform tasks on different modalities such as text, vision, and audio.
These models can be applied on:
Transformer models can also perform tasks on several modalities combined, such as table question answering, optical character recognition, information extraction from scanned documents, video classification, and visual question answering.
Transformers provides APIs to quickly download and use those pretrained models on a given text, fine-tune them on your own datasets and then share them with the community on our model hub. At the same time, each python module defining an architecture is fully standalone and can be modified to enable quick research experiments.
Transformers is backed by the three most popular deep learning libraries Jax, PyTorch and TensorFlow with a seamless integration between them. It's straightforward to train your models with one before loading them for inference with the other.
You can test most of our models directly on their pages from the model hub. We also offer private model hosting, versioning, & an inference API for public and private models.
Here are a few examples:
In Natural Language Processing:
In Computer Vision:
In Audio:
Write With Transformer, built by the Hugging Face team, is the official demo of this repos text generation capabilities.
To immediately use a model on a given input (text, image, audio, ...), we provide the pipeline
API. Pipelines group together a pretrained model with the preprocessing that was used during that model's training. Here is how to quickly use a pipeline to classify positive versus negative texts:
>>> from transformers import pipeline
# Allocate a pipeline for sentiment-analysis
>>> classifier = pipeline('sentiment-analysis')
>>> classifier('We are very happy to introduce pipeline to the transformers repository.')
[{'label': 'POSITIVE', 'score': 0.9996980428695679}]
The second line of code downloads and caches the pretrained model used by the pipeline, while the third evaluates it on the given text. Here the answer is "positive" with a confidence of 99.97%.
Many NLP tasks have a pre-trained pipeline
ready to go. For example, we can easily extract question answers given context:
>>> from transformers import pipeline
# Allocate a pipeline for question-answering
>>> question_answerer = pipeline('question-answering')
>>> question_answerer({
... 'question': 'What is the name of the repository ?',
... 'context': 'Pipeline has been included in the huggingface/transformers repository'
... })
{'score': 0.30970096588134766, 'start': 34, 'end': 58, 'answer': 'huggingface/transformers'}
In addition to the answer, the pretrained model used here returned its confidence score, along with the start position and end position of the answer in the tokenized sentence. You can learn more about the tasks supported by the pipeline
API in this tutorial.
To download and use any of the pretrained models on your given task, all it takes is three lines of code. Here is the PyTorch version:
>>> from transformers import AutoTokenizer, AutoModel
>>> tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
>>> model = AutoModel.from_pretrained("bert-base-uncased")
>>> inputs = tokenizer("Hello world!", return_tensors="pt")
>>> outputs = model(**inputs)
And here is the equivalent code for TensorFlow:
>>> from transformers import AutoTokenizer, TFAutoModel
>>> tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
>>> model = TFAutoModel.from_pretrained("bert-base-uncased")
>>> inputs = tokenizer("Hello world!", return_tensors="tf")
>>> outputs = model(**inputs)
The tokenizer is responsible for all the preprocessing the pretrained model expects, and can be called directly on a single string (as in the above examples) or a list. It will output a dictionary that you can use in downstream code or simply directly pass to your model using the ** argument unpacking operator.
The model itself is a regular Pytorch nn.Module
or a TensorFlow tf.keras.Model
(depending on your backend) which you can use normally. This tutorial explains how to integrate such a model into a classic PyTorch or TensorFlow training loop, or how to use our Trainer
API to quickly fine-tune on a new dataset.
Easy-to-use state-of-the-art models:
Lower compute costs, smaller carbon footprint:
Choose the right framework for every part of a model's lifetime:
Easily customize a model or an example to your needs:
This repository is tested on Python 3.6+, Flax 0.3.2+, PyTorch 1.3.1+ and TensorFlow 2.3+.
You should install Transformers in a virtual environment. If you're unfamiliar with Python virtual environments, check out the user guide.
First, create a virtual environment with the version of Python you're going to use and activate it.
Then, you will need to install at least one of Flax, PyTorch or TensorFlow. Please refer to TensorFlow installation page, PyTorch installation page and/or Flax and Jax installation pages regarding the specific install command for your platform.
When one of those backends has been installed, Transformers can be installed using pip as follows:
pip install transformers
If you'd like to play with the examples or need the bleeding edge of the code and can't wait for a new release, you must install the library from source.
Since Transformers version v4.0.0, we now have a conda channel: huggingface
.
Transformers can be installed using conda as follows:
conda install -c huggingface transformers
Follow the installation pages of Flax, PyTorch or TensorFlow to see how to install them with conda.
All the model checkpoints provided by Transformers are seamlessly integrated from the huggingface.co model hub where they are uploaded directly by users and organizations.
Current number of checkpoints:
Transformers currently provides the following architectures (see here for a high-level summary of each them):
templates
folder of the repository. Be sure to check the contributing guidelines and contact the maintainers or open an issue to collect feedbacks before starting your PR.To check if each model has an implementation in Flax, PyTorch or TensorFlow, or has an associated tokenizer backed by the Tokenizers library, refer to this table.
These implementations have been tested on several datasets (see the example scripts) and should match the performance of the original implementations. You can find more details on performance in the Examples section of the documentation.
Section | Description |
---|---|
Documentation | Full API documentation and tutorials |
Task summary | Tasks supported by Transformers |
Preprocessing tutorial | Using the Tokenizer class to prepare data for the models |
Training and fine-tuning | Using the models provided by Transformers in a PyTorch/TensorFlow training loop and the Trainer API |
Quick tour: Fine-tuning/usage scripts | Example scripts for fine-tuning models on a wide range of tasks |
Model sharing and uploading | Upload and share your fine-tuned models with the community |
Migration | Migrate to Transformers from pytorch-transformers or pytorch-pretrained-bert
|
We now have a paper you can cite for the Transformers library:
@inproceedings{wolf-etal-2020-transformers,
title = "Transformers: State-of-the-Art Natural Language Processing",
author = "Thomas Wolf and Lysandre Debut and Victor Sanh and Julien Chaumond and Clement Delangue and Anthony Moi and Pierric Cistac and Tim Rault and Rmi Louf and Morgan Funtowicz and Joe Davison and Sam Shleifer and Patrick von Platen and Clara Ma and Yacine Jernite and Julien Plu and Canwen Xu and Teven Le Scao and Sylvain Gugger and Mariama Drame and Quentin Lhoest and Alexander M. Rush",
booktitle = "Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations",
month = oct,
year = "2020",
address = "Online",
publisher = "Association for Computational Linguistics",
url = "https://www.aclweb.org/anthology/2020.emnlp-demos.6",
pages = "38--45"
}