HMTL (Hierarchical Multi-Task Learning model)

***** New November 20th, 2018: Online web demo is available *****

We released an online demo (along with pre-trained weights) so that you can play yourself with the model. The code for the web interface is also available in the demo folder.

To download the pre-trained models, please install git lfs and do a git lfs pull. The weights of the model will be saved in the model_dumps folder.

A Hierarchical Multi-Task Approach for Learning Embeddings from Semantic Tasks
Victor SANH, Thomas WOLF, Sebastian RUDER
Accepted at AAAI 2019

About

HMTL is a Hierarchical Multi-Task Learning model which combines a set of four carefully selected semantic tasks (namely Named Entity Recoginition, Entity Mention Detection, Relation Extraction and Coreference Resolution). The model achieves state-of-the-art results on Named Entity Recognition, Entity Mention Detection and Relation Extraction. Using SentEval, we show that as we move from the bottom to the top layers of the model, the model tend to learn more complex semantic representation.

For further details on the results, please refer to our paper.

We released the code for training, fine tuning and evaluating HMTL. We hope that this code will be useful for building your own Multi-Task models (hierarchical or not). The code is written in Python and powered by Pytorch.

Dependecies and installation

The main dependencies are:

• AllenNLP
• PyTorch
• SentEval (only for evaluating the embeddings)

The code works with Python 3.6. A stable version of the dependencies is listed in requirements.txt.

You can quickly setup a working environment by calling the script ./script/machine_setup.sh. It installs Python 3.6, creates a clean virtual environment, and installs all the required dependencies (listed in requirements.txt). Please adapt the script depending on your needs.

Example usage

We based our implementation on the AllenNLP library. For an introduction to this library, you should check these tutorials.

An experiment is defined in a json configuration file (see configs/*.json for examples). The configuration file mainly describes the datasets to load, the model to create along with all the hyper-parameters of the model.

Once you have set up your configuration file (and defined custom classes such DatasetReaders if needed), you can simply launch a training with the following command and arguments:

python train.py --config_file_path configs/hmtl_coref_conll.json --serialization_dir my_first_training

Once the training has started, you can simply follow the training in the terminal or open a Tensorboard (please make sure you have installed Tensorboard and its Tensorflow dependecy before):

tensorboard --logdir my_first_training/log

Evaluating the embeddings with SentEval

We used SentEval to assess the linguistic properties learned by the model. hmtl_senteval.py gives an example of how we can create an interface between SentEval and HMTL. It evaluates the linguistic properties learned by every layer of the hiearchy (shared based word embeddings and encoders).

Data

To download the pre-trained embeddings we used in HMTL, you can simply launch the script ./script/data_setup.sh.

We did not attach the datasets used to train HMTL for licensing reasons, but we invite you to collect them by yourself: OntoNotes 5.0, CoNLL2003, and ACE2005. The configuration files expect the datasets to be placed in the data/ folder.

References

Please consider citing the following paper if you find this repository useful.

@article{sanh2018hmtl,
  title={A Hierarchical Multi-task Approach for Learning Embeddings from Semantic Tasks},
  author={Sanh, Victor and Wolf, Thomas and Ruder, Sebastian},
  journal={arXiv preprint arXiv:1811.06031},
  year={2018}
}