Head Qa

HEAD-QA: A Healthcare Dataset for Complex Reasoning
Alternatives To Head Qa
Project NameStarsDownloadsRepos Using ThisPackages Using ThisMost Recent CommitTotal ReleasesLatest ReleaseOpen IssuesLicenseLanguage
2 days ago22mitPython
Cross-silo Federated Learning playground in Python. Discover 7 real-world federated datasets to test your new FL strategies and try to beat the leaderboard.
Covid Cxr94
2 years ago7mitPython
Neural network model for classifying chest X-rays by presence of COVID-19 features
4 years agoapache-2.0Jupyter Notebook
Detecting cancers on mobile phones
5 months ago4March 14, 20218cc0-1.0R
NHS and healthcare related datasets for training and learning R
Awesome Healthcare Datasets54
4 years agomit
A curated list of awesome healthcare datasets in the public domain.
Epillid Benchmark48
a year ago4mitJupyter Notebook
ePillID Dataset: A Low-Shot Fine-Grained Benchmark for Pill Identification (CVPR 2020 VL3)
Covid Net38
4 months ago
Launched in March 2020 in response to the coronavirus disease 2019 (COVID-19) pandemic, COVID-Net is a global open source, open access initiative dedicated to accelerating advancement in machine learning to aid front-line healthcare workers and clinical institutions around the world fighting the continuing pandemic. Towards this goal, our global multi-disciplinary team of researchers, developers, and clinicians have made publicly available a suite of tailored deep neural network models for tackling different challenges ranging from screening to risk stratification to treatment planning for patients with the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2). Furthermore, we have made available fully curated, open access benchmark datasets comprised of some of the largest, most diverse patient cohorts from around the world.
2 years ago3
Awesome Healthmetrics31
2 years agomit
A curated list of awesome resources at the intersection of healthcare and AI
Covid19 Algeria And World Dataset18
a month agoJupyter Notebook
A COVID-19 dataset with 90 World countries including Algeria.
Alternatives To Head Qa
Select To Compare

Alternative Project Comparisons


NEWS! HEAD-QA can be now imported from huggingface datasets. Thank you very much to Maria Grandury for adding it.

This repository contains the sources used in "HEAD-QA: A Healthcare Dataset for Complex Reasoning" (ACL, 2019)

HEAD-QA is a multi-choice HEAlthcare Dataset. The questions come from exams to access a specialized position in the Spanish healthcare system, and are challenging even for highly specialized humans. They are designed by the Ministerio de Sanidad, Consumo y Bienestar Social, who also provides direct access to the exams of the last 5 years (in Spanish).

Date of the last update of the documents object of the reuse: January, 14th, 2019.

HEAD-QA tries to make these questions accessible for the Natural Language Processing community. We hope it is an useful resource towards achieving better QA systems. The dataset contains questions about the following topics:

  • Medicine.
  • Nursing.
  • Psychology.
  • Chemistry.
  • Pharmacology.
  • Biology.


  • Python 3.6.7
  • DrQA
  • scikit-learn==0.20.2
  • numpy==1.16.0
  • torch==1.0.0
  • torchvision
  • spacy==2.0.0
  • prettytable==0.70.2

Requirements for the ARC-Solvers

  • Python 3.6.7
  • torch==0.3.1
  • torchvision
  • allennlp==0.20.1


We first recommend you to install a virtualenv in the first place (e.g. virtualenv -p python3.6 head-qa) The script install.sh automatically installs the mentioned packages, assuming that you have previously created and activated your virtualenv (tested on Ubuntu 18.04, 64 bits). The script install_arc_solvers.sh install the needed stuff to run the ARC-solvers (Clark et al,2019).

We recommend using a different virtualenv for them as stuff such as the pytorch version might create conflicts.


ES_HEAD dataset EN_HEAD dataset Each dataset contains:

  • *.gold -> A tsv gold file that maps question IDs to the ground truth answer ID to such question. One file per exam.
  • HEAD[_EN].json -> It contains the whole data for HEAD-QA (used in the so-called 'unsupervised' setting).
  • train_HEAD[_EN].json -> It contains the training set of HEAD-QA (used as the training set in the so-called 'supervised' setting)
  • dev_HEAD[_EN].json -> A json file containing the development set of HEAD-QA (used in the 'supervised' setting).
  • test_HEAD[_EN].json -> A json file containing the test set of HEAD-QA (used in the 'supervised' setting).

Data (images, pdfs, etc). Note that these are medical images and some of them might have sensitive content.

Run the baselines: Length, Random, Blind_n, IR and DrQA

Available baselines for Spanish HEAD-QA: Length, Random, Blind_n, IR- Available baselines for English HEAD-QA (HEAD-QA_EN): Length, Random, Blind_n, IR, DrQA-

Description of the baselines:

  • Length: Chooses the longest answer
  • Random: Chooses a random answer.
  • Blind_n: Chooses the nth answer.
  • IR: Chooses the answer based on the relevance of the query: question+nth answer.
  • DrQA: A model based on DrQA's (Chen, D., Fisch, A., Weston, J., & Bordes, A. Reading Wikipedia to Answer Open-Domain Questions)

Creating an inverted index

IR and DrQA require to create an inverted index in advance. This is done using wikiextractor and following DrQa's Document Reader guidelines (visit their README.md for a detailed explanation about how to create the index, we here summarize the main steps):

In this work we used the following Wikipedia dumps:

Alternative, you can try to use the current Wikipedia dump maintained by https://dumps.wikimedia.org/

PYTHONPATH="$HOME/git/wikiextractor" python $HOME/git/wikiextractor/WikiExtractor.py $PATH_WIKIPEDIA_DUMP -o $PATH_WIKI_JSON --json
PYTHONPATH="$HOME/git/DrQA/" python $HOME/git/DrQA/scripts/retriever/build_db.py $PATH_WIKI_JSON $PATH_DB
PYTHONPATH="$HOME/git/DrQA/" python $HOME/git/DrQA/scripts/retriever/build_tfidf.py --num-workers 2 $PATH_DB $PATH_TFIDF

The created model in $PATH_TFIDF it's what will be used as our inverted index. If they are of any help, the indexes we used in our work can be found here.

Updating DrQA's tokenizer

By default, DrQA uses the CoreNLP tokenizer. In this work we used the SpacyTokenizer instead. To use it, go to DrQA/drqa/pipeline/__init__.py and make sure you use the DEFAULT below these lines. Also, we used multitask.mdl as the reader_model. Make sure you have downloaded it when you installed DrQA.

from ..tokenizers import CoreNLPTokenizer, SpacyTokenizer

    'tokenizer': SpacyTokenizer,#CoreNLPTokenizer,
    'ranker': TfidfDocRanker,
    'db': DocDB,
    'reader_model': os.path.join(DATA_DIR, 'reader/multitask.mdl'),

Create a configuration file

#A configuration file for Spanish

#Path to your DrQA's installation
#The folder containing the .gold files

es_head=HEAD/HEAD.json #HEAD-QA in json format
#The inverted index that we have previously created.

After this, you should be abl to run the script run.py:

python run.py --config configs/configuration$LANG.config --answerer $ANSWERER --output $OUTPUT
  • --config A path to a configuration file (see the folder configs for an example)
  • --answerer A string to indicate what 'answerer' to use. Valid values are [length, random, ir, drqa, blind_n] (n is a number to indicate to take as the right answer the nth answer.
  • --output The path to the file to save the results

Running the ARC-solvers

We also run the ARC-Solvers used in the ARC challenge (Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., & Tafjord, O. Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge.). To install and run them follow these steps:

1- Follow the ARC-solvers README.md instructions to create a virtualenv, create the index and download the models and resources:

NOTE that instead of using their ARC_corpus.txt as the inverted index we used again Wikipedia. If you also want to use Wikipedia you need to do two things:

  1. Make sure you have downloaded our Wikipedia corpus in txt format.
  2. Modify the file ARC-Solvers/scripts/download_data.sh and change the argument specifying the corpus ARC_corpus.txt to the path where you have stored the Wikipedia corpus.

NOTE 2 ARC-Solvers need of elasticsearch 6+ to download the data. Download it and to run it execute.

cd elasticsearch-<version>

2 - Convert HEAD_EN.json into the input format for the ARC solvers

PYTHONPATH=. python scripts/head2ARCformat.py --input HEAD_EN/HEAD_EN.json --output HEAD_ARC/

3 - Run the models using the evaluation scripts provided together with the ARC solvers:

cd ARC-Solvers
sh scripts/evaluate_solver.sh ../HEAD_ARC/HEAD_EN.arc.txt data/ARC-V1-Models-Aug2018/dgem/
sh scripts/evaluate_solver.sh ../HEAD_ARC/HEAD_EN.arc.txt data/ARC-V1-Models-Aug2018/decompatt/
sh scripts/evaluate_bidaf.sh ../HEAD_ARC/HEAD_EN.arc.txt data/ARC-V1-Models-Aug2018/bidaf/

4 - Compute the scores for HEAD-QA, based on the ARC-solvers outputs

cd ..
python evaluate_arc_solvers.py --arc_results $PATH_RESULTS --output $PATH_OUTPUT_DIR --disambiguator length --breakdown_results --path_eval eval.py


  • --arc_results Path to the output directory containing the outputs computed in step 3.
  • --output The path to the output directory where to store the results.
  • --disambiguator The strategy to decide the right answer if many answers where selected as valid by an ARC-solver
  • --breakdown_results Activate to report individual results for each exam
  • --path_eval Path to the evaluation script


We had problems running some models, being unable to find the question-tuplizer.jar used in the ARC-solvers. If you experience this error Error: Unable to access jarfile data/ARC-V1-Models-Feb2018/question-tuplizer.jar we recommend you to change in the file scripts/evaluate_solver.sh the line: java -Xmx8G -jar data/ARC-V1-Models-Feb2018/question-tuplizer.jar by java -Xmx8G -jar data/ARC-V1-Models-Aug2018/question-tuplizer.jar

We also had problems ruuning the dgem baseline. The default torch version that is installed if you follow the instructions in the ARC-solvers README.md is the 0.4.1. To make them work we needed to install torch 0.3.1 instead.


This work has received funding from the European Research Council (ERC), under the European Union's Horizon 2020 research and innovation programme (FASTPARSE, grant agreement No 714150).


Vilares, David and Gómez-Rodríguez, Carlos. "HEAD-QA: A Healthcare Dataset for Complex Reasoning", to appear, ACL 2019.

Popular Dataset Projects
Popular Healthcare Projects
Popular Data Processing Categories
Related Searches

Get A Weekly Email With Trending Projects For These Categories
No Spam. Unsubscribe easily at any time.
Questions And Answers