Implement some state-of-the-art text classification models with TensorFlow.
Note: Original code is written in TensorFlow 1.4, while the
VocabularyProcessor is depreciated, updated code changes to use
tf.keras.preprocessing.text to do preprocessing. The new preprocessing function is named
You can load the data with
dbpedia = tf.contrib.learn.datasets.load_dataset('dbpedia', test_with_fake_data=FLAGS.test_with_fake_data)
Or download it from Baidu Yun.
Paper: Attention Is All You Need
Use self-attention where Query = Key = Value = sentence after word embedding
Multihead Attention module is implemented by Kyubyong
IndRNNCell is implemented by batzener
Attention module is implemented by ilivans/tf-rnn-attention .
See: RMDL.py See: RMDL Github
Note: The parameters are not fine-tuned, you can modify the kernel as you want.
|Attention-based Bi-LSTM||98.23 %|
|HAN||89.15%||1080Ti 10 epochs 12 min|
|Adversarial Attention-based Bi-LSTM||98.5%||AWS p2 2 hours|
|IndRNN||98.39%||1080Ti 10 epochs 10 min|
|Attention is All Your Need||97.81%||1080Ti 15 epochs 8 min|
|RMDL||98.91%||2X Tesla Xp (3 RDLs)|
If you have any models implemented with great performance, you're welcome to contribute. Also, I'm glad to help if you have any problems with the project, feel free to raise a issue.