Awesome Open Source
Awesome Open Source

Speech Transformer: End-to-End ASR with Transformer

A PyTorch implementation of Speech Transformer [1], an end-to-end automatic speech recognition with Transformer network, which directly converts acoustic features to character sequence using a single nueral network.

Ad: Welcome to join Kwai Speech Team, make your career great! Send your resume to: xukaituo [at] kuaishou [dot] com!
广告时间:欢迎加入快手语音组,make your career great! 快发送简历到xukaituo [at] kuaishou [dot] com吧!
広告:Kwai チームへようこそ!自分のキャリアを照らそう!レジュメをこちらへ: xukaituo [at] kuaishou [dot] com!


  • Python3 (recommend Anaconda)
  • PyTorch 0.4.1+
  • Kaldi (just for feature extraction)
  • pip install -r requirements.txt
  • cd tools; make KALDI=/path/to/kaldi
  • If you want to run egs/aishell/, download aishell dataset for free.


Quick start

$ cd egs/aishell
# Modify aishell data path to your path in the begining of 
$ bash

That's all!

You can change parameter by $ bash --parameter_name parameter_value, egs, $ bash --stage 3. See parameter name in egs/aishell/ before . utils/


Workflow of egs/aishell/

  • Stage 0: Data Preparation
  • Stage 1: Feature Generation
  • Stage 2: Dictionary and Json Data Preparation
  • Stage 3: Network Training
  • Stage 4: Decoding

More detail

egs/aishell/ provide example usage.

$ cd egs/aishell/; . ./
# Train
$ -h
# Decode
$ -h

How to visualize loss?

If you want to visualize your loss, you can use visdom to do that:

  1. Open a new terminal in your remote server (recommend tmux) and run $ visdom.
  2. Open a new terminal and run $ bash --visdom 1 --visdom_id "<any-string>" or $ ... --visdom 1 --vidsdom_id "<any-string>".
  3. Open your browser and type <your-remote-server-ip>:8097, egs,
  4. In visdom website, chose <any-string> in Environment to see your loss. loss

How to resume training?

$ bash --continue_from <model-path>

How to solve out of memory?

When happened in training, try to reduce batch_size. $ bash --batch_size <lower-value>.


Model CER Config
LSTMP 9.85 4x(1024-512). See kaldi-ktnet1
Listen, Attend and Spell 13.2 See Listen-Attend-Spell's egs/aishell/
SpeechTransformer 12.8 See egs/aishell/


  • [1] Yuanyuan Zhao, Jie Li, Xiaorui Wang, and Yan Li. "The SpeechTransformer for Large-scale Mandarin Chinese Speech Recognition." ICASSP 2019.

Get A Weekly Email With Trending Projects For These Topics
No Spam. Unsubscribe easily at any time.
python (55,393
pytorch (2,454
transformer (214
attention (112
asr (61
end-to-end (35
self-attention (34
attention-is-all-you-need (19