Real Time Voice Cloning

Clone a voice in 5 seconds to generate arbitrary speech in real-time
Alternatives To Real Time Voice Cloning
Project NameStarsDownloadsRepos Using ThisPackages Using ThisMost Recent CommitTotal ReleasesLatest ReleaseOpen IssuesLicenseLanguage
Transformers112,324641,8696 hours ago114July 18, 2023820apache-2.0Python
🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.
Stable Diffusion Webui103,231
20 hours ago2January 17, 20221,509agpl-3.0Python
Stable Diffusion web UI
Pytorch71,0373,3416,7286 hours ago37May 08, 202312,776otherPython
Tensors and Dynamic neural networks in Python with strong GPU acceleration
Real Time Voice Cloning42,864
3 months ago148otherPython
Clone a voice in 5 seconds to generate arbitrary speech in real-time
18 hours ago8September 21, 2021227agpl-3.0Python
YOLOv5 🚀 in PyTorch > ONNX > CoreML > TFLite
Annotated_deep_learning_paper_implementations36,22313 days ago78September 24, 202227mitJupyter Notebook
🧑‍🏫 60 Implementations/tutorials of deep learning papers with side-by-side notes 📝; including transformers (original, xl, switch, feedback, vit, ...), optimizers (adam, adabelief, sophia, ...), gans(cyclegan, stylegan2, ...), 🎮 reinforcement learning (ppo, dqn), capsnet, distillation, ... 🧠
Made With Ml34,182
2 days ago5May 15, 20192mitJupyter Notebook
Learn how to design, develop, deploy and iterate on production-grade ML applications.
Gfpgan32,185911 days ago11September 20, 2022271otherPython
GFPGAN aims at developing Practical Algorithms for Real-world Face Restoration.
19 days ago2February 28, 2022446otherPython
🚀AI拟声: 5秒内克隆您的声音并生成任意语音内容 Clone a voice in 5 seconds to generate arbitrary speech in real-time
Deepspeed28,4955313 hours ago68July 17, 2023794apache-2.0Python
DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.
Alternatives To Real Time Voice Cloning
Select To Compare

Alternative Project Comparisons

Real-Time Voice Cloning

This repository is an implementation of Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis (SV2TTS) with a vocoder that works in real-time. This was my master's thesis.

SV2TTS is a deep learning framework in three stages. In the first stage, one creates a digital representation of a voice from a few seconds of audio. In the second and third stages, this representation is used as reference to generate speech given arbitrary text.

Video demonstration (click the picture):

Toolbox demo

Papers implemented

URL Designation Title Implementation source
1806.04558 SV2TTS Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis This repo
1802.08435 WaveRNN (vocoder) Efficient Neural Audio Synthesis fatchord/WaveRNN
1703.10135 Tacotron (synthesizer) Tacotron: Towards End-to-End Speech Synthesis fatchord/WaveRNN
1710.10467 GE2E (encoder) Generalized End-To-End Loss for Speaker Verification This repo

Heads up

Like everything else in Deep Learning, this repo is quickly getting old. Many other open-source repositories or SaaS apps (often paying) will give you a better audio quality than this repository will. If you care about the fidelity of the voice you're cloning, and its expressivity, here are some personal recommendations of alternative voice cloning solutions:

  • Check out CoquiTTS for an open source repository that is more up-to-date, with a better voice cloning quality and more functionalities.
  • Check out paperswithcode for other repositories and recent research in the field of speech synthesis.
  • Check out (disclaimer: I work there) for state of the art voice cloning with little hassle.


1. Install Requirements

  1. Both Windows and Linux are supported. A GPU is recommended for training and for inference speed, but is not mandatory.
  2. Python 3.7 is recommended. Python 3.5 or greater should work, but you'll probably have to tweak the dependencies' versions. I recommend setting up a virtual environment using venv, but this is optional.
  3. Install ffmpeg. This is necessary for reading audio files.
  4. Install PyTorch. Pick the latest stable version, your operating system, your package manager (pip by default) and finally pick any of the proposed CUDA versions if you have a GPU, otherwise pick CPU. Run the given command.
  5. Install the remaining requirements with pip install -r requirements.txt

2. (Optional) Download Pretrained Models

Pretrained models are now downloaded automatically. If this doesn't work for you, you can manually download them here.

3. (Optional) Test Configuration

Before you download any dataset, you can begin by testing your configuration with:


If all tests pass, you're good to go.

4. (Optional) Download Datasets

For playing with the toolbox alone, I only recommend downloading LibriSpeech/train-clean-100. Extract the contents as <datasets_root>/LibriSpeech/train-clean-100 where <datasets_root> is a directory of your choosing. Other datasets are supported in the toolbox, see here. You're free not to download any dataset, but then you will need your own data as audio files or you will have to record it with the toolbox.

5. Launch the Toolbox

You can then try the toolbox:

python -d <datasets_root>

depending on whether you downloaded any datasets. If you are running an X-server or if you have the error Aborted (core dumped), see this issue.

Popular Deep Learning Projects
Popular Pytorch Projects
Popular Machine Learning Categories
Related Searches

Get A Weekly Email With Trending Projects For These Categories
No Spam. Unsubscribe easily at any time.
Deep Learning