This project uses Python 3.7. There are features used that do not exist in 3.6, so make sure to use Python 3.7.
This guide also assumes you 1) are using Linux and 2) are somewhat capable with the commandline.
It's not impossible to run Nyaa on Windows, but this guide doesn't focus on that.
./dev.py lintbefore committing to see a list of warnings/problems.
./dev.py fix && ./dev.py isortto automatically fix some of the issues reported by the previous command.
tests folder contains tests for the the
nyaa module and the webserver. To run the tests:
./dev.py testwhile in the repository directory.
pyenv eases the use of different Python versions, and as not all Linux distros offer 3.7 packages, it's right up our alley.
pyenvand create a virtualenv for the project:
pyenv install 3.7.2
pyenv virtualenv 3.7.2 nyaa
pyenv activate nyaa
pip install -r requirements.txt
config.pydepending on which instance you want to host
You may use SQLite but the current support for it in this project is outdated and rather unsupported.
USE_MYSQLflag in config.py
mysql Ver 15.1 Distrib 10.0.30-MariaDB, for debian-linux-gnu (x86_64) using readline 5.2
config.pyvalues if desired):
CREATE USER 'test'@'localhost' IDENTIFIED BY 'test123';
GRANT ALL PRIVILEGES ON *.* TO 'test'@'localhost';
CREATE DATABASE nyaav2 DEFAULT CHARACTER SET utf8 COLLATE utf8_bin;
python db_create.pyto create the database and import categories
./db_migrate.py stamp headto mark the database version for Alembic
source deactivate(or just close your shell session)
You're now ready for simple testing and development!
Continue below to learn about database migrations and enabling the advanced search engine, Elasticsearch.
./db_migrate.py stamp headbefore pulling the new migration script(s).
./db_migrate.py historyinstead and choose a hash that matches your current database state, then run
./db_migrate.py stamp <hash>.
git fetch && git rebase origin/master)
./db_migrate.py upgrade headto run the migration. Done!
models.pyand ensure the database schema matches the previous version (ie. your new tables/columns are not added to the live database)
./db_migrate.py migrate -m "Short description of changes"to automatically generate a migration script for the changes
migrations/versions/...) and make sure it works! Alembic may not able to notice all changes.
./db_migrate.py upgradeto run the migration and verify the upgrade works.
./db_migrate.py downgradeto verify the downgrade works as well, then upgrade again)
sudo apt-get install openjdk-8-jdk
sudo systemctl enable elasticsearch.service
sudo systemctl start elasticsearch.service
curl -XGET 'localhost:9200'and make sure ES is running
log-bin server_id=1 log-basename=master1 binlog-format=row
sudo service mysql restart)
es_sync_config.jsonand adjust options in it to your liking (verify the connection options!)
SHOW VARIABLES LIKE 'binlog_format';is
GRANT REPLICATION SLAVE ON *.* TO 'username'@'localhost';to allow your configured user access to the binlog
./create_es.shto create the indices for the torrents:
python import_to_es.pyto import all the torrents (on nyaa and sukebei) into the ES indices.
USE_ELASTIC_SEARCH flag in
config.py and (re)start the application.
Elasticsearch should now be functional! The ES indices won't be updated "live" with the current setup, continue below for instructions on how to hook Elasticsearch up to MySQL binlog.
However, take note that binglog is not necessary for simple ES testing and development; you can simply run
import_to_es.py from time to time to reindex all the torrents.
sync_es.py keeps the Elasticsearch indices updated by reading the binlog and pushing the changes to the ES indices.
es_sync_config.jsonis configured with the user you grated the
import_to_es.pyand copy the outputted JSON into the file specified by
sync_es.pyas-is or, for actual deployment, set it up as a service and run it, preferably as the system/root
sync_es.pyruns within the venv with the right dependencies!
You're done! The script should now be feeding updates from the database to Elasticsearch.
Take note, however, that the specified ES index refresh interval is 30 seconds, which may feel like a long time on local development. Feel free to adjust it or poke Elasticsearch yourself!