Awesome Open Source
Awesome Open Source

Build Status codecov Code Health License Python

-- --
Version 1.0.1
WebSite http://119.23.223.90:8000
Source https://awesomeopensource.com/project/GuozhuHe/webspider
Keywords Python3, Tornado, Celery, Requests

python3``MySQL, celery``requests``Cookies``ECharts + Bootstrap

Alt text

Quick Start

Linux - Ubuntu

git clone [email protected]:GuozhuHe/webspider.git
  • MySQL, Redis, Python3
#  redis
apt-get install redis-server

#  redis-server
nohup redis-server &

#  python3
apt-get install python3

#  MySQL
apt-get install mysql-server

#  MySQL
sudo service mysql start
# 
CREATE DATABASE `spider` CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
#  tests/schema.sql  MySQL 
make
#  env/bin 
make test
make flake8
  • Web Server
env/bin/web
# 
env/bin/celery_beat
#    worker
env/bin/celery_lg_data_worker
#    worker
env/bin/celery_lg_jobs_data_worker
#    worker
env/bin/celery_lg_jobs_count_worker
  • env/bin
# 
env/bin/crawl_lg_jobs_count
# 
env/bin/crawl_lg_data
# celery
env/bin/celery_flower
make clean

TODO

  • [ ]

  • [ ]

  • [x]

  • [x]

  • [x] Fix Bug: MySQL Server has gone away. MR


Get A Weekly Email With Trending Projects For These Topics
No Spam. Unsubscribe easily at any time.
Python (1,140,439
Requests (2,002
Spider (1,563
Celery (690
Tornado (418
Related Projects