Awesome Open Source
Awesome Open Source

awesome_crawl Awesome


爬虫 说明 技术栈
腾讯新闻 采集所有腾讯新闻的链接和新闻详情 scrapy,mongo,redis
知乎话题 从话题广场出发,先采集子话题ID,再采集ID下所有问题 scrapy,mongo,redis
微博粉丝 采集大V的所有粉丝 scrapy,mongo,redis
Tumblr爬虫 下载指定Tumblr博主的资源 requests,concurrent
妹子图爬虫 go语言下载妹子图 goroutine,goquery
全站爬虫 分布式全站爬虫——以"搜狗电视剧"为例 scrapy,redis,分布式

如果能帮助你,那就最好了。欢迎关注公众号:


Get A Weekly Email With Trending Projects For These Topics
No Spam. Unsubscribe easily at any time.
python (50,968
golang (3,683
awesome (1,318
redis (904
websockets (305
scrapy (124
weibo (62
zhihu (55
tumblr (26
douyu (26
crawl (20
tencent (20

Find Open Source By Browsing 7,000 Topics Across 59 Categories