Awesome Open Source
Awesome Open Source

Taiwan-news-crawlers

🐞 Scrapy-based Crawlers for news of Taiwan including 10 media companies:

  1. 蘋果日報
  2. 中國時報
  3. 中央社
  4. 華視
  5. 東森新聞雲
  6. 自由時報
  7. 公視
  8. 三立
  9. TVBS
  10. UDN

Getting Started

$ git clone https://github.com/TaiwanStat/Taiwan-news-crawlers.git
$ cd Taiwan-news-crawlers
$ pip install -r requirements.txt
$ scrapy crawl apple -o apple_news.json

Prerequisites

  • Python3
  • Scrapy 1.3.0

Usage

scrapy crawl <spider> -o <output_name>

Available spiders

  1. apple
  2. appleRealtime
  3. china
  4. cna
  5. cts
  6. ettoday
  7. liberty
  8. libertyRealtime
  9. pts
  10. setn
  11. tvbs
  12. udn

Output

Key Value
website the publisher
url the origin web
title the news title
content the news content
category the category of news

License

The MIT License


Get A Weekly Email With Trending Projects For These Topics
No Spam. Unsubscribe easily at any time.
python (55,394
crawler (379
scrapy (130
news (121
taiwan (20