Examples Of Web Crawlers

一些非常有趣的python爬虫例子,对新手比较友好,主要爬取淘宝、天猫、微信、微信读书、豆瓣、QQ等 interesting examples of python crawlers that are friendly to beginners. )
Alternatives To Examples Of Web Crawlers
Project NameStarsDownloadsRepos Using ThisPackages Using ThisMost Recent CommitTotal ReleasesLatest ReleaseOpen IssuesLicenseLanguage
Examples Of Web Crawlers11,050
a year ago3mitPython
一些非常有趣的python爬虫例子,对新手比较友好,主要爬取淘宝、天猫、微信、微信读书、豆瓣、QQ等网站。(Some interesting examples of python crawlers that are friendly to beginners. )
Python8,683
5 days ago70Python
Python脚本。模拟登录知乎, 爬虫,操作excel,微信公众号,远程开机
Spider Flow8,075
3 months ago20mitJava
新一代爬虫平台,以图形化方式定义爬虫流程,不写代码即可完成爬虫。
Infospider6,649
4 months ago7gpl-3.0Python
INFO-SPIDER 是一个集众多数据源于一身的爬虫工具箱🧰,旨在安全快捷的帮助用户拿回自己的数据,工具代码开源,流程透明。支持数据源包括GitHub、QQ邮箱、网易邮箱、阿里邮箱、新浪邮箱、Hotmail邮箱、Outlook邮箱、京东、淘宝、支付宝、中国移动、中国联通、中国电信、知乎、哔哩哔哩、网易云音乐、QQ好友、QQ群、生成朋友圈相册、浏览器浏览历史、12306、博客园、CSDN博客、开源中国博客、简书。
Python3 Spider2,541
3 years agoPython
Python爬虫实战 - 模拟登陆各大网站 包含但不限于:滑块验证、拼多多、美团、百度、bilibili、大众点评、淘宝,如果喜欢请start ❤️
Python Crawler1,576
2 years ago2HTML
从头开始 系统化的 学习如何写Python爬虫。 Python版本 3.6
Autocrawler1,438
20 days ago13apache-2.0Python
Google, Naver multiprocess image web crawler (Selenium)
Instagram Profilecrawl1,001
5 months ago8mitPython
📝 quickly crawl the information (e.g. followers, tags etc...) of an instagram profile.
Scrapy Selenium699322 years ago6January 24, 201957wtfplPython
Scrapy middleware to handle javascript pages using selenium
Xxl Crawler650216 months ago5October 24, 201820apache-2.0Java
A distributed web crawler framework.(分布式爬虫框架XXL-CRAWLER)
Alternatives To Examples Of Web Crawlers
Select To Compare


Alternative Project Comparisons
Readme

python,


****python



chinese_flag****, Gitee,



1.

  1. chrome
  2. chromechromedriver
  3. pip
    • [x] pip install selenium
  4. mainchromedriver
  5. main
#chromedriver
chromedriver_path = "/Users/bird/Desktop/chromedriver.exe" 
#
weibo_username = ""
#
weibo_password = ""



2.

  1. chrome
  2. chromechromedriver
  3. pip
    • [x] pip install selenium
    • [x] pip install pyquery
  4. mainchromedriver
  5. main
#chromedriver
chromedriver_path = "/Users/bird/Desktop/chromedriver.exe" 
#
weibo_username = ""
#
weibo_password = ""



3.

  1. chrome
  2. chromechromedriver
  3. pip
    • [x] pip install selenium
    • [x] pip install pyquery
  4. mainchromedriver
  5. main
#chromedriver
chromedriver_path = "/Users/bird/Desktop/chromedriver.exe" 
#
weibo_username = ""
#
weibo_password = ""



4.


python********


****

  1. pip
  • [x] pip install wxpy
  • [x] pip install requests
  • [x] config.ini

example1example2

example3

5.5K


MacPap.erMac5KWindowsLinux5K

# 
cd 
# 
pip uninstall -y -r requirement.txt
# 
pip install -r requirement.txt -i https://pypi.tuna.tsinghua.edu.cn/simple
# 
python main.py


6.(GUI)

  1. Chromechrome://version/Chrome
  2. http://chromedriver.storage.googleapis.com/index.htmlchromedriver****
  3. getMovieInRankingList.py``107``executable_path=./chromedriver.exechromedriver
  4. pip install -r requirement.txt
  5. python main.py

  • [x]
  • [x] (TOP250)
  • [x] IMDB
  • [x] vip
  • [x]
  • [x]
  • [ ]


7.+()

ScrapyIP

  • IP

000056,,2019-03-26,1.7740,1.7914,0.98,2019-03-27 15:00

000031,,2019-03-26,1.5650,1.5709,0.38,2019-03-27 15:00

000048,C,2019-03-26,1.2230,1.2236,0.05,2019-03-27 15:00

000008,500ETFA,2019-03-26,1.4417,1.4552,0.93,2019-03-27 15:00

000024,A,2019-03-26,1.1670,1.1674,0.04,2019-03-27 15:00

000054,,2019-03-26,1.1697,1.1693,-0.03,2019-03-27 15:00

000016,C,2019-03-26,1.1790,1.1793,0.03,2019-03-27 15:00

	# python3pip install 
	import requests
	import random
	import re
	import queue
	import threading
	import csv
	import json


8.())

python


# 
cd 
# 
pip uninstall -y -r requirement.txt
# 
pip install -r requirement.txt
# 
python generate_wx_data.py

# pyinstaller
pip install pyinstaller
# 
cd 
# 
pip uninstall -y -r requirement.txt
# 
pip install -r requirement.txt
#  setuptools
pip install --upgrade setuptools
# 
pyinstaller generate_wx_data.py

9.QQ

QQQQQQ

QQQQ****

QQQQQQQQ

# 
cd 
# 
pip uninstall -y -r requirement.txt
# 
pip install -r requirement.txt
# 
python main.py


10.


()

()

# 
cd 
# 
pip uninstall -y -r requirement.txt
# 
pip install -r requirement.txt
# 
python main.py


11.(web)


ChromeChromeChromiumChromiumIEFirefoxSafari

URL

:http://39.106.118.77:8090()

# 
cd 
# 
pip uninstall -y -r requirement.txt
# 
pip install -r requirement.txt
# 
python app.py

# http://localhost:8090


12.

@arry-leewereader


2.150019-3560%80%/80%****

# 
cd 
# 
pip uninstall -y -r requirement.txt
# 
pip install -r requirement.txt -i https://pypi.tuna.tsinghua.edu.cn/simple
# 
python pyqt_gui.py


star



License

The MIT License (MIT)

Popular Crawler Projects
Popular Selenium Projects
Popular Data Processing Categories
Related Searches

Get A Weekly Email With Trending Projects For These Categories
No Spam. Unsubscribe easily at any time.
Python
Crawler
Selenium
Stock
Qq
Wechat
Spider
Multi Threading
Taobao