Awesome Open Source

Programming Languages

Search results for python crawler

1,005 search results found

Scrapy ⭐ 49,918

Scrapy, a fast high-level web crawling & scraping framework for Python.

Proxy_pool ⭐ 19,442

Python ProxyPool for web spider

Pyspider ⭐ 15,943

A Powerful Spider(Web Crawler) System in Python.

Examples Of Web Crawlers ⭐ 13,142

一些非常有趣的python爬虫例子,对新手比较友好,主要爬取淘宝、天猫、微信、微信读书、豆瓣、QQ等 interesting examples of python crawlers that are friendly to beginners. )

Tushare ⭐ 12,165

TuShare is a utility for crawling historical data of China stocks

Photon ⭐ 10,244

Incredibly fast crawler designed for OSINT.

Python ⭐ 9,097

Python脚本。模拟登录知乎，爬虫，操作excel，微信公众号，远程开机

Wechatsogou ⭐ 5,822

基于搜狗微信搜索的微信公众号爬虫接口

Autoscraper ⭐ 5,159

A Smart, Automatic, Fast and Lightweight Web Scraper for Python

Proxypool ⭐ 5,154

An Efficient ProxyPool with Getter, Tester and Server

Douyin_tiktok_download_api ⭐ 4,844

🚀「Douyin_TikTok_Download_API」是一个开箱即用的高性能异步抖音、快手、T

Mygptreader ⭐ 4,267

A community-driven way to read and chat with AI bots - powered by chatGPT.

Scylla ⭐ 3,819

Intelligent proxy pool for Humans™ to extract content from the internet and build your own Large Language Models in this new AI era

Ecommercecrawlers ⭐ 3,724

实战🐍多种网站、电商数据爬虫🕷。包含🕸：淘宝商品、微信公众号、大众点评、企查查、招聘网站、闲鱼

Toapi ⭐ 3,417

Every web site provides APIs.

Distribute_crawler ⭐ 3,176

使用scrapy,redis, mongodb,graphite实现的一个分布式网络爬虫,底层存储mongodb集群,分布式使用re

Toapi ⭐ 3,153

Every web site provides APIs.

Gerapy ⭐ 3,144

Distributed Crawler Management Framework Based on Scrapy, Scrapyd, Django and Vue.js

Weibo Crawler ⭐ 2,820

新浪微博爬虫，用python爬取新浪微博数据，并下载微博图片和微博视频

Python3 Spider ⭐ 2,582

Python爬虫实战 - 模拟登陆各大网站包含但不限于：滑块验证、拼多多、美团、百度、bilibili、大众点评、淘宝，如果喜欢请start ❤️

Googlescraper ⭐ 2,540

A Python module to scrape several search engines (like Google, Yandex, Bing, Duckduckgo, ...). Including asynchronous networking support.

Lianjia Beike Spider ⭐ 2,464

链家网和贝壳网房价爬虫，采集北京上海广州深圳等21个中国主要城市的房价数据（小区，二手房，出租房，新 MongoDB,Excel, json存储，支持Python2和3，图表展示数据，注释丰富，点星支持，仅供学习参考，请勿用于商业用途，后果自负。

Trafilatura ⭐ 2,447

Python & command-line tool to gather text on the Web: web crawling/scraping, extraction of text, metadata, comments

Decryptlogin ⭐ 2,375

DecryptLogin: APIs for loginning some websites by using requests.

Owllook ⭐ 2,340

owllook-小说搜索引擎

Feapder ⭐ 2,333

🚀🚀🚀feapder is an easy to use, powerful crawler framework | feapder是一款上手简单，功能强大的Python爬虫框架。内置AirSpider、Spider、

Web Scraping Framework

Weibo_terminater ⭐ 2,265

Final Weibo Crawler Scrap Anything From Weibo, comments, weibo contents, followers, anything. The Terminator

Finalrecon ⭐ 2,054

All In One Web Recon

Web crawling framework based on asyncio.

Web crawling framework based on asyncio.

Xalpha ⭐ 1,851

基金投资管理回测引擎

News Please ⭐ 1,821

news-please - an integrated web crawler and information extractor for news that just works

Domain_analyzer ⭐ 1,747

Analyze the security of any domain by finding all the information possible. Made in python.

Dirhunt ⭐ 1,747

Find web directories without bruteforce

Async Python 3.6+ web scraping micro-framework based on asyncio

Pspider ⭐ 1,675

简单易用的Python爬虫框架，QQ交流群：597510560

Python Crawler ⭐ 1,576

从头开始系统化的学习如何写Python爬虫。 Python版本 3.6

Autocrawler ⭐ 1,454

Google, Naver multiprocess image web crawler (Selenium)

Bilix ⭐ 1,433

⚡️Lightning-fast async download tool for bilibili and more | 快如闪电的异步下载工具，支持bilibili及更多

Weixin Game Helper ⭐ 1,352

微信小游戏辅助合集（加减大师、包你懂我、大家来找茬腾讯版、头脑王者、好友画我、悦动音符、我最在行、星

Openwpm ⭐ 1,330

A web privacy measurement framework

Jd Autobuy ⭐ 1,300

Python爬虫，京东自动登录，在线抢购商品

Sotawhat ⭐ 1,280

Returns latest research results by crawling arxiv papers and summarizing abstracts. Helps you stay afloat with so many new papers everyday.

Lxspider ⭐ 1,267

爬虫案例合集。包括但不限于《淘宝、京东、天猫、豆瓣、抖音、快手、微博、微信、阿里、头条、pdd、优酷

Frontera ⭐ 1,244

A scalable frontier for web crawlers

Lightnovel Crawler ⭐ 1,185

Generate and download e-books from online sources.

Scrapy Cluster ⭐ 1,137

This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.

Parliament Scraper ⭐ 1,049

Public Data Scraper for Parliament Data for the EU and other Parliaments

Crawler User Agents ⭐ 1,045

Syntactic patterns of HTTP user-agents used by bots / robots / crawlers / scrapers / spiders. pull-request welcome ⭐

Holiday Cn ⭐ 1,018

📅🇨🇳中国法定节假日数据自动每日抓取国务院公告

Instagram Profilecrawl ⭐ 1,001

📝 quickly crawl the information (e.g. followers, tags etc...) of an instagram profile.

Python Seo Analyzer ⭐ 956

An SEO tool that analyzes the structure of a site, crawls the site, count words in the body of the site and warns of any technical SEO issues.

Mlscraper ⭐ 935

🤖 Scrape data from HTML websites automatically by just providing examples

Instagram Crawler ⭐ 922

Get Instagram posts/profile/hashtag data without using Instagram API

Python website crawler.

Baiduspider ⭐ 872

BaiduSpider，一个爬取百度搜索结果的爬虫，目前支持百度网页搜索，百度图片搜索，百度知道搜索

Angrysearch ⭐ 866

Linux file search, instant results as you type

👧 美女写真套图爬虫（二）

Scrapy Selenium ⭐ 842

Scrapy middleware to handle javascript pages using selenium

Scrapyrt ⭐ 793

HTTP API for Scrapy spiders

Icrawler ⭐ 792

A multi-thread crawler framework with many builtin image crawlers provided.

Baiduimagespider ⭐ 774

一个超级轻量的百度图片爬虫

Computerstudent ⭐ 764

计算机专业系统性学习资料（python,c,c++,计算机组成，计算机网络，编译原理，电路，谷歌插件

Spider_collection ⭐ 754

python爬虫，目前库存：网易云音乐歌曲爬取，B站视频爬取，知乎问答爬取，壁纸爬取，xvideos

[Unmaintained] A simple and clean video/music/image downloader 👾

Packtpub Crawler ⭐ 701

Download your daily free Packt Publishing eBook https://www.packtpub.com/packt/offers/free-learnin

Bookcorpus ⭐ 698

Crawl BookCorpus

Tweetscraper ⭐ 698

TweetScraper is a simple crawler/spider for Twitter Search without using API

Listed Company News Crawl And Text Analysis ⭐ 689

从新浪财经、每经网、金融界、中国证券网、证券时报网上，爬取上市公司（个股）的历史新闻文本数据进行文本

One Python ⭐ 655

We don't need a lot of libraries. We just need the best ones. | Unofficial recommended first choice.

Word2vec Graph ⭐ 650

Exploring word2vec embeddings as a graph of nearest neighbors

Public Amazon Crawler ⭐ 645

Google Play Scraper ⭐ 645

Google play scraper for Python inspired by <facundoolano/google-play-scraper>

Course Crawler ⭐ 633

🎓 中国大学MOOC、学堂在线、网易云课堂、好大学在线、爱课程 MOOC 课程下载。

英语字典英语词库字典词库四级单词六级单词考研单词雅思托福 SAT GMAT TOEFL GRE

Easy Scraping Tutorial ⭐ 618

Simple but useful Python web scraping tutorial code.

Brozzler ⭐ 613

brozzler - distributed browser-based web crawler

Magnet Dht ⭐ 591

✌️ Python3 BitTorrent DHT crawler

Python Fxxk Spider ⭐ 571

收集各种免费的 Python 爬虫项目

API of DouYin for Humans used to Crawl Popular Videos and Musics

基于小红书 Web 端进行的请求封装。https://reajason.github.io/xhs/

a new crawler based on python with more function including Network fingerprint search

swiss army knife for hackers

Personrelationknowledgegraph ⭐ 480

ChinesePersonRelationGraph, person relationship extraction based on nlp methods.中文人物关系知识图谱项目,内容包括中文人物关系图谱构建,基于知识库的数据回标,基于远

Sentiment Analysis In Event Driven Stock Price Movement Prediction ⭐ 462

Use NLP to predict stock price movement associated with news

Convert HTML to Markdown.

Pywebcopy ⭐ 455

Locally saves webpages to your hard disk with images, css, js & links as is.

Scrapple ⭐ 452

A framework for creating semi-automatic web content extractors

Awesome Scrapy ⭐ 450

A curated list of awesome packages, articles, and other cool resources from the Scrapy community.

Learnpython ⭐ 437

Python的基础练习代码与各种爬虫代码

Movie metadata scraper

《爬虫逆向进阶实战》书籍代码库

Malspider ⭐ 425

Malspider is a web spidering framework that detects characteristics of web compromises.

Fbcrawl ⭐ 415

A Facebook crawler

dude uncomplicated data extraction: A simple framework for writing web scrapers using Python decorators

Music Recover ⭐ 397

🎵 缓存文件转换为 MP3 文件

Scrapybook ⭐ 378

Scrapy Book Code

Search Engines Scraper ⭐ 377

Search google, bing, yahoo, and other search engines with python

Instagramcrawler ⭐ 373

A non API python program to crawl public photos, posts or followers

Related Searches

Python Django (28,897)

Python Deep Learning (20,143)

Python Flask (17,643)

Python Docker (14,113)

Python Machine Learning (14,099)

Python Command Line (13,351)

Python Network (11,495)

Python Html (10,924)

Python Algorithms (10,033)

Python Natural Language Processing (9,064)

1-100 of 1,005 search results

Privacy | About | Terms | Follow Us On Twitter

Copyright 2018-2024 Awesome Open Source. All rights reserved.