Awesome Open Source

Programming Languages

Search results for python crawler

1,245 search results found

Scrapy ⭐ 49,918

Scrapy, a fast high-level web crawling & scraping framework for Python.

Proxy_pool ⭐ 19,442

Python ProxyPool for web spider

Pyspider ⭐ 15,943

A Powerful Spider(Web Crawler) System in Python.

Newspaper ⭐ 13,147

News, full-text, and article metadata extraction in Python 3. Advanced docs:

Examples Of Web Crawlers ⭐ 13,142

一些非常有趣的python爬虫例子,对新手比较友好,主要爬取淘宝、天猫、微信、微信读书、豆瓣、QQ等 interesting examples of python crawlers that are friendly to beginners. )

Tushare ⭐ 12,165

TuShare is a utility for crawling historical data of China stocks

Photon ⭐ 10,244

Incredibly fast crawler designed for OSINT.

Python ⭐ 9,097

Python脚本。模拟登录知乎，爬虫，操作excel，微信公众号，远程开机

Infospider ⭐ 6,856

INFO-SPIDER 是一个集众多数据源于一身的爬虫工具箱🧰，旨在安全快捷的帮助用户拿回自己的数据，工具代码开源，流程透

Wechatsogou ⭐ 5,777

基于搜狗微信搜索的微信公众号爬虫接口

Autoscraper ⭐ 5,159

A Smart, Automatic, Fast and Lightweight Web Scraper for Python

Proxypool ⭐ 5,154

An Efficient ProxyPool with Getter, Tester and Server

Douyin_tiktok_download_api ⭐ 4,844

🚀「Douyin_TikTok_Download_API」是一个开箱即用的高性能异步抖音、快手、T

Mygptreader ⭐ 4,267

A community-driven way to read and chat with AI bots - powered by chatGPT.

Scylla ⭐ 3,819

Intelligent proxy pool for Humans™ to extract content from the internet and build your own Large Language Models in this new AI era

Interesting Python ⭐ 3,748

有趣的Python爬虫和Python数据分析小项目(Some interesting Python crawlers and data analysis projects)

Ecommercecrawlers ⭐ 3,724

实战🐍多种网站、电商数据爬虫🕷。包含🕸：淘宝商品、微信公众号、大众点评、企查查、招聘网站、闲鱼

Proxybroker ⭐ 3,561

Proxy [Finder | Checker | Server]. HTTP(S) & SOCKS 🎭

Toapi ⭐ 3,417

Every web site provides APIs.

Distribute_crawler ⭐ 3,176

使用scrapy,redis, mongodb,graphite实现的一个分布式网络爬虫,底层存储mongodb集群,分布式使用re

Toapi ⭐ 3,153

Every web site provides APIs.

Gerapy ⭐ 3,144

Distributed Crawler Management Framework Based on Scrapy, Scrapyd, Django and Vue.js

Weibo Crawler ⭐ 2,820

新浪微博爬虫，用python爬取新浪微博数据，并下载微博图片和微博视频

Python3 Spider ⭐ 2,582

Python爬虫实战 - 模拟登陆各大网站包含但不限于：滑块验证、拼多多、美团、百度、bilibili、大众点评、淘宝，如果喜欢请start ❤️

Googlescraper ⭐ 2,540

A Python module to scrape several search engines (like Google, Yandex, Bing, Duckduckgo, ...). Including asynchronous networking support.

Lianjia Beike Spider ⭐ 2,464

链家网和贝壳网房价爬虫，采集北京上海广州深圳等21个中国主要城市的房价数据（小区，二手房，出租房，新 MongoDB,Excel, json存储，支持Python2和3，图表展示数据，注释丰富，点星支持，仅供学习参考，请勿用于商业用途，后果自负。

Trafilatura ⭐ 2,447

Python & command-line tool to gather text on the Web: web crawling/scraping, extraction of text, metadata, comments

Decryptlogin ⭐ 2,375

DecryptLogin: APIs for loginning some websites by using requests.

Owllook ⭐ 2,340

owllook-小说搜索引擎

Torbot ⭐ 2,338

Dark Web OSINT Tool

Feapder ⭐ 2,333

🚀🚀🚀feapder is an easy to use, powerful crawler framework | feapder是一款上手简单，功能强大的Python爬虫框架。内置AirSpider、Spider、

Web Scraping Framework

Weibo_terminater ⭐ 2,265

Final Weibo Crawler Scrap Anything From Weibo, comments, weibo contents, followers, anything. The Terminator

Web crawling framework based on asyncio.

Web crawling framework based on asyncio.

Finalrecon ⭐ 1,949

All In One Web Recon

Xalpha ⭐ 1,851

基金投资管理回测引擎

News Please ⭐ 1,821

news-please - an integrated web crawler and information extractor for news that just works

Domain_analyzer ⭐ 1,747

Analyze the security of any domain by finding all the information possible. Made in python.

Async Python 3.6+ web scraping micro-framework based on asyncio

Dirhunt ⭐ 1,677

Find web directories without bruteforce

Pspider ⭐ 1,675

简单易用的Python爬虫框架，QQ交流群：597510560

Python Crawler ⭐ 1,576

从头开始系统化的学习如何写Python爬虫。 Python版本 3.6

Autocrawler ⭐ 1,454

Google, Naver multiprocess image web crawler (Selenium)

Bilix ⭐ 1,433

⚡️Lightning-fast async download tool for bilibili and more | 快如闪电的异步下载工具，支持bilibili及更多

Xsscrapy ⭐ 1,398

XSS spider - 66/66 wavsep XSS detected

Diskover Community ⭐ 1,391

Diskover Community Edition - Open source file indexer, file search engine and data management and analytics powered by Elasticsearch

Weixin Game Helper ⭐ 1,352

微信小游戏辅助合集（加减大师、包你懂我、大家来找茬腾讯版、头脑王者、好友画我、悦动音符、我最在行、星

Openwpm ⭐ 1,307

A web privacy measurement framework

Jd Autobuy ⭐ 1,291

Python爬虫，京东自动登录，在线抢购商品

Sotawhat ⭐ 1,280

Returns latest research results by crawling arxiv papers and summarizing abstracts. Helps you stay afloat with so many new papers everyday.

Lxspider ⭐ 1,267

爬虫案例合集。包括但不限于《淘宝、京东、天猫、豆瓣、抖音、快手、微博、微信、阿里、头条、pdd、优酷

Frontera ⭐ 1,244

A scalable frontier for web crawlers

Lightnovel Crawler ⭐ 1,185

Generate and download e-books from online sources.

Scrapy Cluster ⭐ 1,137

This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.

Tumblr Crawler ⭐ 1,105

Easily download all the photos/videos from tumblr blogs. 下载指定的 Tumblr 博客中的图片，视频

Parliament Scraper ⭐ 1,049

Public Data Scraper for Parliament Data for the EU and other Parliaments

Crawler User Agents ⭐ 1,045

Syntactic patterns of HTTP user-agents used by bots / robots / crawlers / scrapers / spiders. pull-request welcome ⭐

Holiday Cn ⭐ 1,018

📅🇨🇳中国法定节假日数据自动每日抓取国务院公告

Instagram Profilecrawl ⭐ 1,001

📝 quickly crawl the information (e.g. followers, tags etc...) of an instagram profile.

🍻 bilibili video (including bangumi) and danmaku downloader | B站视频（含番剧）、弹幕下载器

Python Seo Analyzer ⭐ 956

An SEO tool that analyzes the structure of a site, crawls the site, count words in the body of the site and warns of any technical SEO issues.

Querido Diario ⭐ 944

📰 Diários oficiais brasileiros acessíveis a todos | 📰 Brazilian government gazettes, accessible to everyone.

Mlscraper ⭐ 935

🤖 Scrape data from HTML websites automatically by just providing examples

Instagram Crawler ⭐ 922

Get Instagram posts/profile/hashtag data without using Instagram API

Python website crawler.

Crawlergo_x_xray ⭐ 915

360/0Kee-Team/crawlergo动态爬虫结合长亭XRAY扫描器的被动扫描功能

Bhban_rpa ⭐ 903

<6개월 치 업무를 하루 만에 끝내는 업무 자동화(생능출판사, 2020)>의 예제 코드입니다. 파이썬을 한 번도 배워본 적 없는 분들을 위한 예제이며, 엑셀부터 디자인, 매크로, 크롤링까지 업무 자동화와 관련된 다양한 분야 예제가 제공됩니다.

Baiduspider ⭐ 872

BaiduSpider，一个爬取百度搜索结果的爬虫，目前支持百度网页搜索，百度图片搜索，百度知道搜索

Angrysearch ⭐ 866

Linux file search, instant results as you type

👧 美女写真套图爬虫（二）

Scrapy Selenium ⭐ 842

Scrapy middleware to handle javascript pages using selenium

Scrapyrt ⭐ 793

HTTP API for Scrapy spiders

Icrawler ⭐ 792

A multi-thread crawler framework with many builtin image crawlers provided.

Baiduimagespider ⭐ 774

一个超级轻量的百度图片爬虫

Computerstudent ⭐ 764

计算机专业系统性学习资料（python,c,c++,计算机组成，计算机网络，编译原理，电路，谷歌插件

Spider_collection ⭐ 754

python爬虫，目前库存：网易云音乐歌曲爬取，B站视频爬取，知乎问答爬取，壁纸爬取，xvideos

[Unmaintained] A simple and clean video/music/image downloader 👾

Packtpub Crawler ⭐ 701

Download your daily free Packt Publishing eBook https://www.packtpub.com/packt/offers/free-learnin

Bookcorpus ⭐ 698

Crawl BookCorpus

Tweetscraper ⭐ 698

TweetScraper is a simple crawler/spider for Twitter Search without using API

Listed Company News Crawl And Text Analysis ⭐ 689

从新浪财经、每经网、金融界、中国证券网、证券时报网上，爬取上市公司（个股）的历史新闻文本数据进行文本

直接連線登入的 PTT library，支援 PTT, PTT2

One Python ⭐ 655

We don't need a lot of libraries. We just need the best ones. | Unofficial recommended first choice.

Word2vec Graph ⭐ 650

Exploring word2vec embeddings as a graph of nearest neighbors

Google Play Scraper ⭐ 645

Google play scraper for Python inspired by <facundoolano/google-play-scraper>

Public Amazon Crawler ⭐ 645

Course Crawler ⭐ 633

🎓 中国大学MOOC、学堂在线、网易云课堂、好大学在线、爱课程 MOOC 课程下载。

英语字典英语词库字典词库四级单词六级单词考研单词雅思托福 SAT GMAT TOEFL GRE

Hacker News Digest ⭐ 620

📰 Let ChatGPT Summarize Hacker News for You

Easy Scraping Tutorial ⭐ 618

Simple but useful Python web scraping tutorial code.

Brozzler ⭐ 613

brozzler - distributed browser-based web crawler

Xehentai ⭐ 591

Doujinshi downloader 绅士漫画下载

Magnet Dht ⭐ 591

✌️ Python3 BitTorrent DHT crawler

Chatweb ⭐ 573

ChatWeb can crawl web pages, read PDF, DOCX, TXT, and extract the main content, then answer your questions based on the content, or summarize the key points.

Python Fxxk Spider ⭐ 571

收集各种免费的 Python 爬虫项目

API of DouYin for Humans used to Crawl Popular Videos and Musics

Python Gems ⭐ 544

Beautifully constructed python scripts

基于小红书 Web 端进行的请求封装。https://reajason.github.io/xhs/

a new crawler based on python with more function including Network fingerprint search

Related Searches

Python Django (28,897)

Python Deep Learning (20,143)

Python Flask (17,643)

Python Docker (14,113)

Python Machine Learning (14,099)

Python Command Line (13,351)

Python Network (11,495)

Python Html (10,924)

Python Algorithms (10,033)

Python Natural Language Processing (9,064)

1-100 of 1,245 search results

Privacy | About | Terms | Follow Us On Twitter

Copyright 2018-2024 Awesome Open Source. All rights reserved.