Awesome Open Source

Programming Languages

Search results for python crawl

2,523 search results found

Scrapy ⭐ 47,417

Scrapy, a fast high-level web crawling & scraping framework for Python.

Proxy_pool ⭐ 18,014

Python爬虫代理IP池(proxy pool)

Pyspider ⭐ 15,819

A Powerful Spider(Web Crawler) System in Python.

Newspaper ⭐ 12,678

News, full-text, and article metadata extraction in Python 3. Advanced docs:

Tushare ⭐ 12,165

TuShare is a utility for crawling historical data of China stocks

Examples Of Web Crawlers ⭐ 11,050

一些非常有趣的python爬虫例子,对新手比较友好,主要爬取淘宝、天猫、微信、微信读书、豆瓣、QQ等 interesting examples of python crawlers that are friendly to beginners. )

Photon ⭐ 9,272

Incredibly fast crawler designed for OSINT.

Python ⭐ 8,296

Python脚本。模拟登录知乎，爬虫，操作excel，微信公众号，远程开机

Infospider ⭐ 6,649

INFO-SPIDER 是一个集众多数据源于一身的爬虫工具箱🧰，旨在安全快捷的帮助用户拿回自己的数据，工具代码开源，流程透

Wechatsogou ⭐ 5,495

基于搜狗微信搜索的微信公众号爬虫接口

Scrapy Redis ⭐ 5,286

Redis-based components for Scrapy.

Autoscraper ⭐ 5,159

A Smart, Automatic, Fast and Lightweight Web Scraper for Python

Proxypool ⭐ 4,411

An Efficient ProxyPool with Getter, Tester and Server

Mygptreader ⭐ 4,019

A community-driven way to read and chat with AI bots - powered by chatGPT.

Interesting Python ⭐ 3,748

有趣的Python爬虫和Python数据分析小项目(Some interesting Python crawlers and data analysis projects)

Scylla ⭐ 3,729

Intelligent proxy pool for Humans™

Ecommercecrawlers ⭐ 3,724

实战🐍多种网站、电商数据爬虫🕷。包含🕸：淘宝商品、微信公众号、大众点评、企查查、招聘网站、闲鱼

Proxybroker ⭐ 3,367

Proxy [Finder | Checker | Server]. HTTP(S) & SOCKS 🎭

Distribute_crawler ⭐ 3,176

使用scrapy,redis, mongodb,graphite实现的一个分布式网络爬虫,底层存储mongodb集群,分布式使用re

Toapi ⭐ 3,153

Every web site provides APIs.

Gerapy ⭐ 2,987

Distributed Crawler Management Framework Based on Scrapy, Scrapyd, Django and Vue.js

Douyin_tiktok_download_api ⭐ 2,816

🚀「Douyin_TikTok_Download_API」是一个开箱即用的高性能异步抖音|TikT

Googlescraper ⭐ 2,495

A Python module to scrape several search engines (like Google, Yandex, Bing, Duckduckgo, ...). Including asynchronous networking support.

Python3 Spider ⭐ 2,440

Python爬虫实战 - 模拟登陆各大网站包含但不限于：滑块验证、拼多多、美团、百度、bilibili、大众点评、淘宝，如果喜欢请start ❤️

Weibo Crawler ⭐ 2,391

新浪微博爬虫，用python爬取新浪微博数据，并下载微博图片和微博视频

Decryptlogin ⭐ 2,375

DecryptLogin: APIs for loginning some websites by using requests.

Owllook ⭐ 2,340

owllook-小说搜索引擎

Dirmap ⭐ 2,325

An advanced web directory & file scanning tool that will be more powerful than DirBuster, Dirsearch, cansina, and Yu Jian.一个高级web目录、文件扫描工具，功能将会强于DirBuster、Dirsearch、ca

Weibo_terminater ⭐ 2,265

Final Weibo Crawler Scrap Anything From Weibo, comments, weibo contents, followers, anything. The Terminator

Web Scraping Framework

Torbot ⭐ 1,980

Dark Web OSINT Tool

Web crawling framework based on asyncio.

Feapder ⭐ 1,815

🚀🚀🚀feapder is an easy to use, powerful crawler framework | feapder是一款上手简单，功能强大的Python爬虫框架。内置AirSpider、Spider、

Domain_analyzer ⭐ 1,747

Analyze the security of any domain by finding all the information possible. Made in python.

Xalpha ⭐ 1,736

基金投资管理回测引擎

Finalrecon ⭐ 1,728

The Last Web Recon Tool You'll Need

Lianjia Beike Spider ⭐ 1,695

链家网和贝壳网房价爬虫，采集北京上海广州深圳等21个中国主要城市的房价数据（小区，二手房，出租房，新 MongoDB,Excel, json存储，支持Python2和3，图表展示数据，注释丰富 🚁，点星支持

Pspider ⭐ 1,675

简单易用的Python爬虫框架，QQ交流群：597510560

Async Python 3.6+ web scraping micro-framework based on asyncio

Vulnx ⭐ 1,632

vulnx 🕷️ an intelligent Bot, Shell can achieve automatic injection, and help researchers detect security vulnerabilities CMS system. It can perform a quick CMS security detection, information collection (including sub-domain name, ip address, country information, organizational information and time zone, etc.) and vulnerability scanning.

News Please ⭐ 1,623

news-please - an integrated web crawler and information extractor for news that just works

Python Crawler ⭐ 1,576

从头开始系统化的学习如何写Python爬虫。 Python版本 3.6

Dirhunt ⭐ 1,450

Find web directories without bruteforce

Xsscrapy ⭐ 1,398

XSS spider - 66/66 wavsep XSS detected

Autocrawler ⭐ 1,387

Google, Naver multiprocess image web crawler (Selenium)

Diskover Community ⭐ 1,286

Diskover Community Edition - Open source file indexer, file search engine and data management and analytics powered by Elasticsearch

Openwpm ⭐ 1,278

A web privacy measurement framework

Jd Autobuy ⭐ 1,263

Python爬虫，京东自动登录，在线抢购商品

Bilix ⭐ 1,160

⚡️Lightning-fast async download tool for bilibili and more | 快如闪电的异步下载工具，支持bilibili及更多

Weixin ⭐ 1,154

微信小游戏辅助合集（加减大师、包你懂我、大家来找茬腾讯版、头脑王者、好友画我、悦动音符、我最在行、星

Sotawhat ⭐ 1,154

Returns latest research results by crawling arxiv papers and summarizing abstracts. Helps you stay afloat with so many new papers everyday.

Tumblr Crawler ⭐ 1,105

Easily download all the photos/videos from tumblr blogs. 下载指定的 Tumblr 博客中的图片，视频

Frontera ⭐ 1,084

A scalable frontier for web crawlers

Parliament Scraper ⭐ 1,049

Public Data Scraper for Parliament Data for the EU and other Parliaments

Trafilatura ⭐ 1,044

Python & command-line tool to gather text on the Web: web crawling/scraping, extraction of text, metadata, comments

Scrapy Cluster ⭐ 1,016

This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.

Instagram Profilecrawl ⭐ 1,001

📝 quickly crawl the information (e.g. followers, tags etc...) of an instagram profile.

Crawler User Agents ⭐ 956

Syntactic patterns of HTTP user-agents used by bots / robots / crawlers / scrapers / spiders. pull-request welcome ⭐️

Lightnovel Crawler ⭐ 954

Generate and download e-books from online sources.

Lxspider ⭐ 938

爬虫案例合集。包括但不限于《淘宝、京东、天猫、豆瓣、抖音、快手、微博、微信、阿里、头条、pdd、优酷

Mlscraper ⭐ 935

🤖 Scrape data from HTML websites automatically by just providing examples

Grab Site ⭐ 935

The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns

Instagram Crawler ⭐ 922

Get Instagram posts/profile/hashtag data without using Instagram API

Python website crawler.

Crawlergo_x_xray ⭐ 915

360/0Kee-Team/crawlergo动态爬虫结合长亭XRAY扫描器的被动扫描功能

🍻 bilibili video (including bangumi) and danmaku downloader | B站视频（含番剧）、弹幕下载器

Python Seo Analyzer ⭐ 878

An SEO tool that analyzes the structure of a site, crawls the site, count words in the body of the site and warns of any technical SEO issues.

Angrysearch ⭐ 866

Linux file search, instant results as you type

Querido Diario ⭐ 860

📰 Brazilian government gazettes, accessible to everyone.

👧 美女写真套图爬虫（二）

Bhban_rpa ⭐ 793

<6개월 치 업무를 하루 만에 끝내는 업무 자동화(생능출판사, 2020)>의 예제 코드입니다. 파이썬을 한 번도 배워본 적 없는 분들을 위한 예제이며, 엑셀부터 디자인, 매크로, 크롤링까지 업무 자동화와 관련된 다양한 분야 예제가 제공됩니다.

Baiduimagespider ⭐ 774

一个超级轻量的百度图片爬虫

Holiday Cn ⭐ 761

📅🇨🇳中国法定节假日数据自动每日抓取国务院公告

[Unmaintained] A simple and clean video/music/image downloader 👾

Icrawler ⭐ 749

A multi-thread crawler framework with many builtin image crawlers provided.

Packtpub Crawler ⭐ 701

Download your daily free Packt Publishing eBook https://www.packtpub.com/packt/offers/free-learnin

Scrapyrt ⭐ 701

HTTP API for Scrapy spiders

Scrapy Selenium ⭐ 699

Scrapy middleware to handle javascript pages using selenium

Tweetscraper ⭐ 698

TweetScraper is a simple crawler/spider for Twitter Search without using API

Listed Company News Crawl And Text Analysis ⭐ 689

从新浪财经、每经网、金融界、中国证券网、证券时报网上，爬取上市公司（个股）的历史新闻文本数据进行文本

Gyoithon ⭐ 683

GyoiThon is a growing penetration test tool using Machine Learning.

直接連線登入的 PTT library，支援 PTT, PTT2

One Python ⭐ 655

We don't need a lot of libraries. We just need the best ones. | Unofficial recommended first choice.

Word2vec Graph ⭐ 650

Exploring word2vec embeddings as a graph of nearest neighbors

Public Amazon Crawler ⭐ 645

Course Crawler ⭐ 633

🎓 中国大学MOOC、学堂在线、网易云课堂、好大学在线、爱课程 MOOC 课程下载。

英语字典英语词库字典词库四级单词六级单词考研单词雅思托福 SAT GMAT TOEFL GRE

Easy Scraping Tutorial ⭐ 618

Simple but useful Python web scraping tutorial code.

Baiduspider ⭐ 611

BaiduSpider，一个爬取百度搜索结果的爬虫，目前支持百度网页搜索，百度图片搜索，百度知道搜索

Tools to download and cleanup Common Crawl data

Xehentai ⭐ 591

Doujinshi downloader 绅士漫画下载

Magnet Dht ⭐ 591

✌️ Python3 BitTorrent DHT crawler

Brozzler ⭐ 578

brozzler - distributed browser-based web crawler

Chatweb ⭐ 573

ChatWeb can crawl web pages, read PDF, DOCX, TXT, and extract the main content, then answer your questions based on the content, or summarize the key points.

API of DouYin for Humans used to Crawl Popular Videos and Musics

Python Gems ⭐ 544

Beautifully constructed python scripts

Google Play Scraper ⭐ 543

Google play scraper for Python inspired by <facundoolano/google-play-scraper>

Bookcorpus ⭐ 536

Crawl BookCorpus

a new crawler based on python with more function including Network fingerprint search

Computerstudent ⭐ 496

计算机专业系统性学习资料（python,c,c++,计算机组成，计算机网络，编译原理，电路，谷歌插件

Related Searches

Python Python3 (857,414)

Python Flask (16,475)

Python Docker (14,113)

Python Machine Learning (14,099)

Python Command Line (13,209)

Python Deep Learning (13,092)

Python Network (11,547)

Python Algorithms (9,749)

Python Django (8,165)

Python Server (7,730)

1-100 of 2,523 search results

Privacy | About | Terms | Follow Us On Twitter

Copyright 2018-2023 Awesome Open Source. All rights reserved.