Awesome Open Source

Programming Languages

Search results for php crawler

188 search results found

Avbook ⭐ 8,777

AV 电影管理系统， avmoo , javbus , javlibrary 爬虫，线上 AV 影片图书馆，AV 磁力链接数据库，Japanese Adult Video Library,Adult Video Magnet Links - Japanese Adult Video Database

Wooyun_public ⭐ 3,701

This repo is archived. Thanks for wooyun! 乌云公开漏洞、知识库爬虫和搜索 crawl and search for wooyun.org public bug(vulnerability) and drops

Crawler ⭐ 2,417

An easy to use, powerful crawler implemented in PHP. Can execute Javascript.

Laravel Sitemap ⭐ 2,122

Create and generate sitemaps with ease

Crawler Detect ⭐ 1,842

🕷 CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent

Diskover Community ⭐ 1,391

Diskover Community Edition - Open source file indexer, file search engine and data management and analytics powered by Elasticsearch

Php Spider ⭐ 1,316

A configurable and extensible PHP web spider

The complete web scraping toolkit for PHP.

Beanbun ⭐ 1,195

Beanbun 是用 PHP 编写的多进程网络爬虫框架，具有良好的开放性、高可扩展性，基于 Workerman。

Http Status Check ⭐ 587

CLI tool to crawl a website and check HTTP status codes

Blackfire Player is a powerful Web Crawling, Web Testing, and Web Scraper application. It provides a nice DSL to crawl HTTP services, assert responses, and extract data from HTML/XML/JSON responses.

Hquery.php ⭐ 345

An extremely fast web scraper that parses megabytes of invalid HTML in a blink of an eye. PHP5.3+, no dependencies.

Dhtcrawler ⭐ 339

python 编写的DHT Crawler 网络爬虫，抓取磁力链接

Crawler ⭐ 285

Library for Rapid (Web) Crawler and Scraper Development

Curl Easy ⭐ 282

cURL wrapper for PHP. Supports parallel and non-blocking requests. For high speed crawling, see stil/curl-robot

Laravel Crawler Detect ⭐ 262

A Laravel wrapper for CrawlerDetect - the web crawler detection library

Laravel Site Search ⭐ 248

Create a full-text search index by crawling your site

Arachnid ⭐ 246

Crawl all unique internal links found on a given website, and extract SEO related information - supports javascript based sites

Awesome Crawler Cn ⭐ 243

互联网爬虫，蜘蛛，数据采集器，网页解析器的汇总，因新技术不断发展，新框架层出不穷，此文会不断更新..

Laravel ⭐ 238

Laravel adapter for Roach, the complete web scraping toolkit for PHP.

Sitemap Generator Crawler ⭐ 208

PHP script to recursively crawl websites and generate a sitemap. Zero dependencies.

Robots Txt ⭐ 201

Determine if a page may be crawled from robots.txt, robots meta tags and robot headers

Packagist Mirror ⭐ 189

📦✂️📋📦 Create a mirror of packagist.org metadata for use locally with composer

PHP Link Checker

Laravel Seo Scanner ⭐ 157

Scan your Laravel application routes for SEO improvements suggestions.

Grawler ⭐ 128

Grawler is a tool written in PHP which comes with a web interface that automates the task of using google dorks, scrapes the results, and stores them in a file.

An Open Source Search Engine

Php Crawler ⭐ 121

A php crawler that finds emails on the internets

一只优雅的正方教务系统爬虫。

Woothee Php ⭐ 112

Woothee PHP implementation

Phpcreeper ⭐ 108

A new generation of multi-process asynchronous event-driven spider engine based on Workerman. http://www.phpcreeper.com

Crawl Anywhere ⭐ 98

Crawl-Anywhere - Web Crawler and document processing pipeline with Solr integration.

Mixed Content Scanner ⭐ 95

Scan a HTTPS-site for mixed content

Crawler is a bare-bones spider designed to quickly and effectively build an index of all files and pages on a given Web site as well as the link relationship (both incoming and outgoing) between each page.

Xvideos Pornhub Redtube Api ⭐ 84

This script scrapes the HTML from different web pages to get the information from the video (XVideos, PornHub, RedTube) and you can use it in your own video player.

Instagram Parser ⭐ 82

📷 Crawl the Instagram website tags and users like if it was a simple API client.

Cakephp Hybridauth ⭐ 81

CakePHP plugin for HybridAuth

Tiktok Scraper Php ⭐ 69

Tiktok (Musically) PHP scraper

Sitemap Cache Warmer ⭐ 63

Visits pages based on a sitemap to keep your cache warm

Crawlzone ⭐ 60

Crawlzone is a fast asynchronous internet crawling framework for PHP.

Yuccaprerenderbundle ⭐ 56

Symfony2 Bundle to use prerender.io

Php Xml Sitemap Generator ⭐ 56

PHP Script that generates a sitemap by crawling a given URL.

Phpcrawl ⭐ 54

Copy of http://phpcrawl.cuab.de/ for using with composer

Diffbot Php Client ⭐ 53

[Deprecated - Maintenance mode - use APIs directly please!] The official Diffbot client library

Simple PHP function to detect search engines bots

Metacritic_api ⭐ 52

PHP Metacritic API - Mirror from my GitLab

Self-hosted search engine for your static blog

Libraries and scripts for crawling the TYPO3 page tree. Used for re-caching, re-indexing, publishing applications etc.

Tikscraperphp ⭐ 50

Wrapper for TikTok API

Packagist Crawler ⭐ 50

make mirror of https://packagist.org

Alipay Crawler ⭐ 47

支付宝账单爬虫

Dom Query ⭐ 46

A jQuery-like interface for DOM Crawling

Php Crawler ⭐ 45

PHP crawler and spider. working with UTF8, MySQL, Random host, Supports robots.txt and many more surprises

Querylist Phantomjs ⭐ 45

QueryList Plugin: Use PhantomJS to crawl Javascript dynamically rendered pages.(headless WebKit ) 使用PhantomJS采集JavaScript动态渲染的页面

Bear.resource ⭐ 45

A hypermedia framework for an object as a service

Opensearchserver Php Client ⭐ 44

OpenSearchServer PHP Client

Silverstripe Seo ⭐ 44

A SilverStripe module to optimise the Meta, crawling, indexing, and sharing of your website content

Instagram Php Crawler ⭐ 44

A simple PHP Crawler for Instagram

Asynccontrol ⭐ 43

Trait for asynchronous control rendering

Gwt_crawlerrors Php ⭐ 42

Download website crawl errors from Google Webmaster Tools as CSV.

Escargot ⭐ 41

A web crawler or spider library based on Symfony components

Piratebay Alfred Workflow ⭐ 40

workflow for Alfred App to crawl piratebay and return torrent files / magnet links

Caterpillar ⭐ 39

Caterpillar is a PHP library intended for website crawling and screen scraping. It handles parallel requests using the curl_multi functions.

PJAX for Laravel 4

Siteone Crawler ⭐ 38

SiteOne Crawler is a website analyzer and exporter you'll ♥ as a Dev/DevOps, QA engineer, website owner or consultant. Works on all popular platforms - Windows, macOS and Linux (x64 and arm64 too).

Cache Warmup ⭐ 37

🔥 PHP library to warm up caches of URLs located in XML sitemaps

Centipede Crawler ⭐ 35

Crawls all unique links.

Recipe Scraper ⭐ 35

A library for scraping recipes from popular recipe sites.

Laravel Block Bots ⭐ 34

Block crawlers and high traffic users on your site by IP using Redis

Podcastcrawler ⭐ 33

PHP library to find podcasts

Leboncoin Crawler ⭐ 32

Crawler for leboncoin.fr

Silverstripe Googleanalytics ⭐ 31

Google Analytics module for SilverStripe CMS

Googleplaywebserviceapi ⭐ 28

Tiny script to crawl information of a specific application in the Google play/store base on PHP.

Yurun Crawler ⭐ 28

宇润爬虫框架(Yurun Crawler) 是一个低代码、高性能、分布式爬虫采集框架，基于 imi 框架开发，运行在 Swoole 常驻内存的协程环境。

A middleware for rendering static pages when crawled by search engines

Codeigniter Sitemap Generator ⭐ 27

Generate a sitemap or sitemap index file for your web application with the CodeIgniter PHP framework.

Crazydhtspider ⭐ 25

Based on Swoole,a PHP DHT crawler, which have insane productivity（依托于swoole的PHP版本的DHT爬虫，有着奇高的效率）

Pimcore Lucene Search ⭐ 25

Pimcore Website Indexer (powered by Zend Search Lucene)

Crawl IMDb movie rating, content rating, release date, poster, presentation, director, actor, duration and other information

Zhihu Crawler ⭐ 23

轻量级知乎爬虫，支持问题、收藏夹和本月最热

php爬虫系统

Sundayforum ⭐ 23

a Forum Web Application base on Symfony framework

Doogle is a search engine and web crawler which can search indexed websites and images

Nivinedu ⭐ 22

拟物校园，一个开源的高校教务移动化解决方案。

Loghappens ⭐ 21

Bug happens. Every developer knows that. The bad thing is that if you want to see what happened you have to analyze hundreds of lines of log files, written in a format that is anything but human friendly... Know the feeling? Then logHappens is here for you!

Covid 19 Api ⭐ 20

Crawlerdetectbundle ⭐ 19

A Symfony bundle for the Crawler-Detect library (detects bots/crawlers/spiders via the user agent)

Mageruntools ⭐ 19

A module for Magerun to check the search engine optimization of a Magento website.

Proxycrawl Php ⭐ 19

ProxyCrawl PHP library for scraping and crawling websites

Dont Log Crawlers ⭐ 18

YOURLS plugin "Don't Log Crawlers"

Sitesearch ⭐ 18

Crawl and index a whole site

Php Sitemap ⭐ 18

PHP Simple Sitemap Generator

Mage2_ext_bot_sess ⭐ 18

Magento2: prevent session creation for bots & crawlers

Google Crawler ⭐ 18

A simple Crawler for getting Google results

Yii Community: Functional tests extension

A little crawler/sdk for retrieve in real time information about transport in Paris

Php Google ⭐ 17

Google search results crawler, get google search results that you need - php

Crawl your own website with various clients for SEO and indexing purposes.

Magento Full Page Cache Crawler ⭐ 17

Magento Full Page Cache Crawler

Water Melon ⭐ 16

Water Melon is simple melon.com api sdk for php

Related Searches

Php Laravel (34,445)

Php Wordpress (20,474)

Php Plugin (20,330)

Php Security (10,260)

Php Vulnerabilities (8,873)

Php Mysql (8,136)

Php Database (7,936)

Php Theme (6,771)

Javascript Php (6,406)

Php Docker (5,987)

1-100 of 188 search results

Privacy | About | Terms | Follow Us On Twitter

Copyright 2018-2024 Awesome Open Source. All rights reserved.