Frontendmasters Crawler
A demo of a serverless crawler built on AWS Lambda (scheduled tasks) and store results in S3
| Project Name | Stars | Downloads | Repos Using This | Packages Using This | Most Recent Commit | Total Releases | Latest Release | Open Issues | License | Language |
|---|---|---|---|---|---|---|---|---|---|---|
| Festin | 150 | 3 years ago | 15 | December 04, 2020 | bsd-3-clause | Python | ||||
| FestIn - S3 Bucket Weakness Discovery | ||||||||||
| Bucketlist | 68 | 7 years ago | 1 | mit | HTML | |||||
| Amazon S3 bucket spelunking! | ||||||||||
| Browsercrawler | 54 | 13 years ago | JavaScript | |||||||
| Crawl websites from your browser and save them in S3 | ||||||||||
| S3recon | 53 | 4 years ago | 12 | March 22, 2020 | 3 | mit | Python | |||
| Amazon S3 bucket finder and crawler. | ||||||||||
| Common_crawl_types | 28 | 12 years ago | Ruby | |||||||
| A simple Ruby example of how to process Common Crawl files using Elastic MapReduce | ||||||||||
| Staticizer | 27 | 5 years ago | 7 | December 05, 2018 | 3 | mit | Ruby | |||
| A tool to create a static version of a website for hosting on S3. | ||||||||||
| Nutch Aws | 23 | 9 years ago | 1 | Makefile | ||||||
| Amazon S3 Step Functions Ingestion Orchestration | 19 | 5 years ago | apache-2.0 | Python | ||||||
| Design pattern for orchestrating an incremental data ingestion pipeline using AWS Step Functions from an on premise location into an Amazon S3 datalake bucket | ||||||||||
| Googleplay Web Crawler | 15 | 7 years ago | Java | |||||||
| Mapreduce project by Hadoop, Nutch, AWS EMR, Pig, Tez, Hive | ||||||||||
| Pandemic Knowledge | 8 | 3 years ago | 1 | mit | Python | |||||
| A fully-featured multi-source data pipeline for continuously extracting knowledge from COVID-19 data. | ||||||||||
Alternatives To Frontendmasters CrawlerSelect To Compare
Alternative Project Comparisons
Popular S3 Projects
Popular Crawler Projects
Popular Cloud Computing Categories
Related Searches
Get A Weekly Email With Trending Projects For These Categories
No Spam. Unsubscribe easily at any time.
Javascript
Amazon Web Services
Lambda Functions
Serverless
Crawler
S3