Crawling Infrastructure
Distributed crawling infrastructure running on top of severless computation, cloud storage (such as S3) and sophisticated queues.
| Project Name | Stars | Downloads | Repos Using This | Packages Using This | Most Recent Commit | Total Releases | Latest Release | Open Issues | License | Language |
|---|---|---|---|---|---|---|---|---|---|---|
| Crawling Infrastructure | 321 | 2 years ago | 22 | agpl-3.0 | TypeScript | |||||
| Distributed crawling infrastructure running on top of severless computation, cloud storage (such as S3) and sophisticated queues. | ||||||||||
| Stopstalk Deployment | 306 | 5 months ago | 92 | mit | Python | |||||
| Stop stalking and start StopStalking :wink: | ||||||||||
| Cc Pyspark | 280 | a year ago | 4 | mit | Python | |||||
| Process Common Crawl data with Python and Spark | ||||||||||
| Intoli Article Materials | 255 | a year ago | 85 | other | JavaScript | |||||
| All of the supporting materials for articles from Intoli's blog. | ||||||||||
| Awsets | 184 | 1 | a year ago | 35 | May 19, 2022 | 6 | mit | Go | ||
| A utility for crawling an AWS account and exporting all its resources for further analysis. | ||||||||||
| Serverlesscrawler Vancouverrealstate | 66 | 7 years ago | 1 | mit | Python | |||||
| A Serverless Crawler For Real State Data in Vancouver Using AWS Lambda, Dynamo, RDS MySQL and CloudWatch | ||||||||||
| Serverless Web Differ | 60 | 2 years ago | mit | Python | ||||||
| A serverless web browser which crawls websites and compares pages by schedule. | ||||||||||
| Blog | 59 | 3 months ago | 99 | SCSS | ||||||
| Your internal mediocrity is the moment when you lost the faith of being excellent. Just do it. | ||||||||||
| Elasticrawl | 50 | 1 | 7 years ago | 10 | February 15, 2017 | 1 | mit | Ruby | ||
| Launch AWS Elastic MapReduce jobs that process Common Crawl data. | ||||||||||
| Browser As A Service | 43 | a year ago | 30 | mit | JavaScript | |||||
| A web browser :earth_americas: hosted as a service, to render your JavaScript web pages as HTML | ||||||||||
Alternatives To Crawling InfrastructureSelect To Compare
Alternative Project Comparisons
Popular Crawler Projects
Popular Amazon Web Services Projects
Popular Data Processing Categories
Get A Weekly Email With Trending Projects For These Categories
No Spam. Unsubscribe easily at any time.
Typescript
Amazon Web Services
Cloud Computing
Crawler
Puppeteer