Daily Scraper
| Project Name | Stars | Downloads | Repos Using This | Packages Using This | Most Recent Commit | Total Releases | Latest Release | Open Issues | License | Language |
|---|---|---|---|---|---|---|---|---|---|---|
| Daily Scraper | 86 | 20 days ago | 16 | agpl-3.0 | HTML | |||||
| Fetches information about every webpage 🤖 | ||||||||||
| Webpage Rs | 27 |  | 3 | 3 months ago | 10 | December 12, 2021 | 2 | Rust | ||
| Rust library to fetch info from a webpage | ||||||||||
| Awesome Seo Scripts | 22 | 2 years ago | JavaScript | |||||||
| Random SEO scripts | ||||||||||
| Metatags | 19 | 5 years ago | PHP | |||||||
| A Laravel package to fetch webpage metadata ( Open Graph | Twitter | Facebook | Article ) | ||||||||||
| Webpage Scraper | 15 | 7 years ago | 1 | Python | ||||||
| This is a flask based application which fetches images, hyperlinks, indented source code and text after stripping the html tags from a given webpage and allows you to save them onto your system in a directory or text file with a name of your choice. | ||||||||||
| Web2db | 13 | 3 years ago | 6 | September 22, 2020 | Python | |||||
| Fetch webpage full-text, persist link and full text to SQLITE3 db, resumable with tqdm progressbar. | ||||||||||
| Gangsta | 8 | 6 years ago | apache-2.0 | PHP | ||||||
| Fetch OpenGraph data from a url and display in ExpressionEngine templates | ||||||||||
| Metadog | 7 |  | 2 | 6 years ago | 8 | November 02, 2016 | mit | JavaScript | ||
| Sniffs out and fetches open graph and schema.org metadata from webpages. | ||||||||||
| Webpage | 6 | 2 months ago | 13 | HTML | ||||||
| Sources for the JMLR webpage | ||||||||||
| Xst Google Calendar Events | 4 | 2 years ago | 3 | mit | JavaScript | |||||
| A react Component, which fetch Calendar entries from Google Calender and output them to your Webpage. It's based on Javascript (react) and create an sortable Event-List Table. No need for PHP or Database-Connection. | ||||||||||
Daily Scraper
Fetches information about every webpage 🤖
The service uses Puppeteer, a headless Chrome, to scrape webpages. Currently it's only purpose is to provide information when a user suggests a new source. The scraper can find the icon, rss feed, name, and other relevant information for every page.
Stack
- Node v12.19.0 (a
.nvmrcis presented for nvm users). - NPM for managing dependencies.
- Fastify as the web framework
Project structure
-
__tests__- There you can find all the tests and fixtures. Tests are written usingjest. -
helm- The home of the service helm chart for easily deploying it to kubernetes. -
src- This is obviously the place where you can find the source files.-
scrape- Stores many utility functions to scrape information from a webpage.
-
Local environment
Daily Scraper requires nothing to run. It doesn't need any database or a service.
.env is used to set the required environment variables. It is loaded automatically by the project.
Finally run npm run dev to run the service and listen to port 5001.
Want to Help?
So you want to contribute to Daily Scraper and make an impact, we are glad to hear it. 😍
Before you proceed we have a few guidelines for contribution that will make everything much easier. We would appreciate if you dedicate the time and read them carefully: https://github.com/dailydotdev/.github/blob/master/CONTRIBUTING.md
