Scrapy Heroku
| Project Name | Stars | Downloads | Repos Using This | Packages Using This | Most Recent Commit | Total Releases | Latest Release | Open Issues | License | Language |
|---|---|---|---|---|---|---|---|---|---|---|
| Metabase | 32,623 | 19 hours ago | 1 | June 08, 2022 | 3,027 | other | Clojure | |||
| The simplest, fastest way to get business intelligence and analytics to everyone in your company :yum: | ||||||||||
| Dbeaver | 32,272 | a day ago | 1,744 | apache-2.0 | Java | |||||
| Free universal database tool and SQL client | ||||||||||
| Prisma | 31,833 |  | 442 | 20 hours ago | 4,993 | September 24, 2022 | 2,920 | apache-2.0 | TypeScript | |
| Next-generation ORM for Node.js & TypeScript | PostgreSQL, MySQL, MariaDB, SQL Server, SQLite, MongoDB and CockroachDB | ||||||||||
| Typeorm | 31,387 |  | 1,994 | 2,164 | a day ago | 650 | September 20, 2022 | 1,975 | mit | TypeScript |
| ORM for TypeScript and JavaScript. Supports MySQL, PostgreSQL, MariaDB, SQLite, MS SQL Server, Oracle, SAP Hana, WebSQL databases. Works in NodeJS, Browser, Ionic, Cordova and Electron platforms. | ||||||||||
| Directus | 21,780 |  | 50 | 20 hours ago | 55 | September 22, 2022 | 225 | other | TypeScript | |
| The Modern Data Stack 🐰 — Directus is an instant REST+GraphQL API and intuitive no-code data collaboration app for any SQL database. | ||||||||||
| Postgrest | 20,617 | 4 | 4 days ago | 37 | July 12, 2022 | 207 | mit | Haskell | ||
| REST API for any Postgres database | ||||||||||
| Shardingsphere | 18,460 | 8 | a day ago | 7 | June 04, 2020 | 667 | apache-2.0 | Java | ||
| Ecosystem to transform any database into a distributed database system, and enhance it with sharding, elastic scaling, encryption features & more | ||||||||||
| Mindsdb | 16,410 |  | 3 | 1 | 19 hours ago | 42 | March 19, 2019 | 620 | gpl-3.0 | Python |
| MindsDB is a Server for Artificial Intelligence Logic. Enabling developers to ship AI powered projects to production in a fast and scalable way. | ||||||||||
| Timescaledb | 15,046 | a day ago | 531 | other | C | |||||
| An open-source time-series SQL database optimized for fast ingest and complex queries. Packaged as a PostgreSQL extension. | ||||||||||
| Beekeeper Studio | 12,980 | 12 days ago | 488 | gpl-3.0 | Vue | |||||
| Modern and easy to use SQL client for MySQL, Postgres, SQLite, SQL Server, and more. Linux, MacOS, and Windows. | ||||||||||
Scrapy-Heroku
A package to assist with running scrapy on heroku. This is accomplished by providing
a custom application configuration at scrapy_heroku.app.application that launches
the scrapyd web service using the PORT environment variable and a multi-process work
queue implemented on a Postgres database specified by the DATABASE_URL environment
variable.
Configuration
Create a git repo that has a scrapy project at the root (scrapy.cfg should be at the top level). Edit your scrapy.cfg to include the following::
[scrapyd]
application = scrapy_heroku.app.application
[deploy]
url = http://<YOUR_HEROKU_APP_NAME>.herokuapp.com:80/
project = <YOUR_PROJECT_NAME>
username = <A_USER_NAME>
password = <A_PASSWORD>
Add a requirements.txt file that includes scrapy, scrapy-heroku, and scrapyd.
It is strongly recommended that you version pin scrapy-heroku as well as the version of scrapy that
your project is developed against (pip freeze > requirements.txt).
For Example:
# requirements.txt
Scrapy==0.24.4
scrapyd==1.0.1
scrapy-heroku==0.7.1
Finally create a Procfile that consists of::
web: scrapyd
Make sure you have a postgres database with the DATABASE_URL env parameter set.
- Project page: dmclain/scrapy-heroku
