Awesome Open Source
Awesome Open Source

Smart Cities Data Platform

Project Description

The platform is a combination of Elixir micro services custom built to ingest, normalize, transform, persist, and stream data from numerous sources, orchestrated via Kubernetes in any cloud provider or on-prem Kubernetes deployment. The loosely coupled services pass data across the pipeline via Kafka message queues and persist data to any hyper-scalable object store providing the S3 standard. They coordinate and communicate via a single event bus, also running on top of Kafka. The distributed data files are persisted and retrieved via SQL queries processed by the PrestoDB engine. Finally, user access, discovery, and analysis is facilitated by a ReactJS web application user interface, a RESTful API, or a web socket API for streaming data feeds.

scdp architecture diagram

Microservices

Application Short Description Build Status
Andi Admin Interface for creating/editing datasets to be ingested
Discovery API API to search for and query datasets
Discovery Streams Websocket connection to listen to streaming data
Estuary Microservice to persist event stream events
Forklift Microservice for saving data to Presto DB
Odo Microservice to convert Shapefiles to GeoJSON
Reaper Microservice to retrieve data
Valkyrie Microservice to validate data structure during ingestion

Prerequisites

General Prerequisites

  • Elixir - The primary language that all of the microservices are written in
  • Docker - All microservices are built as docker images
  • Apache Kafka - Communication mechanism between microservices
  • Redis - General purpose storage and caching
  • Elasticsearch - Used by Discovery API for search
  • PostgreSQL - General purpoase storage
  • Presto - Big Data storage of ingested data
  • Vault - Secure storage of secrets

Development Enviornment Prerequisites

Usage

The microservices written in Elixir use Mix as the build tool.

Building

Each microservice under the apps/ directory has a Dockerfile that can be used to build that microservice individually by running the following command:

docker build .

Testing

  • Unit Tests. Can be executed from the root of this repository or a specific application under the apps/ directory
mix test
  • Integration Tests Can be executed from the root of this repository or a specific application under the apps/ directory
mix test.integration
  • End to End (E2E) Tests. Can be executed from the root of this repository.
mix test.e2e

Execution

How to run and use the code

Additional Notes

Version History and Retention

Each microservice is released independently and can be found here in the Releases section. All releases will be kept indefinitely.

License

Released under Apache 2 license.

Contributions

How to contribute

Contact Information

Acknowledgements


Get A Weekly Email With Trending Projects For These Topics
No Spam. Unsubscribe easily at any time.
Hacktoberfest (33,731
Elixir (18,709
Data Visualization (4,285
Data Analytics (407
Data Processing (322
Elixir Phoenix (231
Related Projects