Hailstorm
| Project Name | Stars | Downloads | Repos Using This | Packages Using This | Most Recent Commit | Total Releases | Latest Release | Open Issues | License | Language |
|---|---|---|---|---|---|---|---|---|---|---|
| Tokio Tungstenite | 1,232 |  | 27 | 229 | 3 days ago | 23 | July 13, 2022 | 12 | mit | Rust |
| Future-based Tungstenite for Tokio. Lightweight stream-based WebSocket implementation | ||||||||||
| Node Libspotify | 103 |  | 10 | 4 | 7 years ago | 4 | September 25, 2013 | 29 | C++ | |
| Node bindings for the libspotify C library | ||||||||||
| Bencode Go | 101 | 9 | 27 | 3 years ago | 1 | August 13, 2018 | 4 | other | Go | |
| A Go language binding for encodeing and decoding data in the bencode format that is used by the BitTorrent peer-to-peer file sharing protocol. | ||||||||||
| Reason Nodejs | 98 |  | 2 | 2 years ago | 9 | May 14, 2020 | 15 | mit | Reason | |
| Node bindings for Reason and Bucklescript | ||||||||||
| Hailstorm | 85 | 9 years ago | 1 | apache-2.0 | Haskell | |||||
| Haskell distributed stream processing with exactly-once semantics | ||||||||||
| Ruby Bencode | 76 |  | 66 | 14 | 4 years ago | 13 | September 04, 2014 | mit | Ruby | |
| Ruby bindings for the bencode data serialization format | ||||||||||
| Node Ogg | 63 |  | 23 | 14 | 2 years ago | 12 | March 01, 2019 | 7 | mit | JavaScript |
| Node.js native binding to libogg | ||||||||||
| Minizlib | 59 |  | 172,982 | 86 | a year ago | 20 | August 14, 2020 | 5 | other | JavaScript |
| A smaller, faster, zlib stream built on http://npm.im/minipass and Node.js's zlib binding. | ||||||||||
| Xz2 Rs | 57 |  | 148 | 93 | 4 months ago | 8 | June 06, 2022 | 28 | apache-2.0 | Rust |
| Bindings to liblzma in Rust (xz streams in Rust) | ||||||||||
| Rsoundio | 43 |  | 1 | 1 | 2 years ago | 6 | May 03, 2016 | 9 | other | Rust |
| Rust binding for libsound.io | ||||||||||
Hailstorm
Hailstorm is a distributed stream computation system that uses exactly once semantics.
Written by Thomas Dimson (@cosbynator) and Milind Ganjoo (@mganjoo).
References
The architecture of Hailstorm is based on Apache Storm (which is also the inspiration for the name).
The exactly-one semantics implemented in Hailstorm are based on a high-level description in an essay by @jasonjckn.
Dependencies
Zookeeper
Hailstorm requires Apache Zookeeper, its C
bindings, and its Haskell bindings
hzk to run.
On OSX, the zookeeper package on homebrew contains the
binaries and C bindings for Zookeeper. You can install it as follows:
brew install --c zookeeper
On Ubuntu, we recommend following the official instructions to obtain and set up the Zookeeper binaries. To install the C bindings:
sudo apt-get install libzookeeper-mt-dev
Finally, to build and install hzk on Mac OS X, run the following command in
your cabal sandbox:
cabal install --extra-include-dirs=/usr/local/include/zookeeper hzk
The above command is required on Mac OS X because of the non-standard include
directory location. On Ubuntu, cabal install hzk should work.
Kafka
Hailstorm requires Apache Kafka to be installed and operating. See the official instructions for details.
Haskakafka
Hailstorm uses
Haskakafka, the Haskell bindings
written by our very own @cosbynator. Haskakfka, in
turn, depends on librdkafka (see the Haskakafka project page for installation
instructions).
Haskakafka itself is not available yet on Cabal, so install
cabalg and
c2hs into your sandbox:
cabal install c2hs
cabal install cabalg
.cabal-sandbox/bin/cabalg https://github.com/cosbynator/haskakafka.git
On OS X, you may get 'stdio.h' errors, in which case you should try:
.cabal-sandbox/bin/cabalg https://github.com/cosbynator/haskakafka.git -- --with-gcc=gcc-4.8
And you are done!
Running
First, start an instance of Zookeeper (zkServer start on Mac or zkServer.sh start on Ubuntu).
Before running Hailstorm for the first time, you will have to initialize your
topology using the zk_init subcommand:
hailstorm zk_init
Finally, run a sample topology:
hailstorm -f data/test.txt run_sample
While it is running, you can extract debug metadata by executing
hailstorm zk_show
