Reference application developed in the Functional event-driven architecture: Powered by Scala 3 book.
The web application allows users to subscribe/unsubscribe to/from symbol alerts such as
EURUSD, which are emitted in real-time via Web Sockets.
It is written in Elm and can be built as follows.
$ cd web-app && nix-build $ xdg-open result/index.html # or specify browser
There's also a
shell.nix handy for local development.
$ cd web-app && nix-shell $ elm make src/Main.elm --output=Main.js $ xdg-open index.html # or specify browser
If Nix is not your jam, you can install Elm by following the official instructions and then compile as usual.
$ cd web-app $ elm make src/Main.elm --output=Main.js $ xdg-open index.html # or specify browser
ws-client module (this last task is only required for
$ cd modules/ws-client $ sbt 'webapp/fastOptJS;webapp/copyJsFileTask'
You can then run it via Nix as follows (it requires flakes).
$ nix run Using cache dir: /home/gvolpe/workspace/trading/modules/ws-client/nix-parcel-cache Server running at http://localhost:1234 Built in 7ms
nix run comand will create a directory for the Parcel cache, which needs write permissions.
For development iterations, it may be more convenient to use
$ nix develop $ yarn install $ yarn build $ yarn start yarn run v1.22.17 parcel index.html --no-cache --dist-dir dist --log-level info Server running at http://localhost:1234 Built in 1.82s
However, this is not fully reproducible and can't be guaranteed this will work in the future.
Without Nix, you need to install
parcel, and use
yarn as shown above.
Here's an overview of all the components of the system.
The back-end application is structured as a mono-repo, and it requires both Apache Pulsar and Redis up and running. To make things easier, you can use the provided
docker-compose file depends on declared services to be published on the local docker server. All docker builds are handled within the
To build all of the service images, we have a few options.
The first one via the given Dockerfile.
$ docker build -t jdk17-curl modules/
The second one via Nix, from where we can build a slim image also based on
$ nix build .#slimDocker -o result-jre $ docker load -i result-jre
The third one also via Nix, though building a layered image based on the same JDK we use for development.
$ nix build .#docker -o result-jre $ docker load -i result-jre
The main difference between these three options is the resulting image size.
$ docker images | rg jdk17 jdk17-curl latest 0ed94a723ce3 10 months ago 422MB jdk17-curl-nix latest c28f54e42c21 52 years ago 557MB jdk17-curl-slim latest dbe24e7a7163 52 years ago 465MB
Any image is valid. Feel free to pick your preferred method.
NOTE: As of September 2022, the Docker image resulting from
nix build .#docker is no longer compatible with
sbt-native-packager, so either go for
nix build (defaults to the slim image) or build it directly via Docker with the given Dockerfile.
Once the base
jdk17-curl image has been built, we can proceed with building all our services' images.
$ sbt docker:publishLocal
$ docker-compose up -d pulsar redis
To run the Kafka Demo (see more below in X Demo), only Zookeeper and Kafka are needed.
$ docker-compose -f kafka.yml up
If we don't specify any arguments, then all the containers will be started, including all our services (except
feed), Prometheus, Grafana, and Pulsar Manager.
$ docker-compose up Creating network "trading_app" with the default driver Creating trading_pulsar_1 ... done Creating trading_redis_1 ... done Creating trading_ws-server_1 ... done Creating trading_pulsar-manager_1 ... done Creating trading_alerts_1 ... done Creating trading_processor_1 ... done Creating trading_snapshots_1 ... done Creating trading_forecasts_1 ... done Creating trading_tracing_1 ... done Creating trading_prometheus_1 ... done Creating trading_grafana_1 ... done
It is recommended to run the
feed service directly from
sbt whenever necessary, which publishes random data to the topics where other services are consuming messages from.
The back-end application consists of 9 modules, from which 5 are deployable applications, and 3 are just shared modules. There's also a demo module and a web application.
modules alerts core domain feed forecasts it lib processor snapshots tracing ws-client ws-server x-demo
Capability traits such as
GenUUID, and potential library abstractions such as
Producer, which abstract over different implementations such as Kafka and Pulsar.
Commands, events, state, and all business-related data modeling.
Core functionality that needs to be shared across different modules such as snapshots,
ForecastCommands followed by publishing them to the corresponding topics. In the absence of real input data, this random feed puts the entire system to work.
Registers new authors and forecasts, while calculating the author's reputation.
The brain of the trading application. It consumes
TradeCommands, processes them to generate a
TradeState and emitting
TradeEvents via the
TradeEvents and recreates the
TradeState that is persisted as a snapshot, running as a single instance in fail-over mode.
The alerts engine consumes
TradeEvents and emits
Alert messages such as
Sell via the
trading-alerts topic, according to the configured parameters.
Alert messages and sends them over Web Sockets whenever there's an active subscription for the alert.
A decentralized application that hooks up on multiple topics and creates traces via the Open Tracing protocol, using the Natchez library and Honeycomb.
All unit tests can be executed via
sbt test. There's also a small suite of integration tests that can be executed via
sbt it/test (it requires Redis to be up).
It contains all the standalone examples shown in the book. It also showcases both
MemDemo programs that use the same
Producer abstractions defined in the
To run the Pulsar CDC Demo, you need Postgres and Pulsar (make sure no other instances are running). Before running them, we need to download the connector NAR file.
$ mkdir -p pulsarconf/connectors && cd pulsarconf/connectors $ wget https://archive.apache.org/dist/pulsar/pulsar-2.10.1/connectors/pulsar-io-debezium-postgres-2.10.1.nar $ docker-compose -f pulsar-cdc.yml up
Once both instances are up and healthy, we can run the Pulsar Debezium connector.
$ docker-compose exec -T pulsar bin/pulsar-admin source localrun --source-config-file /pulsar/conf/debezium-pg.yaml
You should see this in the logs.
Snapshot step 3 - Locking captured tables [public.authors]
It contains the
smokey project that models the smoke test for trading.
JVM stats are provided for every service via Prometheus and Grafana.
Two Pulsar topics can be compacted to speed-up reads on startup, corresponding to
To compact a topic on demand (useful for manual testing), run these commands.
$ docker-compose exec pulsar bin/pulsar-admin topics compact persistent://public/default/trading-alerts Topic compaction requested for persistent://public/default/trading-alerts. $ docker-compose exec pulsar bin/pulsar-admin topics compact persistent://public/default/trading-switch-events Topic compaction requested for persistent://public/default/trading-switch-events
In production, one would configure topic compaction to be triggered automatically at the namespace level when certain threshold is reached. For example, to trigger compaction when the backlog reaches 10MB:
$ docker-compose exec pulsar bin/pulsar-admin namespaces set-compaction-threshold --threshold 10M public/default