dali (named after the great painter Salvador Dali) is a service which performs image transformations. It is used by OLX to process and serve images for users in around 40 countries. The application supports:
All configuration should be provided through either a json config file or environment variables. The application will search first by a file named
config/default.json, then if the environment variable
RUN_MODE is set it will search for
config/RUN_MODE, override the values and lastly it will override with environment variables.
||Enum(trace, debug, info, warn, error)||Logging level for the application||N||
||Default value is
||integer||Port which the web server listens to for requests||Y||-|
||integer||Port which the web server listens to for the health requests||Y||-|
||integer||Defines a timeout for reading client request header. If a client does not transmit the entire set headers within this time, the request is terminated with the 408 (Request Time-out) error.||N||-||-|
||integer||Defines a timeout for shutdown connection. If a shutdown procedure does not complete within this time, the request is dropped.||N||-||-|
||integer||Server keep alive value||N||-||If not specified, it will use the OS's configuration|
||integer||Set the timeout for connecting to a URL||N||-||Default is no timeout|
||integer||Set the timeout for the response||N||-||Default is no timeout|
||integer||Set the timeout for the request||N||-||Default is no timeout|
||integer||Max size of response payload (for
||N||-||Default is 256Kb|
||integer||Max number of threads the application will spawn for serving requests and processing images||N||-||If not specified it will take the number of physical CPUs from the machine|
||integer||Max number of threads for image processing that will be used||N||-||if not specified it will take
||integer||Max number of threads for serving requests that will be used||N||-||if not specified it will take
||integer||Max number of threads for serving the
||N||-||Defaults to 1|
The application will compute the number of threads by the following formula:
pod_number_of_cpus * cpu_usage_percentage / 100. This number will be divided by 2 and half will be assigned to the HTTP connection listener and half will be assigned to
libvips (the image library). An extra worker will be created to listen to the
health endpoint (this was done to be sure the application won't block the
health endpoint even when overloaded).
This application relies on C libvips library. That means it has to be previously installed into the system before compiling and/or running.
For installation follow this instructions. (Required minimum version 8.8.3)
rustup is the recommended way to install
rust. It is a tool that manages and updates rust versions (like
nvm for node for example). To install it, simply run
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh. Then run
To build and run the application, run the following command:
Note: if you're building the application on a
linux musl system, you'll need to set this env variable:
if it fails to compile because of the linker, try to add the result of
pkg-config --libs vips to
RUSTFLAGS= env variable`
Alternatively, it is possible to run both components inside docker (once you build them):
There are 3 kinds of tests: unit, integration and benchmark tests.
At the moment, the only logic the application has is regarding resizing and watermark positioning. There are some unit tests around these use cases in the package
image_processor. All other functions are deeply tied to the
libvips library which complicates testing since it is not easily mockable.
To run them, simply execute:
cargo test --bin dali. The parameter
--bin is needed because otherwise it would run also integration tests.
This tests run over a running application.
To run the tests, the script will start the containers through
docker-compose, copy some sample files to the
http container and run the tests over the application, checking the array of bytes from the responses against expected result images stored in the
tests/resources/results directory. To run the whole flow, simply run:
This is an experimental feature from rust, so in order to use, the nightly rust toolchain has to be enabled. To do that run:
rustup toolchain install nightly rustup default nightly
To rollback to stable, run
rustup default stable.
To run the benchmark, simply call
cargo bench (application must be running at localhost on port 8080).
It outputs the average time per iteration and the deviation between max and min. Example output from a 4 core 2.3GHz MacBook (application running inside docker limited to 4 cores and with 4 workers):
test bench_highhes ... bench: 71,112,344 ns/iter (+/- 7,798,699)
The application supports the following endpoints.
Signifies the application is healthy by returning a HTTP Status OK - 200 return code.
Prometheus formatted metrics. Currently exposes request count and duration per endpoint
Fetches and processes an image file. The only mandatory parameter is the
||The address for the Image. Should be a HTTP, HTTPS or HTTP valid URI.|
||desired image format. Possible values are
||desired quality for the image. For Jpeg, it goes from 0 to 100 (defaults to 75)|
||desired width for the image. Images won't get upscaled or have their aspect ratio changed by variations on parameters for width and height.|
||desired height for the image. Images won't get upscaled or have their aspect ratio changed by variations on parameters for width and height.|
||optional rotation of the image. Possible values are
Watermarks is an array parameter and therefore, must be indexed when informed (0 indexed).
||watermark file. File has to be smaller than original file. Should be a HTTP, HTTPS or HTTP valid URI.|
||opacity from the watermark over the original image. it is a floating point number from 0 to 1.|
||identifier to position the watermark based on a point or centered (X axis). Possible values: Left (default), Right, Center.|
||identifier to position the watermark based on a point or centered (Y axis). Possible values: Top (default), Bottom, Center.|
||position of the watermark in the X axis. Value in pixels.|
||position of the watermark in the Y axis. Value in pixels.|
||optional size of the watermark. It should be a value between 1 and 100 representing a percentage from the original image.|