rrinlog is my attempt at Replacing Elasticsearch with Rust and SQLite for my nginx access logs, as Elasticsearch is a resource hungry application even at idle. rrinlog's success has been outstanding, with a 100x reduction in memory, 1000x reduction in CPU usage, and 100x reduction in disk usage.
This project contains two binaries:
rrinlogis for consuming nginx acces logs and storing them in a SQLite database. This binary may be built on Rust stable.
rrinlog-serverexposes this SQLite database according to Grafana's JSON API datasource. This binary requires Rust nightly. Current Rust web framework is actix web, but there is a Rocket branch as well
This project currently isn't meant at replacing Elasticsearch for the general populous for the following reasons:
rrinlog ingests a custom nginx access log format:
log_format vhost '$remote_addr - $remote_user [$time_local] ' '"$request" $status $body_bytes_sent ' '"$http_referer" "$http_user_agent" "$host"';
Any other format would likely result in parsing errors.
rrinlog-server let's me know what my top blog articles with the following SQL query:
SELECT referer, Count(*) AS views FROM logs WHERE host = 'comments.nbsoftsolutions.com' AND method = 'GET' AND path <> '/js/embed.min.js' AND epoch >= ? AND epoch < ? AND referer <> '-' AND remote_addr <> ? GROUP BY referer ORDER BY views DESC
This SQL query is tailored to me and how my blog is setup, so make no mistake that the intended audience with this query is solely me 😄
These hardcoded SQL queries are needed as Grafana doesn't support SQLite as a native datasource. One day it may be supported like Mysql and Postgres, but until that day,
rrinlog-server contains only a limited set of visualizations:
Elasticsearch has the ability to take an IP address and turn it into a location. This is called GeoIP. I had a Grafana panel showing the top visiting cities, which is novel but not critical to monitor. Migrating from Elasticsearch meant I had to remove the visualization.