Awesome Open Source
Awesome Open Source

Build Status


rrinlog is my attempt at Replacing Elasticsearch with Rust and SQLite for my nginx access logs, as Elasticsearch is a resource hungry application even at idle. rrinlog's success has been outstanding, with a 100x reduction in memory, 1000x reduction in CPU usage, and 100x reduction in disk usage.

This project contains two binaries:

  • rrinlog is for consuming nginx acces logs and storing them in a SQLite database. This binary may be built on Rust stable.
  • rrinlog-server exposes this SQLite database according to Grafana's JSON API datasource. This binary requires Rust nightly. Current Rust web framework is actix web, but there is a Rocket branch as well

This project currently isn't meant at replacing Elasticsearch for the general populous for the following reasons:

Custon Nginx Access Log

rrinlog ingests a custom nginx access log format:

log_format vhost    '$remote_addr - $remote_user [$time_local] '
                    '"$request" $status $body_bytes_sent '
                    '"$http_referer" "$http_user_agent" "$host"';

Any other format would likely result in parsing errors.

Hardcoded SQL Queries

rrinlog-server let's me know what my top blog articles with the following SQL query:

SELECT referer,
       Count(*) AS views
FROM   logs
WHERE  host = ''
       AND method = 'GET'
       AND path <> '/js/embed.min.js'
       AND epoch >= ?
       AND epoch < ?
       AND referer <> '-'
       AND remote_addr <> ?
GROUP  BY referer

This SQL query is tailored to me and how my blog is setup, so make no mistake that the intended audience with this query is solely me 😄

Limited Endpoints

These hardcoded SQL queries are needed as Grafana doesn't support SQLite as a native datasource. One day it may be supported like Mysql and Postgres, but until that day, rrinlog-server contains only a limited set of visualizations:

  • What are my top blog articles
  • How much outbound web data is leaving the server to other external IPs
  • How many requests are being serviced by other virtual hosts

No GeoIP Capabilities

Elasticsearch has the ability to take an IP address and turn it into a location. This is called GeoIP. I had a Grafana panel showing the top visiting cities, which is novel but not critical to monitor. Migrating from Elasticsearch meant I had to remove the visualization.

Get A Weekly Email With Trending Projects For These Topics
No Spam. Unsubscribe easily at any time.
rust (4,331
elasticsearch (532
nginx (471
sqlite (299
grafana (168
actix-web (20
rocket (20

Find Open Source By Browsing 7,000 Topics Across 59 Categories