file.d is a blazing fast tool for building data pipelines: read, process, and output events. Primarily developed to read from files, but also supports numerous input/action/output plugins.
⚠ Although we use it in production,
it still isn't v1.0.0. Please, test your pipelines carefully on dev/stage environments.
Well, we already have several similar tools: vector, filebeat, logstash, fluend-d, fluent-bit, etc.
Performance tests state that best ones achieve a throughput of roughly 100MB/sec. Guys, it's 2020 now. HDDs and NICs can handle the throughput of a few GB/sec and CPUs processes dozens of GB/sec. Are you sure 100MB/sec is what we deserve? Are you sure it is fast?
On MacBook Pro 2017 with two physical cores
file.d can achieve the following throughput:
files > devnullcase
files > json decode > devnullcase
TBD: throughput on production servers.
Generated using insane-doc