Top 5 Open-Source Log Shippers (alternatives to Logstash) in 2022

That is our checklist of the highest 5 open-source log shippers to suit your wants.

Many of those might be thought-about as “Logstash alternate options” in several methods. For these of you who don’t understand it but, Logstash is very widespread amongst DevOps for the potential for ingesting knowledge from totally different sources, dynamic knowledge transformation, and way more.

Nonetheless, it won’t be the best alternative for everybody. In case you are searching for open-source alternate options and to decide on what’s greatest for you – this information is right here that will help you.

Right here is the short checklist:

1. Logstash
2. Vector
3. Filebeat
4. FluentD
5. Promtail

For every log shipper, we additionally took among the core standards, like variety of plugins and configuration problem, and you will discover the total comparability desk on the finish of this weblog submit.

Logstash

The problem with Logstash

Let’s begin with the king of log collectors. If you understand about Logstash, be happy to scroll decrease for different choices.

As generally recognized, Logstash is a part of the well-known ELK stack and acts as a workhorse in it. It’s used to gather, parse, and ship your logs. Nonetheless, as a result of its excessive workload, it reveals low figures in efficiency checks (see ref.1 under) in comparison with different, extra light-weight choices.

As famous in the blog post by Logz.io, one of many causes for low efficiency is that Logstash requires JVM to run, this dependency causes vital reminiscence consumption. That is very true when you’ve many knowledge pipelines and superior filtering. That is likely one of the the reason why Filebeat was created, and we are going to focus on extra on that in a while, nonetheless be happy to dive into the weblog submit by Logz.io, which compares Filebeat and Logstash intimately.

As for the efficiency comparability of Logstash vs others, Vector has an insightful benchmark TCP To Blackhole Performance Test (ref.1) that compares Vector, Logstash, FluentD, and FluentBit.

$ bin/examine -t tcp_to_blackhole_performance

| Metric          | fluentbit | fluentd   | logstash  | vector    |
|:----------------|:----------|:----------|:----------|:----------|
| IO Thrpt (avg)  | 64.4MiB/s | 27.7MiB/s | 40.6MiB/s | 86MiB/s W |
| CPU sys (max)   | 4         | 3.5 W     | 6.1       | 6.5       |
| CPU usr (max)   | 53.2      | 50.8 W    | 91.5      | 96.5      |
| Load 1m (avg)   | 0.5 W     | 0.8       | 1.8       | 1.7       |
| Mem used (max)  | 614.8MiB  | 294MiB    | 742.5MiB  | 181MiB W  |
| Disk learn (sum) | 9MiB      | 2.6MiB W  | 2.6MiB    | 2.6MiB    |
| Disk writ (sum) | 14.8MiB   | 13.7MiB   | 11.6MiB   | 11MiB W   |
| Internet recv (sum)  | 3.9gib    | 1.7gib    | 2.4gib    | 5.1gib W  |
| Internet ship (sum)  | 7.9MiB    | 5.7MiB    | 2.6MiB    | 9MiB      |
| TCP estab (avg) | 663       | 664       | 665       | 664       |
| TCP sync (avg)  | 0         | 0         | 0         | 0         |
| TCP shut (avg) | 1         | 2         | 7         | 4         |
-------------------------------------------------------------------------------------------------------------
W = winner
fluentbit = 1.1.0
fluentd = 3.3.0-1
logstash = 7.0.1
vector = 0.2.0-6-g434bed8

Reference 1, TCP to Blackhole Efficiency Take a look at, Supply: Vector Repo

Logstash Execs & Cons

Here’s a checklist of professionals & cons for Logstash (compared to different log shippers on this information):

Vector

Vector is a instrument used to gather, rework, and route all of your logs and metrics. It was created by TimberIO in 2019, and purchased by Datadog in 2021.

Though Vector is an end-to-end agent & aggregator, it may well nonetheless be used as a log shipper/collector, and with its impressive benchmarks, it might turn out to be a big instrument in your stack.

The good half about Vector is that it’s written in Rust, which is thought for its efficiency, reminiscence security, and is designed for high-intensity work. Vector additionally offers distributed, centralized, and stream-based deployment.

To match Vector based mostly on options with different log shippers, the Vector repo has a good side-by-side comparison (reference 2 under) with Beats, Fluentbit, FluentD, Logstash, Splunk UF, and Splunk HF, which I added under for the reference.

Reference 2, Vector’s Options, Supply: Vector Repo

The previously discussed benchmark particularly caught our eye and it is an amazing basis to match Vector’s efficiency in opposition to different log shippers. In brief, Vector wins Logstash, FluentD, and Fluentbit in IO Thrpt (avg), Mem used (max), Disk writ (sum), and Internet recv (sum) in TCP to Blackhole check.

Vector Execs & Cons

Here’s a checklist of professionals & cons for Vector (compared to different log shippers on this information):

Filebeat

An alternative choice is Filebeat. This one was created with a purpose in thoughts to be a light-weight various to Logstash, in the event you don’t want superior tuning. Though you need to use each Logstash and Filebeat, you do have an choice to ship your logs immediately with Filebeat to your centralized logging platform. As you get extra acquainted or require extra options, you may all the time join Logstash in a while to your course of.

If you wish to accumulate logs on distant machines, Filebeat is a superb possibility. And in the event you don’t must make transformations to your knowledge then you might be free to ship it straight to Elasticsearch! However as a rule, in the event you want extra than simply timestamp and message fields, you’ll need Logstash.

General, it’s a a lot less complicated possibility in comparison with Logstash. This makes it extra dependable for brand new customers, as you’ve much less gears and handles to spin.

For scaling Filebeat, most definitely than not you’d nonetheless require Logstash in your stack. Nonetheless, on this case, you will need to have a well-designed / architectured cluster.

On your stack, you won’t solely use Filebeat. That is particularly helpful if you’re contemplating scaling. Beneath is a illustration of what your course of would possibly appear like at scale with Beats, Logstash, Elasticsearch, and Kibana.

Source: Elastic.co, Information on Deploying and Scaling Logstash

Filebeat Execs & Cons

Here’s a checklist of professionals & cons for Filebeat (compared to different log shippers on this information):

FluentD

This one was constructed with the thought in thoughts to construction knowledge in JSON as a lot as attainable. FluentD claims that this technique permits for a Unified Logging Layer (in different phrases, unifying logging infrastructure).

As a result of an unlimited variety of plugins, FluentD is a superb alternative for people who have knowledge from totally different or distinctive sources.

Fluentbit can also be an possibility to think about if you’re searching for one thing that matches extra small units & distributed programs. Identical to FluentBit, it was created by Treasure Knowledge, however in a while in 2015. There’s a Side by Side comparison of FluentD and Fluentbit by Logz.io, which we advise you to take a look at.

There’s an insightful article with a number of checks evaluating FluentD and FluentBit, one in all which is Forwarding 5,000 1KB occasions per second, which resulted in 80% CPU and 120MB Reminiscence for FluentD, however 27% CPU and 26MB Reminiscence for FluentBit.

Nonetheless, when evaluating FluentD to different log shippers on this information, it performs fairly properly.

FluentD Execs & Cons

Here’s a checklist of professionals & cons for FluentD (compared to different log shippers on this information):

Promtail

For people who know Loki, you’ve most likely heard of Promtail. Its use case is particularly tailor-made to Loki, and it may well accumulate logs each regionally and for Kubernetes pods.

As for comparability, you can check with an article by CrashLaker, which compares Loki vs ELK vs Splunk. On prime of it, there’s additionally an insightful Medium article by Ronen Schaffer, which analyses Promtail together with write path efficiency, learn path efficiency, and way more.

Promtail Execs & Cons

Here’s a checklist of professionals & cons for Promtail (compared to different log shippers on this information):

Conclusion

Let’s conclude with a comparability desk that features the entire above talked about log shippers:

We hope this information helped you to analyse simply the floor of options for prime log shippers, as to see which answer you need to discover additional and add to your stack.

Let me know if in case you have any questions on any of the instruments, I’ll see the remark and reply asap! Thanks for studying.