Control Fluent Bit Backpressure With Prometheus

Fluent Bit is one of the most widely used open source data collection agents for logs, metrics and traces. It’s lightweight, high-performance and easily extensible, making it ideal for modern observability pipelines.

At its core, Fluent Bit is a simple data pipeline comprising various stages, as illustrated in the diagram below.

However, even the most efficient pipeline can hit a bottleneck known as backpressure, which occurs when data is ingested at a rate that exceeds the system’s ability to process and flush it. Backpressure causes…