Fluentd Log Collector

Fluentd is a free and open-source unified logging layer and data collector that decouples data sources from backend systems, collecting, processing, and routing logs and event data from various sources to multiple destinations. Created by Sadayuki Furuhashi at Treasure Data (headquartered in Mountain View, California, and Tokyo, Japan) in 2011, Fluentd is written in Ruby and C, and became a CNCF graduated project in 2019. Key features: unified logging: Fluentd acts as a universal logging layer, allowing applications to emit logs once (in JSON format) and have Fluentd route them to multiple backends (files, databases, message queues, cloud services) without modifying application code. Plugin architecture: over 800 community plugins. Input plugins collect data (tail for log files, http, tcp/udp, syslog). Output plugins send data to destinations (elasticsearch, s3, kafka, mongodb, cloudwatch). Filter plugins transform or filter records. Parser plugins parse raw data into structured JSON. Reliability: at-least-once delivery semantics. Events are buffered in memory or on disk before being sent to destinations, with retry logic for failed sends. Buffer types: file-backed buffers persist events to disk, preventing data loss during crashes, while memory buffers provide higher throughput. Tag-based routing: each event is tagged with a string, and routing rules match tags with regular expressions to determine which output plugins process the event. Multi-worker processes: for horizontal scaling. High availability: active-standby pairs and load-balanced clusters. Cross-platform: Linux, macOS, Windows. Apache-2.0.