Install and configure Fluent Bit #

Fluent Bit is a log shipping tool. This means that it is a service that collects logs from a device and sends them to an external storage.

In Fluent Bit’s Settings, you specify a source of the logs you need, for example, logs of a specific service, internal storage, TCP port, or OS events. You must indicate a destination for log export—VNETWORK’s Logging servers. After that, Fluent Bit works automatically: Once a required log is recorded in the system, the log shipper transfers it to our storage, and the log appears on OpenSearch Dashboards.

Install #

Fluent Bit’s official documentation provides up-to-date installation guides for different operating systems. Open this link and select your OS. Simply follow the steps in the relevant installation manual.

Configure #

The Fluent Bit configuration file consists of two sections: INPUT and OUTPUT. INPUT determines the source of the logs you need to collect, and OUTPUT indicates the destination where they will be sent. Here’s how to set up the configuration file correctly:

Open the fluent-bit.conf file and add the following data:

[INPUT] Name tail Path /var/log/syslog [OUTPUT] name kafka brokers laas-example.gcore.com:443 topics yourlogin.yourtopic rdkafka.security.protocol SASL_SSL rdkafka.sasl.mechanism SCRAM-SHA-512 rdkafka.sasl.username yourlogin rdkafka.sasl.password yourpassword

Customize the highlighted values:

tail: Data source
/var/log/syslog: File path
laas-example.gcore.com:443: Kafka Endpoint on the Logging page
yourlogin.yourtopic: Your username on the Logging page and your topic name separated with a dot (.)
yourlogin: Your username on the Logging page
yourpassword: Your password

For more information on how to add INPUT for different log sources, go to the “Input” section of the Fluent Bit documentation and click the log source you need.

For example, if you want to gather logs from a specific file, open the guide for Tail. Tail is a utility on UNIX-like systems used to display the tail end of a file. It helps Fluent Bit to read changes in the log file. You need to fill in the INPUT so that it corresponds with Tail.

[OUTPUT]: Type of process (export).
name: Servers where logs will be delivered (Kafka servers).
brokers: Server(s) where logs will be exported to.
topics: Topic(s) where logs will be exported to.
rdkafka.security.protocol: Security protocol that encrypts data to protect it from theft.
rdkafka.sasl.mechanism: Authentication mechanism that helps to verify a login and a password entered to sign into your logs storage.
rdkafka.sasl.username: The username that helps to verify the sender.
rdkafka.sasl.password: The password that helps to verify the sender.

Save the changes in the fluent-bit.conf file.
Restart Fluent Bit, and it will begin sending logs to your VNETWORK Logging storage.

No timestamps #

Some services (for example, nginx) transmit timestamps of logs in a non-standard format. In this case, OpenSearch Dashboards will show logs without their date and time. If this happens, copy the string below to the OUTPUT section. This will ensure that Fluent Bit forwards date and time in a different format that is suitable for services like nginx.

    timestamp_format        iso8601