Install and configure Fluentd #
Fluentd is a data collection and processing tool that supports multiple input sources, data formats, and output destinations. This allows you to stream and consolidate logs from different applications into one unified format and send this information to the VNETWORK Managed Logging.
Install Fluentd #
Download Fluentd from the official Fluentd website and install it using your preferred method.
Configure Fluentd #
A standard Fluentd configuration consists of three sections: input, filter, and output. The input and filter sections depend on the sources of logs.
To send logs to VNETWORK Managed Logging, configure Fluentd with the Kafka Integration Plugin in your Fluentd installation.
1. Configure Fuentd with Kafka output by adding the following data to the fluent.conf file:
<match pattern> @type kafka2
# list of seed brokers brokers laas-example.gcore.com:443 username: username password: password scram_mechanism: “sha512” default_topic: namespace_username.topic_name sasl_over_ssl: true
# buffer settings <buffer> @type memory ## OPTIONAL PARAMS timekey: 30s timekey_wait: 10s timekey_use_utc: true chunk_limit_size: 4MB total_limit_size: 64MB chunk_full_threshold: “0.9” queued_chunks_limit_size: 12 flush_thread_count: 12 </buffer>
# data type settings <format> @type json </format> </match>
2. Customize the highlighted values:
brokers: Kafka servers where logs will be exported to.
username: Your username
password: Your password
default_topic: Your username and topic name, separated with a dot (.)
You can find your username, password, and topic name information in the VNETWORK Customer Portal on the Logging page. Learn more about logging configuration in our dedicated guide.
For more settings, check out the official Fluentd documentation.
- brokers: Kafka servers where Fluentd logs will be sent to.
- scram_mechanism: Authentication mechanism that verifies a username and password entered to sign in your logs storage.
- default_topic: Kafka topic where logs will be exported to.
- sasl_over_ssl: When this parameter is set to “true”, the system will use SASL for authentication and SSL for secure communication.
3. Save the changes in the Fluentd configuration file.
4. Restart Fluentd. It will start sending logs to the VNETWORK Managed Logging.