Hi,
this seems a fluentd issue not a kafka one.
maybe you can you try another log shipper to validate this assumption;
there are plenty and fluentd is already an old (and buggy and slow one);
you can try fluent-bit or vector maybe.
--
Raphael Mazelier
On 12/08/2022 04:58, pengyh wrote:
do you konw if the current fluentd supports kafka 3 (my version is
3.2.1)?
fluentd dry-run outputs:
2022-08-12 10:47:24 +0800 [info]: gem
'fluent-plugin-calyptia-monitoring' version '0.1.3'
2022-08-12 10:47:24 +0800 [info]: gem 'fluent-plugin-elasticsearch'
version '5.2.3'
2022-08-12 10:47:24 +0800 [info]: gem
'fluent-plugin-flowcounter-simple' version '0.1.0'
2022-08-12 10:47:24 +0800 [info]: gem 'fluent-plugin-kafka' version
'0.18.0'
2022-08-12 10:47:24 +0800 [info]: gem 'fluent-plugin-metrics-cmetrics'
version '0.1.2'
2022-08-12 10:47:24 +0800 [info]: gem 'fluent-plugin-opensearch'
version '1.0.7'
2022-08-12 10:47:24 +0800 [info]: gem 'fluent-plugin-prometheus'
version '2.0.3'
2022-08-12 10:47:24 +0800 [info]: gem
'fluent-plugin-prometheus_pushgateway' version '0.1.0'
2022-08-12 10:47:24 +0800 [info]: gem 'fluent-plugin-record-modifier'
version '2.1.0'
2022-08-12 10:47:24 +0800 [info]: gem
'fluent-plugin-rewrite-tag-filter' version '2.4.0'
2022-08-12 10:47:24 +0800 [info]: gem 'fluent-plugin-s3' version '1.7.1'
2022-08-12 10:47:24 +0800 [info]: gem 'fluent-plugin-sd-dns' version
'0.1.0'
2022-08-12 10:47:24 +0800 [info]: gem 'fluent-plugin-systemd' version
'1.0.5'
2022-08-12 10:47:24 +0800 [info]: gem 'fluent-plugin-td' version '1.2.0'
2022-08-12 10:47:24 +0800 [info]: gem 'fluent-plugin-utmpx' version
'0.5.0'
2022-08-12 10:47:24 +0800 [info]: gem 'fluent-plugin-webhdfs' version
'1.5.0'
2022-08-12 10:47:24 +0800 [info]: gem 'fluentd' version '1.15.1'
2022-08-12 10:47:24 +0800 [info]: starting fluentd-1.15.1 as dry run
mode ruby="2.7.6"
2022-08-12 10:47:24 +0800 [info]: brokers has been set:
["127.0.0.1:9092"]
my pipeline is: apache2 weblogs -> fluentd -> kafka -> spark
But the issue I met is I can't write the output to the correct topic
from fluentd, the messages always fall into the default topic, not the
one I specified.
The plugin I am using is:
https://github.com/fluent/fluent-plugin-kafka
which is built-in by td-agent (the latest version as well).
Thanks.
