Hi Users,
In Google Cloud monitoring there is a limit of 100 metrics and when we are
using KafkaIO, the library publishes a bunch of metrics per topic. With our
use we will easily run out of 100 metric limit.
We want to stop KafkaIO from publishing metrics and I do not see this is
configurable. So I am trying to write a metric filtering logic (we are
using beam version 2.55.1).
I wrote a Sink but when I try to find a way to register the sink I cannot
see a way to do the following in this beam version,
*MetricsEnvironment.setMetricsSink(new
CustomMetricsSink(options.getProject()));*
Then I tried to register it like this,
PipelineResult results = run(options);
results.waitUntilFinish();
* MetricQueryResults metricQueryResults =
results.metrics().queryMetrics(MetricsFilter.builder().build());
CustomMetricSink reporter = new CustomMetricSink(options.getProject());
reporter.writeMetrics(metricQueryResults);*
With the above code pipeline is failing to start with the
error(java.lang.UnsupportedOperationException:
The result of template creation should not be used.)
Do you suggest another solution for this problem (it sounds like a quite
common problem when using kafkaio). Or do you have any suggestion about my
attempts ?
Regards
Lahiru
