Hello all,
I am trying to run Flink 1.7.1 on EMR and having some trouble with metric
reporting.
I was using the DataDogHttpReporter, but have also tried the
StatsDReporter, but with both was seeing no metrics being collected.
To debug this I implemented my own reporter (based on StatsDReporter) and
logged the name of the metric being sent:
private void send(final String name, final String value) {
log.info("STATSD SENDING: ", name, value);
try {
String formatted = String.format("%s:%s|g", name, value);
byte[] data = formatted.getBytes(StandardCharsets.UTF_8);
socket.send(new DatagramPacket(data, data.length, this.address));
}
catch (IOException e) {
LOG.error("unable to send packet to statsd at '{}:{}'",
address.getHostName(), address.getPort());
}
}
This code is certainly reached, because in my log I see a lot of this:
2019-05-30 10:18:40,351 INFO com.grab.statsd.StatsDReporter
- STATSD SENDING:
2019-05-30 10:18:40,351 INFO com.grab.statsd.StatsDReporter
- STATSD SENDING:
2019-05-30 10:18:40,351 INFO com.grab.statsd.StatsDReporter
- STATSD SENDING:
2019-05-30 10:18:40,351 INFO com.grab.statsd.StatsDReporter
- STATSD SENDING:
2019-05-30 10:18:40,351 INFO com.grab.statsd.StatsDReporter
- STATSD SENDING:
2019-05-30 10:18:40,351 INFO com.grab.statsd.StatsDReporter
- STATSD SENDING:
2019-05-30 10:18:40,352 INFO com.grab.statsd.StatsDReporter
- STATSD SENDING:
As you can see, the name and value for the metric being reported is empty.
And the logs show everything initialized fine with no error:
2019-05-30 10:18:30,342 INFO
org.apache.flink.runtime.metrics.MetricRegistryImpl -
Configuring stsd with {port=8125, host=127.0.0.1,
class=com.grab.statsd.StatsDReporter}.
2019-05-30 10:18:30,344 INFO com.grab.statsd.StatsDReporter
- Configured StatsDReporter with {host:127.0.0.1,
port:8125}
2019-05-30 10:18:30,344 INFO
org.apache.flink.runtime.metrics.MetricRegistryImpl -
Periodically reporting metrics in intervals of 10 SECONDS for reporter
stsd of type com.grab.statsd.StatsDReporter.
Has anyone else tried to work with Flink and metrics on EMR 1.7.1
(latest version on EMR). If so, any pointers as to what could be set
up incorrectly?
--
*_Grab is hiring. Learn more at _**https://grab.careers
<https://grab.careers/>*
By communicating with Grab Inc and/or its
subsidiaries, associate companies and jointly controlled entities (“Grab
Group”), you are deemed to have consented to the processing of your
personal data as set out in the Privacy Notice which can be viewed at
https://grab.com/privacy/ <https://grab.com/privacy/>
This email contains
confidential information and is only for the intended recipient(s). If you
are not the intended recipient(s), please do not disseminate, distribute or
copy this email Please notify Grab Group immediately if you have received
this by mistake and delete this email from your system. Email transmission
cannot be guaranteed to be secure or error-free as any information therein
could be intercepted, corrupted, lost, destroyed, delayed or incomplete, or
contain viruses. Grab Group do not accept liability for any errors or
omissions in the contents of this email arises as a result of email
transmission. All intellectual property rights in this email and
attachments therein shall remain vested in Grab Group, unless otherwise
provided by law.
