Hello Philipp,
there is certainly something very wrong here.
What you _should_ see is 6 entries, 1 for each operator; 2-3 more for
the tasks the operators are executed in and the taskmanager stuff.
Usually, operator metrics use the name that you configured, like
"TokenMapStream", whereas tasks use the concatenation of all operator
names joined with =>, as in "KafkaSource => TokenMapStream".
I will look into this, I've never seen these issues before.
One more thing, which version of Flink are you currently using?
Regards,
Chesnay
On 15.10.2016 00:07, Philipp Bussche wrote:
Hi there,
I am struggeling to understand what I am looking at after enabling JMX
metric reporting on my taskmanager.
The job I am trying this out with has 1 source, 2 map functions (where one
is a RichMap) and 3 sinks.
This is how I have written my Job:
DataStream<Invitation> invitations = streaming
.addSource(new FlinkKafkaConsumer09<>(
dsp.getPropertyAsString("kafka.invitation.topic"),
new InvitationSchema(),
kafkaProps)).name("KafkaSource");
invitations.addSink(new
PostgresqlInvitationDetailsSink<>(psqlConfig)).name("InvitationDetailSink");
DataStream<Tuple2<String, String>> tokens = invitations
.map(new
TokenExtractor()).name("TokenMapStream");
tokens.addSink(new
PostgresqlInvitationTokenSink<>(psqlConfig)).name("InvitationTokenSink");
DataStream<Tuple4<String, String, String, String>> invitationResponses =
invitations
.map(new InvitationDetailsExtractor(psqlConfig,
tokenToSenderMapping)).name("InvitationDetailsRichMapStream");
invitationResponses.addSink(new
Neo4JInvitationSink<>(neo4jConfig)).name("InvitationRelationshipSink");
streaming.execute("InvitationJob");
Somehow I was expecting to have metrics representing the source, the sinks
and the operators, however instead of 6 entries in my JMX tree I only have
4. Please see screenshot attached. Also I was somehow expecting the JMX
objects to be named like my task / operator names but it has all sorts of
prefix/suffix magic around the names. Finally I have one custom metric which
obviously is attached to my RichMapFunction (InvitationDetailsExtractor).
However the custom metric (invitationDetailsAdded) shows up under an object
where one of the keys (which I would expect to be set to the operation name)
is a combination of the prefix "Sink" plus the name of the first sink that I
am using plus the name of the first map function (which is not the
RichMapFunction actually: my custom metric "invitationDetailsAdded" shows up
under "(Sink-_InvitationDetailSink-_TokenMapStream" which is very confusing
because this metric is actually incremented as part of the
InvitationDetailsRichMapStream).
Can somebody please explain what I can expect from metrics exposed via JMX
(should they really represent my tasks and operations) and why the naming is
so strange ?
Thanks
Philipp
<http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/file/n9560/Screen_Shot_2016-10-15_at_00.png>
--
View this message in context:
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Task-and-Operator-Monitoring-via-JMX-naming-tp9560.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at
Nabble.com.
