An update in the yarn logs I could see the below -
Classpath:
*lib/flink-metrics-influxdb-1.9.0.jar:lib/flink-shaded-hadoop-2-uber-2.8.5-amzn-5-7.0.jar:lib/flink-table-blink_2.11-1.9.0.jar:lib/flink-table_2.11-1.9.0.jar:lib/log4j-1.2.17.jar:lib/slf4j-log4j12-1.7.15.jar:log4j.properties:plugins/influxdb/flink-metrics-influxdb-1.9.0.jar....*
*..........*
*......*
This means the jar is getting loaded, in the logs I could also see -
2020-08-12 15:28:51,505 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
- Registered UNIX signal handlers for [TERM, HUP, I
NT]
2020-08-12 15:28:51,508 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
- Current working Directory: /mnt/yarn/usercache/ha
doop/appcache/application_1595767096609_0013/container_1595767096609_0013_01_000004
*2020-08-12 15:28:51,512 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: metrics.reporter.influxdb.interval, 60 SECONDS*
*2020-08-12 15:28:51,512 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: env.yarn.conf.dir, /etc/hadoop/conf*
2020-08-12 15:28:51,513 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: metrics.reporter.
influxdb.host, xx.xxx.xxx.xx
2020-08-12 15:28:51,513 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: high-availability
.cluster-id, application_1595767096609_0013
2020-08-12 15:28:51,513 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: jobmanager.rpc.ad
dress, ip-xx-x-xx-xxx.ec2.internal
2020-08-12 15:28:51,513 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: metrics.reporter.
influxdb.password, ******
*2020-08-12 15:28:51,513 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: FLINK_PLUGINS_DIR, /usr/lib/flink/plugins*
2020-08-12 15:28:51,513 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: metrics.reporter.
influxdb.db, xxxxxx
2020-08-12 15:28:51,520 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: metrics.reporter.
influxdb.connectTimeout, 60000
2020-08-12 15:28:51,520 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: env.hadoop.conf.d
ir, /etc/hadoop/conf
2020-08-12 15:28:51,521 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: taskmanager.numbe
rOfTaskSlots, 1
2020-08-12 15:28:51,521 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: web.port, 0
2020-08-12 15:28:51,521 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: metrics.reporter.influxdb.username, xxxx
2020-08-12 15:28:51,521 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: taskmanager.memory.size, 264241152b
2020-08-12 15:28:51,521 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: web.tmpdir,
/tmp/flink-web-5562f065-6020-4c38-8260-3aea434bf285
2020-08-12 15:28:51,521 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: jobmanager.rpc.port, 32777
2020-08-12 15:28:51,521 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: metrics.reporter.influxdb.port, 8086
2020-08-12 15:28:51,521 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: metrics.reporter.influxdb.retentionPolicy, one_hour
2020-08-12 15:28:51,522 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: internal.cluster.execution-mode, NORMAL
2020-08-12 15:28:51,522 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: metrics.reporter.influxdb.writeTimeout, 60000
2020-08-12 15:28:51,522 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: metrics.reporter.influxdb.consistency, ONE
2020-08-12 15:28:51,522 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: rest.address, ip-xx-x-xx-xxx.ec2.internal
*2020-08-12 15:28:51,522 INFO
org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: metrics.reporter.influxdb.factory.class,
org.apache.flink.metrics.influxdb.InfluxdbReporterFactory*
.......
but then below I could see -
*2020-08-12 15:28:51,523 WARN org.apache.flink.core.plugin.PluginConfig
- Environment variable [FLINK_PLUGINS_DIR] is set to
[/usr/lib/flink/plugins] but the directory doesn't exist*
2020-08-12 15:28:51,561 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
- Current working/local Directory:
/mnt/yarn/usercache/hadoop/appcache/application_1595767096609_0013,/mnt1/yarn/usercache/hadoop/appcache/application_1595767096609_0013
2020-08-12 15:28:51,564 INFO
org.apache.flink.runtime.clusterframework.BootstrapTools - Setting
directories for temporary files to:
/mnt/yarn/usercache/hadoop/appcache/application_1595767096609_0013,/mnt1/yarn/usercache/hadoop/appcache/application_1595767096609_0013
2020-08-12 15:28:51,564 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
- TM: remote keytab path obtained null
2020-08-12 15:28:51,564 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
- TM: remote keytab principal obtained null
2020-08-12 15:28:51,566 INFO org.apache.flink.yarn.YarnTaskExecutorRunner
- YARN daemon is running as: hadoop Yarn client user
obtainer: hadoop
2020-08-12 15:28:51,675 INFO
org.apache.flink.runtime.security.modules.HadoopModule - Hadoop
user set to hadoop (auth:xxxxxx)
2020-08-12 15:28:51,984 WARN org.apache.flink.configuration.Configuration
- Config uses deprecated configuration key 'web.port'
instead of proper key 'rest.port'
2020-08-12 15:28:51,987 INFO
org.apache.flink.runtime.taskexecutor.TaskManagerRunner - Using
configured hostname/address for TaskManager: ip-xx-x-xx-xxx.ec2.internal.
2020-08-12 15:28:51,996 INFO
org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils - Trying to
start actor system at ip-xx-x-xx-xxx.ec2.internal:0
2020-08-12 15:28:52,823 INFO akka.event.slf4j.Slf4jLogger
- Slf4jLogger started
2020-08-12 15:28:52,854 INFO akka.remote.Remoting
- Starting remoting
2020-08-12 15:28:53,061 INFO akka.remote.Remoting
- Remoting started; listening on addresses
:[akka.tcp://flink@ip-xx-x-xx-xxx.ec2.internal:37937]
2020-08-12 15:28:53,563 INFO
org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils - Actor
system started at akka.tcp://flink@iip-xx-x-xx-xxx.ec2.ec2.internal:37937
*2020-08-12 15:28:53,593 WARN
org.apache.flink.runtime.metrics.ReporterSetup - The
reporter factory
(org.apache.flink.metrics.influxdb.InfluxdbReporterFactory) could not be
found for reporter influxdb. Available factories:*
*2020-08-12 15:28:53,597 INFO
org.apache.flink.runtime.metrics.MetricRegistryImpl - No metrics
reporter configured, no metrics will be exposed/reported.*2020-08-12
15:28:53,599 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils
- Trying to start actor system at ip-xx-x-xx-xxx.ec2.ec2.internal:0
So at one place org.apache.flink.configuration.GlobalConfiguration refers
to the properties and metrics reported but
then org.apache.flink.runtime.metrics.ReporterSetup complains of not
finding it.
Can anyone guide what I am missing here.
Thanks,
Hemant
On Wed, Aug 12, 2020 at 9:15 PM bat man <tintin0...@gmail.com> wrote:
> Hello Experts,
>
> I am running Flink - 1.9.0 on AWS EMR(emr-5.28.1). I want to push
> metrics to Influxdb. I followed the documentation[1]. I added the
> configuration to /usr/lib/flink/conf/flink-conf.yaml and copied the jar to
> /usr/lib/flink//lib folder on master node. However, I also
> understand that the cluster might need a re-start as only with these steps
> when I run the job I don't see any measurement(table) created in my influx
> db. I am not able to find any documentation on how to restart the cluster
> on EMR.
> Anyone who has configured to push metrics to InfluxDB from AWS EMR could
> you share the steps please.
>
> [1]
> https://ci.apache.org/projects/flink/flink-docs-release-1.9/monitoring/metrics.html#influxdb-orgapacheflinkmetricsinfluxdbinfluxdbreporter
>
> Thanks,
> Hemant
>
