Also, you are right that the plugin did not have anything by default when
we created EMR 5.31 with Flink 1.11.
In opt/ I see:
[hadoop@ip-10-223-71-70 flink]$ pwd
/usr/lib/flink
[hadoop@ip-10-223-71-70 flink]$ ll opt/
total 172860
-rw-r--r-- 1 root root 24029243 Sep 19 03:08
flink-azure-fs-hadoop-1.11.0.jar
-rw-r--r-- 1 root root 185395 Sep 19 03:11 flink-cep_2.11-1.11.0.jar
-rw-r--r-- 1 root root 53473 Sep 19 03:17 flink-cep-scala_2.11-1.11.0.jar
-rw-r--r-- 1 root root 640604 Sep 19 03:16 flink-gelly_2.11-1.11.0.jar
-rw-r--r-- 1 root root 764049 Sep 19 03:16
flink-gelly-scala_2.11-1.11.0.jar
-rw-r--r-- 1 root root 268951 Sep 19 03:17 flink-ml_2.11-1.11.0.jar
-rw-r--r-- 1 root root 22316430 Sep 19 03:08 flink-oss-fs-hadoop-1.11.0.jar
-rw-r--r-- 1 root root 37228704 Sep 19 03:17 flink-python_2.11-1.11.0.jar
-rw-r--r-- 1 root root 22155 Sep 19 03:16
flink-queryable-state-runtime_2.11-1.11.0.jar
-rw-r--r-- 1 root root 19985454 Sep 19 03:08 flink-s3-fs-hadoop-1.11.0.jar
-rw-r--r-- 1 root root 36173428 Sep 19 03:08 flink-s3-fs-presto-1.11.0.jar
-rw-r--r-- 1 root root 194834 Aug 28 16:51
flink-shaded-netty-tcnative-dynamic-2.0.25.Final-11.0.jar
-rw-r--r-- 1 root root 8028165 Aug 28 17:04
flink-shaded-zookeeper-3.5.6.jar
-rw-r--r-- 1 root root 544183 Sep 19 03:17
flink-sql-client_2.11-1.11.0.jar
-rw-r--r-- 1 root root 103766 Sep 19 03:17
flink-state-processor-api_2.11-1.11.0.jar
-rw-r--r-- 1 root root 26428976 Sep 19 03:08
flink-swift-fs-hadoop-1.11.0.jar
drwxr-xr-x 2 root root 134 Oct 13 18:01 python
in lib/ I see:
[hadoop@ip-10-223-71-70 flink]$ ll lib/
total 190304
-rw-r--r-- 1 root root 90784 Sep 19 03:14 flink-csv-1.11.0.jar
-rw-r--r-- 1 root root 114256876 Sep 19 03:17 flink-dist_2.11-1.11.0.jar
-rw-r--r-- 1 root root 94866 Sep 19 03:14 flink-json-1.11.0.jar
-rw-r--r-- 1 root root 7712156 Aug 28 16:51
flink-shaded-zookeeper-3.4.14.jar
-rw-r--r-- 1 root root 33325748 Sep 19 03:17 flink-table_2.11-1.11.0.jar
-rw-r--r-- 1 root root 37330514 Sep 19 03:17
flink-table-blink_2.11-1.11.0.jar
-rw-r--r-- 1 root root 67114 Aug 28 16:50 log4j-1.2-api-2.12.1.jar
-rw-r--r-- 1 root root 276771 Aug 28 16:50 log4j-api-2.12.1.jar
-rw-r--r-- 1 root root 1674433 Aug 28 16:50 log4j-core-2.12.1.jar
-rw-r--r-- 1 root root 23518 Aug 28 16:50 log4j-slf4j-impl-2.12.1.jar
Regards,
Vijay
On Tue, Oct 27, 2020 at 10:57 AM Vijayendra Yadav <contact....@gmail.com>
wrote:
> Hi Chesnay,
>
> Steps to upgrade are as follows:
>
> 1) Created EMR 5.31 Cluster which comes with Flink 1.11
> 2) Copied flink-s3-fs-hadoop-1.11.0.jar to plugin folder for application.
>
> cd /usr/lib/flink/
>
> mkdir -p ./plugins/s3-fs-hadoop
>
> cp ./opt/flink-s3-fs-hadoop-1.11.0.jar ./plugins/s3-fs-hadoop/
>
> 3) Recompiled Application with Flink 1.11 dependency.
> 4) Updated Graphite plugin class in config
>
> That is all I did.
>
> Regards,
> Vijay
>
>
> On Tue, Oct 27, 2020 at 10:00 AM Chesnay Schepler <ches...@apache.org>
> wrote:
>
>> How exactly did you do the upgrade? Did you copy some files from 1.11
>> into an existing 1.10 distribution?
>>
>> The configuration is correct, but it appears as if the entire plugins
>> directory is either a) empty or b) not shipped.
>>
>> On 10/27/2020 5:22 PM, Vijayendra Yadav wrote:
>>
>> Hi Robert and Chesnay,
>>
>> Only thing changed is I upgraded from Flink 1.10 to 1.11 and to support
>> that updated conf yaml with factory class.
>>
>> Here is attached Full Log with classpath etc. (log.txt)
>>
>> Regards,
>> Vijay
>>
>>
>>
>> On Tue, Oct 27, 2020 at 9:31 AM Chesnay Schepler <ches...@apache.org>
>> wrote:
>>
>>> Are you writing a test? (otherwise the ReporterSetupTest reporters
>>> wouldn't be around)
>>> Do you have a dependency on the graphite reporter?
>>>
>>> On 10/27/2020 8:27 AM, Robert Metzger wrote:
>>>
>>> Hi Vijayendra,
>>> can you post or upload the entire logs, so that we can see the Classpath
>>> logged on startup, as well as the effective configuration parameters?
>>>
>>> On Tue, Oct 27, 2020 at 12:49 AM Vijayendra Yadav <contact....@gmail.com>
>>> wrote:
>>>
>>>> Hi Chesnay,
>>>>
>>>> Another log message:
>>>>
>>>> 2020-10-26 23:33:08,516 WARN
>>>> org.apache.flink.runtime.metrics.ReporterSetup - The reporter factory
>>>> (org.apache.flink.metrics.graphite.GraphiteReporterFactory) could not be
>>>> found for reporter grph. Available factories:
>>>> [org.apache.flink.runtime.metrics.ReporterSetupTest$ConfigExposingReporterFactory,
>>>> org.apache.flink.runtime.metrics.ReporterSetupTest$TestReporterFactory,
>>>> org.apache.flink.runtime.metrics.ReporterSetupTest$InstantiationTypeTrackingTestReporterFactory,
>>>> org.apache.flink.runtime.metrics.ReporterSetupTest$FailingFactory].
>>>> 2020-10-26 23:33:08,517 INFO
>>>> org.apache.flink.runtime.metrics.MetricRegistryImpl - No metrics reporter
>>>> configured, no metrics will be exposed/reported.
>>>> Regards,
>>>> Vijay
>>>>
>>>> On Mon, Oct 26, 2020 at 2:34 PM Vijayendra Yadav <contact....@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi Chesnay,
>>>>>
>>>>> I have the same, and I am exporting the flinkconf like below, where i
>>>>> have flink-conf.yaml with configuration you have given.What else can I
>>>>> try ?
>>>>>
>>>>> export FLINK_CONF_DIR=${app_install_path}/flinkconf/
>>>>>
>>>>> regards,
>>>>> Vijay
>>>>>
>>>>> On Sun, Oct 25, 2020 at 8:03 AM Chesnay Schepler <ches...@apache.org>
>>>>> wrote:
>>>>>
>>>>>> Ah wait, in 1.11 it should not longer be necessary to explicitly copy
>>>>>> the reporter jar.
>>>>>>
>>>>>> Please update your reporter configuration to this:
>>>>>>
>>>>>> metrics.reporter.grph.factory.class:
>>>>>> org.apache.flink.metrics.graphite.GraphiteReporterFactory
>>>>>>
>>>>>> On 10/25/2020 4:00 PM, Chesnay Schepler wrote:
>>>>>>
>>>>>> Have you followed the documentation, specifically this bit?
>>>>>>
>>>>>> > In order to use this reporter you must copy
>>>>>> /opt/flink-metrics-influxdb-1.11.2.jar into the plugins/influxdb
>>>>>> folder of your Flink distribution.
>>>>>>
>>>>>> On 10/24/2020 12:17 AM, Vijayendra Yadav wrote:
>>>>>>
>>>>>> Hi Team,
>>>>>>
>>>>>> for Flink 1.11 Graphite Metrics. I see the following Error in the
>>>>>> log.
>>>>>> Any suggestions?
>>>>>>
>>>>>> 020-10-23 21:55:14,652 ERROR
>>>>>> org.apache.flink.runtime.metrics.ReporterSetup - Could
>>>>>> not instantiate metrics reporter grph. Metrics might not be
>>>>>> exposed/reported.
>>>>>> java.lang.ClassNotFoundException:
>>>>>> org.apache.flink.metrics.graphite.GraphiteReporter
>>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
>>>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
>>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
>>>>>> at java.lang.Class.forName0(Native Method)
>>>>>> at java.lang.Class.forName(Class.java:264)
>>>>>> at
>>>>>> org.apache.flink.runtime.metrics.ReporterSetup.loadViaReflection(ReporterSetup.java:313)
>>>>>> at
>>>>>> org.apache.flink.runtime.metrics.ReporterSetup.loadReporter(ReporterSetup.java:274)
>>>>>> at
>>>>>> org.apache.flink.runtime.metrics.ReporterSetup.setupReporters(ReporterSetup.java:235)
>>>>>> at
>>>>>> org.apache.flink.runtime.metrics.ReporterSetup.fromConfiguration(ReporterSetup.java:148)
>>>>>> at
>>>>>> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.createMetricRegistry(ClusterEntrypoint.java:316)
>>>>>> at
>>>>>> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.initializeServices(ClusterEntrypoint.java:270)
>>>>>> at
>>>>>> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runCluster(ClusterEntrypoint.java:208)
>>>>>> at
>>>>>> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.lambda$startCluster$0(ClusterEntrypoint.java:169)
>>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>>> at javax.security.auth.Subject.doAs(Subject.java:422)
>>>>>> at
>>>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1893)
>>>>>> at
>>>>>> org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
>>>>>> at
>>>>>> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:168)
>>>>>> at
>>>>>> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runClusterEntrypoint(ClusterEntrypoint.java:517)
>>>>>> at
>>>>>> org.apache.flink.yarn.entrypoint.YarnJobClusterEntrypoint.main(YarnJobClusterEntrypoint.java:89)
>>>>>>
>>>>>>
>>>>>> Regards,
>>>>>> Vijay
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>
>>
