Hi Chesnay,
Steps to upgrade are as follows:
1) Created EMR 5.31 Cluster which comes with Flink 1.11
2) Copied flink-s3-fs-hadoop-1.11.0.jar to plugin folder for application.
cd /usr/lib/flink/
mkdir -p ./plugins/s3-fs-hadoop
cp ./opt/flink-s3-fs-hadoop-1.11.0.jar ./plugins/s3-fs-hadoop/
3) Recompiled Application with Flink 1.11 dependency.
4) Updated Graphite plugin class in config
That is all I did.
Regards,
Vijay
On Tue, Oct 27, 2020 at 10:00 AM Chesnay Schepler <[email protected]>
wrote:
> How exactly did you do the upgrade? Did you copy some files from 1.11 into
> an existing 1.10 distribution?
>
> The configuration is correct, but it appears as if the entire plugins
> directory is either a) empty or b) not shipped.
>
> On 10/27/2020 5:22 PM, Vijayendra Yadav wrote:
>
> Hi Robert and Chesnay,
>
> Only thing changed is I upgraded from Flink 1.10 to 1.11 and to support
> that updated conf yaml with factory class.
>
> Here is attached Full Log with classpath etc. (log.txt)
>
> Regards,
> Vijay
>
>
>
> On Tue, Oct 27, 2020 at 9:31 AM Chesnay Schepler <[email protected]>
> wrote:
>
>> Are you writing a test? (otherwise the ReporterSetupTest reporters
>> wouldn't be around)
>> Do you have a dependency on the graphite reporter?
>>
>> On 10/27/2020 8:27 AM, Robert Metzger wrote:
>>
>> Hi Vijayendra,
>> can you post or upload the entire logs, so that we can see the Classpath
>> logged on startup, as well as the effective configuration parameters?
>>
>> On Tue, Oct 27, 2020 at 12:49 AM Vijayendra Yadav <[email protected]>
>> wrote:
>>
>>> Hi Chesnay,
>>>
>>> Another log message:
>>>
>>> 2020-10-26 23:33:08,516 WARN
>>> org.apache.flink.runtime.metrics.ReporterSetup - The reporter factory
>>> (org.apache.flink.metrics.graphite.GraphiteReporterFactory) could not be
>>> found for reporter grph. Available factories:
>>> [org.apache.flink.runtime.metrics.ReporterSetupTest$ConfigExposingReporterFactory,
>>> org.apache.flink.runtime.metrics.ReporterSetupTest$TestReporterFactory,
>>> org.apache.flink.runtime.metrics.ReporterSetupTest$InstantiationTypeTrackingTestReporterFactory,
>>> org.apache.flink.runtime.metrics.ReporterSetupTest$FailingFactory].
>>> 2020-10-26 23:33:08,517 INFO
>>> org.apache.flink.runtime.metrics.MetricRegistryImpl - No metrics reporter
>>> configured, no metrics will be exposed/reported.
>>> Regards,
>>> Vijay
>>>
>>> On Mon, Oct 26, 2020 at 2:34 PM Vijayendra Yadav <[email protected]>
>>> wrote:
>>>
>>>> Hi Chesnay,
>>>>
>>>> I have the same, and I am exporting the flinkconf like below, where i
>>>> have flink-conf.yaml with configuration you have given.What else can I try
>>>> ?
>>>>
>>>> export FLINK_CONF_DIR=${app_install_path}/flinkconf/
>>>>
>>>> regards,
>>>> Vijay
>>>>
>>>> On Sun, Oct 25, 2020 at 8:03 AM Chesnay Schepler <[email protected]>
>>>> wrote:
>>>>
>>>>> Ah wait, in 1.11 it should not longer be necessary to explicitly copy
>>>>> the reporter jar.
>>>>>
>>>>> Please update your reporter configuration to this:
>>>>>
>>>>> metrics.reporter.grph.factory.class:
>>>>> org.apache.flink.metrics.graphite.GraphiteReporterFactory
>>>>>
>>>>> On 10/25/2020 4:00 PM, Chesnay Schepler wrote:
>>>>>
>>>>> Have you followed the documentation, specifically this bit?
>>>>>
>>>>> > In order to use this reporter you must copy
>>>>> /opt/flink-metrics-influxdb-1.11.2.jar into the plugins/influxdb
>>>>> folder of your Flink distribution.
>>>>>
>>>>> On 10/24/2020 12:17 AM, Vijayendra Yadav wrote:
>>>>>
>>>>> Hi Team,
>>>>>
>>>>> for Flink 1.11 Graphite Metrics. I see the following Error in the log.
>>>>> Any suggestions?
>>>>>
>>>>> 020-10-23 21:55:14,652 ERROR
>>>>> org.apache.flink.runtime.metrics.ReporterSetup - Could not
>>>>> instantiate metrics reporter grph. Metrics might not be exposed/reported.
>>>>> java.lang.ClassNotFoundException:
>>>>> org.apache.flink.metrics.graphite.GraphiteReporter
>>>>> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
>>>>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
>>>>> at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
>>>>> at java.lang.Class.forName0(Native Method)
>>>>> at java.lang.Class.forName(Class.java:264)
>>>>> at
>>>>> org.apache.flink.runtime.metrics.ReporterSetup.loadViaReflection(ReporterSetup.java:313)
>>>>> at
>>>>> org.apache.flink.runtime.metrics.ReporterSetup.loadReporter(ReporterSetup.java:274)
>>>>> at
>>>>> org.apache.flink.runtime.metrics.ReporterSetup.setupReporters(ReporterSetup.java:235)
>>>>> at
>>>>> org.apache.flink.runtime.metrics.ReporterSetup.fromConfiguration(ReporterSetup.java:148)
>>>>> at
>>>>> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.createMetricRegistry(ClusterEntrypoint.java:316)
>>>>> at
>>>>> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.initializeServices(ClusterEntrypoint.java:270)
>>>>> at
>>>>> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runCluster(ClusterEntrypoint.java:208)
>>>>> at
>>>>> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.lambda$startCluster$0(ClusterEntrypoint.java:169)
>>>>> at java.security.AccessController.doPrivileged(Native Method)
>>>>> at javax.security.auth.Subject.doAs(Subject.java:422)
>>>>> at
>>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1893)
>>>>> at
>>>>> org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
>>>>> at
>>>>> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:168)
>>>>> at
>>>>> org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runClusterEntrypoint(ClusterEntrypoint.java:517)
>>>>> at
>>>>> org.apache.flink.yarn.entrypoint.YarnJobClusterEntrypoint.main(YarnJobClusterEntrypoint.java:89)
>>>>>
>>>>>
>>>>> Regards,
>>>>> Vijay
>>>>>
>>>>>
>>>>>
>>>>>
>>
>
