Re: Working with bounded Datastreams - Flink 1.11.1

Hi All, Request your inputs please. Regards,Sunitha On Tuesday, October 27, 2020, 01:01:41 PM GMT+5:30, s_penakalap...@yahoo.com wrote: Hi Team, I want to use Flink Datastream for Batch operations which involves huge data, I did try to calculate count and average on the whole Datastrea

Re: How to understand NOW() in SQL when using Table & SQL API to develop a streaming app?

Our behavior also conflicts with the SQL standard, we should also mention this in the document. Till Rohrmann 于2020年10月27日周二 下午10:37写道： > Thanks for the clarification. This improvement would be helpful, I believe. > > Cheers, > Till > > On Tue, Oct 27, 2020 at 1:19 PM Jark Wu wrote: > >> Hi Til

Re: ValidationException using DataTypeHint in Scalar Function

Hi Dawid, I added `*bridgedTo = Map.class*` as you suggested and got a slightly different exception. I also tried passing a rawSerializer (an implementation similar to MapSerializer[1] with String type key and value) but got the same exception as without it. I am using Flink v1.11 for reference.

Re: FLINK 1.11 Graphite Metrics

Thank You for help, Let's raise a case with AWS. On Tue, Oct 27, 2020 at 1:58 PM Chesnay Schepler wrote: > In the normal Flink distribution these jars were moved from opt/ to > plugins/ so that they are available by default without having to mess > around with any jars. > I don't think anyone wa

Re: FLINK 1.11 Graphite Metrics

In the normal Flink distribution these jars were moved from opt/ to plugins/ so that they are available by default without having to mess around with any jars. I don't think anyone was aware that the plugin directory is not populated on EMR. On 10/27/2020 9:53 PM, Vijayendra Yadav wrote: Perfe

Re: FLINK 1.11 Graphite Metrics

Perfect after downloading it into the plugin, it is working well. I am wondering why these jars have been removed from opt/ folder, earlier I was able to copy from opt/ to lib/ folder for 1.10. For now I just downloaded from Maven for 1.11 and copied in plugin/. Regards, Vijay On Tue, Oct 27, 202

Re: how to enable metrics in Flink 1.11

Hi Robert, Could please correct me. I'm not able to stop the app. Also, i stopped flink job already. sh-4.2$ yarn app -stop application_1603649952937_0002 2020-10-27 20:04:25,543 INFO client.RMProxy: Connecting to ResourceManager at ip-10-0-55-50.ec2.internal/10.0.55.50:8032 2020-10-27 20:04:25,71

Re: FLINK 1.11 Graphite Metrics

So the plugins directory is completely empty? In that case, please download the flink-metrics-graphite jar and also copy it into the plugins directory. On 10/27/2020 7:04 PM, Vijayendra Yadav wrote: Also, you

Re: FLINK 1.11 Graphite Metrics

Also, you are right that the plugin did not have anything by default when we created EMR 5.31 with Flink 1.11. In opt/ I see: [hadoop@ip-10-223-71-70 flink]$ pwd /usr/lib/flink [hadoop@ip-10-223-71-70 flink]$ ll opt/ total 172860 -rw-r--r-- 1 root root 24029243 Sep 19 03:08 flink-azure-fs-hadoop-

Re: FLINK 1.11 Graphite Metrics

Hi Chesnay, Steps to upgrade are as follows: 1) Created EMR 5.31 Cluster which comes with Flink 1.11 2) Copied flink-s3-fs-hadoop-1.11.0.jar to plugin folder for application. cd /usr/lib/flink/ mkdir -p ./plugins/s3-fs-hadoop cp ./opt/flink-s3-fs-hadoop-1.11.0.jar ./plugins/s3

Re: EMR Logging Woes

Thanks! I'll check these out. On Tue, Oct 27, 2020 at 12:58 AM Robert Metzger wrote: > Hi Rex, > > 1. You can also use the Flink UI for retrieving logs. That usually works > quite fast (unless your logs are huge). > > 2. These are the correct configuration files for setting the log level. > Are

Re: FLINK 1.11 Graphite Metrics

How exactly did you do the upgrade? Did you copy some files from 1.11 into an existing 1.10 distribution? The configuration is correct, but it appears as if the entire plugins directory is either a) empty or b) not shipped. On 10/27/2020 5:22 PM, Vijayendra Yadav wrote: Hi Robert and Chesnay

RE: Dependency vulnerabilities with flink 1.11.1 version

Thanks Robert. Regards, Suchithra From: Robert Metzger Sent: Tuesday, October 27, 2020 9:10 PM To: Till Rohrmann Cc: V N, Suchithra (Nokia - IN/Bangalore) ; user@flink.apache.org Subject: Re: Dependency vulnerabilities with flink 1.11.1 version FYI: For the sake of completeness, I have added

Re: Dependency vulnerabilities with flink 1.11.1 version

FYI: For the sake of completeness, I have added some reasoning to all the JIRA tickets why we are not backporting fixes to the 1.11-line of Flink. On Mon, Oct 26, 2020 at 4:51 PM Robert Metzger wrote: > Hey Suchithra, > thanks a lot for this report. I'm in the process of closing all the > ticket

Re: Flink checkpointing state

Thanks Yun, This refers to Flip144 https://cwiki.apache.org/confluence/display/FLINK/FLIP-144%3A+Native+Kubernetes+HA+for+Flink Flip contains 2 parts - leader election and HA information persistence and

Re: How to understand NOW() in SQL when using Table & SQL API to develop a streaming app?

Thanks for the clarification. This improvement would be helpful, I believe. Cheers, Till On Tue, Oct 27, 2020 at 1:19 PM Jark Wu wrote: > Hi Till, > > The documentation mentions that "this function is not deterministic" where > the "not deterministic" means the value of this function is not > d

[ANNOUNCE] Weekly Community Update 2020/39-43

Dear community, happy to finally share another community update with you. With Flink Forward and the release of Flink SQL on Ververica Platform I was a bit too busy the previous weeks. I'll try to return to the weekly cadence going forward again. This time we have a lot of ground to cover. With th

Re: How to understand NOW() in SQL when using Table & SQL API to develop a streaming app?

Hi Till, The documentation mentions that "this function is not deterministic" where the "not deterministic" means the value of this function is not deterministic for every record. However, this is not very clear for users. I think we can improve the documentation. Best, Jark On Tue, 27 Oct 2020

Fwd: Flink memory usage monitoring

I missed adding the mailing list in my previous email. -- Forwarded message - From: Matthias Pohl Date: Tue, Oct 27, 2020 at 12:39 PM Subject: Re: Flink memory usage monitoring To: Rajesh Payyappilly Jose Hi Rajesh, thanks for reaching out to us. We worked on providing metrics

Re: [BULK]Re: [SURVEY] Remove Mesos support

Hello Xintong, Thanks for the insights and support. Browsing the Mesos backlog and didn't identify anything critical, which is left there. I see that there are were quite a lot of contributions to the Flink Mesos in the recent version: https://github.com/apache/flink/commits/master/flink-mesos

Re: adding core-site xml to flink1.11

Hi, it seems that this is what you have to do for now. However, I see that it would be nice if Flink would allow reading from multiple configuration files, so that you can have a "common configuration" and a "per cluster" configuration. I filed a JIRA ticket for a feature request: https://issues.

Native memory allocation (mmap) failed to map 1006567424 bytes

After the job is running for 10 days in production, TaskManagers start failing with: Connection unexpectedly closed by remote task manager Looking in the machine logs, I can see the following error: = Java processes for user hadoop = OpenJDK 64-Bit Server VM warning: INFO

Re: ValidationException using DataTypeHint in Scalar Function

Hey Steve, You should be able to do via the bridgedTo parameter. You can additionally specify a serializer you want to use via rawSerializer parameter:         @FunctionHint(                 input = {                         @DataTypeHint(value = "RAW", bridgedTo = Map.class[, rawSerializer = ...

Re: RestClusterClient and classpath

In the logs I see that the jar is the classpath (I'm trying to debug the program from the IDE)..isn'it? Classpath: [file:/tmp/job-bundle.jar] ... Best, Flavio On Tue, Oct 27, 2020 at 10:39 AM Chesnay Schepler wrote: > * your JobExecutor is _not_ putting it on the classpath. > > On 10/27/2020 1

Re: adding core-site xml to flink1.11

Hi, Thank you for your reply, WE are deploying on kubernetes and the xml is part of the common config map to all flink jobs we have(or at least was for previous versions) This means that we need to duplicate the configuration in the flink-conf.yaml for each job instead of having a common confi

Re: Feature request: Removing state from operators

Hi Peter, I'm adding two committers to this thread who can help answering your question. On Mon, Oct 26, 2020 at 3:22 PM Peter Westermann wrote: > We use the feature for removing stateful operators via the > *allowNonRestoredState* relatively often and it works great. However, > there doesn’t s

Re: RestClusterClient and classpath

* your JobExecutor is _not_ putting it on the classpath. On 10/27/2020 10:36 AM, Chesnay Schepler wrote: Well it happens on the client before you even hit the RestClusterClient, so I assume that either your jar is not packaged correctly or you your JobExecutor is putting it on the classpath.

Re: RestClusterClient and classpath

Well it happens on the client before you even hit the RestClusterClient, so I assume that either your jar is not packaged correctly or you your JobExecutor is putting it on the classpath. On 10/27/2020 9:42 AM, Flavio Pompermaier wrote: Sure. Here it is (org.apache.flink.client.cli.JobExecutor

Re: FLINK 1.11 Graphite Metrics

Are you writing a test? (otherwise the ReporterSetupTest reporters wouldn't be around) Do you have a dependency on the graphite reporter? On 10/27/2020 8:27 AM, Robert Metzger wrote: Hi Vijayendra, can you post or upload the entire logs, so that we can see the Classpath logged on startup, as w

Re: adding core-site xml to flink1.11

Hi Shachar, Why do you want to use the core-site.xml to configure the file system? Since we are adding the file systems as plugins, their initialization is customized. It might be the case that we are intentionally ignoring xml configurations from the classpath. You can configure the filesystem i

Re: RestClusterClient and classpath

Sure. Here it is (org.apache.flink.client.cli.JobExecutor is my main class I'm trying to use as a client towards the Flink cluster - session mode). it/test/MyOb is within the fat jar (/tmp/job-bundle.jar). The code of getBatchEnv is: @Deprecated public static BatchEnv getBatchEnv() { // TOD

Re: HA on AWS EMR

Hello Robert, Thanks for the info. That makes sense. I will save and cancel my jobs with 1.10, upgrade to 1.11, and restore the jobs from the savepoints. Thanks and regards, Averell -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: RestClusterClient and classpath

Hi Flavio, can you share the full stacktrace you are seeing? I'm wondering if the error happens on the client or server side (among other questions I have). On Mon, Oct 26, 2020 at 5:58 PM Flavio Pompermaier wrote: > Hi to all, > I was trying to use the RestClusterClient to submit my job to the

Re: [BULK]Re: Support of composite data types in flink-parquet

Indeed, thanks Andrey! From: Andrey Zagrebin Date: Tuesday, October 20, 2020 at 6:03 PM To: Jon Alberdi Cc: user@flink.apache.org Subject: [BULK]Re: Support of composite data types in flink-parquet Hi Jon, I have found this ticket [1]. It looks like what you are looking for. Best, Andrey [1]

Re: How to understand NOW() in SQL when using Table & SQL API to develop a streaming app?

Quick question Jark: Is this difference in behaviour documented? I couldn't find it in the docs. Cheers, Till On Tue, Oct 27, 2020 at 7:30 AM Jark Wu wrote: > Hi Longdexin, > > In traditional batch sql, NOW() is executed and determined before the job > is submitted and will not change for every

Re: EMR Logging Woes

Hi Rex, 1. You can also use the Flink UI for retrieving logs. That usually works quite fast (unless your logs are huge). 2. These are the correct configuration files for setting the log level. Are you running on a vanilla EMR cluster, or are there modifications? The "problem" is that Flink on YAR

Working with bounded Datastreams - Flink 1.11.1

Hi Team, I want to use Flink Datastream for Batch operations which involves huge data, I did try to calculate count and average on the whole Datastream with out using window function.  Approach I tried to calculate count on the datastream:1> Read data from table (say past 2 days of data) as Data

Re: FLINK 1.11 Graphite Metrics

Hi Vijayendra, can you post or upload the entire logs, so that we can see the Classpath logged on startup, as well as the effective configuration parameters? On Tue, Oct 27, 2020 at 12:49 AM Vijayendra Yadav wrote: > Hi Chesnay, > > Another log message: > > 2020-10-26 23:33:08,516 WARN > org.apa

Re: how to enable metrics in Flink 1.11

Hey Diwakar, how are you deploying Flink on EMR? Are you using YARN? If so, you could also use log aggregation to see all the logs at once (from both JobManager and TaskManagers). (yarn logs -applicationId ) Could you post (or upload somewhere) all logs you have of one run? It is much easier for

[Flink::Test] access registered accumulators via harness

Hi mates ! I guess that I'm doing something wrong, but I couldn't find a way to access registered accumulators and their values via *org.apache.flink.streaming.util.**ProcessFunctionTestHarness *function wrapper that I'm using to test my functions. During the code research I've found, that requir

Re: HA on AWS EMR

Hey Averell, to clarify: You should be able to migrate using a savepoint from 1.10 to 1.11. Restoring from the state stored in Zookeeper (for HA) with a newer Flink version won't work. On Mon, Oct 26, 2020 at 5:05 PM Robert Metzger wrote: > Hey Averell, > > you should be able to migrate savepoi