Re: problem with increase job parallelism

Hi Lei, setting explicit operator ID should solve this issue. As far as I know, the auto-generated operator id also depended on the operator parallelism in previous versions of Flink (not sure until which point). Which version are you running? Best, Fabian 2017-10-17 3:15 GMT+02:00 Lei Chen :

Re: HBase config settings go missing within Yarn.

Is this /etc/hbase/conf/hbase-site.xml file is present on all of the machines? If yes, could you share your code? > On 20 Oct 2017, at 16:29, Niels Basjes wrote: > > I look at the logfiles from the Hadoop Yarn webinterface. I.e. actually > looking in the jobmanager.log of the container running

BucketingSink with disabled checkpointing will never clean up it's state

Hi, got one more little question about BucketingSink with disabled checkpointing. In terms of my current task, I’m looking through sources of BucketingSink and it seem’s that I found an issue for the case, when checkpointing is disabled. BucketingSink - is a flink rich function, that also implem

Re: Avoid duplicate messages while restarting a job for an application upgrade

Hi Piotrek, I come back to you with a Jira ticket that I created and a proposal the ticket : https://issues.apache.org/jira/browse/FLINK-7883 the proposal : https://github.com/aphilippot/flink/commit/9c58c95bb4b68ea337f7c583b7e039d86f3142a6 I'am open to any comments or suggestions Antoine Le m

Re: SLF4j logging system gets clobbered?

This is with Flink 1.3.2. I’ll paste the full contents of the flink lib dir below, as well as the maven shade config for our job jar. lib/ -rw-r--r-- 1 root root62983 Oct 18 18:24 activation-1.1.jar -rw-r--r-- 1 root root44925 Oct 18 18:24 apacheds-i18n-2.0.0-M15.jar -rw-r--r-- 1 root

Re: HBase config settings go missing within Yarn.

I look at the logfiles from the Hadoop Yarn webinterface. I.e. actually looking in the jobmanager.log of the container running the Flink task. That is where I was able to find these messages . I do the hbaseConfig.addResource(new Path("file:/etc/hbase/conf/hbase-site.xml")); in all places directl

Re:

Hi, Only batch API is using managed memory. If you are using streaming API, you can do two things: - estimate max cache size based on for example fraction of max heap size - use WeakReference to implement your cache In batch API, you could estimate max cache size based on: - fraction of (heapSi

Re: Question about configuring Rich Functions

This thread [1] is where I asked about it as well. [1] http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/LocalStreamEnvironment-configuration-doesn-t-seem-to-be-used-in-RichFunction-operators-td15751.html#a15777 On Fri, Oct 13, 2017 at 9:50 PM Tony Wei wrote: > Hi Steve, > >

Re: flink can't read hdfs namenode logical url

Hi, Please double check the content of config files in YARN_CONF_DIR and HADOOP_CONF_DIR (the first one has a priority over the latter one) and that they are pointing to correct files. Also check logs (WARN and INFO) for any relevant entries. Piotrek > On 20 Oct 2017, at 06:07, 邓俊华 wrote: >

Re: java.lang.NoSuchMethodError and dependencies problem

That’s good to hear :) Unfortunately at this moment dependencies can pollute class path in both ways (Flink’s can mess with user’s application and also the other way around). Cheers, Piotrek > On 20 Oct 2017, at 15:11, r. r. wrote: > > By Ali Baba's beard and his forty bandits, Piotrek, this

Re: java.lang.NoSuchMethodError and dependencies problem

By Ali Baba's beard and his forty bandits, Piotrek, this worked! My understanding was that I have to prevent Flink from loading the older compress.jar and force the newer one. One I shade-relocated org.apache.commons.compress for my project the problem went away Many thanks! > Ори

Re: Does heap memory gets released after computing windows?

Hi, Memory used by session windows should be released once window is triggered (allowedLateness can prolong window’s life). Unless your code introduces some memory leak (by not releasing references) everything should be garbage collected. Keep in mind that session windows with time gap of 10 m

Re: HBase config settings go missing within Yarn.

Hi, What do you mean by saying: > When I open the logfiles on the Hadoop cluster I see this: The error doesn’t come from Flink? Where do you execute hbaseConfig.addResource(new Path("file:/etc/hbase/conf/hbase-site.xml")); ? To me it seems like it is a problem with misconfigured HBase and n

Re: Flink BucketingSink, subscribe on moving of file into final state

You’re welcome. Unfortunately I am not aware about a such use case before Piotrek > On 20 Oct 2017, at 13:47, Rinat wrote: > > Piotrek, thanks for your reply. > > Yes, now I’m looking for the most suitable way to extend BucketingSink > functionality, to handle moments of moving the file into

Re: Parallelism, registerEventTimeTimer and watermark problem

Hi Fritz, If the watermark is not updating this usually means that one of the input partitions (if you're using Kafka) is not carrying data. In that case, the watermark/timestamp assigner will have no data on which to base an updated watermark. For such use cases I recently implemented a specia

Re: Custom Sink Checkpointing errors

Thanks Stefan for the answers. The serialization is happening during the creation of snapshot state. I have added a gist with a larger stacktrace( https://gist.github.com/neoeahit/aee5562bf0b8d8d02e2a012f6735d850). I am not using any serializer, in the custom sink. We have src.keyBy(m => (m.topic

Re: Flink BucketingSink, subscribe on moving of file into final state

Piotrek, thanks for your reply. Yes, now I’m looking for the most suitable way to extend BucketingSink functionality, to handle moments of moving the file into final state. I thought, that maybe someone has already implemented such thing or knows any other approaches that will help me to not cop

Re: java.lang.NoSuchMethodError and dependencies problem

But you said > this seems to work as mvn dependency:tree -Ddetail=true only shows 1.14 To avoid this error that you describe I think that you have to ensure, that no 1.14 commons-compress comes from your application, because it can conflict with 1.4.1 used by flink cluster. By shading I meant

Re: HBase config settings go missing within Yarn.

To facilitate you guys helping me I put this test project on github: https://github.com/nielsbasjes/FlinkHBaseConnectProblem Niels Basjes On Fri, Oct 20, 2017 at 1:32 PM, Niels Basjes wrote: > Hi, > > Ik have a Flink 1.3.2 application that I want to run on a Hadoop yarn > cluster where I need t

Re: Flink BucketingSink, subscribe on moving of file into final state

Hi, Maybe you can just list files in your basePath and filter out those that have inProgress or pending suffixes? I think you could wrap/implement your own Bucketer and track all the paths that it returns. However some of those might be pending or in progress files that will be committed in t

HBase config settings go missing within Yarn.

Hi, Ik have a Flink 1.3.2 application that I want to run on a Hadoop yarn cluster where I need to connect to HBase. What I have: In my environment: HADOOP_CONF_DIR=/etc/hadoop/conf/ HBASE_CONF_DIR=/etc/hbase/conf/ HIVE_CONF_DIR=/etc/hive/conf/ YARN_CONF_DIR=/etc/hadoop/conf/ In /etc/hbase/conf/

Does heap memory gets released after computing windows?

Hi All, I have been facing "GC Overhead limit exceeded" exception for quite a time with my flink program. In my flink program, I have created a session window with time gap of 10 minutes and I am performing some custom reduce operation using Window function. The current statebackend is File System

Re: Custom Sink Checkpointing errors

Hi, the crash looks unrelated to Flink code from the dump’s trace. Since it happens somewhere in managing a jar file, it might be related to this: https://bugs.openjdk.java.net/browse/JDK-8142508 , point (2). Maybe your jar gets overwritten wh

Re: Flink Streaming example: Kafka010Example.scala doesn't work

Hi, Thanks for reporting back! That’s good to know. Gordon On 20 October 2017 at 3:51:21 PM, Wojtkowski, Michal (michal.wojtkowski@roche.com) wrote: Hi Gordon,  Thanks for finding time to write back! I managed to solve the issue and it turned out to be entirely related to kafka instance

Flink BucketingSink, subscribe on moving of file into final state

Hi All ! I’m trying to create a meta-info file, that contains link to file, created by Flink BucketingSink. At first I was trying to implement my own org.apache.flink.streaming.connectors.fs.Writer, that creates a meta-file on close method call. But I understood, that it’s not completely right