Re: Why I am getting Null pointer exception while accessing RuntimeContext in FlinkKafkaProducer010 ?

I’m afraid you would have to copy the code and to the edits until we resolve the issue. > On 19. Apr 2017, at 06:38, sohimankotia wrote: > > Thanks Aljoscha for reply . > > Is there any way where I can add custom metrics or counter in > FlinkKafkaProducer010 ? > > > > > > -- > View this

Re: Why I am getting Null pointer exception while accessing RuntimeContext in FlinkKafkaProducer010 ?

Thanks Aljoscha for reply . Is there any way where I can add custom metrics or counter in FlinkKafkaProducer010 ? -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Why-I-am-getting-Null-pointer-exception-while-accessing-RuntimeContext-in-Fl

Re: Key by Task number

Do you have a small example or a link to a doc that has one ? From: Chesnay Schepler To: user@flink.apache.org Sent: Tuesday, April 18, 2017 8:16 AM Subject: Re: Key by Task number If the number of combinations between partition and schemaID is limited then the subtask index cou

Yarn terminating TM for pmem limit cascades causing all jobs to fail

I'm on Flink 1.1.4. We had yet another occurrence of Yarn killing a TM due to exceeding pmem limits and all jobs failing as a result. I thought I had successfully disabled that check, but apparently the property doesn't work as expected in EMR. From what I can tell in the logs, it looks like af

Re: Window Functions and Empty Panes

I forgot to say that timers are fault-tolerant. You set them, and Flink takes care of checkpointing and restoring them after failure. The flag will also be fault-tolerant as, i suppose, you will use Flink’s keyed state. For more info, you can check the ProcessFunction documentation that Konstant

Re: Window Functions and Empty Panes

No problem! Glad I could help! Kostas > On Apr 18, 2017, at 11:01 PM, Ryan Conway > wrote: > > Hi Kostas, > > Re restarting: I missed that ProcessFunction.OnTimerContext extends > ProcessFunction.Context! Until now my thought was that OnTimerContext did not > provide a means of restarting a

Re: Flink Kafka Consumer Behaviour

Is this fixed now? If not, is there any way to monitor kafka offset that is being processed by Flink? This should be a use case for everyone who uses Flink with Kafka. -- View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-Kafka-Consumer-Behav

Re: Window Functions and Empty Panes

Hi Kostas, Re restarting: I missed that ProcessFunction.OnTimerContext extends ProcessFunction.Context! Until now my thought was that OnTimerContext did not provide a means of restarting a timer. Re initial timer, you're right, I'll just need to track a boolean in a state variable that notes whet

Re: Window Functions and Empty Panes

Hi Ryan, “A periodic window like this requires the ability to start a timer without an element and to restart a timer when fired.” For the second part, i.e. “to restart a timer when fired”, you can re-register the timer in the onTimer() method (set a new timer for “now + T"), so that the next

Re: Window Functions and Empty Panes

Hi Konstantin, Thank you for your response. That is unfortunate as this is a mission-critical component of our application. I tried process functions first, but ran into problems because their timers can only be controlled when processing an element. A periodic window like this requires the abili

CEP Pattern detection progressively slower

I just realized that the conversation was not sent to the Mailing List, so I am resending it. Kostas > Begin forwarded message: > > From: Kostas Kloudas > Subject: Re: CEP Pattern detection progressively slower > Date: April 11, 2017 at 11:40:25 PM GMT+2 > To: Dawid Wysakowicz > Cc: vijayakum

Re: CEP timeout does not trigger under certain conditions

I just realized that the conversation was not sent to the Mailing List, so I am resending it. Kostas > On Apr 11, 2017, at 7:30 PM, vijayakumar palaniappan > wrote: > > Sure Thanks > > On Tue, Apr 11, 2017 at 1:28 PM, Kostas Kloudas > wrote: > You are cor

Re: Why I am getting Null pointer exception while accessing RuntimeContext in FlinkKafkaProducer010 ?

The reason for this is that FlinkKafkaProducer010 is a peculiar hybrid of a StreamSink (a subclass of StreamOperator) and SinkFunction/RichFunction (the interface for user functions). The runtime context of this class is only set when you use it as an operator, which happens when you use FlinkK

Re: CEP Pattern detection progressively slower

Yes, as far is I know this issue is know. +Kostas might have some more info regarding the plans there. Could you please comment? > On 11. Apr 2017, at 19:57, vijayakumar palaniappan > wrote: > > While trying to use Pattern detection on KeyedStream, list of unique keys > encountered, keeps ac

Re: CEP timeout does not trigger under certain conditions

+Kostas + Dawid, could you please have a look at this? You are kind of the experts on this topic. > On 11. Apr 2017, at 18:09, vijayakumarpl wrote: > > This is with 1.2.0 version of Flink libraries > > > > -- > View this message in context: > http://apache-flink-user-mailing-list-archive.23

Re: Flink slots, threads, task, etc

Hi, sorry for not getting any responses but I think everyone was quite busy with Flink Forward SF. I’m also no expert on the topic but I’ll try and give some answers. Regarding a Google Doc version, I don’t think that there is any. You would have to modify the Markdown version we have in the do

Re: Flink to S3 streaming

Hi, You would have to write your own SinkFunction that uses the AWS S3 sdk to write to S3. You might be interested in the work proposed in this Jira: https://issues.apache.org/jira/browse/FLINK-6306 As to buffering elements, I’m afraid you woul

Re: Signal Trigger

Ah sorry, you’re right. You add the delay to the signal timestamp. > On 4. Apr 2017, at 19:25, nragon wrote: > > Hi, > > I believe this "timestamp + this.delay" is the signal event timestamp + the > allowed lateness which in this case an configuring it as > EventTimeSessionSignalTrigger.of(this

RE: Kafka offset commits

Thanks for your answer. Does that means that flink does not rely on the offset in written to zookeeper anymore, but relies on the snapshots data, implying that it’s crucial to keep the same snapshot folder before and after the migration to Flink 1.2 ? We were also wondering if the flink consumer

Re: Key by Task number

If the number of combinations between partition and schemaID is limited then the subtask index could actually improve the distribution of values. In any case, the easiest way to do this is to add a RichMapFunction after the flatMap, or modify the flatMap, to also include the subtask index. Typi

Re: Disk I/O in Flink

Hi, I have already looked at the UnilateralSortMerger, concluding that all I/O eventually goes via SegmentReadRequest and SegmentWriteRequest (which in turn use java.nio.channels.FileChannel) in AsynchronousFileIOChannel. Are there more interaction points between Flink and the underlying file syst

Re: Key by Task number

I am not sure if you really need a keyby, your load will be distributed among your map function without it. But could you explain a bit what is your sink doing? As for setting parallelism on the consumer remember that you wont have higher parallelism than number of partitions in your topic. If y

Key by Task number

I am trying to use the task number as a keyby value to help fan out the work load reading from kafka. Given:        DataStream stream =                env.addSource(new FlinkKafkaConsumer010("topicA", schema, properties)                ).setParallelism(240).flatMap(new SchemaRecordSplit()).se

Re: Parallel execution but keep order of kafka messages

Hi Gordon, Thank you for your explanation! It has helped a lot. Just for learning completeness: Here in the docs it says “the parallelism of a stream is always that of its producing op