Accessing Global State when processing KeyedStreams

2018-09-19 Thread Scott Sue
Hi, In our application, we receive Orders and Prices via two KafkaSources. What I want to do is to perform calculations for a given Order against a stream of Prices for the same securityId, i.e. same identifier between the Order and stream of Prices. Naturally this is a perfect fit for a KeyedSt

Re: Accessing Global State when processing KeyedStreams

2018-09-19 Thread Scott Sue
Hi Rong, Thanks for your suggestion, I'll give that a go. I just found a great article on this as well that explains the functionality https://data-artisans.com/blog/a-practical-guide-to-broadcast-state-in-apache-flink Regards, Scott -- Sent from: http://apache-flink-user-mailing-list-archi

Logging Kafka during exceptions

2018-11-21 Thread Scott Sue
Hi all, When I'm running my jobs I am consuming data from Kafka to process in my job. Unfortunately my job receives unexpected data from time to time which I'm trying to find the root cause of the issue. Ideally, I want to be able to have a way to know when the job has failed due to an exception

Re: Logging Kafka during exceptions

2018-11-21 Thread Scott Sue
operator I create must have a try catch block? Regards, Scott SCOTT SUE CHIEF TECHNOLOGY OFFICER Support Line : +44(0) 2031 371 603 Mobile : +852 9611 3969 9/F, 33 Lockhart Road, Wan Chai, Hong Kong www.celer-tech.com <http://www.celer-tech.com/> > On 22 Nov 2018, at 13:17, Paul L

Re: Logging Kafka during exceptions

2018-11-21 Thread Scott Sue
. Thoughts? Regards, Scott SCOTT SUE CHIEF TECHNOLOGY OFFICER Support Line : +44(0) 2031 371 603 Mobile : +852 9611 3969 9/F, 33 Lockhart Road, Wan Chai, Hong Kong www.celer-tech.com <http://www.celer-tech.com/> > On 22 Nov 2018, at 15:04, Paul Lam wrote: > > > a

Re: Logging Kafka during exceptions

2018-11-21 Thread Scott Sue
Yeah I think that would work for incorrect data consumed, but not for if deserialization passes correctly, but one of my custom functions post deserialization generates an error? Regards, Scott SCOTT SUE CHIEF TECHNOLOGY OFFICER Support Line : +44(0) 2031 371 603 Mobile : +852 9611 3969 9/F

Re: Logging Kafka during exceptions

2018-11-22 Thread Scott Sue
there is an unexpected exception. Regards, Scott SCOTT SUE CHIEF TECHNOLOGY OFFICER Support Line : +44(0) 2031 371 603 Mobile : +852 9611 3969 9/F, 33 Lockhart Road, Wan Chai, Hong Kong www.celer-tech.com <http://www.celer-tech.com/> > On 23 Nov 2018, at 00:12, Till Rohrmann wrote

Re: Logging Kafka during exceptions

2018-11-22 Thread Scott Sue
Thanks Till, I’ve raised a JIRA for this: https://issues.apache.org/jira/browse/FLINK-10988. Let me know if theres anything else I can add to the JIRA to help Regards, Scott SCOTT SUE CHIEF TECHNOLOGY OFFICER Support Line : +44(0) 2031 371 603 Mobile : +852 9611 3969 9/F, 33 Lockhart Road

Intermittent KryoException

2019-03-08 Thread Scott Sue
Hi, When running our job, we’re seeing sporadic instances of when we have KryoExceptions. I’m new to this area of Flink so I’m not exactly too sure what I could look out for. From my understanding, Kryo is the default serializer for generic types, and whilst there is a potential performance p