Re: Can I only use checkpoints instead of savepoints in production?

2018-08-26 Thread vino yang
Hi Averell, What is the error message? Do you seem to forget to post it? As far as I know, if you enable checkpoints, it will automatically resume when the job fails. Thanks, vino. Averell 于2018年8月27日周一 下午1:21写道: > Thank you Vino. > > I sometimes got the error message like the one below. It lo

Re: [DISCUSS] Remove the slides under "Community & Project Info"

2018-08-26 Thread Renjie Liu
Hi, Stephan: Can we put project wiki in some place? I think it's a great place to find flip lists On Mon, Aug 27, 2018 at 9:56 AM vino yang wrote: > +1 > The reason is the same as Hequn, because we have given a link to > SlideShare under the "Flink Forward" section. > > Thanks, vino. > > Hequn C

Re: Low Performance in High Cardinality Big Window Application

2018-08-26 Thread Jörn Franke
If you have a window larger than hours then you need to rethink your architecture - this is not streaming anymore. Only because you receive events in a streamed fashion you don’t need to do all the processing in a streamed fashion. Can you store the events in a file or a database and then do aft

Re: Can I only use checkpoints instead of savepoints in production?

2018-08-26 Thread Averell
Thank you Vino. I sometimes got the error message like the one below. It looks like my executors got overloaded. Here I have another question: is there any existing solution that allows me to have the job restored automatically? Thanks and best regards, Averell -- Sent from: http://apache-fl

Low Performance in High Cardinality Big Window Application

2018-08-26 Thread Ning Shi
The application consumes from a single Kafka topic, deserializes the JSON payload into POJOs and use a big keyed window (30+ days) for deduplication, then emits the result for every single event to four other keyed windows for aggregation. It looks roughly like the following. Source->KeyBy(A,B,C)

Re: [DISCUSS] Remove the slides under "Community & Project Info"

2018-08-26 Thread vino yang
+1 The reason is the same as Hequn, because we have given a link to SlideShare under the "Flink Forward" section. Thanks, vino. Hequn Cheng 于2018年8月27日周一 上午9:31写道: > Hi Stephan, > > Thanks for bringing up this discussion. > I think we can just remove it, because slides have already be provided

Re: [DISCUSS] Remove the slides under "Community & Project Info"

2018-08-26 Thread Hequn Cheng
Hi Stephan, Thanks for bringing up this discussion. I think we can just remove it, because slides have already be provided from the `Flink Forward` section in this page. Adding another slides section not only brings redundancy but also introduces problems you listed above. Best, Hequn On Sun, Au

Dealing with Not Serializable classes in Java

2018-08-26 Thread Dominik Wosiński
Hey, I was wondering how do You normally deal with fields that contain references that are not serializable. Say, we have a custom serialization schema in Java that needs to serialize *LocalDateTime* field with *ObjectMapper.* This requires registering specific module for *ObjectMapper* and this

Re: AvroSchemaConverter and Tuple classes

2018-08-26 Thread Rong Rong
Yes you should be able to use Row instead of Tuple in your BatchTableSink. There's sections in Flink documentation regarding mapping of data types to table schemas [1]. and table can be converted into various typed DataStream [2] as well. Hope these are helpful. Thanks, Rong [1] https://ci.apache

Re: Kryo Serialization Issue

2018-08-26 Thread Rong Rong
This seems to be irrelevant to the issue for KyroSerializer in recent discussions [1]. which has been fixed in 1.4.3, 1.5.0 and 1.6.0. On a quick glance, this might have been a corrupted message in your decoding, for example a malformed JSON string. -- Rong [1] https://issues.apache.org/jira/brow

[DISCUSS] Remove the slides under "Community & Project Info"

2018-08-26 Thread Stephan Ewen
Hi all! In the past, we collected slide sets under the "Community & Project Info" side. I would like to see what the community thinks about removing them. There are currently several issues: - The list is not well maintained. Tthere are for example no 2018 slides at all. - Many slide sets ar

Re: would you join a Slack workspace for Flink?

2018-08-26 Thread Nicos Maris
Hi Dominik, I was writing about gitter just now :) If searchability is an issue, then indeed we could consider the free plan of gitter: https://billing.gitter.im In any case, we should instruct users who paste code snippets at the mailing list to use http://gist.github.com On Sun, Aug 26, 201

Fwd: would you join a Slack workspace for Flink?

2018-08-26 Thread Dominik Wosiński
-- Forwarded message - From: Dominik Wosiński Date: niedz., 26 sie 2018 o 15:12 Subject: ODP: would you join a Slack workspace for Flink? To: Hequn Cheng Hey, I have been facing this issue for multiple open source projects and discussions. Slack in my opinion has two main issues

Re: would you join a Slack workspace for Flink?

2018-08-26 Thread Hequn Cheng
Hi Nicos, Thanks for bring up this discussion. :-) Slack is a good way to communicate, but it seems not very fit for the open source field. The messages on Slack are mixed up and can not be searched through search engine. Best, Hequn On Sun, Aug 26, 2018 at 7:22 PM Nicos Maris wrote: > Chesnay

Re: would you join a Slack workspace for Flink?

2018-08-26 Thread Nicos Maris
Chesnay can you take a look at the following PR? https://github.com/apache/flink-web/pull/120 On Sun, Aug 26, 2018 at 1:09 PM Chesnay Schepler wrote: > There have been previous discussions around using slack and they were > rejected. > > Personally I would just remove the IRC channel; I'm not a

Re: would you join a Slack workspace for Flink?

2018-08-26 Thread Chesnay Schepler
There have been previous discussions around using slack and they were rejected. Personally I would just remove the IRC channel; I'm not aware of any committer actually spending time there. On 25.08.2018 17:07, Nicos Maris wrote: Hi all, This mailing list is for user support and question