Re: Ingesting a table from a retraction stream !!

Hi Puneet, I agree with Timo, such a feature should be carefully designed. Besides, I'm very interested with your business scenario. I am wondering where your retraction stream come from (Maybe mysql binlog? or ?). It would be great if you can tell more about the details. Maybe we can figure out

Re: ElasticsearchSink in Flink 1.4.0 with Elasticsearch 5.2+

Ok, I got it "done". I have a PR for ES5.3 (FLINK-7386) just rebasing the original one that was never merged (#4675). And added ES 6.X through RestHighLevelClient on top (FLINK-8101). This is: https://github.com/apache/flink/pull/5374. And believe it or not but someone else submitted a PR for thos

RE: How to deal with dynamic types

We’ve been using genericRecords with custom serializers to do exactly this. We need to run the same flink pipeline for 10s of thousands of different schemas for our use cases and code gening or building that many different jars just isn’t practical. From: madan [mailto:madan.yella...@gmail.com

RE: Two operators consuming from same stream

We've seen the same thing here. We read files twice for the same reason, it's simply faster to do that than to connect the two pipes to the same input. From: Sofer, Tovi [mailto:tovi.so...@citi.com] Sent: Monday, January 01, 2018 8:51 AM To: user@flink.apache.org Subject: Two operators consuming

Re: Ingesting a table from a retraction stream !!

Hi Puneet, I'm very sure that this feature will be included in upcoming releases. However, such an interface has to be carefully designed because built-in operators assume that only records that have been emitted previously are retracted. Maybe it will be part of Flink 1.6 but 1.7 should defin

Ingesting a table from a retraction stream !!

Hi I know currently Ingesting a table from a retraction stream is not supported yet. is there any plan to include in upcoming releases. -- *Cheers * *Puneet Kinra* *Mobile:+918800167808 | Skype : puneet.ki...@customercentria.com * *e-mail :puneet.ki...@customercentria.com *

Re: Flip-6 + Dynamic scaling

Hi Jayant, I am working on FLIP-6, and I can tell you that we are aiming at shipping it with Flink 1.5. For the scope and release timeline see this blog post: https://flink.apache.org/news/2017/11/22/release-1.4-and-1.5-timeline.html Best, Gary On Wed, Jan 24, 2018 at 7:02 AM, Jayant Ameta w

Re: Trigger not firing when using BoundedOutOfOrdernessTimestampExtractor

A while back I wrote this slightly more elaborate extractor that will advance the watermark independently after the stream is idle for a while: https://github.com/aljoscha/flink/blob/6e4419e550caa0e5b162bc0d2ccc43f6b0b3860f/flink-streaming-java/src/main/java/org/apache/flink/streaming/api/functio

Re: Low throughput when trying to send data with Sockets

Taking a step back: why do you want to manually implement communication via sockets in the first place? With this you will not get any fault-tolerance guarantees and I would guess that maintaining a custom solution is more difficult than using, say, Kafka. Best, Aljoscha > On 16. Jan 2018, at

Re: Scheduling of GroupByKey and CombinePerKey operations

Hi, What are the other stages in that program? Best, Aljoscha > On 18. Jan 2018, at 16:22, Fabian Hueske wrote: > > Hi Pawel, > > This question might be better suited for the Beam user list. > Beam includes the Beam Flink runner which translates Beam programs into Flink > programs. > > Best

Re: Forcing consuming one stream completely prior to another starting

Hi, How can you determine whether the required decoration data for an event from the main stream is there? If it works via event-time you could think about buffering main-input events in the operator until the corresponding decoration arrives. On a side note, we're currently working on broadca

Re: flink read hdfs file error

Hi, It seems you are using Akka in your user program and the reference.conf in your Jar is clashing with the same File from Flink. What is the reason for having Akka (or reference.conf) in your user jar? Best, Aljoscha > On 22. Jan 2018, at 11:09, 韩宁宁 <453673...@qq.com> wrote: > > Dear All >

Re: Understanding Restart Strategy

Thanks for the update! > On 25. Jan 2018, at 04:12, Ashish Pokharel wrote: > > FYI, > > I think I have gotten to the bottom this situation. For anyone who might be > in situation hopefully my observations will help. > > In my case, it had nothing to do with Flink Restart Strategy, it was doin

Re: Bootstrapping

Hi, I see this coming up more and more often these days. For now, the solution of doing a savepoint and switching sources should work but I've had it in my head for a while now to add functionality for bootstrapping inputs in the API. An operator would read from the bootstrap stream (which is f

Re: ElasticsearchSink in Flink 1.4.0 with Elasticsearch 5.2+

Great, thank you! Hopefully, this pushes the PR forward. Thanks, Fabian 2018-01-25 22:30 GMT+01:00 Christophe Jolif : > Hi Fabian, > > FYI I rebased the branch and tested it and it worked OK on a sample. > > -- > Christophe > > On Mon, Jan 22, 2018 at 2:53 PM, Fabian Hueske wrote: > >> Hi Adria