Re: WindowOperator TestHarness

2021-12-12 Thread Timo Walther
Hi Lars, you can take a look at how org.apache.flink.streaming.api.datastream.WindowedStream#WindowedStream constructs the graph under the hood. In particular, it uses org.apache.flink.streaming.runtime.operators.windowing.WindowOperatorBuilder which constructs the InternalWindowFunction you

Re: CoGroupedStreams and disableAutoGeneratedUIDs

2021-12-12 Thread Timo Walther
Hi Dan, if there is no way of setting a uid(), then it sounds like a bug in the API that should be fixed. Feel free to open an issue for it. Regards, Timo On 13.12.21 08:19, Schwalbe Matthias wrote: Hi Dan, When I run into such problem I consider using the not so @public api levels: * F

RE: CoGroupedStreams and disableAutoGeneratedUIDs

2021-12-12 Thread Schwalbe Matthias
Hi Dan, When I run into such problem I consider using the not so @public api levels: * First of all uids are especially needed for operator that hold state and is not so important for operators that don’t hold state primitives, not sure of the implications created by disableAutoGeneratedUID

IOException/StacklessClosedChannelException on flink-connector-kinesis trigger job to restart

2021-12-12 Thread Leon Xu
Hi Flink users, I used flink-1.12.5 kinesis connector to consume data from kinesis. >From time to time I am getting IOException or StacklessClosedChannelException, which will fail the Flink operator when it by default reaches 10 times and trigger the entire job to restart. I have two questions:

CoGroupedStreams and disableAutoGeneratedUIDs

2021-12-12 Thread Dan Hill
Hi. I tried to use CoGroupedStreams w/ disableAutoGeneratedUIDs. CoGroupedStreams creates two map operators without the ability to set uids on them. These appear as "Map" in my operator graph. I noticed that the CoGroupedStreams.apply function has two map calls without setting uids. If I try to

Re: Unable to create new native thread error

2021-12-12 Thread Ilan Huchansky
Hi David, Sorry for the previous mail, sent it before it was finished, please ignore. We made the changes, now submitting the jobs using flink CLI. To be more specific – We use a docker, with a flink image containing flink CLI. We submit the jobs with the run command specifying the job manager w

Re: Unable to create new native thread error

2021-12-12 Thread Ilan Huchansky
Hi David, We made the changes, now submitting the jobs using flink CLI. To be more specific Unfortunately, we are still seeing the same error. Caused by: java.lang.OutOfMemoryError: unable to create new native thread. The behavior is the following: One task manager crashes, from that point submit