Data type serialization and testing

2021-04-30 Thread Dave Maughan
Hi, I recently encountered a scenario where the data type being passed between operators in my streaming job was modified such that it broke serialization. This was due to a non-Avro top-level data type containing an Avro field. The existing integration test (mini cluster) continued to work and un

Re: Data type serialization and testing

2021-04-30 Thread Dave Maughan
force serialization. > > Maybe others have better ideas. > > Regards, > Timo > > On 30.04.21 10:25, Dave Maughan wrote: > > Hi, > > > > I recently encountered a scenario where the data type being passed > > between operators in my streaming job was modified suc

Flink operator stuck on created

2021-09-20 Thread Dave Maughan
Hi, I have a Flink job on EMR with an operator stuck on CREATED. The subtasks are not being assigned to task manager slots. The previous operator is running and has non-zero Bytes Sent and Records Sent. When the job started the Job manager requested new workers to start a bunch of the operators bu

Re: Flink operator stuck on created

2021-09-20 Thread Dave Maughan
I should note - this job is being run in batch mode. Could there be a deadlock related to FLINK-16430? On Mon, 20 Sept 2021 at 11:26, Dave Maughan wrote: > Hi, > > I have a Flink job on EMR with an operator stuck on CREATED. The subtasks > are not being assigned to task manage