Re: Windowing

2021-05-28 Thread Sozonoff Serge
Hi Kenneth, Thanks, I can eventually share the code but it needs some work to remove all references to any sensitive data. The way I have gone about fixing this so far is executing the code and dealing with the error messages related to window/trigger.  The two messages which come up are the f

Re: Is there a way (seetings) to limit the number of element per worker machine

2021-05-28 Thread Eila Oriel Research
Hi Brian, Thanks for your response. I am reading genomic data. I am using a research tool (software) that was built to process the files - it is not built to work on multiple machines. I dont usually work with Splittable DoFn - so I hope that I understand the concept properly. Please let me know i

Re: Is there a way (seetings) to limit the number of element per worker machine

2021-05-28 Thread Eila Oriel Research
Thanks Robert. I found the following explanation for the number of threads for 4 cores: You have *4* CPU sockets, each CPU *can* have, up to, 12 *cores* and each *core can* have two *threads*. Your *max thread* count is, *4* CPU x 12 *cores* x 2 *threads* per *core*, so 12 x *4* x 2 is 96 Can I lim

[DISCUSS] Drop support for Flink 1.10

2021-05-28 Thread Ismaël Mejía
Hello, With Beam support for Flink 1.13 just merged it is the time to discuss the end of support for Flink 1.10 following the agreed policy on supporting only the latest three Flink releases [1]. I would like to propose that for Beam 2.31.0 we stop supporting Flink 1.10 [2]. I prepared a PR for t