Re: Parallelism of state processor jobs

2024-07-06 Thread Alexis Sarda-Espinosa
Hi Junrui, I think you understood correctly. What I'm seeing is that each vertex has a single subtask, but multiple vertices are started in parallel in different slots. That is not a problem in my case, I _want_ to parallelize the work, it's just that this mechanism is very different from streamin

Re: Parallelism of state processor jobs

2024-07-06 Thread Junrui Lee
Hi Alexis, Could you clarify what you mean by "If I add more slots to the task manager, I see the transformations actually start in parallel even though I submit the job with 'flink run -p 1'"? Are you asking if multiple slots are working simultaneously, or if a single JobVertex contains multiple

flinkcdc????postgrep????????checkpoint????????????????????????

2024-07-06 Thread Eleven
PostgresSourceBuilder

Re: Parallelism of state processor jobs

2024-07-06 Thread Alexis Sarda-Espinosa
Hi Junrui, Thanks for the confirmation. I tested some more and I'm seeing a strange behavior. I'm currently testing a single source stream that is fed to 6 identical transformations. The state processor api requires batch mode and, from what I can tell, I must specify a parallelism of 1 in the jo