Re: Question about POJO rules - why fields should be public or have public setter/getter?

Hi Dawid, Thanks for your reply. Good to know it is due to historic and compatibility reasons. The reason why I started looking into POJO rules is to understand if Scala Case Class can conform to POJO rules to support schema evolution. In our case, we store several Scala Case Classes to RocksDB s

Re: Jobmanager Crashes with Kubernetes HA When Restart Kubernetes Master Node

Hi Yang, I tried to upgrade to the latest v1.13.1 the bug logs seem different. But the jobmanager still crashes when I restart one of my three master nodes. The logs from job manager is: 2021-07-12 19:50:04,854 [ERROR] io.fabric8.kubernetes.client.extended.leaderelection.LeaderElector - Except

Re: ValueState is null; checkpointing issues

Hi, > 1) The ValueState can only return a non-null value if a prior value with the > same key (in your case, "x.id") has been received. Have you double-checked > that this is the case? I use the payloads in each operator to make post requests. So the payloads are not null and I keyBy using the

Re: key_by problem in Pyflink

Thanks. In addition, I run the program in a local mini cluster mode, not sure if it would affect the results. Xingbo Huang 于2021年7月12日周一 下午9:02写道： > Hi, > > I think your understanding is correct. The results seem a little wired. > I'm looking into this and will let you know when there are any fi

Re: key_by problem in Pyflink

Hi, I think your understanding is correct. The results seem a little wired. I'm looking into this and will let you know when there are any findings. Best, Xingbo 赵飞 于2021年7月12日周一 下午4:48写道： > Hi all, > I'm using pyflink to develop a module, whose main functionality is > processing user data bas

Process finite stream and notify upon completion

Hey Community, I'm working on a stream job that should aggregate a bounded data and notify upon completion. (It works in Batch mode; however, I'm trying to achieve the same results in Stream mode, if possible). Source: Kafka Sink: PostgresDB I'm looking for an elegant way to notify upon comple

Re: Job Recovery Time on TM Lost

Yes, time is main when detecting the TM's liveness. The count method will check by certain intervals. Gen Luo 于2021年7月9日周五 上午10:37写道： > @刘建刚 > Welcome to join the discuss and thanks for sharing your experience. > > I have a minor question. In my experience, network failures in a certain > cluste

key_by problem in Pyflink

Hi all, I'm using pyflink to develop a module, whose main functionality is processing user data based on specific rules. The program involves two datastreams: data and rule. They have different types, so I connect them and use a field 'product_id' as the key for key_by method. The code is as follow

Re: Error when trying to setup and run wordcount example on dataproc

There is already a follow up discussion on Beam ML [1]. [1] https://lists.apache.org/x/thread.html/r25910bf9ef5bc4b5f3c3d1c99071cb6d43103898a054dcd2a14d8da8@%3Cuser.beam.apache.org%3E Best, D. On Mon, Jul 12, 2021 at 9:27 AM Chesnay Schepler wrote: > I would recommend reaching out to the Beam

Re: Error when trying to setup and run wordcount example on dataproc

I would recommend reaching out to the Beam project instead, because this seems like an issue that is entirely on their side. As for the listed workaround, they are suggesting to modify the wordcount example to also bundle the listed jackson dependencies. Maybe you could use a different example

Re: ValueState is null; checkpointing issues

1) The ValueState can only return a non-null value if a prior value with the same key (in your case, "x.id") has been received. Have you double-checked that this is the case? 2) Checkpointing does not alleviate the need to restart all operators, it alleviates having to reprocess all data. It i