Flink OpenSearch Connector - Avro JSON to JSON

Hello Team Please helps to answer the below query. 1. OpenSearch supports writing data in the JSON format, but in Flink its default data format is Avro JSON. What is the best practice to write data to Open Search using Flink OpenSearch Connector? Do we need to manually convert Avro JSON to

Re: Flink Kubernetes operator keeps reporting REST client timeout.

Seems the operator didn't get restarted automatically after the configmap is changed. After a roll-out restart, the exception disappeared. Never mind this issue. Thanks. On Tue, Nov 21, 2023 at 11:31 AM Xiaolong Wang wrote: > Hi, > > Recently I upgraded the flink-kubernetes-operator from 1.4.0 t

Flink Kubernetes operator keeps reporting REST client timeout.

Hi, Recently I upgraded the flink-kubernetes-operator from 1.4.0 to 1.6.1 to use Flink 1.18. After that, the operator kept reporting the following exception: 2023-11-21 03:26:50,505 o.a.f.k.o.r.d.AbstractFlinkResourceReconciler [INFO > ][sn-push/sn-push-decision-maker-log-s3-hive-prd] Resource fu

Handling Errors and Message Replay in Flink

Hello Flink community , We are currently working on a Flink job that consumes messages from RabbitMQ, with checkpointing configured to at-least-once mode. In our job, we make external API requests to retrieve information. If the external api is down or a timeout is occured, we currently throw an

Re: Avoid dynamic classloading in native mode with Kubernetes Operator

Hi Trystan, I'm actually not very familiar with the operator's internals, but I'd guess that limitation is in Flink itself - application mode is a feature from core Flink, the operator just configures it based on the CRDs it defines. Maybe one of the maintainers can confirm. Regards, Alexis. On M

Re: Avoid dynamic classloading in native mode with Kubernetes Operator

Thanks Alexis, I can give that a try. However, that seems less than ideal from the user's perspective. Is there a technical reason why the operator can't support this combination of modes? I'd really like to just let the system do its thing rather than build a complicated two-jar approach. Thanks

Re: [DISCUSS] FLIP-391: Deprecate RuntimeContext#getExecutionConfig

Hi Junrui, Thanks for the clarification. On one hand, adding more methods into the RuntimeContext flat will increase the effort for users who will use RuntimeContext. But the impact is limited. It is fine. The big impact is, on the other hand, for users who want to focus on the execution config, t