Re: Flink 1.12 StreamRecordQueueEntry is not public class

2022-07-01 Thread Martijn Visser
What I'm interested in is understanding why you're relying on this class; what is the problem you're trying to solve? That input could be useful to consider making that interface public or there could be another way to solve your problem. Best regards, Martijn Op wo 29 jun. 2022 om 18:12 schreef

Re: Flink 1.12 StreamRecordQueueEntry is not public class

2022-06-29 Thread Milind Vaidya
Thanks Xuyang, I did something similar to unblock myself. - Milind On Wed, Jun 29, 2022 at 8:40 PM Xuyang wrote: > Hi, Milind. You may notice that these classes are tagged with 'Internal' > and that mean they are may only used in flink itself. But I think you may > do some retrofit work on fli

Re: Flink 1.12.x DataSet --> Flink 1.14.x DataStream

2022-02-15 Thread saravana...@gmail.com
Thanks Zhipeng. Working as expected. Thanks once again. Saravanan On Tue, Feb 15, 2022 at 3:23 AM Zhipeng Zhang wrote: > Hi Saravanan, > > One solution could be using a streamOperator to implement `BoundedOneInput` > interface. > An example code could be found here [1]. > > [1] > https://gith

Re: Flink 1.12.x DataSet --> Flink 1.14.x DataStream

2022-02-15 Thread Zhipeng Zhang
Hi Saravanan, One solution could be using a streamOperator to implement `BoundedOneInput` interface. An example code could be found here [1]. [1] https://github.com/apache/flink-ml/blob/56b441d85c3356c0ffedeef9c27969aee5b3ecfc/flink-ml-core/src/main/java/org/apache/flink/ml/common/datastream/Data

Re: Flink 1.12.x DataSet --> Flink 1.14.x DataStream

2022-02-14 Thread saravana...@gmail.com
Hi Niklas, Thanks for your reply. Approach [1] works only if operators are chained (in order words, operators executed within the same task). Since mapPartition operator parallelism is different from previous operator parallelism, it doesn't fall under the same task(or not chained) . https://

Re: Flink 1.12.x DataSet --> Flink 1.14.x DataStream

2022-02-14 Thread Niklas Semmler
Hi Saravanan, AFAIK the last record is not treated differently. Does the approach in [1] not work? Best regards, Niklas https://github.com/dmvk/flink/blob/2f1b573cd57e95ecac13c8c57c0356fb281fd753/flink-runtime/src/test/java/org/apache/flink/runtime/operators/chaining/ChainTaskTest.java#L279

Re: Flink 1.12.x DataSet --> Flink 1.14.x DataStream

2022-02-09 Thread saravana...@gmail.com
Is there any way to identify the last message inside RichFunction in BATCH mode ? On Wed, Feb 9, 2022 at 8:56 AM saravana...@gmail.com wrote: > I am trying to migrate from Flink 1.12.x DataSet api to Flink 1.14.x > DataStream api. mapPartition is not available in Flink DataStream. > *Current C

Re: Flink-1.12 Sql on Job two SQL sink control order

2021-10-18 Thread Arvid Heise
Are you running in Batch? Then you probably need to write 2 SQL jobs (or statements). In streaming, the notion of order doesn't make much sense. But maybe I misunderstood your use case. On Thu, Oct 14, 2021 at 11:37 AM Francesco Guardiani < france...@ververica.com> wrote: > I'm not aware of any

Re: Flink-1.12 Sql on Job two SQL sink control order

2021-10-14 Thread Francesco Guardiani
I'm not aware of any way to control the sink order, afaik each Table#executeInsert will generate a separate job on its own. You may be able to hack it around by having a custom DynamicTableSink that for each record sends it to tidb and then to kafka. May I ask why you need that? If the notificatio

Re: Flink 1.12 Compatibility with hbase-shaded-client 2.1 in application jar

2021-03-03 Thread Debraj Manna
The issue is resolved. org.apache.hbase exclusion was missing on my application pom while creating the uber jar. diff --git a/map/engine/pom.xml b/map/engine/pom.xml index 8337be031d1..8eceb721fa7 100644 --- a/map/engine/pom.xml +++ b/map/engine/pom.xml @@ -203,6 +203,7 @@

Re: Flink 1.12 Kryo Serialization Error

2021-01-18 Thread Timo Walther
Forgot to add the link: https://github.com/twalthr/flink/tree/kryoBug_ser Regards, Timo On 18.01.21 14:11, Timo Walther wrote: I ported the code to the Flink code base. Because I had issues with SBT and Scala 2.12. Note it uses an older version of circe. I'm just pasting it here in case it h

Re: Flink 1.12 Kryo Serialization Error

2021-01-18 Thread Timo Walther
I ported the code to the Flink code base. Because I had issues with SBT and Scala 2.12. Note it uses an older version of circe. I'm just pasting it here in case it helps someone. Regards, Timo On 18.01.21 13:51, Timo Walther wrote: Hi Yuval, thanks for sharing some code with us. I scanned th

Re: Flink 1.12 Kryo Serialization Error

2021-01-18 Thread Timo Walther
Hi Yuval, thanks for sharing some code with us. I scanned the code but could not find anything suspicious from an API perspective. By using the full RAW serializable string, we should actually be on the save side when it comes to configure the Kryo serializer. I would suggest to further inve

Re: Flink 1.12 Kryo Serialization Error

2021-01-13 Thread Timo Walther
Hi Yuval, could you share a reproducible example with us? I see you are using SQL / Table API with a RAW type. I could imagine that the KryoSerializer is configured differently when serializing and when deserializing. This might be due to `ExecutionConfig` not shipped (or copied) through the

Re: Flink 1.12

2020-12-20 Thread Yang Wang
IIUC, "state.checkpoints.dir" is specifying an external checkpoint path, which will not be cleaned up unless the users configured it explicitly[1]. However, for "high-availability.storageDir", it will be cleaned up automatically when all the jobs in the application reaches to the terminal state. M

Re: Flink 1.12

2020-12-20 Thread Boris Lublinsky
I understand this. State storage Is defined defined by state.checkpointing.dir, for example state.checkpoints.dir: file:///mnt/flink/storage/checkpoints I am talking about reference defined in 2 places > On Dec 20, 2020, at 8:05 PM, Yang Wang wrote: > > I am afraid only the state handle is sto

Re: Flink 1.12

2020-12-20 Thread Yang Wang
I am afraid only the state handle is stored in the ConfigMap. The real state data is stored in the distributed storage configured via "high-availability.storageDir". I believe you could find more information in this class KubernetesStateHandleStore[1]. How could you find that the checkpointing inf

Re: Flink 1.12

2020-12-19 Thread Boris Lublinsky
Thanks Yang, This is still confusing. I did more experiments and see that checkpointing information is stored twice - in config map and in high-availability.storageDir Do we need this duplication? Do we need to specify high-availability.storageDir as defined in https://ci.apache.org/projects/flin

Re: Flink 1.12

2020-12-17 Thread Yang Wang
The latest successful checkpoint pointer is stored in the ConfigMap, as well as the JobGraph pointer. They could help us recover the running jobs before you delete the K8s deployment. If the HA ConfigMaps are deleted, then when you create a Flink cluster with the same cluster-id, it could not recov

Re: Flink 1.12

2020-12-17 Thread Boris Lublinsky
Also re reading https://ci.apache.org/projects/flink/flink-docs-release-1.12/deployment/ha/kubernetes_ha.html#high-availability-data-clean-up This does not seem right:

Re: Flink 1.12

2020-12-17 Thread Boris Lublinsky
Thanks Yang, > On Dec 17, 2020, at 8:49 PM, Yang Wang wrote: > > Hi Boris, > > Thanks for your follow up response and trying the new KubernetesHAService. > > 1. It is a valid bug. We are not setting the service account for TaskManager > pod. Before the KubernetesHAService is introduced, it w

Re: Flink 1.12

2020-12-17 Thread Yang Wang
Hi Boris, Thanks for your follow up response and trying the new KubernetesHAService. 1. It is a valid bug. We are not setting the service account for TaskManager pod. Before the KubernetesHAService is introduced, it works fine because the TaskManager does not need to access the K8s resource(e.g.

Re: Flink 1.12

2020-12-17 Thread Boris Lublinsky
And K8 native HA works, But there are 2 bugs in this implementation. 1. Task manager pods are running as default user account, which fails because it does not have access to config maps to get endpoint’s information. I had to add permissions to default service account to make it work. Ideally bo

Re: Flink 1.12

2020-12-16 Thread Boris Lublinsky
Thanks guys, The reason I am interested in rolling update is to avoid complete restarts in the case of parameter (for example parallelism) changes. > On Dec 15, 2020, at 8:40 PM, Yang Wang wrote: > > Hi Boris, > > What is -p 10? > It is same to --parallelism 10. Set the default parallelism to

Re: Flink 1.12 and Stateful Functions

2020-12-16 Thread Igal Shilman
Hello Jan, The next release candidate for stateful functions is expected in mid February, and the release itself would follow shortly thereafter. This release will be based on Flink 1.12. Cheers, Igal. On Tue, Dec 15, 2020 at 4:19 PM Jan Brusch wrote: > Hi, > > just a quick question: Is there

Re: Flink 1.12

2020-12-15 Thread Yang Wang
Hi Boris, What is -p 10? It is same to --parallelism 10. Set the default parallelism to 10. does it require a special container build? No, the official flink docker image could be used directly. Unfortunately, we do not have the image now. And we are trying to figure out. You could follow the i

Re: Flink 1.12

2020-12-15 Thread Boris Lublinsky
Thanks Chesney for your quick response, I read documentation https://cwiki.apache.org/confluence/display/FLINK/FLIP-144%3A+Native+Kubernetes+HA+for+Flink#FLIP144:NativeKubernetesHAforFlink-NativeK8s

Re: Flink 1.12

2020-12-15 Thread Chesnay Schepler
Unfortunately no; there are some discussions going on in the docker-library/official-images PR that have to be resolved first, but currently these would require changes on the Flink side that we cannot do (because it is already relea

Re: Flink 1.12

2020-12-15 Thread Boris Lublinsky
Thanks. Do you have ETA for docker images? > On Dec 14, 2020, at 3:43 AM, Chesnay Schepler wrote: > > 1) It is compiled with Java 8 but runs on Java 8 & 11. > 2) Docker images are not yet published. > 3) It is mentioned at the top of the Kubernetes HA Services documentation > that it also work

Re: Flink 1.12

2020-12-14 Thread Chesnay Schepler
1) It is compiled with Java 8 but runs on Java 8 & 11. 2) Docker images are not yet published. 3) It is mentioned at the top of the Kubernetes HA Services documentation that it also works for the native Kubernetes integration. /Kubernetes high availability services can only be used when d

Re: Flink 1.12, Kafka Connector, JSON format - "Could not find a suitable table factory"

2020-12-11 Thread abelm
Hi again! I did not realise both the new and old codes are supported in the later versions of the connector, but that was indeed the problem. Updated all of the options to use the new syntax and everything works like a charm. I should've definitely been a bit more careful about that 😅 Thanks a l

Re: Flink 1.12, Kafka Connector, JSON format - "Could not find a suitable table factory"

2020-12-10 Thread Danny Chan
One thing needs to note is that the old connectors are still in the release-1.11/release-1.12 jars. So the old option still works but with the old connector codes. You may need to find the root cause why the new options do not work, maybe some stack trace here ? abelm 于2020年12月10日周四 下午10:54写道:

Re: Flink 1.12, Kafka Connector, JSON format - "Could not find a suitable table factory"

2020-12-10 Thread abelm
Hi! Thank you for the reply! I understand that the metadata syntax is only available as of 1.12, but I am indeed trying to use Flink 1.12. Regarding the option for ignoring parse errors: I have already noticed from before that, according to the docs, even in Flink 1.11 (which is the version that

Re: Flink 1.12, Kafka Connector, JSON format - "Could not find a suitable table factory"

2020-12-09 Thread Danny Chan
Hi, abelm ~ Which version Flink did you use? We did some refactoring for the connector options since Flink 1.11. The METADATA syntax is only supported since version 1.12. In 1.11, to ignore the parse errors, you need to use option "json.ignore-parse-error" [1] [1] https://ci.apache.org/projects/

Re: Flink 1.12 cannot handle large schema

2020-10-02 Thread Lian Jiang
Appreciate Arvid for the jira and the workaround. I will monitor the jira status and retry when the fix is available. I can help test the fix when it is in a private branch. Thanks. Regards! On Fri, Oct 2, 2020 at 3:57 AM Arvid Heise wrote: > Hi Lian, > > Thank you for reporting. It looks like a

Re: Flink 1.12 cannot handle large schema

2020-10-02 Thread Arvid Heise
Hi Lian, Thank you for reporting. It looks like a bug to me and I created a ticket [1]. You have two options: wait for the fix or implement the fix yourself (copy AvroSerializerSnapshot and use another way to write/read the schema), then subclass AvroSerializer to use your snapshot. Of course, we

Re: Flink 1.12 snapshot throws ClassNotFoundException

2020-10-02 Thread Till Rohrmann
Great to hear that it works now :-) On Fri, Oct 2, 2020 at 2:17 AM Lian Jiang wrote: > Thanks Till. Making the scala version consistent using 2.11 solved the > ClassNotFoundException. > > On Tue, Sep 29, 2020 at 11:58 PM Till Rohrmann > wrote: > >> Hi Lian, >> >> I suspect that it is caused by

Re: Flink 1.12 snapshot throws ClassNotFoundException

2020-10-01 Thread Lian Jiang
Thanks Till. Making the scala version consistent using 2.11 solved the ClassNotFoundException. On Tue, Sep 29, 2020 at 11:58 PM Till Rohrmann wrote: > Hi Lian, > > I suspect that it is caused by an incompatible Akka version. Flink uses > Akka 2.5.21 instead of 2.5.12. Moreover, you are mixing F

Re: Flink 1.12 snapshot throws ClassNotFoundException

2020-09-29 Thread Till Rohrmann
Hi Lian, I suspect that it is caused by an incompatible Akka version. Flink uses Akka 2.5.21 instead of 2.5.12. Moreover, you are mixing Flink jars which use Scala 2.11 with Akka dependencies which are built against Scala 2.12. I am not an Gradle expert but can't Gradle simply pull in the transit