Re: Jobmanager stopped because uncaught exception

I see there's a related issue https://issues.apache.org/jira/browse/FLINK-21053 which is still open. Does it mean the similar issue will still exist even if i upgrade to 1.12.2 ? Thanks, Lei On Mon, Feb 8, 2021 at 3:54 PM Yang Wang wrote: > Maybe it is a known issue[1] and has already been re

Re: Kafka connector doesn't support consuming update and delete changes in Table SQL API

Any help please? Is there a way to use the "Last row" from a deduplication in an append-only stream or tell upsert-kafka to not produce *null* records in the sink? Thank you On Thu, Feb 4, 2021 at 1:22 PM meneldor wrote: > Hello, > Flink 1.12.1(pyflink) > I am deduplicating CDC records coming f

Re: Flink upset-kaka connector not working with Avro confluent

Hi Shamit, thanks for reaching out to the community. I am pulling in Timo who might know more about this problem. Cheers, Till On Sun, Feb 7, 2021 at 6:22 AM shamit jain wrote: > Hello Team, > > I am facing issue with "upsert-kafka" connector which should read the Avro > message serialized usi

Re: Jobmanager stopped because uncaught exception

Hi, The open issue you mentioned (FLINK-21053) is about preventing potential issues in the future. The issue you are experiencing is most likely FLINK-20992 as Yang Wang said. So upgrading to 1.12.2 should solve the problem. Regards, Roman On Mon, Feb 8, 2021 at 9:05 AM Lei Wang wrote: > I se

Re: Table Cache Problem

Hi Yongsong, in newer Flink versions we introduced the concept of statament sets, which are available via `TableEnvironment.createStatementSet()`. They allow you to opimized a branching pipeline as a whole with reusing subplans. In older Flink versions, you can convert the Table to a DataStre

Re: Sliding Window Count: Tricky Edge Case / Count Zero Problem

Hi Yun, thanks for your reply. I do agree with your point about standard windows being for high level operations and the lower-level apis offering a rich toolset for most advanced use cases. I have tried to solve my problem with keyedProcessFunctions also but was not able to get it to work

Re: Kafka connector doesn't support consuming update and delete changes in Table SQL API

Hi, AFAIK this should be supported in 1.12 via FLINK-19568 [1] I'm pulling in Timo and Jark who might know better. https://issues.apache.org/jira/browse/FLINK-19857 Regards, Roman On Mon, Feb 8, 2021 at 9:14 AM meneldor wrote: > Any help please? Is there a way to use the "Last row" from a de

Re: question on ValueState

Hi, I think Yun Tang is right, HeapStateBackend doesn't (de)serialize the value on update. As for "value()", it may (de)serialize it and return a copy if there is an ongoing async snapshot in progress (to protect from modifications). This shouldn't happen often though. Regards, Roman On Mon, Fe

Re: Cannot connect to queryable state proxy

Hi ChangZhuo, Queryable state is exposed on the same address as the TM RPC. You can change this address by modifying taskmanager.host [1]. However, I'm not sure if setting it to 127.0.0.1 or localhost will not break connectivity with the other components. [1] https://ci.apache.org/projects/flink/

Native Kubernetes annotation parsing problem

Hi team, I'm using Native Kubernetes annotation config *kubernetes.jobmanager.annotations* and I'm facing some problem with parsing. I use annotation *iam.amazonaws.com/role:'arn:aws:iam:::role/XX/ '* but seems no

Enabling allowNonRestoredState when resuming from a savepoint causes ClassNotFoundException

Hi, I have an original job (say v1) and I want to start a new job (say v2) from a savepoint of v1. An operator of v1 used to have per-key states of a POJO type, but I want to remove the states together with the definition of the POJO type. When I start v2 from a savepoint of v1, I specified "--a

Re: Table Cache Problem

thanks for your help，Timo，it is very helpful 在 2021年2月8日星期一，Timo Walther 写道： > Hi Yongsong, > > in newer Flink versions we introduced the concept of statament sets, which > are available via `TableEnvironment.createStatementSet()`. They allow you > to opimized a branching pipeline as a whole wit

Joining and Grouping Flink Tables with Java API

Hi, We're trying to use Flink 1.11 Java tables API to process a streaming use case: We have 2 streams, each one with different structures. Both events, coming from Kafka, can be: - A new event (not in the system already) - An updated event (updating an event that previously was inserted) so we on

Re: Enabling allowNonRestoredState when resuming from a savepoint causes ClassNotFoundException

Hi 张静, Q1: By default, a savepoint restore will try to match all state > back to the restored job. `AllowNonRestoredState` cannot avoid > recovery all state from savepoint, but only skip match all of the > restore state back to the restored job. So `ClassNotFoundException ` > could not be avoi

Re: Native Kubernetes annotation parsing problem

I think it will be more generic question of how I inject IAM roles in Native Kubernetes pods I'm using Kubeiam and seems the namespace annotation doesn't work On Mon, Feb 8, 2021 at 2:30 PM Kevin Kwon wrote: > Hi team, I'm using Native Kubernetes annotation config > > > *kubernetes.jobmanager.a

Re: Kafka connector doesn't support consuming update and delete changes in Table SQL API

Hi, could the problem be that you are mixing OVER and TUMBLE window with each other? The TUMBLE is correctly defined over time attribute `row_ts` but the OVER window is defined using a regular column `upd_ts`. This might be the case why the query is not append-only but updating. Maybe you ca

Re: Kafka connector doesn't support consuming update and delete changes in Table SQL API

Thanks for the quick reply, Timo. Ill test with the row_ts and compaction mode suggestions. However, ive read somewhere in the archives that the append only stream is only possible if i extract "the first" record from the ranking only which in my case is the oldest record. Regards On Mon, Feb 8,

Re: Enabling allowNonRestoredState when resuming from a savepoint causes ClassNotFoundException

Hi, I'm pulling Yun Tang who is familiar with StateBackends and RocksDB in particular. >From what I see, the 2nd snapshot (sp2) is built using the same set of states obtained from the starting savepoint/checkpoint (sp1) to write its metadata. This metadata includes serializers snapshots, includin

Re: Connecting Pyflink DDL to Azure Eventhubs using Kafka connector

Hi, Could you provide the exception stack trace? Regards, Roman On Mon, Feb 8, 2021 at 3:46 PM joris.vanagtmaal < joris.vanagtm...@wartsila.com> wrote: > I'm trying to read data from my eventhub in Azure, but i end up with the > Flink error message 'findAndCreateTableSource failed' > > using F

Any plans to make Flink configurable with pure data?

To a naive Flink newcomer (me) it's a little surprising that there is no pure "data" mechanism for specifying a Flink pipeline, only "code" interfaces. With the DataStream interface I can use Java, Scala or Python to set up a pipeline and then execute it - but that doesn't really seem to *need *a p

Re: Sliding Window Count: Tricky Edge Case / Count Zero Problem

Hi, Probably another solution would be to register a timer (using KeyedProcessFunction) once we see an element after keyBy. The timer will fire in windowIntervalMs. Upon firing, it will emit a dummy element which will be ignored (or subtracted) in the end. Upon receiving each new element, the func

Connecting Pyflink DDL to Azure Eventhubs using Kafka connector

I'm trying to read data from my eventhub in Azure, but i end up with the Flink error message 'findAndCreateTableSource failed' using Flink 1.13-Snapshot source_ddl = f"""CREATE TABLE dms_source( x_value VARCHAR ) WITH (

Re: Connecting Pyflink DDL to Azure Eventhubs using Kafka connector

Traceback (most recent call last): File "streaming-dms.py", line 309, in anomalies() File "streaming-dms.py", line 142, in anomalies t_env.sql_query(query).insert_into("ark_sink") File "/Users/jag002/anaconda3/lib/python3.7/site-packages/pyflink/table/table_environment.py", line 748,

Re: Enabling allowNonRestoredState when resuming from a savepoint causes ClassNotFoundException

Hi Khachatryan, Thanks for the explanation and the input! 1. Use the State Processor API to create a new snapshot [1] I haven't used it. but does the API prevent the class of a specific serializer from being loaded? 2. If the operator has only this state then changing uid (together with > allow

Re: Native Kubernetes annotation parsing problem

If you are setting the config options in flink-conf.yaml, then you could directly add the following example. *kubernetes.jobmanager.annotations: iam.amazonaws.com/role:'arn:aws:iam:::role/XX/ '* However, if you are usin

Re: Re: Connecting Pyflink DDL to Azure Eventhubs using Kafka connector

Hi, Have you also include the kakfa-connector related jar in the classpath? Best, Yun --Original Mail -- Sender:joris.vanagtmaal Send Date:Tue Feb 9 03:16:52 2021 Recipients:User-Flink Subject:Re: Connecting Pyflink DDL to Azure Eventhubs using Kafka connector T

Re: Re: Sliding Window Count: Tricky Edge Case / Count Zero Problem

Hi, I also think there should be different ways to achieve the target. For the first option listed previously, the pseudo-code roughly like class MyFunciton extends KeyedProcessFunction { ValueState count; void open() { count = ... // Create the value state } ​void pr

Re: Re: Sliding Window Count: Tricky Edge Case / Count Zero Problem

Hi Jan, Another solution is to insert Heartbeat-events at the source for each sensor. The solution is very similar to how to advance watermarks when there are no elements in the respective source partition. However, it's only easy to implement if you have your own source and know all sensors on a

Re: question on ValueState

What I am interested in is whether I should use rocksDB to replace fileBackend. RocksDB's performance is not good, while it's state size can be very large. Currently, my job's state is about 10GB, and I use 10 TaskManagers in different machines, each 100G memory. I do not think I should use rocksDB

Re: question on ValueState

I have a related question. Since fileStateBackend uses heap as the state storage and the checkpoint is finally stored in the filesystem, so whether the JobManager/TaskManager memory will limit the state size? The state size is limited by TM's memory * number of TMs? or limited by JM's memory. Kha