Re: [DISCUSSION] Using `DataStreamUtils.reinterpretAsKeyedStream` on a pre-partitioned Kafka topic

Hi! As far as I know, even if you prepartition the data exactly the same way in kafka using the key groups, you have no guarantee that the kafka consumer source would pick up the right partitions. Maybe if you have exactly as many kafka partitions as keygroups/max parallelism, partitioned corre

Table/SQL API to read and parse JSON, Java.

Hi there, I'm following the link to read JSON data from Kafka and convert to table, programmatically. I'd try and succeed declarative using SQL client. My Json data is nested like: {a:1,b,2,c:{x:1,y:2}}. Code: >

Re: Using MapState clear, put methods in snapshotState within KeyedCoProcessFunction, valid or not?

Hi Salva The root cause is that you did not figure out the difference between keyed state and operator state. There is no ‘currentKey’ in operator state, which means PartitionableListState#clear() will clear the whole state. However, there is always a ‘currentKey’ in keyed state, which means ‘

Re: Temporary failure in name resolution on JobManager

Hi David, Do you mean when the JobManager starts, the dns has some problem and the service could not be resolved? The dns restores to normal, and the JobManager jvm could not look up the dns. I think it may because the jvm dns cache. You could set the ttl and have a try. sun.net.inetaddr.ttl sun.n

Re: [DISCUSS] Disable conversion between TIMESTAMP and Long in parameters and results of UDXs

Since it is unanimously agreed that we should disable conversion between Timestmap and long in parameters and results of UDXs, in PR [1] we will disable it in blink planner. And we will add a release note in FLINK-14599 [2] of this incompatible modification.

Is it possible to recover from a checkpoint after modify program?

Hi, Here is our case: a job, reading data from Kafka, doing some process and writing to HDFS, has been running quite a while and has completed checkpoints. Now we'd like to add a new phase in process and want to recover from a checkpoint taken before. The new phase may or may not be stateful. Is i

Re: [DISCUSSION] Using `DataStreamUtils.reinterpretAsKeyedStream` on a pre-partitioned Kafka topic

Hi >From the doc[1], the DataStream MUST already be pre-partitioned in EXACTLY the same way Flink’s keyBy would partition the data in a shuffle w.r.t. key-group assignment. you should make sure that the key locates in the right key-group, and the key-group locates in the right parallelism. you can

Re: Using MapState clear, put methods in snapshotState within KeyedCoProcessFunction, valid or not?

Hi >From the exception `No key set. This method should not be called outside of a keyed context.` it means that the key current passed in is null. In my opinion, it's something wrong here if there will throw an exception when no data arrive. could you please share the whole stack and a minimal rep

Re: Read multiline JSON/XML

Also, say sorry to Flavio! Best, Vino vino yang 于2019年12月2日周一 上午10:29写道： > Hi Chesnay, > > Sorry, yes, I lost the "like" keyword. I mistakenly thought he wanted to > ask how to use Spark to accomplish this job. > > Best, > Vino > > Chesnay Schepler 于2019年11月29日周五 下午10:01写道： > >> Why vino? >> >

[ANNOUNCE] Weekly Community Update 2019/48

Dear community, happy to share a short community update this week. With one week to go to the planned feature freeze for Flink 1.10 and Flink Forward Asia in Beijing the dev@ mailing list pretty quiet these days. Flink Development == * [releases] Hequn has started the vote on RC1 for

Re: Idiomatic way to split pipeline

Thanks Arvid, The problem is that I will get an exception on non unique uid on the *stream* . On Thu, Nov 28, 2019 at 2:45 PM Arvid Heise wrote: > *This Message originated outside your organization.* > -- > Hi Avi, > > it seems to me that you are not really needing an