Re: Flink Serialization as stable (kafka) output format?

2020-04-18 Thread Robert Metzger
Hey Theo, we recently published a blog post that answers your request for a comparison between Kryo and Avro in Flink: https://flink.apache.org/news/2020/04/15/flink-serialization-tuning-vol-1.html On Tue, Mar 10, 2020 at 9:27 AM Arvid Heise wrote: > Hi Theo, > > I strongly discourage the use o

Re: Question about Writing Incremental Graph Algorithms using Apache Flink Gelly

2020-04-18 Thread Kaan Sancak
Thanks that worked! I wonder what will be the performance difference if I implement this with Stateful Functions. Does anyone knows recent works/papers on similar approach? Best Kaan > On 16 Apr 2020, at 10:00, Yun Gao wrote: > >  > Hi Kaan, > >For the first issue, I think the two impl

FlinkKafakaProducer with Confluent SchemaRegistry and KafkaSerializationSchema

2020-04-18 Thread Anil K
Hi, What is the best way to use Confluent SchemaRegistry with FlinkKafkaProducer? What I have right now is as follows. SerializationSchema serializationSchema = ConfluentRegistryAvroSerializationSchema.forGeneric(topic, schema, schemaRegistryUrl); FlinkKafkaProducer kafkaProducer = new

Re: Re: Flink streaming sql是否支持两层group by聚合

2020-04-18 Thread dixingxin...@163.com
@Benchao @Jark thank you very much. We have use flink 1.9 for a while , and we will try 1.9 + minibatch. dixingxin...@163.com Sender: Jark Wu Send Time: 2020-04-18 21:38 Receiver: Benchao Li cc: dixingxing85; user; user-zh Subject: Re: Flink streaming sql是否支持两层group by聚合 Hi, I will use

Re: Flink upgrade to 1.10: function

2020-04-18 Thread Jark Wu
Hi, I think it is not caused by SqlStdOperatorTable, but the Calcite parser. So it is tricky to work around it. Maybe it needs to be fixed from Calcite side. Best, Jark On Sat, 18 Apr 2020 at 01:05, seeksst wrote: > Hi, > > >Thank you for reply. > > >I find it caused by SqlStdOperatorT

Re: multi-sql checkpoint fail

2020-04-18 Thread Jark Wu
Hi, What's the statebackend are you using? Is it Heap statebackend? Best, Jark On Sat, 18 Apr 2020 at 07:06, tison wrote: > Hi, > > Could you share the stack traces? > > Best, > tison. > > > forideal 于2020年4月18日周六 上午12:33写道: > >> Hello friend >> I have two SQL, checkpoint fails all the time.

Re: Flink streaming sql是否支持两层group by聚合

2020-04-18 Thread Jark Wu
Hi, I will use English because we are also sending to user@ ML. This behavior is as expected, not a bug. Benchao gave a good explanation about the reason. I will give some further explanation. In Flink SQL, we will split an update operation (such as uv from 100 -> 101) into two separate messages,

Re: Checkpoints for kafka source sometimes get 55 GB size (instead of 2 MB) and flink job fails during restoring from such checkpoint

2020-04-18 Thread Yun Tang
Hi Oleg If your can only view the log of "snapshotState fetcher: partition: KafkaTopicPartition{topic\u003d\u0027dsp-producer-z-clickstream-web-raw\u0027, partition\u003d2} offset:1091528771\n" when the checkpoint of that subtask has reached to 1GB+. This is really weird, as the state in unionO