Request for realistic Flink programs to check correct using differential testing

Hi all, I am part of a research group at the University of Pennsylvania. We are researching distributed stream processing systems; in particular, we are developing a framework for checking correctness of Flink programs through differential testing. What I want to ask you as a community of Flink u

How to handle avro BYTES type in flink

Hi fellow streamers, I'm trying to support avro BYTES type in my flink application. Since ByteBuffer isn't a supported type, I'm converting the field to an Array[Byte]: case Type.BYTES => (avroObj: AnyRef) => { if (avroObj == null) { null } else { val byteBuffer = avroOb

Flink SQL problem

Hi I have encountered a problem with Flink SQL. My code: DataSet dataSet0 = env.fromCollection( infos0 ); tableEnv.registerDataSet( "table0", dataSet0 ); String sql = "select closePrice from table0" Table table = tableEnv.sql( sql ); tableEnv.registerTable( tableName, table );

Using FlinkKafkaConsumer API

I am using flink-kafka-connector and this is my dependency "org.apache.flink" %% "flink-connector-kafka" % *"1.7.0"*, Whe I look at my dependency tree the kafka client version is -org.apache.kafka:kafka-clients:2.0.1 which comes from the above package. However when I run my code in the clust

StreamingFileSink rolling callback Inbox

Hello, I’m writing a Flink job that reads heterogenius (one row contains several types that need to be partitioned downstream) data from AWS Kinesis and writes to S3 directory structure like s3://bucket/year/month/day/hour/type, this all works great with StreamingFileSink in Flink 1.9, but problem

Re: Will there be a Flink 1.9.1 release ?

Thanks Kurt. I was just asking as it would help us a lot with the issue ( https://github.com/apache/flink/pull/9565) that I mentioned in my mail. It got merged recently (after the 1.9.0 release). BTW, the issue you mentioned didn't fixed in 1.9.1. Will u please explain what u mean by this. The i

Re: Will there be a Flink 1.9.1 release ?

Hi Debasish, I think there is a good chance to have 1.9.1, the only question is when. 1.9.0 released ~2 weeks ago, and I think some users are still under the migration if they want to use 1.9.0. Wait another 1 or 2 weeks and also see whether there are some critical bugs in 1.9.0 sounds reasonable

Re: Will there be a Flink 1.9.1 release ?

Hi Debasish, So far I am not aware of any concrete timeline for Flink 1.9.1 but I think that Gordon and Kurt (cc'ed) who were the release-1.9 managers are the best to answer this question. Cheers, Kostas On Mon, Sep 9, 2019 at 9:38 AM Debasish Ghosh wrote: > > Hello - > > Is there a plan for a

Will there be a Flink 1.9.1 release ?

Hello - Is there a plan for a Flink 1.9.1 release in the short term ? We are using Flink and Avro with Avrohugger generating Scala case classes form Avro schema. Hence we need https://github.com/apache/flink/pull/9565 which has been closed recently. regards. -- Debasish Ghosh http://manning.co

Re: suggestion of FLINK-10868

Hi Anyang, I think we cannot take your proposal because this means that whenever we want to call notifyAllocationFailure when there is a connection problem between the RM and the JM, then we fail the whole cluster. This is something a robust and resilient system should not do because connection pr