flink not able to get scheme for S3

Hey All, I am running flink in docker containers (image Tag :flink:scala_2.11-java11) on EC2 and getting exception as I am trying to submit a job through the local ./opt/flink/bin *org.apache.flink.client.program.ProgramInvocationException: The main method caused an error: No FileSystem for schem

[ANNOUNCE] Apache Flink 1.13.2 released

The Apache Flink community is very happy to announce the release of Apache Flink 1.13.2, which is the second bugfix release for the Apache Flink 1.13 series. Apache Flink® is an open-source stream processing framework for distributed, high-performing, always-available, and accurate data stream

Re: Implement task local recovery on TaskManager restart for Signifyd

Adding Sonam Hi Colman, I work for the Streaming SQL team at LinkedIn with Sonam, We have this in our backlog. We are interested in solving this as well. If you are equally interested, We can try to collaborate and solve this problem together. Thanks, Srini. On Wed, Aug 4, 2021 at 8:05 PM Colm

Re: Table API Throws Calcite Exception CannotPlanException When Tumbling Window is Used

Hi Joe, Window TVF is supported since Flink 1.13, while 1.12 does not support yet. Please upgrade to 1.13 version, or use the old Group Window Aggregate [1] syntax in 1.12. [1] https://ci.apache.org/projects/flink/flink-docs-release-1.12/dev/table/sql/queries.html#group-windows Best, JING ZHANG

Re: Best Practice of Using HashSet State

Hi Jerome, The type of value, list and map means that the structure of value to the primary key. I am not sure what the set structure you mean here, if you want to let the value as a set, and you can just leverage map state. As you might know, java actually use HashMap to implement the HashSet.

Re: Bloom Filter - RocksDB - LinkageError Classloading

Hi Sandeep, If you set the flink-statebackend-rocksdb as provided scope, it should not include the org.rocksdb classes, have you ever checked your application jar package directly just as what I described? Best Yun Tang From: Sandeep khanzode Sent: Friday, Aug

Table API Throws Calcite Exception CannotPlanException When Tumbling Window is Used

Hi all,   I am on flink 1.12.3. I am trying to get a tumbling window work with the table API as documented here:   https://ci.apache.org/projects/flink/flink-docs-master/docs/dev/table/sql/queries/window-tvf/#tumble   I have a kafka topic as a flink source. I convert the stream into

Best Practice of Using HashSet State

Hi, I am new to Flink and state backend. I find Flink does provide ValueState, ListState, and MapState. But it does not provide State object for HashSet. What is the best practice of storing HashSet State in Flink? Should we use ValueState and set the value to be HashSet? Or should we use ListS

Re: Bloom Filter - RocksDB - LinkageError Classloading

Hello Stephan, Yun, Thanks for your insights. All I have added is this: org.apache.flink flink-statebackend-rocksdb_2.12 ${flink.version} provided No other library explicitly added. I am assuming, as mentioned, is that the flink-dist.jar already contains the relevant classes a

write into parquet with variable number of columns

Hi mates ! I'm trying to find the best way to persist data into columnar format (parquet) using Flink. Each event contains a fixed list of properties and a variable list of properties, defined by the user. And I would like to save user defined properties into separate columns on the fly. Here is

Using POJOs with the table API

Hi everyone, I had been using the DataSet API until now, but since that's been deprecated, I started looking into the Table API. In my DataSet job I have a lot of POJOs, all of which are even annotated with @TypeInfo and provide the corresponding factories. The Table API documentation talks abo

Re: Bloom Filter - RocksDB - LinkageError Classloading

@Yun Tang Our FRocksDB has the same java package names (org.rocksdb.). Adding 'org.rocksdb' to parent-first patterns ensures it will be loaded only once, and not accidentally multiple times (as Child-first classloading does). The RocksDB code here is a bit like Flink internal components, which we

Re:Is FlinkKafkaConsumer setStartFromLatest() method needed when we use auto.offset.reset=latest kafka properties

Hi suman, FlinkKafkaConsumer.setStartFromLatest() means you always consume messages from the latest whenever you restart the flink job，the consumer ignore any committed group offsets. auto.offset.reset=latest means the consumer fetch messages from the latest if you never committed

Re: Flink k8 HA mode + checkpoint management

FLINK-19358[1] might be related and we already have some discussion there. [1]. https://issues.apache.org/jira/browse/FLINK-19358 Best, Yang Yun Tang 于2021年8月4日周三 上午11:50写道： > Hi Harsh, > > The job id would be fixed as if using HA > mode with native k8s, which

Re: Avro SpecificRecordBase question

Hi Kirill, as far as I know SpecificRecordBase should work in Flink, I don't know if there's any limitation in StateFun. It seems that the typeClass passed to the generateFieldsFromAvroSchema from the PravegaDeserializationSchema.. Maybe the pravega.LoadsSource does not bind correctly the Avro clas

Re: Bloom Filter - RocksDB - LinkageError Classloading

Hi Stephan, Since we use our own FRocksDB instead of the original RocksDB as dependency, I am not sure whether this problem has relationship with this. From my knowledge, more customers would include Flink classes within the application jar package, and it might cause problems if the client has

Re: [ANNOUNCE] RocksDB Version Upgrade and Performance

Hi Piotr, 1. Can we push for better benchmark coverage in the RocksDB project in the future? 2. Sure, I think we could contribute what we did in flink-benchmarks to improve their JMH benchmark [1]. And I will ask them how often will they run the benchmark. 1. Can we try to catch this