Compressing Avro entities in State

Hello, I am using the Maven Avro plugin to create SpecificRecord classes for a bunch of entities specified in .avsc files. I use the generated Java classes directly in a few MapStates. My state is now growing very large. I wanted to know whether there is any config or code to specify Snappy o

Re: Bloom Filter - RocksDB - LinkageError Classloading

Hi Yun, Yes, I will check how these classes are getting included. I do see the below coming in from a Kafka-Streams dependency being used for another module. org.rocksdb:rocksdbjni:jar:5.18.4:compile Thanks, Sandip > On 10-Aug-2021, at 8:22 AM, Yun Tang wrote: > > Hi Sandeep, > > I'm afr

Re: [External] : Big data architecture

Hello Aissa, I guess you might be interested in this video: https://www.youtube.com/watch?v=X3L75Rz64Ns&list=PL2oL9cdRCATGOSFvG3O5QbSuAcvkmr_KV&index=19 Thanks, Fuyao From: Aissa Elaffani Date: Thursday, July 15, 2021 at 03:55 To: user@flink.apache.org Subject: [External] : Big data architect

Apache/flink 1.13.1 images re-pushed and in arm64 arch?

Our 1.13.1 k8s deployments started failing with tandard_init_linux.go:228: exec user process caused: exec format error Not understanding how an old image with an old depoyment starts failing like that i looked at the apache/flink dockerhub page. Although 1.13.1 was released more than a month ago,

Re: Approach to test custom Source/Sink

Hi Bin, We could try the following method to cover the source/sink test. Unit test: To verify whether the behavior of each method in custom source or sink is expected. You could mock interactions with external storage (database, IO, etc.) in this part. Integration test: To test whether the source/s

Re: [External] Re: KafkaDeserializationSchema.open() is not called after task state change

In this case there seems to be a bug. Please open a ticket in JIRA for that and please describe the situation more thoroughly in the ticket. If possible please also provide your user code and the exception message. Thanks. Gil Amsalem 于2021年8月9日周一 下午5:20写道： > Hi Caizhi, > Thanks for your reply.

Re: Approach to test custom Source/Sink

Hi! Currently there is no general principle for testing sources and sinks. However you might want to check out the unit tests and IT cases for Flink connectors. For example flink-connector-jdbc module has a lot of tests for the JDBC source and sink. Follow the ideas in these tests should be enough

Re: [ANNOUNCE] Apache Flink 1.13.2 released

Thanks Yun and everyone~! Thank you~ Xintong Song On Mon, Aug 9, 2021 at 10:14 PM Till Rohrmann wrote: > Thanks Yun Tang for being our release manager and the great work! Also > thanks a lot to everyone who contributed to this release. > > Cheers, > Till > > On Mon, Aug 9, 2021 at 9:48 AM Yu

s3 access denied with flink-s3-fs-presto

We are using Flink version 1.13.0 on Kubernetes. For checkpointing we have configured fs.s3 flink-s3-fs-presto. We have enabled sse on our buckets with kms cmk. flink-conf.yaml is configured as below. s3.entropy.key: _entropy_ s3.entropy.length: 4 s3.path.style.access: true s3.ssl.enabled: true s3

Approach to test custom Source/Sink

Hi team, I'm currently implementing a custom source and sink, and I'm trying to find a way to test these implementations. The testing section in the of

Re: Flink application question

FYI, I've responded to this on stack overflow: https://stackoverflow.com/questions/68715430/apache-flink-accessing-keyed-state-from-late-window On Mon, Aug 9, 2021 at 3:16 AM suman shil wrote: > I am writing a Flink application which consumes time series data from > kafka topic. Time series dat

Re: [ANNOUNCE] Apache Flink 1.13.2 released

Thanks Yun Tang for being our release manager and the great work! Also thanks a lot to everyone who contributed to this release. Cheers, Till On Mon, Aug 9, 2021 at 9:48 AM Yu Li wrote: > Thanks Yun Tang for being our release manager and everyone else who made > the release possible! > > Best R

RE: Unable to use custom AWS credentials provider - 1.9.2

Hi Arvid, no. We are leveraging it as part of our application code, but not Kinesis – after finding and excluding duplicates of this package in our classpath, we are able to submit a job. Thanks. // ah From: Arvid Heise Sent: Friday, July 30, 2021 1:34 PM To: Hailu, Andreas [Engineering] Cc:

Re: Production Grade GitOps Based Kubernetes Setup

Hi Yuval, thank you for sharing all the information. I forgot to mention the Lyft operator. Thanks for "adding" it to the list. About the dual cluster approach during upgrade I have some doubts about the resource usage. If you are operating some "big" jobs that would mean you always have to pro

How to deserialize Avro enum type in Flink SQL?

Hi community, I have a Kafka topic where the schema of its values is defined by the "MyRecord" record in the following Avro IDL and registered to the Confluent Schema Registry. > @namespace("my.type.avro") > protocol MyProtocol { > enum MyEnumType { > TypeVal1, TypeVal2 > } > record MyE

Re: Production Grade GitOps Based Kubernetes Setup

Hi Maciek, thanks for sharing your insights. It is highly appreciated. Regards, Niklas UNIBERG GmbH Simon-von-Utrecht-Straße 85a 20359 Hamburg niklas.wil...@uniberg.com Mobile: +49 160 9793 2593 Office: +49 40 2380 6523 UNIBERG GmbH, Dorfstraße 3, 23816 Bebensee Registergericht / Register

Re: [External] Re: KafkaDeserializationSchema.open() is not called after task state change

Hi Caizhi, Thanks for your reply. I already have some traces in the open() methods, and I do not see that it is being called. We are using Flink version 1.11.2. Should I open a bug for that? On Mon, Aug 9, 2021 at 5:24 AM Caizhi Weng wrote: > Hi! > > This does not sound like an expected behavio

Re: [ANNOUNCE] Apache Flink 1.13.2 released

Thanks Yun Tang for being our release manager and everyone else who made the release possible! Best Regards, Yu On Fri, 6 Aug 2021 at 13:52, Yun Tang wrote: > > The Apache Flink community is very happy to announce the release of Apache > Flink 1.13.2, which is the second bugfix release for the

Flink application question

I am writing a Flink application which consumes time series data from kafka topic. Time series data has components like metric name, tag key value pair, timestamp and a value. I have created a tumbling window to aggregate data based on a metric key (which is a combination of metric name, key value

RE: Inspecting SST state of rocksdb

Good morning Kai, Did you consider using the ‘State Processor API’ yet? : https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/libs/state_processor_api/ This one allows you to load any state primitive as a batch source, and … Best greetings Thias From: Kai Fu Sent: Montag, 9. A

Re: Inspecting SST state of rocksdb

Hi Kai, Since the stored key and value in RocksDB are serialized bytes, it's not easy to read directly. You could consider to use State Processor API [1]. However, this needs you to know what the state name of the SQL operator. You could analysis the checkpoing _metadata [2] to know state name