Re: How to use RichAsyncFunction with Test harness in flink 1.13.1?

Thanks again for the explanation. On Fri, 6 Aug 2021, 15:39 Arvid Heise, wrote: > User function from the point of view of Flink is any function that you as > a user write. In this case, MetadataEnrichment.AsyncFlowLookup is a > user-function of the Async Operator. > > I don't recommend harness f

Re: Support for authenticated schema registry in debezium registry

Hi Joe, there was a follow-up issue, FLINK-23450, which was only fixed for 1.13.3 (not yet released) which I think is what you're seeing? Best Ingo On Fri, Aug 6, 2021, 21:17 Joseph Lorenzini wrote: > Hi all, > > > > I am on flink 1.13.2. I set up create table like so: > > > > CREATE TABLE le

Support for authenticated schema registry in debezium registry

Hi all,   I am on flink 1.13.2. I set up create table like so:   CREATE TABLE lead_buffer (   `id` INT NOT NULL,   `state` STRING NOT NULL,    PRIMARY KEY (`id`) NOT ENFORCED ) WITH (   'connector'= 'kafka',   'topic' = 'buffer',   'format'= 'debezium-avro-confluent',   'debeziu

StreamFileSink not closing file

Hey guys, I wrote a simple DataStream that counts up some numbers into a SideOutput which I am trying to sink into a StreamFileSink so that I can write the results to disk and read them from there. I'm running my little test locally and I can see that the data is being written to hidden "inproress"

1.13 Flamegraphs

Hi all, Does the sample processing also sample threads that do not belong to the Flink framework? For example, a background thread that is created by and managed by the user? Best, Mason

Re: Production Grade GitOps Based Kubernetes Setup

Hi Niklas, We are currently using the Lyft operator for Flink in production ( https://github.com/lyft/flinkk8soperator), which is additional alternative. The project itself is pretty much in Zombie state, but commits happen every now and then. 1. Native Kubernetes could definitely work with GitOp

Re: Production Grade GitOps Based Kubernetes Setup

Hi Niklas, We had the same problem one year ago and we choose Ververica Platform Community Edttion. Pros: - support for jobs on Session Clusters - good support for restoring jobs from checkpoints and savepoints - support for even hundreds of jobs Cons: - state in SQLite (we've already corrupted db

Re: [EXT] Re: Reusing savepoints from a streaming job in a batch job

Sorry to hear that. Did I got that right that there is no way to save and recover state in batch mode? Tobi On 6. Aug 2021, 13:20 +0200, Chesnay Schepler , wrote: > All checkpointing-related features do not work in batch mode. > > https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/

Re: custom flink image error

finally, I work out how to build a custom flink image, the Dockerfile just as: > > FROM flink:1.13.1-scala_2.11 > ADD ./flink-s3-fs-hadoop-1.13.1.jar /opt/flink/plugins > ADD ./flink-s3-fs-presto-1.13.1.jar /opt/flink/plugins > the wrong Docker file is : > FROM apache/flink:1.13.1-scala_2.11

Production Grade GitOps Based Kubernetes Setup

Hi Flink Community, I'm currently assessing the situation about how to properly deploy Flink on Kubernetes via GitOps. There are some options available to deploy Flink on Kubernetes, which I would like to discuss. In general we are looking for an open source or at least unpaid solution, but I

Re: flink not able to get scheme for S3

The reason this doesn't work is that your application works directly against Hadoop. The filesystems in the plugins directory are only loaded via specific code-paths, specifically when the Flink FileSystem class is used. Since you are using Hadoop directly you are side-stepping the plugin mechan

Re: Reusing savepoints from a streaming job in a batch job

All checkpointing-related features do not work in batch mode. https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/dev/datastream/execution_mode/#important-considerations On 04/08/2021 21:23, tobias.schu...@xing.com wrote: He folks, This is a crosspost of a stack overflow question

Re: Is FlinkKafkaConsumer setStartFromLatest() method needed when we use auto.offset.reset=latest kafka properties

it is not the same. Kafka's 'auto.offset.reset' is used when the configured consumer group id does not have offset info stored in kafka. not exist. If you want to consume from latest no matter whether there is group offset info in kafka, you should use flink's setStartFromLatest. suman shil 于2021

Re: How to use RichAsyncFunction with Test harness in flink 1.13.1?

User function from the point of view of Flink is any function that you as a user write. In this case, MetadataEnrichment.AsyncFlowLookup is a user-function of the Async Operator. I don't recommend harness for several reasons: - It's not Public API, we will adjust it and we will introduce breaking

Re: [ANNOUNCE] RocksDB Version Upgrade and Performance

I am a heavy user of rocksdb, and my application works well on flink-1.10,but I am interst in the new features.I want to know if I can increase the manage memory to eliminate the impact of the performance regression In a certain degree。 Can you show me the params about rocksdb of your task?

Re: How to use RichAsyncFunction with Test harness in flink 1.13.1?

Thanks it worked. Can you please let me know what are you referring to as user function and why it is not recommended to not use harness with it? On Wed, 4 Aug 2021, 22:43 Arvid Heise, wrote: > I would strongly recommend not using the harness for testing user > functions. > > Instead I'd just c