unsubscribe (退订)

Hi, To unsubscribe emails from Flink dev mail list, send an email to dev-unsubscr...@flink.apache.org To unsubscribe emails from Flink user mail list, send an email to user-unsubscr...@flink.apache.org To unsubscribe emails from Flink user -zh mail list, send an email to user-zh-unsubscr..

Re: asyhcnrouonous io question

> > 1 . If the future completes exceptionally, ie resultFuture > .completeExceptionally(throwable); > > does the input message get discarded? > If you add an exception that will bubble up and let the task fail. Fail-over then determines what happens but in most cases the job will restart to an earl

jdbc connector configuration

Hi, I am trying to read data from database with JDBC driver. From [1], I have to config below parameters. I am not quite sure if I understand it correctly. lower-bound is smallest value of the first partition, upper-bound is largest value of the last partition. For example, if the db table has 100

Deploying python statefun program on standalone Flink cluster

Hello! I was wondering if there is an example to deploy Python statefun program directly on standalone cluster (or if this is supported at all). Right now I found an early write up here saying t

Can BroadcastProcessFunction invoke both methods concurrently?

I couldn't find any explicit documentation on whether a broadcast operator might invoke processElement and processBroadcastElement concurrently. At first I suspected it can, hence the different Contexts (read-write, read-only). But the TwoInputStreamOperator

Re: RocksDB: Spike in Memory Usage Post Restart

Hi Fabian, Yes I can tell you a bit more about the job we are seeing the problem with. I'll simplify things a bit but this captures the essence: 1. Input datastreams are from a few kafka sources that we intend to join. 2. We wrap the datastreams we want to join into a common container class and k

Re: Snapshot method for custom keyed state checkpointing ?

Hello Nicolaus, Thank you for your quick feedback, sorry if I am not clear enough. Actually in the documented example, the state which is updated in the snapshotState method is an operator state and not a keyed state: *public void initializeState(FunctionInitializationContext context) throws Exce

Re: RocksDB: Spike in Memory Usage Post Restart

Hi Kevin, Since you are seeing the problem across multiple Flink versions and with the default RocksDb and custom configuration it might be related to something else. A lot of different components can allocate direct memory i.e. some filesystem implementations, the connectors or some user grpc

Re: RocksDB: Spike in Memory Usage Post Restart

Hi Fabian, Thanks for collecting feedback. Here's the answers to your questions: 1. Yes, we enabled incremental checkpoints for our job by setting `state.backend.incremental` to true. As for whether the checkpoint we recover from is incremental or not, I'm not sure how to determine that. It's wha

Re: Issues while upgrading from 1.12.1 to 1.14.0

Hello Parag, Looking at the last command you sent, it seems like you are not passing the savepoint path for the savepoint instance, but just passing the savepoint directory while restarting the job. When a savepoint is completed, it is usually materialized under //savepoint-. Can you please try

Re: Issues while upgrading from 1.12.1 to 1.14.0

Hi Parag, When you restore from a savepoint do you see a line like: "Restoring job {} from {}" in jobmanagers logs? Is the entire state lost or just part of it? Could you explain a bit what does your job look like and how do you check that the state is lost? Sorry if too obvious, but what are the

Re: Snapshot method for custom keyed state checkpointing ?

Hi Marc, I think you can just use keyed state in a CheckpointedFunction. FunctionInitializationContext gives you access to both keyed state and operator state (your stream needs to be keyed, of course). So you could just update your local custom state on regular invocations and update keyed state

Re: In flight records on Flink : Newbie question

Many thanks Fabian for your prompt replies much appreciated Thanks Declan On Wed, Oct 6, 2021 at 8:38 AM Fabian Paul wrote: > Hi Declan, > > As far as I know the FileSink does not buffer records but writes the > records to temporary files which are bucketed later. For the Elasticsearch > sink >

Re: In flight records on Flink : Newbie question

Hi Declan, As far as I know the FileSink does not buffer records but writes the records to temporary files which are bucketed later. For the Elasticsearch sink you are right it buffers the records before flushing them to ElasticSearch but you can control the flushing behaviour based on a given

Re: RocksDB: Spike in Memory Usage Post Restart

Hi Kevin, Sorry for the late reply. I collected some feedback from other folks and have two more questions. 1. Did you enable incremental checkpoints for your job and is the checkpoint you recover from incremental? 2. I saw in your configuration that you set `state.backend.rocksdb.block.cach