Re: State Processor API with EmbeddedRocksDBStateBackend

Hi Xianwen, Could you also attach the full stack of the exception~? Very thanks -- Sender:xianwen jin Date:2021/08/10 21:03:44 Recipient: Theme:State Processor API with EmbeddedRocksDBStateBackend Hi Flink Community, I have an iss

Re: Sliding window with filtering

Hi! It seems that you want to filter some record out before they go into the window. So why not filter them before applying the window? Dario Heinisch 于2021年8月10日周二 下午6:26写道： > Hey there, > > So I have a stream of data, let the stream be a_1, a_2, a_3, a_4, a_5. > Now I would like to have a sli

Re: Questions on reading JDBC data with Flink Streaming API

Hello James, To stream real time data out of the database. You need to spin up a CDC instance. For example, Debezium[1]. With the CDC engine, it streams out changed data to Kafka (for example). You can consume the message from Kafka using FlinkKafkaConsumer. For history data, it could be consid

Re: Apache/flink 1.13.1 images re-pushed and in arm64 arch?

Are you sure you are looking at the correct repo? flink has 2 docker repos for it's images - flink and apache/flink for flink, everything is fine https://hub.docker.com/_/flink?tab=tags&page=1&ordering=last_updated&name=1.13.1 But for apache/flink (which is still official, and updated faster then

Re: how to emit a deletion event for all data in iterating of production logic

Hi vtygoss, You could use `Insert overwrite`[1] syntax if the sink connector supports `overwrite`. Otherwise, please try the following 2 solution: Option A: Define custom source to read the data, and update the header of `RowData` to `Delete` Option B: Do a map function which update the header of `

Questions on reading JDBC data with Flink Streaming API

Hello, I'm starting a new Flink application to allow my company to perform lots of reporting. We have an existing legacy system with most the data we need held in SQL Server databases. We will need to consume data from these databases initially before starting to consume more data from newly

Re: Allowed lateness in Flink SQL

Thanks Ingo. I will look into CURRENT_WATERMARK. About allow-lateness. I can imagine the following situation. Let's have a left interval join between two streams. I want to wait 15 minutes for events from the right stream to arrive (this can be done by setting a watermark on the right stream). Als

Re: Apache/flink 1.13.1 images re-pushed and in arm64 arch?

Ah, yes I was looking at the wrong one. I'll figure out what happened and get the situation fixed. Overall though, I'd recommend to rely on https://hub.docker.com/_/flink. On 10/08/2021 16:04, Oran Shuster wrote: Are you sure you are looking at the correct repo? flink has 2 docker repos for it

Re: Allowed lateness in Flink SQL

Hi Maciej, The pr is related to FLINK-21301 [1]. Sets the time by which elements are allowed to be late. Elements that arrive behind the watermark by more than the specified time " + "will be dropped. " + "Note: use the value if it is set

Re: Allowed lateness in Flink SQL

Hi Maciej, there is no documentation for it (besides in the code itself) because it's an experimental flag. What would you expect allow-lateness to do outside the context of a window? Maybe you'd also be interested in CURRENT_WATERMARK()[1][2] which will be released with 1.14 and allows some level

Allowed lateness in Flink SQL

Hi Guys, I was checking if anything changed recently with allowed lateness support in Flink SQL and I found this PR: https://github.com/apache/flink/pull/16022 Is there any documentation for table.exec.emit.allow-lateness ? Is this option working only in window agregation? Regards, -- Maciek Bry

State Processor API with EmbeddedRocksDBStateBackend

Hi Flink Community, I have an issue regarding create a savepoint with EmbeddedRocksDBStateBackend. Since I have to use dataset for state processor api, it seems that the checkpoint storage is always set to jobmanager even though I pass: state.checkpoint-storage: filesystem state.checkpoints.dir: '

Re: Re: Flink 1.12.5: The heartbeat of JobManager/TaskManager with id xxx timed out

Hi Chenyu, The tipically reasons for the heartbeat timeout includes: 1. Long GC time in TM / JM 2. Network instability Thus does the GC log or network monitor metrics could give some hints ? Best, Yun -- Sender:Chenyu Zheng Date:

Re: [ANNOUNCE] Apache Flink 1.12.5 released

Thank you. On Tue, Aug 10, 2021 at 11:44 AM Till Rohrmann wrote: > This is great news. Thanks a lot for being our release manager Jingsong > and also to everyone who made this release possible. > > Cheers, > Till > > On Tue, Aug 10, 2021 at 10:57 AM Jingsong Lee > wrote: > >> The Apache Flink c

Re: [ANNOUNCE] Apache Flink 1.11.4 released

Awesome! Thank you for driving this. On Tue, Aug 10, 2021 at 11:45 AM Till Rohrmann wrote: > This is great news. Thanks a lot for being our release manager Godfrey and > also to everyone who made this release possible. > > Cheers, > Till > > On Tue, Aug 10, 2021 at 11:09 AM godfrey he wrote: >

Re: [ANNOUNCE] Apache Flink 1.13.2 released

Thank you! On Tue, Aug 10, 2021 at 11:04 AM Jingsong Li wrote: > Thanks Yun Tang and everyone! > > Best, > Jingsong > > On Tue, Aug 10, 2021 at 9:37 AM Xintong Song > wrote: > >> Thanks Yun and everyone~! >> >> Thank you~ >> >> Xintong Song >> >> >> >> On Mon, Aug 9, 2021 at 10:14 PM Till Rohrm

how to emit a deletion event for all data in iterating of production logic

Hi, Flink community! I have a problem when iterating the data production logic. e.g. data production procedure: …-> Table A(change log stream) -> Table B(change log stream) ->Table C(change log stream) …. production logic of Table B: insert into table B select * from Table A where a>0;

Re: Flink 1.12.5: The heartbeat of JobManager/TaskManager with id xxx timed out

JobManager timeout error: 2021-08-10 09:58:35,350 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Sink: Print to Std. Out (79/128) (b498a5b17c87eb70c3da9aea93890e25) switched from DEPLOYING to FAILED on stream-93072a8b402f49cca9c134a6e8b4887a-taskmanager-1-121 @ 10.50.15

Flink 1.12.5: The heartbeat of JobManager/TaskManager with id xxx timed out

Hi there, I’m trying to run my flink job on Kubernetes cluster, but when I try to give my job a larger parallelism (128) I get an error said “java.util.concurrent.TimeoutException: The heartbeat of JobManager with id 56ad1a5ded99f9f16ec1c786ad299159 timed out.” And then my job is cancelled. We

Sliding window with filtering

Hey there, So I have a stream of data, let the stream be a_1, a_2, a_3, a_4, a_5. Now I would like to have a sliding window which slides by 1 second and takes the data of 1 second. But here comes the caveat: - I only want to keep the data in the window that does not have matching elements.

Re: [ANNOUNCE] Apache Flink 1.11.4 released

This is great news. Thanks a lot for being our release manager Godfrey and also to everyone who made this release possible. Cheers, Till On Tue, Aug 10, 2021 at 11:09 AM godfrey he wrote: > The Apache Flink community is very happy to announce the release of Apache > Flink 1.11.4, which is the f

Re: [ANNOUNCE] Apache Flink 1.12.5 released

This is great news. Thanks a lot for being our release manager Jingsong and also to everyone who made this release possible. Cheers, Till On Tue, Aug 10, 2021 at 10:57 AM Jingsong Lee wrote: > The Apache Flink community is very happy to announce the release of Apache > Flink 1.12.5, which is th

[ANNOUNCE] Apache Flink 1.11.4 released

The Apache Flink community is very happy to announce the release of Apache Flink 1.11.4, which is the fourth bugfix release for the Apache Flink 1.11 series. Apache Flink® is an open-source stream processing framework for distributed, high-performing, always-available, and accurate data streamin

Re: [ANNOUNCE] Apache Flink 1.13.2 released

Thanks Yun Tang and everyone! Best, Jingsong On Tue, Aug 10, 2021 at 9:37 AM Xintong Song wrote: > Thanks Yun and everyone~! > > Thank you~ > > Xintong Song > > > > On Mon, Aug 9, 2021 at 10:14 PM Till Rohrmann > wrote: > > > Thanks Yun Tang for being our release manager and the great work! Al

[ANNOUNCE] Apache Flink 1.12.5 released

The Apache Flink community is very happy to announce the release of Apache Flink 1.12.5, which is the fifth bugfix release for the Apache Flink 1.12 series. Apache Flink® is an open-source stream processing framework for distributed, high-performing, always-available, and accurate data streaming

Re: Production Grade GitOps Based Kubernetes Setup

Hi all, I reached out [1] to Filipe Regadas, the author of the Spotify fork of the GCP k8s operator you linked in your message. He confirms he's actively working on it and would welcome PR and community input. I have a modest PR I'll submit to him some time this week already. This seems to me

Re: Apache/flink 1.13.1 images re-pushed and in arm64 arch?

Looking at the dockerhub page know, all 1.13.1 images are listed as amd64, with the last push being 5 days ago. Perhaps some error happened during the rebuild, which has since been corrected? On 10/08/2021 00:43, Oran Shuster wrote: Our 1.13.1 k8s deployments started failing with tandard_init_