Re: Broadcast State + Stateful Operator + Async IO

Yes. You have explained my requirements exactly as they are. My operator will talk to multiple databases and a couple of web services to enrich incoming input streams. I cannot think of a way to use the async IO operator. So I thought maybe convert these 7-10 calls into async calls and chain the Fu

Re: How to define event time and Watermark for intermediary joined table in FlinkSQL?

Hi Shengkai, Thank you for the reply. The UDF getEventTimeInNS uses timestamps of both streamA and streamB to calculate the true event time for streamB events. For illustrating purpose, we can consider it to be like this: public Long eval( Long baseTimeStampFromA, Long timestamp

Re: AvroRowDeserializationSchema

Oh, I just missed your last question, sorry for that. The offset is stored in the checkpoint and it will recover the offset from the checkpoint when the job failover. Things which you may need to pay attention to: 1) Enable the checkpoint and configure it if necessary [1] 2) Specify the start up m

Re: Broadcast State + Stateful Operator + Async IO

Hi Vishal I want to understand your needs first. Your requirements are: After a stateful operator receives a notification, it needs to traverse all the data stored in the operator state, communicate with an external system during the traversal process (maybe similar to join?). In order to improve

Re: NO jobmanager.log for yarn-per-job submitted with sql-client.sh

Hi ruiyun, Is there any log4j configuration file or logback.xml in your Flink configuration directory? Yarn should check whether the file exists first and set relative log options. > On 26 Apr 2022, at 5:21 PM, ruiyun wan wrote: > > Flink Version (1.13) > There is not jobmanager.log file

Re: How to dynamically modify the schema information of a table

Thank you for your reporting. I created a ticket in JIRA. You can track this problem in this link: https://issues.apache.org/jira/browse/FLINK-27449 . > On 22 Apr 2022, at 10:44 AM, 草莓 wrote: > > The following is the Java code > > @Test >

Re: Checkpoint Timeout Troubleshooting

Hi Sam I think the first step is to see which part of your Flink APP is blocking the completion of Checkpoint. Specifically, you can refer to the "Checkpoint Details" section of the document [1]. Using these methods, you should be able to observe where the checkpoint is blocked, for example, it ma

Checkpoint Timeout Troubleshooting

Hello, I am running into checkpoint timeouts and am looking for guidance on troubleshooting. What should I be looking at? What configuration parameters would affect this? I am afraid I am a Flink newbie so I am still picking up the concepts. Additional notes are below, anything else I can provide?

RE: Flink-Iceberg Table Sink failing with org.apache.hadoop.fs.s3a.S3AStorageStatistics Cast exception

I solved this problem by removing the Hadoop classpath from Flink cluster deployment. On 2022/04/28 09:04:50 Terry Heathcote wrote: > Hi > > We are running a Flink job that delivers Kafka data to an Iceberg table. > The job uses the *org.apache.iceberg.flink.CatalogLoader* and > *org.apache.iceber

Broadcast State + Stateful Operator + Async IO

Hello, My application has a stateful operator which leverages RocksDB to store a large amount of state. It, along with other operators receive configuration as a broadcast stream (KeyedBroadcastProcessFunction). The operator depends upon another input stream that triggers some communication with ex

Re:How to dynamically modify the schema information of a table

I tried this and found that the comment information will be lost forever when using "fromDataStream". I think you can create a issue to report this bug. At 2022-04-22 10:44:47, "草莓" wrote: The following is the Java code @Test public void test(){ StreamExecutionEnvironment env = StreamExecu

Re: Flink team staffing

Thousands of events per second depending on their size and complexity isn’t a big overhead. If you’re in AWS you might want to look at Kinesis Data Analytics. On Wed, Apr 27, 2022 at 19:32, Wei Liu wrote: > Hi everyone, > > I've been thinking about running some production-critical applications >

Re: How to debug Metaspace exception?

I assume you will take action on your side to track and fix the doc? :) On Thu, Apr 28, 2022 at 11:12 AM John Smith wrote: > Ok so to summarize... > > - Build my job jar and have the JDBC driver as a compile only > dependency and copy the JDBC driver to flink lib folder. > > Or > > - Build my jo

Re: How to debug Metaspace exception?

Ok so to summarize... - Build my job jar and have the JDBC driver as a compile only dependency and copy the JDBC driver to flink lib folder. Or - Build my job jar and include JDBC driver in the shadow, plus copy the JDBC driver in the flink lib folder, plus make an entry in config for classload

Re:Temporal join fails with "unexpected correlate variable $cor0 in the plan"

Hi, can you provide the version of the Flink with this exception? I test your SQL in Flink/master and it works. My test SQL is the following. create table table_1 ( `f0` STRING NOT NULL, `f1` DOUBLE NOT NULL, `rowtime` TIMESTAMP(3), WATERMARK FOR `rowtime` AS rowtime - INTERVAL '10' SECOND ) with(

Re: How to debug Metaspace exception?

I think what I meant was "either add it to /lib, or [if it is already in /lib but also bundled in the jar] add it to the parent-first patterns." On 28/04/2022 15:56, Chesnay Schepler wrote: Pretty sure, even though I seemingly documented it incorrectly :) On 28/04/2022 15:49, John Smith wrote:

Re: How to debug Metaspace exception?

Pretty sure, even though I seemingly documented it incorrectly :) On 28/04/2022 15:49, John Smith wrote: You sure? * /JDBC/: JDBC drivers leak references outside the user code classloader. To ensure that these classes are only loaded once you should either add the driver jars to F

Re: How to debug Metaspace exception?

You sure? - *JDBC*: JDBC drivers leak references outside the user code classloader. To ensure that these classes are only loaded once you should either add the driver jars to Flink’s lib/ folder, or add the driver classes to the list of parent-first loaded class via classloader.

Re: Unsubscribe

To unsubscribe you need to send a mail to user-unsubscr...@flink.apache.org . > 2022年4月27日 下午12:03，Amit Bhatia 写道： > > Unsubscribe > > Regards, > Amit Bhatia

Re: Unsubscribe

To unsubscribe you need to send a mail to user-unsubscr...@flink.apache.org . > 2022年4月28日 下午6:45，bhawana gupta 写道： > > Unsubscribe > > Regards, > Bhawana

Re: Using the official flink operator and kubernetes secrets

Also, just declaring it in the flink configs should be sufficient, no need to define it in the pod templates: flinkConfiguration: kubernetes.env.secretKeyRef: "env:DJANGO_TOKEN,secret:switchdin-django-token,key:token" Best, Matyas On Thu, Apr 28, 2022 at 1:17 PM Őrhidi Mátyás wrote: > Hi

Re: Using the official flink operator and kubernetes secrets

Hi Francis, I suggest accessing the environment variables directly, no need to pass them as command arguments I guess. Best, Matyas On Thu, Apr 28, 2022 at 11:31 AM Francis Conroy < francis.con...@switchdin.com> wrote: > Hi all, > > I'm trying to use a kubernetes secret as a command line argume

Unsubscribe

Unsubscribe Regards, Bhawana

RE: AvroRowDeserializationSchema

Don’t expect that answer =))However, I am very appreciate everything you did Thanks again for helping me out.Best,Quynh. Sent from Mail for Windows From: Dian FuSent: Thursday, April 28, 2022 2:59 PMTo: lan tranCc: user@flink.apache.orgSubject: Re: AvroRowDeserializationSchema Yes, I think so~ On T

Using the official flink operator and kubernetes secrets

Hi all, I'm trying to use a kubernetes secret as a command line argument in my job and the text replacement doesn't seem to be happening. I've verified passing the custom args via the command line on my local flink cluster but can't seem to get the environment var replacement to work. apiVersion:

Re: [EXT] Vertica jdbc sink error

Hi Martin, here is a Jira ticket I created: https://issues.apache.org/jira/browse/FLINK-27429 I guess you now assign it to me, right? :) Best Regards, Jasmin On 19.04.2022., at 22:34, Jasmin Redzepovic mailto:jasmin.redzepo...@superbet.com>> wrote: Hi Martin, Thanks for your answer. Regardin

Flink-Iceberg Table Sink failing with org.apache.hadoop.fs.s3a.S3AStorageStatistics Cast exception

Hi We are running a Flink job that delivers Kafka data to an Iceberg table. The job uses the *org.apache.iceberg.flink.CatalogLoader* and *org.apache.iceberg.flink.TableLoader *interfaces in combination with *org.apache.iceberg.flink.sink.FlinkSink *where the catalog type is Hive. We have had suc

Re: AvroRowDeserializationSchema

Yes, I think so~ On Thu, Apr 28, 2022 at 11:00 AM lan tran wrote: > Hi Dian, > > Sorry for missing your mail, so if I did as your suggestion and the Flink > somehow crashed and we have to restart the service, does the Flink job know > the offset where does it read from Kafka ? > > > > Sent from