Re: Custom Sink Object attribute issue

Hi Jigar, I'm moving your user question to the user ML. The best place to initialize transient fields is in private void readObject(java.io.ObjectInputStream in) throws IOException, ClassNotFoundException; as described in [1]: Remember that transient fields will be initialized to their d

Re: Flink 1.11 loses track of event timestamps and watermarks after process function

Hi Ahmad, >From your description, I'd look in a different direction: Could it be that your Sink/Async IO is not processing data (fast enough)? Since you have a bounded watermark strategy, you'd need to see 10s of data being processed before the first watermark is emitted. To test that, can you ple

Re: Timeout settings for Flink jobs?

Thanks Caizhi, our use case is for testing - before running a production job, we want to “dry run” it for a while and output the result to a temp location where we can verify the output. We are running a streaming job but the use case sounds more like a batch job. On Mon, Oct 11, 2021 at 7:34 PM C

Re: Yarn job not exit when flink job exit

Hi, Jake In Flink 1.14, we recommend using "-t yarn-per-job"[1] for starting per-job cluster. Regarding your issue, I could not reproduce it with the Wordcount example. However, I think this is not the right way for Flink's SQL client, which might be the root cause of your issue. Would you like to

Inconsistent parallelism in web UI when using reactive mode

Hi, We found that parallelism in web UI are inconsistent when using reactive mode. As in attachment, in overview page, all parallelism values are 1, which is not correct one. When clicking operator for detail information, the parallelism in detail information is the correct one. Is it possible to

Re: Checkpoint size increasing even i enable increasemental checkpoint

Hi! Checkpoint sizes are highly related to your job. Incremental checkpointing will help only when the values in the state are converging (for example a distinct count aggregation). If possible, could you provide your user code or explain what jobs are you running? Lei Wang 于2021年10月11日周一 下午4:1

Re: Yarn job not exit when flink job exit

Hi! yarn-cluster is the mode for a yarn session cluster, which means the cluster will remain even after the job is finished. If you want to finish the Flink job as well as the yarn job, use yarn-per-job mode instead. Jake 于2021年10月9日周六 下午5:53写道： > Hi > > When submit job in yarn-cluster model, f

Re: Timeout settings for Flink jobs?

Hi! There is currently no such setting. You need to rely on an external system to read the execution time (from Flink's job information, see [1]) and cancel the job once it exceeds the time limit. Could you elaborate more on your use case? Are you running a streaming job or a batch job? For strea

Re: Does the flink sql support checkpoints

(Forwarding this to the user mailing list as this mail is written in English) Hi! I think problem 1 is the expected behavior. Is this behavior inconvenient for you? If yes why it is the case? For problem 2, could you explain in detail how do you run the word count program and where do you store

Re: PyFlink JDBC SQL Connector for SQL Server

Hi, Currently it only supports derby, mysql, postgresql dialect. The dialect 'sqlserver' is still not supported. There is a ticket https://issues.apache.org/jira/browse/FLINK-14101 for this. Regards, Dian On Mon, Oct 11, 2021 at 9:43 PM Schmid Christian wrote: > Hi all > > > > According to the

Flink 1.11 loses track of event timestamps and watermarks after process function

Flink 1.11 I have a simple Flink application that reads from Kafka, uses event timestamps, assigns timestamps and watermarks and then key's by a field and uses a KeyedProcessFunciton. The keyed process function outputs events from with the `processElement` method using `out.collect`. No timers are

Timeout settings for Flink jobs?

Hi there, We have a use case where we want to terminate a job when a time limit is reached. Is there a Flink setting that we can use for this use case? Thanks, Sharon

Re: Empty Kafka topics and watermarks

Great, thanks! pon., 11 paź 2021 o 17:24 James Sandys-Lumsdaine napisał(a): > Ah thanks for the feedback. I can work around for now but will upgrade as > soon as I can to the latest version. > > Thanks very much, > > James. > -- > *From:* Piotr Nowojski > *Sent:* 08

Re: Empty Kafka topics and watermarks

Ah thanks for the feedback. I can work around for now but will upgrade as soon as I can to the latest version. Thanks very much, James. From: Piotr Nowojski Sent: 08 October 2021 13:17 To: James Sandys-Lumsdaine Cc: user@flink.apache.org Subject: Re: Empty Kaf

Re: Reset of transient variables in state to default values.

It would be difficult to provide even a semblance of the complete product , however I could try to provide enough details to reproduce the problem. Standard source would do: DataStream stream = env.addSource( new FlinkKafkaConsumer<>(topic, new AbstractDeserializationSchema() {

Re: Reset of transient variables in state to default values.

Hi Alex, It is a little weird. Would you please provide the program which could reproduce the problem, including DataStream job code and related classes code. I need some debug to find out the reason. Best, JING ZHANG Alex Drobinsky 于2021年10月11日周一 下午5:50写道： > Hi Jing Zhang, > > I'm using the F

Re: Unsubscribe

Hi, To unsubscribe emails from Flink user mail list, send an email to user-unsubscr...@flink.apache.org To unsubscribe emails from Flink dev mail list, send an email to dev-unsubscr...@flink.apache.org To unsubscribe emails from Flink user -zh mail list, send an email to user-zh-unsubscr...@flink

PyFlink JDBC SQL Connector for SQL Server

Hi all According to the official documentation (Table API / JDBC SQL Connector v.1.14.0) "the JDBC connector allows reading data from and writing data into any relational databases with a JDBC driver". At the moment we are using SQL Server in conjunction with Flink and Java, which works perfect

Unsubscribe

Hello i want to unsubscribe

offset of TumblingEventTimeWindows

As in China (UTC+08:00)，we should use Time.hours(-8) as offset when state day's data, // daily tumbling event-time windows offset by -8 hours. input.keyBy().window(TumblingEventTimeWindows.of(Time.days(1),Time.hours(-8))).() shall we also set Time.hours(-8) as offset for minute's state ?

Re: After Receiving Kafka Data ( getting this error) for s3 bucket access

sorry , there was issue with path of s3 bucket, Got this fixed .. Sorry for troubling you guys On Sunday, October 10, 2021, 12:33:16 PM EDT, Dhiru wrote: We have configured s3 bucket s3a://msc-sandbox-test-bucketI am not sure how come some extra characters get added for a bucket? java

Re: Reset of transient variables in state to default values.

Hi Jing Zhang, I'm using the FileSystem backend. I also implemented ReadObject function to support proper restart procedure: private void readObject(ObjectInputStream ois) throws ClassNotFoundException, IOException { ois.defaultReadObject(); logger.info("Deserialized MultiStorePac

Re: Reset of transient variables in state to default values.

Hi, Alex What state backend do you choose? If you choose MemoryStateBackend or FsStateBackend, `transient` keyword may not have effect because MemoryStateBackend does not serialize state for regular read/write accesses but keeps it as objects on the heap. If you choose RocksDBStateBackend, I though

Reset of transient variables in state to default values.

Dear flink community, I have following state class ( irrelevant fields removed ) public class MultiStorePacketState implements Serializable { public transient RandomAccessFile currentFile = null; public long timerValue; public String fileName; public String exportedFileName; p

RE: Flink S3 Presto Checkpointing Permission Forbidden

Hi Rommel, Thanks for getting back to me and for your time. I switched to the Hadoop plugin and used the following authentication method that worked: *fs.s3a.aws.credentials.provider: "com.amazonaws.auth.WebIdentityTokenCredentialsProvider"* Turns out that I was using the wrong credentials pr

Checkpoint size increasing even i enable increasemental checkpoint

[image: image.png] The checkpointed data size became bigger and bigger and the node cpu is very high when the job is doing checkpointing. But I have enabled incremental checkpointing: env.setStateBackend(new RocksDBStateBackend(checkpointDir, true)); I am using flink-1.11.2 and aliyun oss as ch

Re: OVER IGNORE NULLS support

Seems like IGNORE NULL is specified in SQL spec 2008 (paragraph 6.10), the opposite is called RESPECT NULLS: ::= RESPECT NULLS | IGNORE NULLS Perhaps this is worth supporting, I've opened an issue for that FLINK-24499 . @Adrian are you intereste