Re: State Processor API and existing state

Hi Marco, > I assume that all the data within the checkpoint are stored within the given Savepoint. Is that assumption correct? Yes > I have not figured out how to correct / augment / fix the state though. Can somebody please explain? Please try this way. 1. Load old savepoint file, create Savepoin

Re: Savepoint failure with operation not found under key

Thanks for the context Chesnay. Yes, I sent both requests to the same JM. Best regards Rainie On Mon, Jun 28, 2021 at 8:33 AM Chesnay Schepler wrote: > Ordinarily this happens because the status request is sent to a different > JM than the one who received the request for creating a savepoint.

回复: Re: How to convert local to iso 8601 time in flink sql?

Hi Xu I use Elasticsearch as Sink. If a timestamp field is sent into Elasticsearch without any timezone information, then it will be assumed to be UTC time (Coordinated Universal Time)[1]. [1] https://www.elastic.co/blog/converting-local-time-to-iso-8601-time-in-elasticsearch 1095193...@qq.

[DISCUSS] Better user experience in the WindowAggregate upon Changelog (contains update message)

When WindowAggregate works upon Changelog which contains update messages, UPDATE BEFORE message may be dropped as a late message. [1] In order to handle late UB message, user needs to set *all* the following 3 parameters: (1) enable late fire by setting table.exec.emit.late-fire.enabled : true

Re: How to convert local to iso 8601 time in flink sql?

Hi, Unfortunately Flink SQL doesn’t support TIMESTAMP WITH TIME ZONE type yet[1], maybe the you can try write an UDF to convert the timestamp '2021-06-29 09:00:00’ field to String(the string representation like '2021-06-29T09:00:00+08:00’). And could you share your scenario about using TIMESTA

How to convert local to iso 8601 time in flink sql?

Hi community, Now I have a timestamp field with format '-MM-dd HH:mm:ss', such as '2021-06-29 09:00:00'. How to convert this field to iso 8601 time with offset , such as '2021-06-29T09:00:00+08:00'? Thanks. 1095193...@qq.com

State Processor API and existing state

Let's say that a job has operators with UIDs: W, X, Y, and Z, and uses RocksDB as a backend with checkpoint data URI s3://checkpoints" Then I stop the job with a savepoint at s3://savepoint-1. I assume that all the data within the checkpoint are stored within the given Savepoint. Is that assumpti

Re: Looking for example code

Thanks, I'll check them out. Thomas J. Raef Founder, WeWatchYourWebsite.com http://wewatchyourwebsite.com tr...@wewatchyourwebsite.com LinkedIn Facebook On Mon, Jun 28, 2021 at 11:16 AM Piotr Nowoj

Re: Savepoint failure with operation not found under key

Ordinarily this happens because the status request is sent to a different JM than the one who received the request for creating a savepoint. The meta information for such requests is only stored locally on each JM and neither distributed to all JMs nor persisted anywhere. Did you send both requ

Re: Yarn Application Crashed?

Hi, You should still be able to get the Flink logs via: > yarn logs -applicationId application_1623861596410_0010 And it should give you more answers about what has happened. About the Flink and YARN behaviour, have you seen the documentation? [1] Especially this part: > Failed containers (inc

Re: Looking for example code

Have you seen the documents that I linked? Isn't it enough? First pular link that I posted [4] has some example code. Literally the first link inside the second pulsar blog I referenced [5] leads to the pulsar connector repository which also has some examples [6]. Piotrek [6] https://github.com/

Re: Looking for example code

I need it to connect to Pulsar and stream from Pulsar. I could not find any code on how to connect to Pulsar. I've done the WordCount, but I need sample code for how to connect to Pulsar. Thomas J. Raef Founder, WeWatchYourWebsite.com http://wewatchyourwebsite.com tr...@wewatchyourwebsite.com Link

Re: Cancel job error ! Interrupted while waiting for buffer

Hi, It's hard to say from the log fragment, but I presume this task has correctly switched to "CANCELLED" state and this error should not have been logged as an ERROR, right? How did you get this stack trace? Maybe it was logged as a DEBUG message? If not, that would be probably a minor bug in Fli

Re: Looking for example code

Hi, We are glad that you want to try out Flink, but if you would like to get help you need to be a bit more specific. What are you exactly doing, and what, on which step exactly and how is not working (including logs and/or error messages) is necessary for someone to help you. In terms of how to

Re: FW: Hadoop3 with Flink

Sorry for the belated reply. In 1.12, you just need to make sure that the HADOOP_CLASSPATH environment variable is set up. For more details, please refer to [1]. [1] https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/yarn/ Best, Yangze Guo On Mon, Jun 28,

Re: Event Time Timers in Process functions when time characteristic is ProcessingTime

Hi lenduha, > Processing time: We can only use processing time windows & timer. Event time: We are able to use both processing time & event time windows & timers. Processing time is always wall/clock time depending on the host running task manager so independent from the event time? Yes. > Also K

FW: Hadoop3 with Flink

Hi, Can anyone please share inputs on this? Regards, Suchithra From: V N, Suchithra (Nokia - IN/Bangalore) Sent: Thursday, June 24, 2021 2:35 PM To: user@flink.apache.org Subject: Hadoop3 with Flink Hello, We are using Apache flink 1.12.3 and planning to use Hadoop 3 version. Could you please

Re: Event Time Timers in Process functions when time characteristic is ProcessingTime

Thanks for the info Jing, just to clarify one point, so when we set the environment time characteristics to Processing time: We can only use processing time windows & timer. Event time: We are able to use both processing time & event time windows & timers. Processing time is always wall/clock time

Re: Exception in snapshotState suppresses subsequent checkpoints

My job is very simple as you can see from the code I pasted. I simply print out the number to stdout. If you look at the log the number continued to print out after checkpoint 1 which indicated no back pressure was happening. It is very easy to reproduce this if you run the code I provided in IDE

Re: Event Time Timers in Process functions when time characteristic is ProcessingTime

Hi lenduha, > When set the time characteristics to ProcessingTime via setStreamTimeCharacteristic(...) call, I cannot see watermarks in the Flink UI. The watermark would be swallowed in the case. > can I use Event Time Timers even if I set the time characteristics to ProcessingTime (via ctx.time

Re: [Flink SQL] Lookup join hbase problem

Hi houyin, Jark, Sorry I missed Jark's response before. > We should support lookup HBase on multiple fields (by Get#setFilter). Feel free to open issues. I agree with Jark. It's better to support multiple keys for HBase. > How can I solve this problem ? Before Hbase supports multiple keys, we could