Re: Odd Serialization exception

Hi, I think you might be right. So you could try to call the super.open(...) in your LogSessionProducer. Best, Guowei On Thu, Aug 12, 2021 at 2:01 PM Daniel Vol wrote: > Hi Guowei, > > I am running on EMR 5.32.0 with Flink 1.11.2 > > In meanwhile I did some tests and commented out part of the n

Re: Odd Serialization exception

Hi Guowei, I am running on EMR 5.32.0 with Flink 1.11.2 In meanwhile I did some tests and commented out part of the new code - override def invoke(value: OUT, context: SinkFunction.Context[_]): Unit = { try { // val session = value.asInstanceOf[Session] // sessionDuration = 17L //s

Re: StreamFileSink not closing file

Hi Matthias, Sorry for the late reply, this should be a known issue that Flink would lost the last piece of data for bounded dataset with 2pc sink. However, we are expected to fix this issue in the upcoming 1.14 version [1]. Best, Yun [1] https://issues.apache.org/jira/browse/FLINK-2491

Re: Odd Serialization exception

Hi, Daniel Could you tell me the version of Flink you use? I want to look at the corresponding code. Best, Guowei On Wed, Aug 11, 2021 at 11:23 PM Daniel Vol wrote: > Hi Matthias, > > First, thanks for a fast reply. > I am new to Flink, so probably I miss a lot in terms of flow and objects > pa

Re: Questions on usage of SQL hints

Hi Paul, There are Table hints and Query hints. Query hints are on the way, there is a JIRA to track this issue [3]. AFAIK, the issue is almost close to submit a pull request now. Table hints[1][2] are already supported since Flink 1.11. You could find more detail information in [1][2]. For table

Questions on usage of SQL hints

Hi community, I’m trying out SQL hints on DML, but there’s not very much about the supported SQL hints on the docs. Are the SQL hints limited to source/sink tables only at the moment? And where can I find the full list of supported SQL hints? Thanks in advance! Best, Paul Lam

Re: How to check rocksdb log

Hi, Fabian Thanks for your replay, it helps. in 1.13, [state.backend.rocksdb.log.dir] is deleted and I use [state.backend.rocksdb.localdir]. It works fine. On 2021/08/11 19:07:28, Fabian Paul wrote: > Hi Li, > > Flink has disabled the RocksDb logs because sizing problems but you can have > a

Re: Apache/flink 1.13.1 images re-pushed and in arm64 arch?

Ah yes, the M1... There is a way to generate amd64 images on M1 machines https://blog.jaimyn.dev/how-to-build-multi-architecture-docker-images-on-an-m1-mac/#tldr; but yeah, the architecture change makes a lot of mistakes in our company as well As for why i am using the apache/flink repo - there w

Re: How to check rocksdb log

Hi Li, Flink has disabled the RocksDb logs because sizing problems but you can have a look at this link [1] on how to enable them and setting the log directory. Let me know if that answers your question. Best, Fabian [1] https://ververica.zendesk.com/hc/en-us/articles/360015933320-How-to-get

How to check rocksdb log

Hi, everyone, I have a problem with checking rocksdb's log. I set "state.backend.rocksdb.log.level" to INFO_LEVEL but I can't find the rocksdb's log anywhere? Where can I set the log dir or where should I check by default? Thanks for any replys

Re: Savepoint class refactor in 1.11 causing restore from 1.9 savepoint to fail

I am able to reproduce this failure by loading the production savepoint into a locally running 1.11 flink job using the state processor API.The same sequence of events occurs; the Kryo snapshot deserializer stores a null for the refactored Savepoint interface which causes subsequent failures

Re: Odd Serialization exception

Hi Matthias, First, thanks for a fast reply. I am new to Flink, so probably I miss a lot in terms of flow and objects passed. The motivation is to get internal data from the transferred OUT Object to send metrics. So I do downscale it but as per my perspective it is not forwarded (super called wi

Re: State Processor API with EmbeddedRocksDBStateBackend

Hi Xianwen, Looks like the State Processor API needs to be updated for the new state backend factory stack. For now, just use RocksDBStateBackend and it will work as intended. I've opened a ticket: https://issues.apache.org/jira/browse/FLINK-23728 Seth On Wed, Aug 11, 2021 at 2:08 AM xianwen j

RE: Odd Serialization exception

Hi Daniel, On the first look there is one thing that catches my eye: In line ‘val session = value.asInstanceOf[Session]' it looks like you are downcasting the event from OUT to Session. In Flink this is a dangerous thing to do … DataStream[OUT] uses a specific serializer[OUT] to transport events

Odd Serialization exception

I started to get the following exception: 2021-08-11 09:45:30,299 [Window(EventTimeSessionWindows(180), EventTimeTrigger, SessionAggregator, PassThroughWindowFunction) -> Sink: Unnamed (1/8)] INFO o.a.f.s.runtime.tasks.SubtaskCheckpointCoordinatorImpl - Could not complete snapshot 134 for ope

Re: Apache/flink 1.13.1 images re-pushed and in arm64 arch?

Hi Oran and Chesnay, I think it should be my problem. The docker image I generated on the computer with the macbook M1 will lead to the image of arm64 (When releasing 1.12.5). We will regenerate the image of 1.13.1 on the Intel x86 machine. I'm very sorry. Best, Jingsong On Tue, Aug 10, 2021 a

Re: State Processor API with EmbeddedRocksDBStateBackend

Hi Yun, Here's the stack trace java.util.concurrent.ExecutionException: java.io.IOException: Size of the state is larger than the maximum permitted memory-backed state. Size=100091190 , maxSize=5242880 . Consider using a different state backend, like the File System State backend. at j