EOFException on attempt to scale up job with RocksDB state backend

Hello, I was trying to scale job up, took save point, changed parallelism setting from 6 to 8 and started job from savepoint: switched from RUNNING to FAILED on 10.204.2.98:6122-2946e1 @ gsp-tm-0.gsp-headless.gsp.svc.cluster.local (dataPort=41409). java.lang.Exception: Exception while creating S

Re: Relation between Two Phase Commit and Kafka's transaction aware producer

Hi Kevin, Perhaps the easiest way to answer your question, is to go through how the exactly-once FlinkKafkaProducer using a 2PC implementation on top of Flink's checkpointing mechanism. The phases can be broken down as follows (simplified assuming max 1 concurrent checkpoint and that checkpoint c

Re: Upgrade calcite version

Hi Sheng ~ It is a hard work to upgrade Calcite version because that means you need to: - Fix all the bug introduced by the new planner - Fix all the logical plan to have correct semantics - Replace the deprecate APIs to new In order to achieve this, you need to have good knowledge of Calcite ba

Re: Best practices for complex state manipulation

Thanks Gordon and Seth! On Wed, Mar 10, 2021, 21:55 Tzu-Li (Gordon) Tai wrote: > Hi Dan, > > For a deeper dive into state backends and how they manage state, or > performance critical aspects such as state serialization and choosing > appropriate state structures, I highly recommend starting fro

How to do code isolation if muiltple jobs run on the same taskmanager process?

Consider the following situation. Two jobs builed in the same jar, they will share some common code, for example, some static constants variables. Currently they are running on the same task manager process. I killed job1, changed the static variable to another and then submit it again. Does ano

Re: Best practices for complex state manipulation

Hi Dan, For a deeper dive into state backends and how they manage state, or performance critical aspects such as state serialization and choosing appropriate state structures, I highly recommend starting from this webinar done by my colleague Seth Weismann: https://www.youtube.com/watch?v=9GF8Hwqz

Time Temporal Join

Hello folks, I am looking to enrich rows from an unbounded streaming table by joining it with a bounded static table while preserving rowtime for the streaming table. For example, let's consider table two tables F and D, where F is unbounded and D is bounded. The schema for the two tables is the f

Re: entrypoint for executing job in task manager

Hi Steven, Curious how you solved this for you use case. Did you ever find a satisfactory approach? Thanks in advance, Bob -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Re: Problem when restoring from savepoint with missing state & POJO modification

I'm having the same issue and I don't understand the explanations. I'm using Flink 1.9.3 and a KeyedCoProcessFunction. My state's class is still being used (it's a POJO), but I removed one of its fields. According to the documentation

Re: Flink Read S3 Intellij IDEA Error

Also I don't see ConfigConstants.ENV_FLINK_PLUGINS_DIR I only see ConfigConstants.ENV_FLINK_LIB_DIR will this work ? Thanks Sri On Wed, Mar 10, 2021 at 1:23 PM sri hari kali charan Tummala < kali.tumm...@gmail.com> wrote: > I am not getting what you both are talking about lets be clear. > > Plug

Re: Flink Read S3 Intellij IDEA Error

I am not getting what you both are talking about lets be clear. Plugin ? what is it ? Is it a Jar which I have to download from the Internet and place it in a folder ? Is this the Jar which I have to download ? (flink-s3-fs-hadoop) ? Will this belo solution work ? https://stackoverflow.com/questi

mixing java libraries between 1.12.x and 1.11.x

(also updated https://issues.apache.org/jira/browse/FLINK-19955 w/ this question) i'm in the situation where i want to use ParquetProtoWriters found in flink-parquet 1.12.x. the rest of our codebase, anticipating possibly running on the fully-managed aws flink solution for production, is dependin

Re: Python DataStream API Questions -- Java/Scala Interoperability?

A follow-up question--In the example you provided Shuiqiang, there were no arguments passed to the constructor of the custom sink/source. What's the best way to pass arguments to the constructor? On Fri, Mar 5, 2021 at 4:29 PM Kevin Lam wrote: > Thanks Shuiqiang! That's really helpful, we'll gi

Re: Flink Read S3 Intellij IDEA Error

Well, you could do this before running the job: // set the ConfigConstants.ENV_FLINK_PLUGINS_DIR environment variable, pointing to a directory containing the plugins PluginManager pluginManager = PluginUtils.createPluginManagerFromRootFolder(new Configuration()); Filesystem.initialize(new Co

Re: Flink Read S3 Intellij IDEA Error

Hi. I had the same problem. Flink use a plugins to access s3. When you run local it starts a mini cluster and the mini cluster don’t load plugins. So it’s not possible without modifying Flink. In my case I wanted to investigate save points through Flink processor API and the workaround was to

Is there any complete code available for checkpointing

Hi, [1] speaks about enable checkpointing, and [2] speaks about checkpointed function. Do we need both to see a triggered and completed checkpoint (according to the pic) in the apache flink UI? Is there any complete code available for checkpointing so that we can run and see the triggered and com

Re: Flink and Nomad ( from Hashicorp)

Hi Vishal, There is no specific reason why Flink does not have a Nomad HA implementation other than it has not been done yet. As long as Nomad supports leader election, service recovery and persisting small data it should be possible to implement HighAvailabilityServices against it. The best point

Re: Hadoop Integration Link broken in downloads page

Thanks a lot for reporting this problem Debraj. I've created a JIRA issue for it [1]. [1] https://issues.apache.org/jira/browse/FLINK-21723 Cheers, Till On Tue, Mar 9, 2021 at 5:28 AM Debraj Manna wrote: > Hi > > It appears the Hadoop Interation >

Flink Non-Heap Memory Configuration

Hi, i'm using Flink 1.11.3. With Prometheus and Grafana I get some metrics from my standalone cluster. For this simple calculation there will be very less RAM (lee than 10 MB): flink_jobmanager_Status_JVM_Memory_NonHeap_Committed - flink_jobmanager_Status_JVM_Memory_NonHeap_Used Is that the r

Re: idleTimeMsPerSecond on Flink 1.9?

Hi Lakshmi, as you have said the StreamTask code base has evolved quite a bit between Flink 1.9 and Flink 1.12. With the mailbox model it now works quite differently. Moreover, the community no longer actively maintains versions < 1.11. Hence, if possible I would recommend you to upgrade to one of

Evenly Spreading Out Source Tasks

I have a cluster with 18 task managers 4 task slots each running a job whose source/sink(s) are declared with FlinkSQL using the Kafka connector. The topic being read has 36 partitions. The problem I'm observing is that the subtasks for the sources are not evenly distributed. For example, 1 tas

User metrics outside tasks

Hello, Is it possible to register user metric outside task/operator (not in RichMapFunction#open) Thanks, Alexey

Re: Flink Read S3 Intellij IDEA Error

Flink, I am able to access Kinesis from Intellij but not S3 I have edited my stack overflow question with kinesis code , Flink is still having issues reading S3. https://stackoverflow.com/questions/66536868/flink-aws-s3-access-issue-intellij-idea?noredirect=1#comment117656862_66536868 Thanks Sr

Question about Reactive mode support

Hello, We were going through FlIP-159 and FLIP-160 and found this feature of interest to us for auto-scaling purposes. The limitations

Re: Trigger and completed Checkpointing do not appeared

Hi, CheckpointedFunction is needed to trigger and complete the checkpoint, is it right? What is the work for the code in here [1]? it [1] said to enable checkpointing or some other function. What is the difference between [1] and [2]? If we use ExampleCountSource[2] example, what will be the im

How to debug checkpoint/savepoint stuck in Flink 1.12.2

Hi, We have updated our Flink applications to 1.12.2, alone with the following modifications to improve its performance: - Use unaligned checkpoint - Change the following fs config - state.backend.fs.memory-threshold: 1048576 - state.backend.fs.write-buffer-size: 4194304 However, now our Fli

Re: Future of QueryableState

Hi Konstantin, thanks for detailed answer. I also thought about CoFunction, but it is a bit too heavy for us for the moment (each state would have to have additional kafka producer/consumer). Guess we'll use QueryableState for now and try to phase it out slowly... thanks, maciek On 09.03

io.netty.channel.epoll.EpollEventLoopGroup cannot be cast to io.netty.channel.MultithreadEventLoopGroup

Hi, I am using this library -* jasync-postgresql* [1] for async calls to postgres in asyncio operator. I am using running Flink 1.9 on EMR. I am facing this error - Caused by: java.lang.ClassCastException: io.netty.channel.epoll.EpollEventLoopGroup cannot be cast to io.netty.channel.MultithreadEv

Re: Error Starting PyFlink in Kubernetes Session Cluster "Could Not Get Rest Endpoint"

Yang, "... This could be done by starting a pod as the Flink client." Do you have an example yaml you could share? On Sun, Mar 7, 2021 at 9:57 PM Yang Wang wrote: > I think you want to submit a Flink python job to the existing session > cluster. > Please ensure the session cluster is created w

Filtering lines in parquet

Hi all, I am trying to filter lines from parquet files, the problem is that they have different schemas, however the field that I am using to filter exists in all schemas. in spark this is quite straight forward : *val filtered = rawsDF.filter(col("id") != "123")* I tried to do it in flink by ext

Relation between Two Phase Commit and Kafka's transaction aware producer

Hi team, I just have a bit of confusion where Two Phase Commit and Kafka's transaction aware producer using transaction.id and enable.autocommit plays together what I understand of Flink checkpoint (correct me if I'm wrong) is that it saves the transaction ID as well as the consumer's commit offse

Re: apache-flink +spring boot - logs related to spring boot application start up not printed in file in flink 1.12

Hi Abhishek, sorry for the late reply. Did you manage to fix it? One remark: Are you sure you're referring to the right configuration file? log4j-cli.properties is used for the CLI tool [1]. Or do you try to get the logs from within the main of your job? Best, Matthias [1] https://ci.apache.org/p

Re: KafkaSource Problem

Great to hear. Yes, if you can help fix this issue that would be great. Cheers, Till On Tue, Mar 9, 2021 at 3:41 PM Bobby Richard wrote: > Great thanks, I was able to work around the issue by implementing my own > KafkaRecordDeserializer. I will take a stab at a PR to fix the bug, should > be a

Re: Re: Re: Checkpoint Error

Could it be that another process might have deleted the in progress checkpoint file? Cheers, Till On Mon, Mar 8, 2021 at 4:31 PM Yun Gao wrote: > Hi Navneeth, > > Is the attached exception the root cause for the checkpoint failure ? > Namely is it also reported in job manager log? > > Also, hav