Re: Flink Stateful Function: The program's entry point class not found in the jar file

Thanks Igal the application runs perfectly now! Le On Mon, Jan 4, 2021 at 9:13 AM Igal Shilman wrote: > Hi Le, > > Looking at your pom.xml, you are pointing to the wrong main method here: > > https://github.com/flint-stone/flink-statefun/blob/lx-base/statefun-examples/statefun-flink-datastream-

Re: Unexpected latency across operator instances

Hi Antonis, Did you try to profile the “bad” taskmanager to see what the task thread was busy doing? And a possible culprit might be gc, if you haven't checked that. I’ve seen gc threads eating up 30% of cpu. Best, Paul Lam > 2020年12月14日 06:24，Antonis Papaioannou 写道： > > Hi, > > I experien

Question about Java lists in Flink SQL

Dear Flink committers, I have an issue with Flink SQL that I was wondering if you could help me with. I am trying to use Flink in my project and I ran into an issue. I have a DataStream that consists of POJOs. One of the fields in the POJO is of type java.util.List. I'm converting DataStream into

Re: Implementing a TarInputFormat based on FileInputFormat

Hi Arvid, Thanks for the response. I have created a sample application with input data and uploaded it to google drive. The sample data is in the archive... thus the large size. (27 mb) https://drive.google.com/file/d/1dxpnDF3hPUPNlPO5p2tBf-88oOVV0qck/view?usp=sharing To run it: flink run -Dexe

RE: Flink kafka exceptions handling

Thank you for your answer. I have been subscribed. This is the previous topic I’m referring to http://mail-archives.apache.org/mod_mbox/flink-user/202008.mbox/%3CCACzKVZQ093HixMewb_prtP41ceXgmxCv=cmpsbphw-9+h8b...@mail.gmail.com%3E Our flink job manager fails after multiple restarting, when the

Re: Using key.fields in 1.12

Yes, I do have that dependency. I see it in the dependency view of intellij and directly. in the uber jar. Thanks for responding. - Aeden On Wed, Jan 6, 2021 at 8:06 AM Piotr Nowojski wrote: > > Hey, > > have you added Kafka connector as the dependency? [1] > > [1] > https://ci.apache.org/proje

Re: reason for endless backpressure

Hi, If you have an unstable network, which is dropping packets in a weird way (data is lost, but the connection is still kept alive from the perspective of the underlying operating system) it could happen that task will be perpetually blocked. But this is extremely rare. I would first suggest tryi

Re: Roadmap for Execution Mode (Batch/Streaming) and interaction with Table/SQL APIs

Hi, 1. I think those changes will mostly bring new features/functionalities to the existing Streaming APIs in order to fully support batch executions. For example one way or another to better handle "bounded data streams" in the DataStream API. 2. I think there is and there is not going to be one

reason for endless backpressure

I've had a problem many times. When the task suddenly continues to back pressure, the back pressure node will no longer send any records unless the task is restarted. But I can confirm that it's not due to high pressure. During the back pressure period, the CPU utilization of the machine is all red

Re: Using key.fields in 1.12

Hey, have you added Kafka connector as the dependency? [1] [1] https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/connectors/kafka.html#dependencies Best, Piotrek śr., 6 sty 2021 o 04:37 Aeden Jameson napisał(a): > I've upgraded from 1.11.1 to 1.12 in hopes of using the key.fiel

Roadmap for Execution Mode (Batch/Streaming) and interaction with Table/SQL APIs

Hi Flink Community, Really excited for the "true unification" of Batch and Stream APIs, and had a few questions for clarification after reading the release notes, and watching the video by aljoscha here . 1) How do these changes impact the Table an

Re: question regarding flink local buffer pool

Hi Eleanore, first I'd like to point to a related blog post, which explains most concepts in a better way than I could write here [1]. Now let's go through your questions: 1. A buffer pool is just a place where a task gets a buffer from. So pool-A is used by taskA for both reading (from network

Re: "flink list" fails when zk-based HA is enabled in flink-conf.yaml

Thanks Yang for the very detailed explanation! Wow, I really appreciate it. Best, Dongwon On Wed, Jan 6, 2021 at 10:17 PM Yang Wang wrote: > Hi Dongwon, > > Please find the answer inline. > > > why CLI accessed ZK? > This is a good question. Currently, when the HA is enabled, even though we >

Re: "flink list" fails when zk-based HA is enabled in flink-conf.yaml

Hi Dongwon, Please find the answer inline. > why CLI accessed ZK? This is a good question. Currently, when the HA is enabled, even though we could get the JobManager rest endpoint from Yarn application report, we still have to retrieve the leader information from ZooKeeper. Please find more infor

Re: Batch with Flink Steraming API version 1.12.0

Hi Robert, The most reliable way to use batch mode in streaming is to use event time [1]. Processing time windows or ingestion time does not make a whole lot of sense if you want to do some kind of reprocessing (indeterministic results and resource usage because the timestamp of records change wit

Re: "flink list" fails when zk-based HA is enabled in flink-conf.yaml

Hi Yang, I was wondering why CLI accessed ZK because, as shown in the following lines, CLI seemed to know the address of JM by contacting AHS before connecting to ZK. 2021-01-06 18:35:32,351 INFO org.apache.flink.client.cli.CliFrontend [] - Running 'list' command. 2021-01-06 18:

Re: StreamingFileSink.forBulkFormat() with CheckpointRollingPolicy issues

Hi Mahendra, Sorry for the late reply. I noticed that in your code you implement a bucket assigner that reads to switch to a new bucket every minute, does it related to the current problems met ? Since different buckets would use different directories and files, when switching buckets new