Re: Flink running same task on different Task Manager

lism to 12 for the task2 (this real-time task needs to >> read from 12 different Kafka partitions hence setting it to 12) >> 3. set parallelism of task1 to 2 >> 4. then set this cluster.evenly-spread-out-slots: true >> >> Will these methods work? Also, I did not find a

Re: Flink task lifecycle listener/hook/SPI

Hi Allen, >From my experience, you can do your init setup by the following 2 ways: 1. Do your init setup in RichFunction#open method, see [1] for details. 2. Do your init setup in static block, it will be executed when the class is loaded. [1] https://nightlies.apache.org/flink/flink-docs-master/

Re: Migration to application mode

Hi, I think the difference between ApplicationMode and PerJob is just where the main method is executed (ApplicationMode executes on JM, PerJob executes on client side). So I think your original job code should work well under ApplicationMode. Did you encounter any problems? You can get more detail

Re: Issues with Flink scheduler?

Hi, Which version are you using? Has any job failover occurred? It would be better if you can provide the full log of JM. Best, Lijie Hemanga Borah 于2022年8月1日周一 01:47写道： > Hello guys, > We have been seeing an issue with our Flink applications. Our > applications run fine for several hours, an

Re: Flink running same task on different Task Manager

lure > ? > > On Wed, Jun 15, 2022 at 6:13 PM Lijie Wang > wrote: > >> Hi Great, >> >> Do you mean there is a Task1 and a Task2 on each task manager? >> >> If so, I think you can set Task1 and Task2 to the same parallelism and >> set them in the same

Re: Is it possible to save Table to CSV?

You can use the FileSink and set the format to csv. An example of FileSink: https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/filesystem/#full-example Best, Lijie 于2022年7月11日周一 16:16写道： > > If I create dynamic table with: > > > CREATE TABLE some_table (name STRING, scor

Re: ContinuousFileMonitoringFunction retrieved invalid state.

ion > are in very precarious position. With all the other upgrades (rocksdb, > etc.) going into 1.15.0 there's no easy rollback. > > On Fri, Jul 1, 2022 at 8:14 AM Lijie Wang > wrote: > >> Hi, >> Are you using the reactive mode? There is a known issue like that: >&

Re: ContinuousFileMonitoringFunction retrieved invalid state.

Hi, Are you using the reactive mode? There is a known issue like that: https://issues.apache.org/jira/browse/FLINK-28274 Best, Lijie yuxia 于2022年7月1日周五 09:49写道： > I'm not sure why it happened. But from the Flink source code, it seems try > to restore from an invalid state. Seems the state actu

Re: Flink running same task on different Task Manager

Hi Great, Do you mean there is a Task1 and a Task2 on each task manager? If so, I think you can set Task1 and Task2 to the same parallelism and set them in the same slot sharing group. In this way, the Task1 and Task2 will be deployed into the same slot(That is, the same task manager). You can g

Re: exception while activating calculateThroughp

Hi sigalit, It's a known bug that has been fixed in Flink 1.15.0. See [1] and [2] for details. [1] https://issues.apache.org/jira/browse/FLINK-27712 [2] https://issues.apache.org/jira/browse/FLINK-25454 Best, Lijie Sigalit Eliazov 于2022年6月13日周一 20:17写道： > Hi all > > > We are using the flink k8

Re: Metrics for FileSource

Hi Meghajit, As far as I know, currently, the FileSource does not have the metrics you need. You can implement your own source, and register custom metrics via `SplitEnumeratorContext#metricGroup` and `SourceReaderContext#metricGroup`. Best, Lijie Meghajit Mazumdar 于2022年6月10日周五 16:36写道： > He

Re: [ANNOUNCE] Apache Flink Kubernetes Operator 1.0.0 released

Congrats! Thanks Yang for driving the release, and thanks to all contributors! Best, Lijie John Gerassimou 于2022年6月6日周一 22:38写道： > Thank you for all your efforts! > > Thanks > John > > On Sun, Jun 5, 2022 at 10:33 PM Aitozi wrote: > >> Thanks Yang and Nice to see it happen. >> >> Best, >> Aito

Re: Adaptive Batch Configuration Does Not Work

Thanks for feedback! Can you post the jobmanager log? This may be because the source parallelism has been set in some way (Kind of source can infer parallelism according to the catalog). Best, Lijie Edwin 于2022年3月1日周二 19:30写道： > Hi all, > > I was trying to run flink-tpcds tests on flink v1.15-s

Re: [ANNOUNCE] Open source of remote shuffle project for Flink batch data processing

As one of the contributors of flink remote shuffle, I'm glad to hear all the warm responses! Welcome more people to try the flink remote shuffle and look forward to your feedback. Best， Lijie Yingjie Cao 于2021年12月1日周三 17:50写道： > Hi Jiangang, > > Great to hear that, welcome to work together to m

Re: New blog post published - Sort-Based Blocking Shuffle Implementation in Flink

Thanks Daisy and Kevin for bringing this blog, it is very helpful for understanding the principle of sort shuffle. Best, Lijie Guowei Ma 于2021年11月3日周三 下午2:57写道： > > Thank Daisy& Kevin much for your introduction to the improvement of TM > blocking shuffle, credit base+io scheduling is indeed a