Re: Hybrid Source stop processing files after processing 128 SourceFactories

2022-07-26 Thread Benenson, Michael via user
Hi, Mason Think, the problem is related to https://github.com/apache/flink/blob/release-1.14.3-rc1/flink-connectors/flink-connector-base/src/main/java/org/apache/flink/connector/base/source/hybrid/HybridSourceSplitEnumerator.java#L358 if (readerSourceIndex.get(e.getKey()) == (Integer) sourceInde

Re: Hybrid Source stop processing files after processing 128 SourceFactories

2022-07-26 Thread Mason Chen
Hi Michael, I'm glad the CPU fix works for you! Regarding the behavior, HybridSource should only consume from Kafka after it finishes the bounded read of the files. At that time, files will not be read anymore. In addition, there is no limitation where there can only be 128 source factories (the

Re: Is there any Natural Language Processing samples for flink?

2022-07-26 Thread Yunfeng Zhou
Hi John, So far as I know, Flink does not have an official library or sample specializing in NLP cases yet. You can refer to Flink ML[1] for machine learning samples or Deep Learning on Flink[2] for deep learning samples. [1] https://github.com/apache/flink-ml [2] https://github.com/flink-extende

Is there any Natural Language Processing samples for flink?

2022-07-26 Thread John Smith
As the title asks... All I see is Spark examples. Thanks

Re: Why this example does not save anything to file?

2022-07-26 Thread Gil De Grove
Hello, I may be really wrong with this, but from what I get in the source file, you are using a semi-column to separate the value. This probably means that you should set the csv.field-delimiter to `;` to make your example work properly. Have you tried with that configuration in your create table

Re: What do columns for TM memory usage in Flink UI Console mean?

2022-07-26 Thread Weihua Hu
Hi, John This just shows how many direct buffers are allocated through 'Bytebuffer.allocateDirect'. And the Used will equals Capacity because we can not get the real usage of DirectBuffer. Best, Weihua On Thu, Jul 21, 2022 at 12:54 AM John Tipper wrote: > Sorry, pressed send too early. > > W

Re: Why this example does not save anything to file?

2022-07-26 Thread Weihua Hu
Hi, Can you see any exception logs? Where is this code running? is it a standalone cluster with one TaskManager? Best, Weihua On Tue, Jul 26, 2022 at 4:18 AM wrote: > If I get it correctly this is the way how I can save to CSV: > > https://nightlies.apache.org/flink/flink-docs-master/docs/co

Avro 1.11 with Flink 1.14

2022-07-26 Thread Chirag Dewan via user
Hi, Is it possible to use Avro 1.11 with Flink 1.14? I know that Avro version is still at 1.10, but due to my job using Avro 1.11, I was planning to use it in Flink as well.  Also, I know that Avro 1.10 had some performance issues with Flink 1.12 ([FLINK-19440] Performance regression on 15.09.20