Re: Null result cannot be used for atomic types

hi sunfulin, which flink version are you using ? best, godfrey sunfulin 于2020年1月10日周五 下午1:50写道： > Hi, I am running a Flink app while reading Kafka records with JSON format. > And the connect code is like the following: > > > tableEnv.connect( > > new Kafka() > > .version(ka

Null result cannot be used for atomic types

Hi, I am running a Flink app while reading Kafka records with JSON format. And the connect code is like the following: tableEnv.connect( new Kafka() .version(kafkaInstance.getVersion()) .topic(chooseKafkaTopic(initPack.clusterMode)) .pro

Custom File Sink using EventTime and defined custom file name for parquet file

Hi, I'm working for the first time with Flink and I'm trying to create solution that will store events from Kafka into Parquet files in S3. This also should support re-injection of events from Parquet files into a Kafka topic. Here

Re: Elasticsink sometimes gives NoClassDefFoundError

Also, the ES version I'm using is 5.6.7 Jayant On Thu, Jan 9, 2020 at 10:39 AM Jayant Ameta wrote: > Hi, > The elastic connector is packaged in the uber jar that is submitted. There > is only 1 version of the connector: > flink-connector-elasticsearch5_2.11:1.7.1 > I'm using Flink 1.7.1 > > I

Re: How to assign a UID to a KeyedStream?

Hi Ken, This is actually a bug that a Partition should not require a UID. It is fixed in 1.9.2 and 1.10. see FLINK-14910 . Thanks, Zhu Zhu Ken Krugler 于2020年1月10日周五 上午7:51写道： > Hi all, > > [Of course, right after hitting send I realized I could

Re: Flink Job claster scalability

Hi KristoffSC, Did you increase the parallelism of the vertex that has the largest parallelism? Or did you explicitly set tasks to be in different slot sharing group? With the default slot sharing, the number of slots required/used equals to the max parallelism of a JobVertex, which is 6 in your c

[Question] Failed to submit flink job to secure yarn cluster

Hello I was following https://ci.apache.org/projects/flink/flink-docs-release-1.8/ops/deployment/yarn_setup.html#run-a-flink-job-on-yarn and trying to submit a flink job on yar

Please suggest helpful tools

Hi, I'm running Flink job on 1.9 version with blink planner. My checkpoints are timing out intermittently, but as state grows they are timing out more and more often eventually killing the job. Size of the state is large with Minimum=10.2MB and Maximum=49GB (this one is accumulated due to prior

How to assign a UID to a KeyedStream?

Hi all, [Of course, right after hitting send I realized I could just do rides.getTransformation().setUid(“blah”), ditto for the fares stream. Might be something to add to the docs, or provide a .uid() method on KeyedStreams for syntactic sugar] Just for grins, I disabled auto-generated UIDs fo

How to assign a UID to a KeyedStream?

Hi all, Just for grins, I disabled auto-generated UIDs for the taxi rides/fares state example in the online tutorial. env.getConfig().disableAutoGeneratedUIDs(); I then added UIDs for all operators, sources & sinks. But I still get the following when calling env.getExecutionPlan()

Re: How can I find out which key group belongs to which subtask

Thanks Congxian! My purpose is not only make data goes into one same subtask but the specific subtask which belongs to same taskmanager with upstream record. The key idea is to avoid shuffling between taskmanagers. I think the KeyGroupRangeAssignment.java

Running Flink on java 11

Hi guys, well We have requirement in our project to use Java 11, although we would really like to use Flink because it seems to match our needs perfectly. We were testing it on java 1.8 and all looks fine. We tried to run it on Java 11 and also looks fine, at least for now. We were also running

Re: Flink logging issue with logback

FYI, there is also a PR: https://github.com/apache/flink/pull/10811 On 09.01.20 01:53, Bajaj, Abhinav wrote: Thanks Dawid, Max and Yang for confirming the issue and providing potential workaround. On 1/8/20, 3:24 AM, "Maximilian Michels" wrote: Interesting that we came across this prob

Re: Completed job wasn't saved to archive

Hi Pavel, Sorry for bringing this thread up so late. I was digging into the usage of the Flink history server and I found one situation where there would be no logs and no failure/success message from the cluster: In very rare case in our Flink-YARN session cluster: if an application master (JobMa

Re: UnsupportedOperationException from org.apache.flink.shaded.asm6.org.objectweb.asm.ClassVisitor.visitNestHostExperimental using Java 11

Hi, are there any plans to support Java 11? Thanks, Krzysztof -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/

Job Cluster vs Session Cluster deploying and configuration

Hi all, I'm researching docker/k8s deployment possibilities for Flink 1.9.1. I'm after reading/watching [1][2][3][4]. Currently we do think that we will try go with Job Cluster approach although we would like to know what is the community trend with this? We would rather not deploy more than one

Re: Flink Job claster scalability

Thank you David and Zhu Zhu, this helps a lot. I have follow up questions though. Having this /"Instead the Job must be stopped via a savepoint and restarted with a new parallelism"/ and slot sharing [1] feature, I got the impression that if I would start my cluster with more than 6 task slots,

Re: Flink Job claster scalability

Hi KristoffSC, As Zhu Zhu explained, Flink does not currently auto-scale a Job as new resources become available. Instead the Job must be stopped via a savepoint and restarted with a new parallelism (the old rescale CLI experiment use to perform this). Making Flink reactive to new resources and a

Re: How can I find out which key group belongs to which subtask

Hi If you just want to make sure some key goes into the same subtask, does custom key selector[1] help? For the keygroup and subtask information, you can ref to KeyGroupRangeAssignment[2] for more info, and the max parallelism logic you can ref to doc[3] [1] https://ci.apache.org/projects/flink/

When I use flink 1.9.1 and produce data to Kafka 1.1.1, The streamTask checkpoint error .

Hi all: When I use flink 1.9.1 and produce data to Kafka 1.1.1. the error was happen as log-1,code is::input.addSink(new FlinkKafkaProducer(parameterTool.getRequired("bootstrap.servers"),parameterTool.getRequired("output-topic"),

Data overflow in SpillingResettableMutableObjectIterator

Hi all: We are using flink's iteration，and find the SpillingResettableMutableObjectIterator has a data overflow problem if the number of elements in a single input exceeds Integer.MAX_VALUE. The reason is inside the SpillingResettableMutableObjectIterator, it track the total number of elements and