Re: Kinesis Connector - NoClassDefFoundError

Hey, Have you updated the versions both on the environment and the dependency on the job? >From my personal experience, 95 % of such issues is due to the mismatch between Flink versions on the cluster you are using and Your job. Best Regards, Dom. wt., 20 lis 2018 o 07:41 Steve Bistline napisał

Re: Group by with null keys

Hi Flavio, Whether groupBy with null values works or not depends on the type of the key, or more specifically on the TypeComparator and TypeSerializer that are used to serialize, compare, and hash the key type. The processing engine supports null values If the comparator and serializer can handle

Re: Group by with null keys

Sure! The problem is that Dataset API does an implicit conversion to Tuples during chaining and I didn't found any documentation about this (actually I was pleasantly surprised by the fact that the Table API were supporting aggregates on null values..). Here it is: https://issues.apache.org/jira/

Re: Group by with null keys

I assigned the issue to me. Because I wanted to that for a very long time. I already did some prerequisite work for the documentation in `org.apache.flink.api.common.typeinfo.Types`. Thanks, Timo Am 20.11.18 um 11:44 schrieb Flavio Pompermaier: Sure! The problem is that Dataset API does an imp

Tentative release date for 1.6.3

Hi, Is the tentative release date for 1.6.3 decided? Thanks, Shailesh

how to override s3 key config in flink job

Hi, Is there any way to provide s3.access-key and s3.secret-key in flink application, instead of setting them in flink-conf.yaml? In our use case, my team provide a flink standalone cluster to users. However, we don't want to let each user use the same s3 bucket as filesystem to store checkpoints

About the issue caused by flink's dependency jar package submission method

hi all: I know that when submitting flink jobs, flink's official recommendation is to put all the dependencies and business logic into a fat jar, but now our requirement is to separate the business logic and rely on dynamic commits, so I found one. One way, use the -yt and -C parameters to submit

ClassNotFoundException: org.apache.kafka.common.metrics.stats.Rate$1

looking at the log file of the taskexecutor I see this exception *at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1

Re: Tentative release date for 1.6.3

Hi, there is no release date for 1.6.3, yet. Best, Stefan > On 20. Nov 2018, at 12:18, Shailesh Jain wrote: > > Hi, > > Is the tentative release date for 1.6.3 decided? > > Thanks, > Shailesh

[ANNOUNCE] Weekly community update #47

Dear community, this is the weekly community update thread #47. Please post any news and updates you want to share with the community to this thread. # Updates on sharing state between subtasks Jamie opened a first PR to add a first version of sharing state between tasks. It works by using the J

Could not extract key Exception only on runtime not in dev environment

I am running flink locally on my machine , I am getting the exception below when reading from kafka topic. when running from the ide (intellij) it is running perfectly. however when I deploy my jar to flink runtime (locally) using */bin/flink run ~MyApp-1.0-SNAPSHOT.jar* my class looks like this c

Re: Could not extract key Exception only on runtime not in dev environment

What r.id Value ? Are you sure that is not null ? Miki. On Tue, 20 Nov 2018, 17:26 Avi Levi I am running flink locally on my machine , I am getting the exception > below when reading from kafka topic. when running from the ide (intellij) > it is running perfectly. however when I deploy my jar

Re: Could not extract key Exception only on runtime not in dev environment

yes you can also see it in the message (and also it should have crushed also on the ide) further more to be sure I added a filter that looks like *env* *.addSource(kafka_source)* *.filter(_.id != null)* *.keyBy{ r =>* *val h = fastHash(r.id ) % partitionFactor* *

Flink Table Duplicate Evaluation

Hi everybody, I have a question concerning the Flink Table API, more precisely the way the results of tables statements are evaluated. In the following code example, the statement defining the table t1 is evaluated twice, an effect that leads to some issues of performance and logic in the

Re: Flink Table Duplicate Evaluation

Hi Niklas, The workaround that you described should work fine. However, you don't need a custom sink. Converting the Table into a DataSet and registering the DataSet again as a Table is currently the way to solve this issue. Best, Fabian Am Di., 20. Nov. 2018 um 17:13 Uhr schrieb Niklas Teichman

Re: ClassNotFoundException: org.apache.kafka.common.metrics.stats.Rate$1

Hi, It should be as easy as making sure that there is a jar with the missing class in the class path of your user-code class loader. Best, Stefan > On 20. Nov 2018, at 14:32, Avi Levi wrote: > > looking at the log file of the taskexecutor I see this exception > at scala.concurrent.forkjoin.Fo

[ANNOUNCE] Flink Forward San Francisco Call for Presentations closes soon

Hi Everyone, Flink Forward San Francisco will *take place on April 1st and 2nd 2019*. Flink Forward is a community conference organized by data Artisans and gathers many members of the Flink community, including users, contributors, and committers. It is the perfect event to get in touch and conne

Flink JSON (string) to Pojo (and vice versa) example

Hi everybody, since here at Okkam we didn't find any "native" Flink map functions that already permit to pass from JSON strings to POJOs (and vice versa), we decided to share with the Flink community a simple implementation for these 2 tasks: - JSON (string) to POJO [1] - POJO to JSON (string) [2

Reset kafka offets to latest on restart

Is it possible to have checkpointing but reset the kafka offsets to latest on restart on failure ?

Exception occurred while processing valve output watermark & NullPointerException

Any guidance would be most appreciated. Thx Steve === java.lang.RuntimeException: Exception occurred while processing valve output watermark: at org.apache.flink.streaming.runtime.io.StreamInputProcessor$ForwardingValveOutputHandler.handleWatermar

Re: Per job cluster doesn't shut down after the job is canceled

Hi Paul, Sorry for the late reply. I had a look at the attached log. I think FLINK-10482 affects the shut down of the "per-job cluster" after all. Here is the respective stacktrace: 2018-11-06 10:45:17,405 ERROR org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor - Caught exception whil

Re: About the issue caused by flink's dependency jar package submission method

My only guess would be that you have two versions of the Apache Commons jar on your class path, or the version you compiled against doesn’t match what you’re running against, and that’s why you get: Caused by: java.lang.ClassCastException: cannot assign instance of org.apache.commons.collection

Re: Tentative release date for 1.6.3

Hi Shailesh, Flink 1.7 is about to be released, and many of the problems encountered since Flink 1.6.2 have been fixed in version 1.7. So, I suggest you look forward and upgrade to Flink 1.7. Thanks, vino. Stefan Richter 于2018年11月20日周二 下午10:15写道： > Hi, > > there is no release date for 1.6.3, y

Store Predicate or any lambda in MapState

Hi, I want to store a custom POJO in the MapState. One of the fields in the object is a java.util.function.Predicate type. Flink gives ClassNotFoundException exception on the lambda. How do I store this object in the mapState? Marking the predicate field as transient is an option. But in my use-ca

Re: Exception occurred while processing valve output watermark & NullPointerException

Hi Steve, It seems the NPE caused by the property of the IoTEvent's instance. Can you make sure the property is not null? Thanks, vino. Steve Bistline 于2018年11月21日周三 上午2:09写道： > Any guidance would be most appreciated. > > Thx > > Steve > === > > java.lan

Re: About the issue caused by flink's dependency jar package submission method

hi I have checked all the dependences, and don't find the jar with different version, so ,I double the way to submit jar has some issue? my commend is like this: /data/flink1.6/bin/flink run -m yarn-cluster -ytm 8032 -yn 1 -ys 1 -yqu -yt /data/flink1.6//lib -c com.xxx.xxx.xxx.Launch -C

IntervalJoin is stucked in rocksdb'seek for too long time in flink-1.6.2

I am using IntervalJoin function to join two streams within 10 minutes. As below: labelStream.intervalJoin(adLogStream) .between(Time.milliseconds(0), Time.milliseconds(60)) .process(new processFunction()) .sink(kafkaProducer) labelStream and adLogStream are pr

Re: TaskManager & task slots

Hi Fabian, Is below description still remain the same in Flink 1.6? Slots do not guard CPU time, IO, or JVM memory. At the moment they only isolate managed memory which is only used for batch processing. For streaming applications their only purpose is to limit the number of parallel threads that

Re: About the issue caused by flink's dependency jar package submission method

As far as I know, -yt works for both job manager and task manager, -C works for flink cli. Did you consider putting all your jars in /flink/lib? -- Sent from: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/