Passing a custom SourceContext to a SourceFunction

Hi - I have a custom SourceFunction .. class MySourceFunction[T](data: Seq[T]) extends SourceFunction[T] { def run(ctx: SourceContext[T]): Unit = { data.foreach(d ⇒ ctx.collect(d)) } } When this function is run during job execution, the SourceContext that gets passed serializes the data.

Checkpoints periodically fail with hdfs as the state backend - Could not flush and close the file system output stream

Hello, Every once in a while our checkpoints fail with the following exception: /AsynchronousException{java.lang.Exception: Could not materialize checkpoint 65912 for operator AGGREGATION-FILTER (2/2).} at org.apache.flink.streaming.runtime.tasks.StreamTask$AsyncCheckpointExceptionHandler

Re: Flink and Prometheus setup in K8s

Hi all, To answer my own questions I worked on the following solution: 1) Custom Docker image which pulls the Flink image and moves Prometheus jar to the correct folder [1, 2]. 2) I wrote manifests for Kubernetes with service discovery configuration for Kubernetes [3]. Besides the 'official' Flin

FlinkSQL fails when rowtime meets dirty data

I use FlinkSQL to process Kafka data in the following format： | id | server_time | | 1 | 2019-05-15 10:00:00 | | 2 | 2019-05-15 10:00:00 | ... and I define rowtime from the server_time field: new Schema() .field("rowtime", Types.SQL_TIMESTAMP) .rowtime(new Rowtime().timestam

Re: Migrating Existing TTL State to 1.8

Hi Ning, If you have not activated non-incremental checkpointing then taking a savepoint is the only way to trigger the full snapshot. In any case, it will take time. The incremental cleanup strategy is applicable only for heap state backend and does nothing for RocksDB backend. At the moment, yo

Getting java.lang.BootstrapMethodError: java.lang.NoClassDefFoundError when stopping/canceling job.

Hi, I'm using vertx.io as an async JDBC client for a RichAsyncFunction it works fine but when I stop the job I get... java.lang.BootstrapMethodError: java.lang.NoClassDefFoundError: io/vertx/core/impl/VertxImpl$SharedWorkerPool Is there a way to avoid/fix this?

Re: Table program cannot be compiled

Hi, I am looping in Timo and Dawid to look at the problem. On Tue, May 14, 2019 at 9:12 PM shkob1 wrote: > BTW looking at past posts on this issue[1] it should have been fixed? i'm > using version 1.7.2 > Also the recommendation was to use a custom function, though that's exactly > what im doing

Re: flink 1.4.2. java.lang.IllegalStateException: Could not initialize operator state backend

Hi, I am not sure that FLINK-8836 is related to the failure in the stack trace. You say you are using Flink in production, does it mean it always worked and has started to fail recently? >From the stack trace, it looks like the arity of some Tup

Re: Getting java.lang.BootstrapMethodError: java.lang.NoClassDefFoundError when stopping/canceling job.

Hi John, could you share the full stack trace or better logs? It looks like something is trying to be executed in vertx.io code after the local task has been stopped and the class loader for the user code has been unloaded. Maybe from some daemon thread pool. Best, Andrey On Wed, May 15, 2019 a

NPE in Flink 1.8.0

Hi We would highly appreciate any information on following stacktrace. Our flink job is simply writing data to Cassandra using cassandra sink and we are getting following exception - Caused by: java.lang.NullPointerException at org.apache.flink.streaming.runtime.tasks.StreamTask.creat

Re: NPE in Flink 1.8.0

Hi We had the same issue. Make sure everything is using Flink 1.8 and not half on 1.7.2 and the other half on 1.8. Make sure to prune docker images and so on, even maven repo. Farouk Le mer. 15 mai 2019 à 18:08, Akshay Shinde a écrit : > Hi > > > We would highly appreciate any information o

User Interface not showing the actual count received and produced

Hi Flink team, I am developing a flow which uses · FlinkKafkaConsumer010 to consume message from Kafka and · FlinkKinesisProducer to produce the results to Kinesis. In the user interface, I always see Bytes and Record received from Kafka is zero even though it is receiving event

Flink with HBase

Hi, Does flink support exactly once processing using HBase as sink? I am not able to find any documentation supporting the same. Thanks Nikhil

Re: Getting java.lang.BootstrapMethodError: java.lang.NoClassDefFoundError when stopping/canceling job.

So these are the two exceptions I see in the logs... Exception in thread "vert.x-worker-thread-0" Exception in thread "vert.x-internal-blocking-0" java.lang.NoClassDefFoundError: io/netty/util/concurrent/FastThreadLocal at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnabl

Re: Migrating Existing TTL State to 1.8

Hi Andrey, Thank you for the reply. We are using incremental checkpointing. Good to know that the incremental cleanup only applies to the heap state backend. Looks like taking some downtime to take a full savepoint and restore everything is inevitable. Thanks, -- Ning On Wed, 15 May 2019 10:5

Preventing Late events from getting triggered for custom trigger

Im running SQL along with a a custom trigger - so think about this scenario: Select id, count(*) as count_events, atLeastOneIsTrue(booleanField) as shouldTrigger FROM my_table GROUP BY id transforming it to a retracted stream and then filtering by the shouldTrigger field, that works as expected.

Applying multiple calculation on data aggregated on window

Hi, Im my environment I need to collect stream of messages into windows based on some fields as key and then I need to do multiple calculations that will apply on specaified messages. for example if i had the following messages on the window: {ts: 1, key: a, value: 10} {ts: 1, key: b, value: 0} {t

Error While Initializing S3A FileSystem

hey Friends, Thanks for all the work you have been doing on flink, I have been trying to use BucketingSink (backed by S3AFileSystem) to write data to s3 and I'm running into some issues (which I suspect could be dependency/packaging related) that'd try to describe here. The data pipeline is quite

Re: Error While Initializing S3A FileSystem

Hi Manish, Are you sure this is an exception that’s actually killing the job? Asking because https://issues.apache.org/jira/browse/BEANUTILS-477 talks about Commons Beanutils logging this exception, but it’s a warning vs. something being thr

Re: Error While Initializing S3A FileSystem

Hi Ken, Thanks for the quick response, you are actually right, the job seems to be running even after that error appears. It was crashing earlier (due to fs.s3a.multipart.size being too high) and I confused it with this error since that was the first one popping out and OOM wasn't apparent immedia

Re: Error While Initializing S3A FileSystem

Hi Manish, It’s best to start a new thread if you have a new question - see https://home.apache.org/~hossman/#threadhijack for reasons why… Regards, — Ken > On May 15, 2019, at 4:46 PM, Manish Bellani wrote: > > Hi Ken, > > Thanks for the q

Re: Error While Initializing S3A FileSystem

Thanks, Ken. That makes sense! I'll start a new thread. On Wed, May 15, 2019 at 7:12 PM Ken Krugler wrote: > Hi Manish, > > It’s best to start a new thread if you have a new question - see > https://home.apache.org/~hossman/#threadhijack for reasons why… > > Regards, > > — Ken > > > On May 15, 2

Re: flink 1.4.2. java.lang.IllegalStateException: Could not initialize operator state backend

Thank You Andrey. Arity of the job has not changed. Here issue is that job will run for sometime (with checkpoint enabled) and then after some time will get into above exception. The job keeps restarting afterwards. One thing that I want point out here is that we have a custom *serialization sche

Re: User Interface not showing the actual count received and produced

Flink currently does not measure incoming records for sources and outgoing records for sink, see https://issues.apache.org/jira/browse/FLINK-7286. On 15/05/2019 19:43, PoolakkalMukkath, Shakir wrote: Hi Flink team, I am developing a flow which uses ·FlinkKafkaConsumer010 to consume message

Re: Flink with HBase

Unless I'm blind Flink does not provide an HBase sink, so it depends on the implementation you're using. On 15/05/2019 20:17, Nikhil Goyal wrote: Hi, Does flink support exactly once processing using HBase as sink? I am not able to find any documentation supporting the same. Thanks Nikhil