Elasticsink sometimes gives NoClassDefFoundError

Hi, I see the following error sometimes on my flink job, even though the class is present in my uber jar. java.lang.NoClassDefFoundError: org/apache/flink/streaming/connectors/elasticsearch5/shaded/org/jboss/netty/channel/socket/nio/NioClientSocketPipelineSink$1 at org.apache.flink.streaming.conne

Re: Elasticsink sometimes gives NoClassDefFoundError

the job is restarting after some other failure. But, the occurrence of this error and job restarts is correlated. Jayant Ameta On Wed, Jan 8, 2020 at 6:47 PM Arvid Heise wrote: > Hi Jayant, > > if you only see it sometimes that indicates that you have it in two > different versions of the

Re: Elasticsink sometimes gives NoClassDefFoundError

Also, the ES version I'm using is 5.6.7 Jayant On Thu, Jan 9, 2020 at 10:39 AM Jayant Ameta wrote: > Hi, > The elastic connector is packaged in the uber jar that is submitted. There > is only 1 version of the connector: > flink-connector-elasticsearch5_2.11:1.7.1 > I'

Implementation of ElasticsearchSinkFunction, how to handle class level variables

Hi, I'm trying to integrate ElasticsearchSink in my pipeline. The example shows using Anonymous class which implements ElasticsearchSinkFunction. This is passed as a constructor argument to another anonym

Using CheckpointedFunction interface with Keyed state

Hi, I'm trying to understand the state functionality. Is there any case where using CheckpointedFunction with Keyed state makes any sense? Or the CheckpointedFunction is only to be used with operator state? Also, trying to understand the BufferinSink example here: https://ci.apache.org/projects/fl

State sharing across trigger and evictor

te used by the countTrigger from my Evictor class, so that the count always reflects the number of elements in the windows? Jayant Ameta

LoggingFactory: Access to static method across operators

I am using a custom LoggingFactory. Is there a way to provide access to this custom LoggingFactory to all the operators other than adding it to all constructors? This is somewhat related to: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Adding-Context-To-Logs-td7351.html Ja

Skip event in case of key extraction exception

Hi, My key extraction logic requires a conversion from string to UUID. I want to skip the event in case of exception. Is it doable?

WriteTimeoutException in Cassandra sink kill the job

Hi, During high volumes, cassandra sink fails with the following error: com.datastax.driver.core.exceptions.WriteTimeoutException: Cassandra timeout during write query at consistency SERIAL (2 replica were required but only 1 acknowledged the write) Is there a way to configure the sink to ignore/h

Re: This server is not the leader for that topic-partition

I am getting the same error. Is there a way to retry/ignore instead of killing the job? Jayant Ameta On Tue, May 22, 2018 at 7:58 PM gerardg wrote: > I've seen the same error while upgrading Kafka. We are using > FlinkKafkaProducer011 and Kafka version 1.0.0. While upgrading to Ka

Re: This server is not the leader for that topic-partition

Flink: 1.4.2 flink-connector-kafka-0.11_2.11 (1.4.2) Kafka: 0.10.1.0 Jayant Ameta On Tue, Sep 4, 2018 at 10:16 AM vino yang wrote: > Hi Jayant, > > Can you provide more specific information? For example, the version of > your Flink, the version of kafka on which Flink-Kafka-Conne

Re: WriteTimeoutException in Cassandra sink kill the job

Hi Till, I've opened a JIRA issue: https://issues.apache.org/jira/browse/FLINK-10310. Can we discuss it? Jayant Ameta On Thu, Aug 30, 2018 at 4:35 PM Till Rohrmann wrote: > Hi Jayant, > > afaik it is currently not possible to control how failures are handled in > the Cas

Re: WriteTimeoutException in Cassandra sink kill the job

https://issues.apache.org/jira/browse/FLINK-10310?focusedCommentId=16618447&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16618447 On Tue, Sep 11, 2018 at 10:15 AM Jayant Ameta wrote: > Hi Till, > I've opened a JIRA issue: > https://issues.apac

Queryable state when key is UUID - getting Kyro Exception

-d81a-11e8-bb91-7fd1412de84d"), TypeInformation.of(new TypeHint() {}), descriptor); MapState mapState = resultFuture.get(10, TimeUnit.SECONDS); Any better way to query it? Jayant Ameta

Re: Queryable state when key is UUID - getting Kyro Exception

Hi, I've not configured any serializer in the descriptor. (Neither in flink job, nor in state query client). Which serializer should I use? Jayant Ameta On Thu, Oct 25, 2018 at 2:13 PM bupt_ljy wrote: > Hi, > >It seems that your codes are right. Are you sure that you’re us

Re: Queryable state when key is UUID - getting Kyro Exception

utor$Worker.run(ThreadPoolExecutor.java:617)* * at java.lang.Thread.run(Thread.java:748)* I am not using any custom serialize as mentioned by Jiayi. Jayant Ameta On Thu, Oct 25, 2018 at 3:01 PM bupt_ljy wrote: > Hi Jayant, > > There should be a Serializer parameter in the constructor of the >

Re: Queryable state when key is UUID - getting Kyro Exception

Also, I haven't provided any custom serializer in my flink job. Shouldn't the same configuration work for queryable state client? Jayant Ameta On Thu, Oct 25, 2018 at 4:15 PM Jayant Ameta wrote: > Hi Gordon, > Following is the stack trace that I'm getting: > >

Re: Queryable state when key is UUID - getting Kyro Exception

MapStateDescriptor descriptor = new MapStateDescriptor<>("rulePatterns", UUID.class, String.class); Jayant Ameta On Fri, Oct 26, 2018 at 8:19 AM bupt_ljy wrote: > Hi, > >Can you show us the descriptor in the codes below? > > client

Re: Queryable state when key is UUID - getting Kyro Exception

Hi Jiayi, Any further help on this? Jayant Ameta On Fri, Oct 26, 2018 at 9:22 AM Jayant Ameta wrote: > MapStateDescriptor descriptor = new > MapStateDescriptor<>("rulePatterns", UUID.class, > String.class); > > Jayant Ameta > > > On Fri, Oct 26,

Re: Queryable state when key is UUID - getting Kyro Exception

Yeah, it IS using Kryo serializer. Jayant Ameta On Wed, Nov 7, 2018 at 9:57 PM Till Rohrmann wrote: > Hi Jayant, could you check that the UUID key on the TM is actually > serialized using a Kryo serializer? You can do this by setting a breakpoint > in the construct

Linkage error when using DropwizardMeterWrapper

n the signature Jayant Ameta

Re: Linkage error when using DropwizardMeterWrapper

Nevermind. Relocating the dropwizard packages using maven shade plugin fixed it.

Re: Queryable state when key is UUID - getting Kyro Exception

o.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354) at org.apache.flink.shaded.netty4.io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) at java.lang.Thread.run(Thread.java:748) Jayant Ameta On Fri, Nov 9, 2018 at 5:14 PM Till Rohrmann wrote: > Could you send us a

Re: Queryable state when key is UUID - getting Kyro Exception

ion.run(CompletableFuture.java:442) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ... 1 more Jayant Ameta On Tue, Nov 13, 2018 at 11:35 AM bupt_ljy wrote:

Re: Queryable state when key is UUID - getting Kyro Exception

Getting the same error even when I added flink-avro dependency to the client. Jayant Ameta On Tue, Nov 13, 2018 at 2:28 PM bupt_ljy wrote: > Hi Jayant, > >I don’t know why flink uses the Avro serializer, which is usually used > in POJO class, but from the error messages, I t

Re: Queryable state when key is UUID - getting Kyro Exception

I tried to create a sample project, but couldn't reproduce the error! It was working fine. Turns out I was using wrong Tuple2 package in my client code :( After fixing it, the code worked fine. Thanks Till and Jiayi for your help! Jayant Ameta On Tue, Nov 13, 2018 at 4:01 PM Till Roh

Store Predicate or any lambda in MapState

-case, the predicate field is set using another library, and I don't want to call it every time I want. Jayant Ameta

Re: Store Predicate or any lambda in MapState

org.apache.flink.streaming.runtime.tasks.StreamTask.initializeState(StreamTask.java:679) at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:253) ... 2 frames truncated Jayant Ameta On Wed, Nov 21, 2018 at 3:17 PM Dominik Wosiński wrote: > Hey Jayant, > >

Re: Store Predicate or any lambda in MapState

Any further help on this? Jayant Ameta On Wed, Nov 21, 2018 at 4:37 PM Jayant Ameta wrote: > Here are the error logs. > > First error log was encountered when getting the values from the MapState. > > java.lang.ClassNotFoundException: > com.test.MatcherFactory$$Lam

Flink operator UUID and serialVersionUID

Hi all, I've a few questions regarding serial version: 1. The production ready checklist mentions using uuids for operators. How is it different from setting a serialVersionUID on an ope

Re: Flink operator UUID and serialVersionUID

Descriptor, so no need to worry about it. > > > Best, > > Jiayi Liao > > Original Message > *Sender:* Jayant Ameta > *Recipient:* user > *Date:* Wednesday, Nov 28, 2018 15:09 > *Subject:* Flink operator UUID and serialVersionUID > > Hi all, I've a few questions

Re: Flink operator UUID and serialVersionUID

If I upgrade my flink job, and add a field in "MyObject" class. Will the restore fail? If so, how to handle such scenarios? Should I convert the "MyObject" instance in json and store the string? Jayant Ameta On Wed, Nov 28, 2018 at 1:26 PM bupt_ljy wrote: > Hi Jayant, &

Re: Flink operator UUID and serialVersionUID

Thanks, I'll look into the bravo project. Will it just impact the MapState or all the operators? If I have a map operator which converts DataStream to DataStream>. Will this fail to recover as well if a field is added to MyObject? Jayant Ameta On Wed, Nov 28, 2018 at 3:08 PM bupt_lj

Re: Flink operator UUID and serialVersionUID

the program is restarted. As for the stateless operators, > it’s not affected because Flink won’t do a snapshot on them. > > Best, > Jiayi Liao > Original Message > *Sender:* Jayant Ameta > *Recipient:* bupt_ljy > *Cc:* user > *Date:* Wednesday, Nov 28, 2018 17:5

Flink with Ceph as the persistent storage

Hi, Flink documents suggests that Ceph can be used as a persistent storage for states. https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/stream/checkpointing.html Considering that Ceph is a transactional database, wouldn't it have adverse effect on Flink's performance?

Re: Flink with Ceph as the persistent storage

If the checkpointing to Ceph happens asynchronously, does it still have any impact on the stream processing? Jayant Ameta On Tue, Dec 5, 2017 at 4:34 PM, Gyula Fóra wrote: > Hi, > > To my understanding Ceph as in http://ceph.com/ceph-storage/ is a block > based object storage sys

Performance of docker-flink

Hi, I wanted to explore docker-flink (using Ceph for state backend). before opting for a standalone cluster. Has there been any comparative studies on the performance of docker-flink? Would the states be consistent and performant if the docker containers go down and respawn frequently?

Re: Performance of docker-flink

sting users' experience with docker-flink. Jayant Ameta On Thu, Dec 7, 2017 at 4:37 PM, Gary Yao wrote: > Hi Jayant, > > Running Flink in a Docker container should not have an impact on the > performance > in itself. Docker does not employ virtualization. To put it simply

Fwd: Replace classes at runtime

Jayant Ameta -- Forwarded message -- From: Jayant Ameta Date: Thu, Dec 14, 2017 at 4:46 PM Subject: Replace classes at runtime To: user Hi, I need to update a few classes of my flink job at runtime. What would be the best way to achieve this?

How to apply patterns from a source onto another datastream?

I've a datastream of events, and another datastream of patterns. The patterns are provided by users at runtime, and they need to come via a Kafka topic. I need to apply each of the pattern on the event stream using Flink-CEP. Is there a way to get a PatternStream from the DataStream when I don't kn

Re: How to apply patterns from a source onto another datastream?

Would it be possible to get the same result using windows? Jayant Ameta On Tue, Dec 19, 2017 at 3:23 PM, Dawid Wysakowicz < wysakowicz.da...@gmail.com> wrote: > It is not possible at this moment. FlinkCEP can handle only one Pattern > applied statically. There is a JIRA ticket for t

Re: How to apply patterns from a source onto another datastream?

count the number of matching events within a time window. Do you know if there is any similar example in the docs? Jayant Ameta On Fri, Dec 22, 2017 at 1:24 PM, Dawid Wysakowicz < wysakowicz.da...@gmail.com> wrote: > Hi Jayant, > > Could you elaborate a bit more what you mean?

Trigger not firing when using BoundedOutOfOrdernessTimestampExtractor

Hi, When using a BoundedOutOfOrdernessTimestampExtractor, the trigger is not firing. However, the trigger fires when using custom timestamp extractor with similar watermark. Sample code below: 1.Assigner as anonymous class which works fine AssignerWithPeriodicWatermarks> assigner = new AssignerWi

Re: Trigger not firing when using BoundedOutOfOrdernessTimestampExtractor

Thanks Gary, I was only trying with a fixed set of events, so the Watermark was not advancing, like you said. Jayant Ameta On Thu, Jan 11, 2018 at 3:36 PM, Gary Yao wrote: > Hi Jayant, > > The difference is that the Watermarks from > BoundedOutOfOrdernessTimestampExtractor are

Re: Trigger not firing when using BoundedOutOfOrdernessTimestampExtractor

Hi Fabian, I want to extract timestamps from my event. However, the events stream can be sparse at times (e.g. 2 days without any events). What's the best strategy to create watermarks if I want real-time processing of the events which enter the stream? Jayant Ameta On Thu, Jan 11, 2018 at

Re: Fwd: Replace classes at runtime

Hi Gordon, Is there a way to update the job application without any downtime? Jayant Ameta On Tue, Dec 19, 2017 at 12:09 AM, Tzu-Li (Gordon) Tai wrote: > Hi Jayant, > > Updating your job application / operator code at runtime is currently not > available in Flink. > It is how

Flip-6 + Dynamic scaling

Hi, There hasn't been any activity on https://issues.apache.org/jira/browse/FLINK-4319 since September 21. Any ETA on this? I required dynamic upscaling and downscaling based on the volume of data. Jayant Ameta

Rocksdb in production

Hi, I wanted to know how's the online support and resources for RocksDB? I want to use RocksDB as the state backend, but I'm not sure how active the community is. Can anyone vouch for it?

Re: Rocksdb in production

Oh! Somehow I missed while reading the documentation that RocksDB is embedded in Flink. Also, irrespective of state backend being filesystem or rocksdb, I'll have to setup a shared filesystem (HDFS, S3, etc). Is my understanding correct? Jayant Ameta On Mon, Mar 5, 2018 at 9:51 PM, F

Re: Rocksdb in production

Thanks Fabian. Will there be any performance issues if I use NFS as the shared filesystem (as compared to HDFS or S3)? Jayant Ameta On Mon, Mar 5, 2018 at 10:31 PM, Fabian Hueske wrote: > Yes, that is correct. > > 2018-03-05 8:57 GMT-08:00 Jayant Ameta : > >> Oh! Somehow I mi

state backend configuration

Hi, Can I enable incremental checkpoint for rocksdb via flink-conf.yaml file? Also, I can set state.backend via the conf file as well as in my code using env.setStateBackend(). Which of these take precedence? Jayant Ameta

Is Hadoop 3.0 integration planned?

Jayant Ameta