Re: Streams: Why is ForwardingCacheFlushListener internal? Can I replicate its functionality in my code?

2018-01-31 Thread Matthias J. Sax
If you build a store and enable caching, you get the KTable behavior out of the box. No need to write any custom code in the processor itself. StoreBuilder builder = Stores.keyValueStoreBuilder(...).withCachingEnabled(); topology.addStateStore(builder, ...) -Matthias On 1/31/18 6:19 PM, Dmit

RE: NoClassDefFoundError: Could not initialize class com.yammer.metrics.Metrics

2018-01-31 Thread TSANG, Brilly
Just make sure metrics-core-x.x.x.jar is in your class path. That jar should be in your /libs. I am using kafka_2.11-1.0.0 so I don't have the exact version number of metrics-core for you. Regards, Brilly -Original Message- From: ? ?? [mailto:wangchunc...@outlook.com] Sent: Thursday,

NoClassDefFoundError: Could not initialize class com.yammer.metrics.Metrics

2018-01-31 Thread ? ??
Hi, I am using KafkaSpout to ingest data into Storm. The versions are: Storm-1.1.0 Storm-kafka 1.1.0 kafka_2.10-0.8.2.2 The program worked well at the beginning, but at some point the KafkaSpout threw an exception and the program seemed stuck there afterwards. The program can proceed after res

Re: monitor consumer offset lag script/code

2018-01-31 Thread Sunil Parmar
Thanks Kaufman .. Shubhash, We're using burrow but it has two issues. 1) The lag reported from command line tools and burrow is different often by big margin. 2) When consumer goes down it stops reporting the difference and shows constant value. Sunil Parmar On Tue, Jan 30, 2018 at 6:07 AM, Kauf

Streams: Why is ForwardingCacheFlushListener internal? Can I replicate its functionality in my code?

2018-01-31 Thread Dmitry Minkovsky
I am writing a processor and I want its stores to behave like KTables: For consistency, I don't want to forward values until the stores have been flushed. I am looking at `ForwardingCacheFlushListener` and see that it is using `InternalProcessorContext` to change the current node, perform a forwar

Re: Recommended max number of topics (and data separation)

2018-01-31 Thread David Espinosa
I used: -Djute.maxbuffer=50111000 and the gain I had is that I could increment number of topics from 70k to 100k :P 2018-01-30 23:25 GMT+01:00 Andrey Falko : > On Tue, Jan 30, 2018 at 1:38 PM, David Espinosa wrote: > > Hi Andrey, > > My topics are replicated with a replicated factor equals to th

Group consumer cannot consume messages if kafka service on specific node in test cluster is down

2018-01-31 Thread Dmitry Minkovsky
This is not my question, but I saw it on Stack Overflow yesterday and have been wondering about it: https://stackoverflow.com/questions/48523972/group- consumer-cannot-consume-messages-if-kafka-service-on-specific-node-in-test. Anyone else seen behavior like this?