Re: KStreams Rewind Offset

2016-06-02 Thread Michael D. Coon
ntermediate results correctly (also with regard to committing) What is your exact use case? What kind of feature are you looking for? We are always interested to get feedback/idea from users. -Matthias On 06/01/2016 08:21 PM, Michael D. Coon wrote: > All, >  I think it's great that

KStreams Rewind Offset

2016-06-01 Thread Michael D. Coon
All,   I think it's great that the ProcessorContext offers the partition and offset of the current record being processed; however, it offers no way for me to actually use the information. I would like to be able to rewind to a particular offset on a partition if I needed to. The consumer is als

Re: IncompatibleClassChangeError

2016-04-15 Thread Michael D. Coon
than the broker. Do you have a stacktrace of when the exception is thrown? Ismael On Fri, Apr 15, 2016 at 2:39 PM, Michael D. Coon wrote: > We are seeing odd behavior that we need to understand. We are getting > IncompatibleClassChangeErrors and I know that's related to a Scala versi

IncompatibleClassChangeError

2016-04-15 Thread Michael D. Coon
We are seeing odd behavior that we need to understand. We are getting IncompatibleClassChangeErrors and I know that's related to a Scala version mismatch. What's not clear, however, is where or why the mismatch is occurring. We know up front that there were occasions where we ran apps that had S

Re: KStream Close Processor

2016-04-10 Thread Michael D. Coon
we are shooting to have it released end of this month. Guozhang On Sat, Apr 9, 2016 at 5:59 AM, Michael D. Coon wrote: > Guozhang, >    In my processor, I'm buffering up contents of the final messages in > order to make them larger. This is to optimize throughput and avoid tiny

Re: KStream Close Processor

2016-04-09 Thread Michael D. Coon
behavior than the above described? Guozhang On Fri, Apr 8, 2016 at 12:23 PM, Michael D. Coon wrote: > All, >    I'm seeing my processor's "close" method being called AFTER my > downstream producer has been closed. I had assumed that on close I would be > able to

KStream Close Processor

2016-04-08 Thread Michael D. Coon
All,    I'm seeing my processor's "close" method being called AFTER my downstream producer has been closed. I had assumed that on close I would be able to flush whatever I had been buffering up to send to kafka topic. In other words, we've seen significant performance differences in building flo

Re: KStreams Group Rebalance Commit Error

2016-04-07 Thread Michael D. Coon
he code to know what the corresponds to. On Thursday, April 7, 2016 7:35 AM, Michael D. Coon wrote: Guozhang,    Thanks for the advice; however, "max.poll.records" doesn't seem to be supported since it's not affecting how many records are coming back from the consume

Re: KStreams Group Rebalance Commit Error

2016-04-07 Thread Michael D. Coon
Guozhang On Wed, Apr 6, 2016 at 6:26 AM, Michael D. Coon wrote: All,    I'm getting CommitFailedExceptions on a small prototype I built using kafkaStreams. I'm not using the DSL, but the TopologyBuilder with several processors chained together with a sink in between

KStreams Group Rebalance Commit Error

2016-04-06 Thread Michael D. Coon
All,    I'm getting CommitFailedExceptions on a small prototype I built using kafkaStreams. I'm not using the DSL, but the TopologyBuilder with several processors chained together with a sink in between a few of them. When I try committing through the ProcessorContext, I see exceptions being thr

KStreams Partition Assignment

2016-03-20 Thread Michael D. Coon
I'm evaluating whether the KafkaStreams API will be something we can use on my current project. Namely, we want to be able to distribute the consumers on a Mesos/YARN cluster. It's not entirely clear to me in the code what is deciding which partitions get assigned at runtime and whether this is