Re: RocksDB flushing issue on 0.10.2 streams

2017-07-04 Thread Greg Fodor
setting `state.cleanup.delay.ms` to a higher value (default is 10 > minutes), > to try and avoid it happening during a rebalance (I know this isn't a fix, > but will make it less likely to happen). > > Thanks, > Damian > > On Tue, 4 Jul 2017 at 12:43 Greg Fodor wrote:

Re: RocksDB flushing issue on 0.10.2 streams

2017-07-04 Thread Greg Fodor
Also, I am on 0.10.2.1, so poll interval was already set to MAX_VALUE. On Tue, Jul 4, 2017 at 11:28 AM, Greg Fodor wrote: > I've nuked the nodes this happened on, but the job had been running for > about 5-10 minutes across 5 nodes before this happened. Does the log show a >

Re: RocksDB flushing issue on 0.10.2 streams

2017-07-04 Thread Greg Fodor
Upon another run, I see the same error occur during a rebalance, so either my log was showing a rebalance or there is a shared underlying issue with state stores. On Tue, Jul 4, 2017 at 11:35 AM, Greg Fodor wrote: > Also, I am on 0.10.2.1, so poll interval was already set to MAX_VALUE. >

Re: RocksDB flushing issue on 0.10.2 streams

2017-07-05 Thread Greg Fodor
, this does not seem to be happening during a rebalancing event, but after the cluster is stable. On Tue, Jul 4, 2017 at 12:29 PM, Greg Fodor wrote: > Upon another run, I see the same error occur during a rebalance, so either > my log was showing a rebalance or there is a shared underlying

Re: RocksDB flushing issue on 0.10.2 streams

2017-07-05 Thread Greg Fodor
re were some minor changes around this in > > 0.11.0 to make it less likely to happen, but we've only ever seen the > > locking fail in the event of a rebalance. When everything is running > state > > dirs shouldn't be deleted if they are being used as the lock

Re: RocksDB flushing issue on 0.10.2 streams

2017-07-05 Thread Greg Fodor
, Greg Fodor wrote: > The issue I am hitting is not the directory locking issues we've seen in > the past. The issue seems to be, as you mentioned, that the state dir is > getting deleted by the store cleanup process, but there are still tasks > running that are trying to flush th

Re: RocksDB flushing issue on 0.10.2 streams

2017-07-06 Thread Greg Fodor
t;> Thanks Greg. I'll look into it more tomorrow. Just finding it difficult > >> to reproduce in a test. > >> Thanks for providing the sequence, gives me something to try and repo. > >> Appreciated. > >> > >> Thanks, > >> Damian > >

Dealing with noisy timestamps in kafka streams + 10.1

2017-07-06 Thread Greg Fodor
Hey all, we are currently working on migrating our system to kafka 10.2 from 10.0 and one thing that we have hit that I wanted some advice on is dealing with the new log retention/rolling semantics that are based on timestamps. We send telemetry data from installed clients into kafka via kafka RES

Re: RocksDB flushing issue on 0.10.2 streams

2017-07-06 Thread Greg Fodor
threads per host. - Peak TPS seems to be approximately 5k-10k tuples/sec per node. We're reprocessing historical data in kafka. On Thu, Jul 6, 2017 at 10:45 AM, Greg Fodor wrote: > That's great news, thanks! > > On Thu, Jul 6, 2017 at 6:18 AM, Damian Guy wrote: > >> H

Re: Dealing with noisy timestamps in kafka streams + 10.1

2017-07-06 Thread Greg Fodor
https://issues.apache.org/jira/browse/KAFKA-5344 In our case, we can live with setting the retention.ms during backfills. Still would like to know if there are any better practices for dealing with mis-stamped records during backills w/ state store topics. On Thu, Jul 6, 2017 at 12:32 PM, Greg Fodor wrote: >

Possible deadlock during shutdown in setState in kafka streams 10.2

2017-07-07 Thread Greg Fodor
I'm running a 10.2 job across 5 nodes with 32 stream threads on each node and find that when gracefully shutdown all of them at once via an ansible scripts, some of the nodes end up freezing -- at a glance the attached thread dump implies a deadlock between stream threads trying to update their sta

Re: Possible deadlock during shutdown in setState in kafka streams 10.2

2017-07-07 Thread Greg Fodor
Sure thing: https://issues.apache.org/jira/browse/KAFKA-5571 On Fri, Jul 7, 2017 at 2:59 AM, Damian Guy wrote: > Hi Greg, > > Would you mind creating a JIRA for this with the thread dump ( i don't see > it attached to your message). > > Thanks, > Damian > >

Re: Dealing with noisy timestamps in kafka streams + 10.1

2017-07-07 Thread Greg Fodor
topic while allowing CreateTime timestamps to land on the other topics. On Thu, Jul 6, 2017 at 9:32 PM, Greg Fodor wrote: > I managed to answer some of my own questions :) > > For future google'ers: > > to deal with retention.ms see https://issues.apache.org/ > jira/browse/KA

Contention on CachedSchemaRegistryClient#getBy*

2017-07-10 Thread Greg Fodor
We've noticed that there's a lot of contention on get/set of schemas by ID in CachedSchemaRegistryClient if that object is shared across threads. All of the methods on CachedSchemaRegistryClient are synchronized and getBySubjectAndId is called when deserializing a record: https://github.com/conflue

Consumer group suggestion: minimum consumer count

2017-07-10 Thread Greg Fodor
With Kafka Streams, it's common to spin up and shut down clusters of consumers by performing a graceful shutdown and restart during a deploy. One thing we've been running into is that during the startup and shutdown of a kafka streams clusters you often can have multiple rebalances as the consumer

Re: Consumer group suggestion: minimum consumer count

2017-07-10 Thread Greg Fodor
Apologies, just discovered KIP-134, which would address our specific problem directly via delaying consumer join. On Mon, Jul 10, 2017 at 4:53 PM, Greg Fodor wrote: > With Kafka Streams, it's common to spin up and shut down clusters of > consumers by performing a graceful shutdown

Re: [VOTE] KIP-134: Delay initial consumer group rebalance

2017-07-10 Thread Greg Fodor
Found this thread after posting an alternative idea after we starting hitting this issue ourselves for a job that has a lot of state stores and topic partitions. My suggestion was to have consumer groups have a configurable minimum member count before consumption begins, but that has its own trade

KStream/KTable prioritization/initial offset

2016-03-24 Thread Greg Fodor
Really digging Kafka Streams so far, nice work all. I'm interested in being able to materialize one or more KTables in full before the rest of the topology begins processing messages. This seems fundamentally useful since it allows you to get your database tables replicated up off the change stream

Re: KStream/KTable prioritization/initial offset

2016-03-25 Thread Greg Fodor
is to set the record timestamps to zero. This can be done by using a custom > TimestampExtractor. Kafka Streams tries to synchronize multiple streams > using the extracted timestamps. So, records with the timestamp 0 have > greater chance to be processed earlier than others. > > On Thu,

Re: KStream/KTable prioritization/initial offset

2016-03-25 Thread Greg Fodor
d to be able to not > commit offsets against the input used to populate the state store so that > it will always get reprocessed and you need to be able to disable the > changelog. > > -Jay > > On Fri, Mar 25, 2016 at 11:55 AM, Greg Fodor wrote: > >> That's great. I

best way to duplicate a KStream?

2016-04-14 Thread Greg Fodor
Hi all, I'm working on a Kafka Streaming job and I've hit a case I didn't see straightforward support for. I have an input stream topic I'd like to perform two separate aggregations on, but building up two parallel sub-topologies with the same source topic at the source results in an error: "Topic

Re: best way to duplicate a KStream?

2016-04-14 Thread Greg Fodor
1.map(/*set key*/).through("topic2").aggregateByKey(...); > > stream3 = stream1.map(/*a different > key*/).through("topci3").aggregateByKey(...); > > ----- > > Guozhang > > > On Thu, Apr 14, 2016 at 10:05 AM, Greg Fodor wrote: > >> Hi

Kafka missing from ASF Jira?

2016-04-25 Thread Greg Fodor
I am trying to file a bug, but when I go to create a ticket on the ASF Jira, Kafka is not visible in the list of projects in the first field of the ticket. I see the hundred+ other Apache projects, but no Kafka :(

Kafka Streams file handle leak

2016-04-25 Thread Greg Fodor
Apologies for a second posting to the list, but the ASF JIRA is preventing me from creating KAFKA- tickets so I wanted to report this issue. This seems to be a critical bug with KStreams, the .lock files in the state store directories do not seem to be having their file handles freed (despite the l

Re: Kafka missing from ASF Jira?

2016-04-25 Thread Greg Fodor
g/jira/browse/KAFKA/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel > > Thanks, > Liquan > > On Mon, Apr 25, 2016 at 12:04 AM, Greg Fodor wrote: > >> I am trying to file a bug, but when I go to create a ticket on the ASF >> Jira, Kafka is not visible in the l

Re: Kafka missing from ASF Jira?

2016-04-25 Thread Greg Fodor
e contributor list. >> >> >> Guozhang >> >> On Mon, Apr 25, 2016 at 9:14 AM, Greg Fodor wrote: >> >> > If I go there, and hit "Create", the Project List at the top has lots >> > of apache projects does not contain any items contai

Re: Kafka Streams file handle leak

2016-04-25 Thread Greg Fodor
s and change the RB > title with the JIRA number? > > Thanks! > > Guozhang > > On Mon, Apr 25, 2016 at 12:13 AM, Greg Fodor wrote: > >> Apologies for a second posting to the list, but the ASF JIRA is >> preventing me from creating KAFKA- tickets so I wanted to r

[Streams] Reason for swallowing KafkaExceptions?

2016-05-12 Thread Greg Fodor
We noticed that some errors were happening in one of our KafkaStreams jobs but they were not appearing in our logs or being sent to our error reporting service (Airbrake) -- they only became visible on stdout when running from the console. I believe the reason is because of this explicit catch-reth

Re: [Streams] Reason for swallowing KafkaExceptions?

2016-05-12 Thread Greg Fodor
09 PM, Guozhang Wang wrote: > Greg, > > Could you post the output from stdout when running the console as well? > > > Guozhang > > On Thu, May 12, 2016 at 4:52 PM, Greg Fodor wrote: > >> We noticed that some errors were happening in one of our KafkaStreams >>

Re: [Streams] Reason for swallowing KafkaExceptions?

2016-05-12 Thread Greg Fodor
eptions are not logged in log4j yet before it >> pop up to the StreamThread. I will file a JIRA for this issue. >> >> >> Guozhang >> >> On Thu, May 12, 2016 at 5:57 PM, Greg Fodor wrote: >> >>> Hey Guozhang, output was silent (since the log4j is str

Store flushing on commit.interval.ms from KIP-63 introduces aggregation latency

2016-10-09 Thread Greg Fodor
I'm taking 0.10.1 for a spin on our existing Kafka Streams jobs and I'm hitting what seems to be a serious issue (at least, for us) with the changes brought about in KIP-63. In our job, we have a number of steps in the topology where we perform a repartition and aggregation on topics that require l

Re: Store flushing on commit.interval.ms from KIP-63 introduces aggregation latency

2016-10-09 Thread Greg Fodor
records are forwarded immediately it resolves the issue I am seeing. Having it be done on a per-KGroupedStream basis would provide maximum flexibility. On Sun, Oct 9, 2016 at 1:06 AM, Greg Fodor wrote: > I'm taking 0.10.1 for a spin on our existing Kafka Streams jobs and > I'm hitti

Re: Store flushing on commit.interval.ms from KIP-63 introduces aggregation latency

2016-10-09 Thread Greg Fodor
JIRA opened here: https://issues.apache.org/jira/browse/KAFKA-4281 On Sun, Oct 9, 2016 at 2:02 AM, Greg Fodor wrote: > I went ahead and did some more testing, and it feels to me one option > for resolving this issue is having a method on KGroupedStream which > can be used to configu

Re: Store flushing on commit.interval.ms from KIP-63 introduces aggregation latency

2016-10-10 Thread Greg Fodor
size to 0. That should > give you the old behaviour: > streamsConfiguration.put(StreamsConfig.CACHE_MAX_BYTES_BUFFERING_CONFIG, > 0L); > > Your PR is an alternative, but it requires changing the APIs and would > require a KIP. > > Thanks > Eno > > > On 9 Oct 2016, at 23:4

Re: Store flushing on commit.interval.ms from KIP-63 introduces aggregation latency

2016-10-11 Thread Greg Fodor
perations#write-buffer > <https://github.com/facebook/rocksdb/wiki/Basic-Operations#write-buffer> > > They can override these settings by creating an impl of RocksDBConfigSetter > and setting StreamsConfig.ROCKSDB_CONFIG_SETTER_CLASS_CONFIG in Kafka Streams. > > Hope this h

Re: Store flushing on commit.interval.ms from KIP-63 introduces aggregation latency

2016-10-12 Thread Greg Fodor
g those many serde calls per store). But if you do > observe significant performance degradation I'd personally love to learn > more and help on that end. > > > Guozhang > > > > > > On Tue, Oct 11, 2016 at 10:10 AM, Greg Fodor wrote: > > > Thanks Eno -- m

Re: [ANNOUNCE] Apache Kafka 0.10.1.0 Released

2016-10-20 Thread Greg Fodor
loser.cgi?path=/kafka/0.10.1.0/k >> > > afka_2.10-0.10.1.0.tgz >> > > >> > > Thanks to the 115 contributors on this release! >> > > >> > > Alex Glikson, Alex Loddengaard, Alexey Ozeritsky, Alexey Romanchuk, >> > Andrea >> &

Re: [jira] [Commented] (KAFKA-4113) Allow KTable bootstrap

2016-10-21 Thread Greg Fodor
gracefully shut down our jobs in certain cases so there's a chance that might be related. Anyhow, now that I have a better understanding of things I will be able to investigate if we experience missing keys in the future, thanks! On Oct 20, 2016 2:08 PM, "Greg Fodor (JIRA)" wrote

[jira] [Created] (KAFKA-5571) Possible deadlock during shutdown in setState in kafka streams 10.2

2017-07-07 Thread Greg Fodor (JIRA)
Greg Fodor created KAFKA-5571: - Summary: Possible deadlock during shutdown in setState in kafka streams 10.2 Key: KAFKA-5571 URL: https://issues.apache.org/jira/browse/KAFKA-5571 Project: Kafka

[jira] [Created] (KAFKA-3745) Consider adding join key to ValueJoiner interface

2016-05-22 Thread Greg Fodor (JIRA)
Greg Fodor created KAFKA-3745: - Summary: Consider adding join key to ValueJoiner interface Key: KAFKA-3745 URL: https://issues.apache.org/jira/browse/KAFKA-3745 Project: Kafka Issue Type

[jira] [Commented] (KAFKA-3745) Consider adding join key to ValueJoiner interface

2016-05-23 Thread Greg Fodor (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15297250#comment-15297250 ] Greg Fodor commented on KAFKA-3745: --- sure. we are left joining a KTable again

[jira] [Commented] (KAFKA-3745) Consider adding join key to ValueJoiner interface

2016-05-23 Thread Greg Fodor (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15297477#comment-15297477 ] Greg Fodor commented on KAFKA-3745: --- Yep, I admit this is definitely not the

[jira] [Commented] (KAFKA-3745) Consider adding join key to ValueJoiner interface

2016-05-23 Thread Greg Fodor (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15297733#comment-15297733 ] Greg Fodor commented on KAFKA-3745: --- Yes, the join key needs to be added to the f

[jira] [Commented] (KAFKA-3745) Consider adding join key to ValueJoiner interface

2016-05-23 Thread Greg Fodor (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15297745#comment-15297745 ] Greg Fodor commented on KAFKA-3745: --- Actually this seems like a good enough solutio

[jira] [Created] (KAFKA-3758) KStream job fails to recover after Kafka broker stopped

2016-05-26 Thread Greg Fodor (JIRA)
Greg Fodor created KAFKA-3758: - Summary: KStream job fails to recover after Kafka broker stopped Key: KAFKA-3758 URL: https://issues.apache.org/jira/browse/KAFKA-3758 Project: Kafka Issue Type

[jira] [Updated] (KAFKA-3758) KStream job fails to recover after Kafka broker stopped

2016-05-26 Thread Greg Fodor (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Greg Fodor updated KAFKA-3758: -- Attachment: muon.log.1.gz > KStream job fails to recover after Kafka broker stop

[jira] [Commented] (KAFKA-3758) KStream job fails to recover after Kafka broker stopped

2016-05-26 Thread Greg Fodor (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15302542#comment-15302542 ] Greg Fodor commented on KAFKA-3758: --- Also, the log is truncated at the top to the p

[jira] [Commented] (KAFKA-3758) KStream job fails to recover after Kafka broker stopped

2016-05-28 Thread Greg Fodor (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15305501#comment-15305501 ] Greg Fodor commented on KAFKA-3758: --- Ah yes this looks like the same issue, th

[jira] [Resolved] (KAFKA-3758) KStream job fails to recover after Kafka broker stopped

2016-05-28 Thread Greg Fodor (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Greg Fodor resolved KAFKA-3758. --- Resolution: Duplicate > KStream job fails to recover after Kafka broker stop

[jira] [Commented] (KAFKA-3758) KStream job fails to recover after Kafka broker stopped

2016-05-28 Thread Greg Fodor (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15305503#comment-15305503 ] Greg Fodor commented on KAFKA-3758: --- Oh, actually, I'm not so sure. This was n

[jira] [Reopened] (KAFKA-3758) KStream job fails to recover after Kafka broker stopped

2016-05-28 Thread Greg Fodor (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Greg Fodor reopened KAFKA-3758: --- > KStream job fails to recover after Kafka broker stop

[jira] [Created] (KAFKA-3769) KStream job spending 60% of time writing metrics

2016-05-29 Thread Greg Fodor (JIRA)
Greg Fodor created KAFKA-3769: - Summary: KStream job spending 60% of time writing metrics Key: KAFKA-3769 URL: https://issues.apache.org/jira/browse/KAFKA-3769 Project: Kafka Issue Type: Bug

[jira] [Commented] (KAFKA-3769) KStream job spending 60% of time writing metrics

2016-05-29 Thread Greg Fodor (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15306306#comment-15306306 ] Greg Fodor commented on KAFKA-3769: --- https://github.com/apache/kafka/pull/

[jira] [Commented] (KAFKA-3769) KStream job spending 60% of time writing metrics

2016-05-30 Thread Greg Fodor (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15306318#comment-15306318 ] Greg Fodor commented on KAFKA-3769: --- Actually, additionally it looks like the code

[jira] [Commented] (KAFKA-3769) KStream job spending 60% of time writing metrics

2016-05-30 Thread Greg Fodor (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15306321#comment-15306321 ] Greg Fodor commented on KAFKA-3769: --- It seems it might be desirable to have a wa

[jira] [Created] (KAFKA-3770) KStream job should be able to specify linger.ms

2016-05-30 Thread Greg Fodor (JIRA)
Greg Fodor created KAFKA-3770: - Summary: KStream job should be able to specify linger.ms Key: KAFKA-3770 URL: https://issues.apache.org/jira/browse/KAFKA-3770 Project: Kafka Issue Type

[jira] [Commented] (KAFKA-3770) KStream job should be able to specify linger.ms

2016-05-30 Thread Greg Fodor (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15306371#comment-15306371 ] Greg Fodor commented on KAFKA-3770: --- https://github.com/apache/kafka/pull/

[jira] [Commented] (KAFKA-3769) KStream job spending 60% of time writing metrics

2016-05-30 Thread Greg Fodor (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15306381#comment-15306381 ] Greg Fodor commented on KAFKA-3769: --- Consider the PR a first pass, please advise on

[jira] [Commented] (KAFKA-3758) KStream job fails to recover after Kafka broker stopped

2016-05-31 Thread Greg Fodor (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15308306#comment-15308306 ] Greg Fodor commented on KAFKA-3758: --- No, the kstream job was running across 2 ser

[jira] [Commented] (KAFKA-3769) KStream job spending 60% of time writing metrics

2016-06-01 Thread Greg Fodor (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15311179#comment-15311179 ] Greg Fodor commented on KAFKA-3769: --- Thanks Jay! Guozhang, what are your thought

[jira] [Commented] (KAFKA-3758) KStream job fails to recover after Kafka broker stopped

2016-06-02 Thread Greg Fodor (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15312912#comment-15312912 ] Greg Fodor commented on KAFKA-3758: --- Hey, we're running 16 threads -- for th

[jira] [Commented] (KAFKA-3758) KStream job fails to recover after Kafka broker stopped

2016-06-02 Thread Greg Fodor (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3758?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15312913#comment-15312913 ] Greg Fodor commented on KAFKA-3758: --- also, if we did not run at an elevated numbe

[jira] [Commented] (KAFKA-3770) KStream job should be able to specify linger.ms

2016-06-02 Thread Greg Fodor (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15313434#comment-15313434 ] Greg Fodor commented on KAFKA-3770: --- Cut a new PR since I screwed up the rebase on

[jira] [Created] (KAFKA-3785) Fetcher spending unnecessary time during metrics recording

2016-06-02 Thread Greg Fodor (JIRA)
Greg Fodor created KAFKA-3785: - Summary: Fetcher spending unnecessary time during metrics recording Key: KAFKA-3785 URL: https://issues.apache.org/jira/browse/KAFKA-3785 Project: Kafka Issue

[jira] [Updated] (KAFKA-3785) Fetcher spending unnecessary time during metrics recording

2016-06-02 Thread Greg Fodor (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Greg Fodor updated KAFKA-3785: -- Component/s: consumer > Fetcher spending unnecessary time during metrics record

[jira] [Commented] (KAFKA-3785) Fetcher spending unnecessary time during metrics recording

2016-06-02 Thread Greg Fodor (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15313461#comment-15313461 ] Greg Fodor commented on KAFKA-3785: --- https://github.com/apache/kafka/pull/

[jira] [Updated] (KAFKA-3811) Introduce Kafka Streams metrics recording levels

2016-06-09 Thread Greg Fodor (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Greg Fodor updated KAFKA-3811: -- Summary: Introduce Kafka Streams metrics recording levels (was: Introduce KStream metrics recording

[jira] [Created] (KAFKA-3811) Introduce KStream metrics recording levels

2016-06-09 Thread Greg Fodor (JIRA)
Greg Fodor created KAFKA-3811: - Summary: Introduce KStream metrics recording levels Key: KAFKA-3811 URL: https://issues.apache.org/jira/browse/KAFKA-3811 Project: Kafka Issue Type: Improvement

[jira] [Commented] (KAFKA-3811) Introduce Kafka Streams metrics recording levels

2016-06-10 Thread Greg Fodor (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15324838#comment-15324838 ] Greg Fodor commented on KAFKA-3811: --- Hey [~aartigupta], I ran an attached you

[jira] [Updated] (KAFKA-3811) Introduce Kafka Streams metrics recording levels

2016-06-10 Thread Greg Fodor (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Greg Fodor updated KAFKA-3811: -- Attachment: Muon-Snapshot.zip > Introduce Kafka Streams metrics recording lev

[jira] [Updated] (KAFKA-3811) Introduce Kafka Streams metrics recording levels

2016-06-10 Thread Greg Fodor (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Greg Fodor updated KAFKA-3811: -- Attachment: screenshot-1.png > Introduce Kafka Streams metrics recording lev

[jira] [Commented] (KAFKA-3811) Introduce Kafka Streams metrics recording levels

2016-06-10 Thread Greg Fodor (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15324847#comment-15324847 ] Greg Fodor commented on KAFKA-3811: --- I've also attached a screenshot of YourK

[jira] [Updated] (KAFKA-3811) Introduce Kafka Streams metrics recording levels

2016-06-10 Thread Greg Fodor (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Greg Fodor updated KAFKA-3811: -- Attachment: Muon-latency.zip > Introduce Kafka Streams metrics recording lev

[jira] [Commented] (KAFKA-3811) Introduce Kafka Streams metrics recording levels

2016-06-10 Thread Greg Fodor (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15324869#comment-15324869 ] Greg Fodor commented on KAFKA-3811: --- Also, I've attached a screenshot + snap

[jira] [Updated] (KAFKA-3811) Introduce Kafka Streams metrics recording levels

2016-06-10 Thread Greg Fodor (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Greg Fodor updated KAFKA-3811: -- Attachment: screenshot-latency.png > Introduce Kafka Streams metrics recording lev

[jira] [Commented] (KAFKA-3769) KStream job spending 60% of time writing metrics

2016-06-12 Thread Greg Fodor (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15326778#comment-15326778 ] Greg Fodor commented on KAFKA-3769: --- Discussion/resolution moved to: h

[jira] [Resolved] (KAFKA-3770) KStream job should be able to specify linger.ms

2016-08-07 Thread Greg Fodor (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Greg Fodor resolved KAFKA-3770. --- Resolution: Fixed Fixed via KAFKA-3786 > KStream job should be able to specify linger

[jira] [Commented] (KAFKA-3752) Provide a way for KStreams to recover from unclean shutdown

2016-08-10 Thread Greg Fodor (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15415664#comment-15415664 ] Greg Fodor commented on KAFKA-3752: --- Hey [~guozhang], we're hitting this issu

[jira] [Comment Edited] (KAFKA-3360) Add a protocol page/section to the official Kafka documentation

2016-03-19 Thread Greg Fodor (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15198422#comment-15198422 ] Greg Fodor edited comment on KAFKA-3360 at 3/16/16 11:3

[jira] [Commented] (KAFKA-3360) Add a protocol page/section to the official Kafka documentation

2016-03-19 Thread Greg Fodor (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15198422#comment-15198422 ] Greg Fodor commented on KAFKA-3360: --- Apologies as I did not really know the best p

[jira] [Created] (KAFKA-3542) Add "repartition (+ join)" operations to streams

2016-04-11 Thread Greg Fodor (JIRA)
Greg Fodor created KAFKA-3542: - Summary: Add "repartition (+ join)" operations to streams Key: KAFKA-3542 URL: https://issues.apache.org/jira/browse/KAFKA-3542 Project: Kafka

[jira] [Updated] (KAFKA-3543) Allow a variant of transform() which can emit multiple values

2016-04-11 Thread Greg Fodor (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Greg Fodor updated KAFKA-3543: -- Summary: Allow a variant of transform() which can emit multiple values (was: Allow a variant of

[jira] [Created] (KAFKA-3543) Allow a variant of transform() which allows emitting multiple values

2016-04-11 Thread Greg Fodor (JIRA)
Greg Fodor created KAFKA-3543: - Summary: Allow a variant of transform() which allows emitting multiple values Key: KAFKA-3543 URL: https://issues.apache.org/jira/browse/KAFKA-3543 Project: Kafka

[jira] [Comment Edited] (KAFKA-3543) Allow a variant of transform() which can emit multiple values

2016-04-11 Thread Greg Fodor (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15236121#comment-15236121 ] Greg Fodor edited comment on KAFKA-3543 at 4/11/16 10:4

[jira] [Updated] (KAFKA-3543) Allow a variant of transform() which can emit multiple values

2016-04-11 Thread Greg Fodor (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Greg Fodor updated KAFKA-3543: -- Description: Right now it seems that if you want to apply an arbitrary stateful transformation to a

[jira] [Commented] (KAFKA-3543) Allow a variant of transform() which can emit multiple values

2016-04-11 Thread Greg Fodor (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15236121#comment-15236121 ] Greg Fodor commented on KAFKA-3543: --- Also note that in my case the iflatMap() is

[jira] [Created] (KAFKA-3544) Missing topics on startup

2016-04-11 Thread Greg Fodor (JIRA)
Greg Fodor created KAFKA-3544: - Summary: Missing topics on startup Key: KAFKA-3544 URL: https://issues.apache.org/jira/browse/KAFKA-3544 Project: Kafka Issue Type: Bug Components

[jira] [Created] (KAFKA-3545) Generalized Serdes for List/Map

2016-04-11 Thread Greg Fodor (JIRA)
Greg Fodor created KAFKA-3545: - Summary: Generalized Serdes for List/Map Key: KAFKA-3545 URL: https://issues.apache.org/jira/browse/KAFKA-3545 Project: Kafka Issue Type: Improvement

[jira] [Commented] (KAFKA-3542) Add "repartition (+ join)" operations to streams

2016-04-11 Thread Greg Fodor (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15236331#comment-15236331 ] Greg Fodor commented on KAFKA-3542: --- I might just not be understanding, but a si

[jira] [Commented] (KAFKA-3543) Allow a variant of transform() which can emit multiple values

2016-04-11 Thread Greg Fodor (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15236339#comment-15236339 ] Greg Fodor commented on KAFKA-3543: --- Ah interesting, my assumption was that the i

[jira] [Commented] (KAFKA-3542) Add "repartition (+ join)" operations to streams

2016-04-11 Thread Greg Fodor (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15236343#comment-15236343 ] Greg Fodor commented on KAFKA-3542: --- Ah, I may understand what you're gettin

[jira] [Commented] (KAFKA-3542) Add "repartition (+ join)" operations to streams

2016-04-11 Thread Greg Fodor (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15236374#comment-15236374 ] Greg Fodor commented on KAFKA-3542: --- Right, this map approach is what I am doing r

[jira] [Commented] (KAFKA-3543) Allow a variant of transform() which can emit multiple values

2016-04-11 Thread Greg Fodor (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15236375#comment-15236375 ] Greg Fodor commented on KAFKA-3543: --- That makes sense, thanks! It may be usefu

[jira] [Commented] (KAFKA-3542) Add "repartition (+ join)" operations to streams

2016-04-11 Thread Greg Fodor (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15236417#comment-15236417 ] Greg Fodor commented on KAFKA-3542: --- great. feel free to close. > Add "

[jira] [Commented] (KAFKA-3543) Allow a variant of transform() which can emit multiple values

2016-04-11 Thread Greg Fodor (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15236427#comment-15236427 ] Greg Fodor commented on KAFKA-3543: --- Just something akin to a flatTransform where

[jira] [Commented] (KAFKA-3544) Missing topics on startup

2016-04-11 Thread Greg Fodor (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15236512#comment-15236512 ] Greg Fodor commented on KAFKA-3544: --- Not sure of the best way to share the topo

[jira] [Comment Edited] (KAFKA-3544) Missing topics on startup

2016-04-11 Thread Greg Fodor (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15236512#comment-15236512 ] Greg Fodor edited comment on KAFKA-3544 at 4/12/16 3:0

[jira] [Comment Edited] (KAFKA-3544) Missing topics on startup

2016-04-11 Thread Greg Fodor (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15236512#comment-15236512 ] Greg Fodor edited comment on KAFKA-3544 at 4/12/16 3:0

[jira] [Comment Edited] (KAFKA-3544) Missing topics on startup

2016-04-11 Thread Greg Fodor (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15236512#comment-15236512 ] Greg Fodor edited comment on KAFKA-3544 at 4/12/16 3:0

[jira] [Commented] (KAFKA-3544) Missing topics on startup

2016-04-14 Thread Greg Fodor (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15240723#comment-15240723 ] Greg Fodor commented on KAFKA-3544: --- Ah this makes sense. Fortunately thanks to

  1   2   >