[jira] [Created] (KAFKA-3572) Metrics of topics still exist when they have been deleted

2016-04-17 Thread Eric Huang (JIRA)
Eric Huang created KAFKA-3572: - Summary: Metrics of topics still exist when they have been deleted Key: KAFKA-3572 URL: https://issues.apache.org/jira/browse/KAFKA-3572 Project: Kafka Issue Type:

[jira] [Updated] (KAFKA-3421) Update docs with new connector features

2016-04-17 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liquan Pei updated KAFKA-3421: -- Status: Patch Available (was: In Progress) > Update docs with new connector features >

Re: kafka streaming: passing config string to Processor

2016-04-17 Thread Matthias J. Sax
I guess the simplest way would be to use a constructor parameter: > public static class CampaignProcessor implements > ProcessorSupplier, List> > { > private final String jedis_server; > > public CampaignProcessor(String jedisServer) { > this.jedis_server = jedisServ

Re: kafka streaming: passing config string to Processor

2016-04-17 Thread rss rss
Ok... Is it really perform distribution of the field to several instances of a cluster? I expected to see some way via ProcessorContext... But may be this is result of the Flink's experience... Thanks 2016-04-17 12:38 GMT+02:00 Matthias J. Sax : > I guess the simplest way would be to use a const

[jira] [Created] (KAFKA-3573) JSON (de)serialization for kafka-client

2016-04-17 Thread Igor Stepanov (JIRA)
Igor Stepanov created KAFKA-3573: Summary: JSON (de)serialization for kafka-client Key: KAFKA-3573 URL: https://issues.apache.org/jira/browse/KAFKA-3573 Project: Kafka Issue Type: New Feature

[jira] [Commented] (KAFKA-3337) Extract selector as a separate groupBy operator for KTable aggregations

2016-04-17 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15244674#comment-15244674 ] ASF GitHub Bot commented on KAFKA-3337: --- GitHub user mjsax opened a pull request:

[GitHub] kafka pull request: KAFKA-3337: [WIP] Extract selector as a separa...

2016-04-17 Thread mjsax
GitHub user mjsax opened a pull request: https://github.com/apache/kafka/pull/1231 KAFKA-3337: [WIP] Extract selector as a separate groupBy operator for KTable aggregations You can merge this pull request into a Git repository by running: $ git pull https://github.com/mjsax/k

Re: kafka streaming: passing config string to Processor

2016-04-17 Thread Matthias J. Sax
KafkaStreams works quite different than other systems like Flink/Storm. It is not a system but a library. If you start a KafkaStreams application, it runs locally. Scaling/Parallelism comes into place if you start the same application on multiple nodes. For this, Kafka's parallelization model is u

[jira] [Created] (KAFKA-3574) Add missing @returns tags for JavaDoc

2016-04-17 Thread Matthias J. Sax (JIRA)
Matthias J. Sax created KAFKA-3574: -- Summary: Add missing @returns tags for JavaDoc Key: KAFKA-3574 URL: https://issues.apache.org/jira/browse/KAFKA-3574 Project: Kafka Issue Type: Improveme

[jira] [Updated] (KAFKA-3574) Add missing @returns tags for JavaDoc

2016-04-17 Thread Matthias J. Sax (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-3574: --- Issue Type: Sub-task (was: Improvement) Parent: KAFKA-2590 > Add missing @returns tag

[jira] [Assigned] (KAFKA-3429) Remove Serdes needed for repartitioning in KTable stateful operations

2016-04-17 Thread Matthias J. Sax (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax reassigned KAFKA-3429: -- Assignee: Matthias J. Sax > Remove Serdes needed for repartitioning in KTable stateful

KTable.count(...)

2016-04-17 Thread Damian Guy
Hi, I'm slightly confused by KTable.count(..). The javadoc says: Count number of records of this stream by the selected key into a new instance of {@link KTable}. So.. if i send 5 records with the same key to the input topic, as per below final KafkaProducer producer = new KafkaProducer<>(prod

[jira] [Assigned] (KAFKA-3101) Optimize Aggregation Outputs

2016-04-17 Thread Bill Bejeck (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Bejeck reassigned KAFKA-3101: -- Assignee: Bill Bejeck > Optimize Aggregation Outputs > > >

Re: [VOTE] KIP-33 - Add a time based log index

2016-04-17 Thread Liquan Pei
+1 On Sat, Apr 16, 2016 at 10:25 PM, Gwen Shapira wrote: > +1 > > On Fri, Apr 15, 2016 at 9:37 AM, Guozhang Wang wrote: > > +1 from me. Thanks. > > > > On Fri, Apr 15, 2016 at 9:16 AM, Jun Rao wrote: > > > >> Hi, Jiangjie, > >> > >> Thanks for the latest update. +1 on the KIP. > >> > >> Jun >

[jira] [Commented] (KAFKA-3573) JSON (de)serialization for kafka-client

2016-04-17 Thread Liquan Pei (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15244768#comment-15244768 ] Liquan Pei commented on KAFKA-3573: --- Thanks for working on this. Currently, there is a J

[jira] [Commented] (KAFKA-3429) Remove Serdes needed for repartitioning in KTable stateful operations

2016-04-17 Thread Matthias J. Sax (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15244771#comment-15244771 ] Matthias J. Sax commented on KAFKA-3429: [~guozhang] I am a little confused here:

Re: KTable.count(...)

2016-04-17 Thread Liquan Pei
Hi Damin, I am new to KStreams as well, so my answer might not be 100% precise. In KTable, the same key is treated as updates instead of events. Thus aggregation on the same key will do some de-dup. The docs for the tech preview contains some explanation on this behavior: http://docs.confluent.io

Re: kafka streaming: passing config string to Processor

2016-04-17 Thread rss rss
Thanks for the answer. But is it correct in this case to use yahoo streaming benchmark to compare Kafka, Flink and Storm? Or Kafka streaming processor is for other category of customers? Best regards 2016-04-17 16:48 GMT+02:00 Matthias J. Sax : > KafkaStreams works quite different than other sys

[jira] [Commented] (KAFKA-3573) JSON (de)serialization for kafka-client

2016-04-17 Thread Igor Stepanov (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15244831#comment-15244831 ] Igor Stepanov commented on KAFKA-3573: -- Good point, found your code here: https://git

[jira] [Commented] (KAFKA-3565) Producer's throughput lower with compressed data after KIP-31/32

2016-04-17 Thread Jiangjie Qin (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15244946#comment-15244946 ] Jiangjie Qin commented on KAFKA-3565: - [~ijuma] A few comments: 1. The performance im

[jira] [Updated] (KAFKA-3554) Generate actual data with specific compression ratio and add multi-thread support in the ProducerPerformance tool.

2016-04-17 Thread Jiangjie Qin (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiangjie Qin updated KAFKA-3554: Summary: Generate actual data with specific compression ratio and add multi-thread support in the Pr

[jira] [Updated] (KAFKA-3554) Generate actual data with specific compression ratio in the ProducerPerformance tool.

2016-04-17 Thread Jiangjie Qin (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiangjie Qin updated KAFKA-3554: Description: Currently the ProducerPerformance always generate the payload with same bytes. This do

[jira] [Commented] (KAFKA-3565) Producer's throughput lower with compressed data after KIP-31/32

2016-04-17 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15245007#comment-15245007 ] Ismael Juma commented on KAFKA-3565: Becket, 1. Yes, the non-compressed throughput c

[jira] [Comment Edited] (KAFKA-3565) Producer's throughput lower with compressed data after KIP-31/32

2016-04-17 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15245007#comment-15245007 ] Ismael Juma edited comment on KAFKA-3565 at 4/18/16 12:30 AM: --

[jira] [Comment Edited] (KAFKA-3565) Producer's throughput lower with compressed data after KIP-31/32

2016-04-17 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15245007#comment-15245007 ] Ismael Juma edited comment on KAFKA-3565 at 4/18/16 12:29 AM: --

[jira] [Comment Edited] (KAFKA-3565) Producer's throughput lower with compressed data after KIP-31/32

2016-04-17 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15245007#comment-15245007 ] Ismael Juma edited comment on KAFKA-3565 at 4/18/16 12:29 AM: --

[jira] [Commented] (KAFKA-3429) Remove Serdes needed for repartitioning in KTable stateful operations

2016-04-17 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15245142#comment-15245142 ] Guozhang Wang commented on KAFKA-3429: -- The {{originalSerde}} is used to materialize

Re: kafka streaming: passing config string to Processor

2016-04-17 Thread Guozhang Wang
Hello, I think Kafka Streams is better treated as one approach in streaming processing systems for a variety of customers. For example, say if you already have a YARN cluster, and you have a dedicated team operating it and many teams wants to use this for their various streaming jobs, then submitt

[jira] [Commented] (KAFKA-3573) JSON (de)serialization for kafka-client

2016-04-17 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15245148#comment-15245148 ] Guozhang Wang commented on KAFKA-3573: -- We were working on moving the serdes from con

[jira] [Resolved] (KAFKA-119) Avoid duplicated code in ProducerPerformance

2016-04-17 Thread Manikumar Reddy (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar Reddy resolved KAFKA-119. --- Resolution: Won't Fix ProducerPerformance.scala is deprecated. New ProducerPerformance tool is

[jira] [Resolved] (KAFKA-1058) Change the patch review tool to use the reviewboard python client

2016-04-17 Thread Manikumar Reddy (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar Reddy resolved KAFKA-1058. Resolution: Won't Fix Kafka code contribution process is moved to Github PR process. > Cha

[jira] [Resolved] (KAFKA-3005) delete kafka topic without delete.topic.enable=true

2016-04-17 Thread Manikumar Reddy (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar Reddy resolved KAFKA-3005. Resolution: Not A Problem Your approach is error prone. Use rolling restart to set delete.t

[jira] [Resolved] (KAFKA-3050) Space in the value for "host.name" causes "Unresolved address"

2016-04-17 Thread Manikumar Reddy (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar Reddy resolved KAFKA-3050. Resolution: Fixed Fix Version/s: 0.9.0.0 This got fixed in 0.9.0.0 release > Space i