Re: Question how to drop a unused group id?

2018-08-14 Thread Vahid S Hashemian
ommit offsets (automatically or manually) will extend the lifecycle of a group. --Vahid From: "ÅíÅô(Peyton)" To: Vahid S Hashemian Cc: "彭鹏" , users@kafka.apache.org Date: 08/13/2018 07:11 PM Subject:Re: Question how to drop a unused group id? Yes

Re: Question how to drop a unused group id?

2018-08-13 Thread Vahid S Hashemian
consumer-groups.sh --bootstrap-server [broker]:9092 --list --Vahid From: "ÅíÅô(Peyton)" To: Vahid S Hashemian Cc: users@kafka.apache.org, "彭鹏" Date: 08/13/2018 09:18 AM Subject:Re: Question how to drop a unused group id? Thanks! Hmmm.. the last qu

Re: Question how to drop a unused group id?

2018-08-13 Thread Vahid S Hashemian
If you did not set the config, the default will be used. For your version of Kafka (0.10.2.1) the default is 1 day: https://github.com/apache/kafka/blob/0.10.2.1/core/src/main/scala/kafka/server/KafkaConfig.scala#L152 --Vahid From: "ÅíÅô(Peyton)" To: Vahid S Hashemian Cc:

Re: Question how to drop a unused group id?

2018-08-13 Thread Vahid S Hashemian
Hi Peyton, Yes, if you'd like to change the value a broker restart is required for the change to take effect. --Vahid From: "ÅíÅô(Peyton)" To: Vahid S Hashemian Cc: users@kafka.apache.org, "彭鹏" Date: 08/13/2018 08:49 AM Subject:Re: Question h

Re: Question how to drop a unused group id?

2018-08-13 Thread Vahid S Hashemian
Hi Peyton, What version of Kafka are you using? Starting from version 1.1.0 there is a DELETE_GROUP API: https://cwiki.apache.org/confluence/display/KAFKA/KIP-229%3A+DeleteGroups+API You can also use the consumer group command with `--delete` to delete a group. For prior versions, the safest op

Re: ConsumerGroupCommand tool improvement?

2018-08-06 Thread Vahid S Hashemian
l rest my case. Thanks! --Vahid From: Colin McCabe To: users@kafka.apache.org Date: 08/06/2018 12:44 PM Subject:Re: ConsumerGroupCommand tool improvement? On Mon, Aug 6, 2018, at 12:04, Vahid S Hashemian wrote: > Hi Colin, > > Thanks for considering the idea and sh

Re: ConsumerGroupCommand tool improvement?

2018-08-06 Thread Vahid S Hashemian
d? best, Colin On Thu, Aug 2, 2018, at 14:23, Vahid S Hashemian wrote: > Hi all, > > A requirement has been raised by a colleague and I wanted to see if there > is any interest in the community in adding the functionality to Apache > Kafka. > > ConsumerGroupComma

ConsumerGroupCommand tool improvement?

2018-08-02 Thread Vahid S Hashemian
Hi all, A requirement has been raised by a colleague and I wanted to see if there is any interest in the community in adding the functionality to Apache Kafka. ConsumerGroupCommand tool in describe ('--describe' or '--describe --offsets') mode currently lists all topics the group has consumed

Re: [ANNOUNCE] Apache Kafka 2.0.0 Released

2018-07-30 Thread Vahid S Hashemian
Such a good news on a Monday morning ... Thank you Rajini for driving the release! --Vahid From: Mickael Maison To: Users Cc: dev , annou...@apache.org, kafka-clients Date: 07/30/2018 04:37 AM Subject:Re: [ANNOUNCE] Apache Kafka 2.0.0 Released Great news! Thanks for

Re: [VOTE] 2.0.0 RC3

2018-07-24 Thread Vahid S Hashemian
+1 (non-binding) Built from source and ran quickstart successfully with both Java 8 and Java 9 on Ubuntu. Thanks Rajini! --Vahid From: Rajini Sivaram To: dev , Users , kafka-clients Date: 07/24/2018 08:33 AM Subject:[VOTE] 2.0.0 RC3 Hello Kafka users, developers and cli

Re: [VOTE] 2.0.0 RC2

2018-07-11 Thread Vahid S Hashemian
+1 (non-binding) Built executables from source and ran quickstart (Ubuntu / Java 8) Thanks! --Vahid From: Brett Rann To: d...@kafka.apache.org Cc: Users , kafka-clients Date: 07/10/2018 09:53 PM Subject:Re: [VOTE] 2.0.0 RC2 +1 (non binding) rolling upgrade of tiny sh

Re: Apache Kafka QuickStart

2018-07-11 Thread Vahid S Hashemian
Hi Nicholas, The quickstart is meant to run in terminals. The two commands in Step 2 should be run in different terminals unless you're sending the ZooKeeper process to the backgroud. If you are facing particular errors please share so we can better assist you. Thanks. --Vahid From: Nich

Re: [VOTE] 2.0.0 RC1

2018-07-02 Thread Vahid S Hashemian
+1 (non-binding) Built from source and ran quickstart successfully on Ubuntu (with Java 8). Minor: It seems this doc update PR is not included in the RC: https://github.com/apache/kafka/pull/5280 Guozhang seems to have wanted to cherry-pick it to 2.0. Thanks Rajini! --Vahid From: Rajini S

Re: [VOTE] 0.11.0.3 RC0

2018-06-22 Thread Vahid S Hashemian
0.11.0-jdk7/385/ System tests: https://jenkins.confluent.io/job/system-test-kafka/job/0.11.0/217/ /** Thanks, -Matthias [attachment "signature.asc" deleted by Vahid S Hashemian/Silicon Valley/IBM]

Re: [VOTE] 2.0.0 RC0

2018-06-22 Thread Vahid S Hashemian
+1 (non-binding) Built from source and ran quickstart successfully on Ubuntu (with Java 8 and Java 9). Thanks Rajini! --Vahid

Re: [VOTE] 1.1.1 RC1

2018-06-22 Thread Vahid S Hashemian
+1 (non-binding) Built from source and ran quickstart successfully on Ubuntu (with Java 8). Thanks Dong! --Vahid From: Dong Lin To: d...@kafka.apache.org, users@kafka.apache.org, kafka-clie...@googlegroups.com Date: 06/22/2018 10:10 AM Subject:[VOTE] 1.1.1 RC1 Hello Kafka

Re: [VOTE] 1.0.2 RC0

2018-06-22 Thread Vahid S Hashemian
1/ System tests: https://jenkins.confluent.io/job/system-test-kafka/job/1.0/217/ /** Thanks, -Matthias [attachment "signature.asc" deleted by Vahid S Hashemian/Silicon Valley/IBM]

Re: Multiple consumers subscribing to a topic

2018-06-18 Thread Vahid S Hashemian
Hi Nitin, 1) A Kafka Consumer uses a poll loop to pull messages from the topics it is subscribed to. You can see examples of how this can be implemented in Java here: https://www.confluent.io/blog/tutorial-getting-started-with-the-new-apache-kafka-0-9-consumer-client/ 2,3) Consumers normally c

Re: [kafka-clients] Re: [VOTE] 1.1.0 RC4

2018-03-27 Thread Vahid S Hashemian
Hi Rajini, +1 (non-binding) Built from source on Linux and Windows (Java 8), and tested quickstart on both platforms. Connect quickstart on Windows is not working as per my note on RC3: https://www.mail-archive.com/dev@kafka.apache.org/msg86138.html No other issues detected. Thanks! --Vahid

RE: Kafka consumer issue

2018-03-20 Thread Vahid S Hashemian
Id=console-consumer-35712] Connection to node -1 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient) -Original Message- From: Vahid S Hashemian [mailto:vahidhashem...@us.ibm.com] Sent: Tuesday, March 20, 2018 1:55 PM To: users@kafka.apache.

RE: Kafka consumer issue

2018-03-20 Thread Vahid S Hashemian
4 log.retention.check.interval.ms=30 zookeeper.connect=zkp02.mp.com:2181 zookeeper.connection.timeout.ms=6 -Original Message- From: Vahid S Hashemian [mailto:vahidhashem...@us.ibm.com] Sent: Tuesday, March 20, 2018 1:31 PM To: users@kafka.apache.org Subject: RE: Kafka consumer issue EXTERNAL EMA

RE: Kafka consumer issue

2018-03-20 Thread Vahid S Hashemian
ogs num.partitions=1 num.recovery.threads.per.data.dir=1 log.retention.hours=1440 log.segment.bytes=1073741824 log.retention.check.interval.ms=30 zookeeper.connect=zkp02.mp.com:2181 zookeeper.connection.timeout.ms=6 -Original Message- From: Vahid S Hashemian [mailto:vahidhashem.

RE: Kafka consumer issue

2018-03-20 Thread Vahid S Hashemian
Could you paste the server.properties content? It wasn't attached to the original note. Thanks. --Vahid From: "Anand, Uttam" To: "users@kafka.apache.org" Date: 03/20/2018 11:10 AM Subject:RE: Kakfa consumer issue Yes checked everything. No errors. Replication factor is 1 o

Re: Subject: [VOTE] 1.1.0 RC3

2018-03-16 Thread Vahid S Hashemian
Hi Damian, Thanks for running the release. I tried building from source and running the quick start on Linux & Windows with both Java 8 & 9. Here's the result: +-+-+-+ | | Linux | Windows | + +-+-+ |

Re: [VOTE] 1.1.0 RC0

2018-02-26 Thread Vahid S Hashemian
+1 (non-binding) Built the source and ran quickstart (including streams) successfully on Ubuntu (with both Java 8 and Java 9). I understand the Windows platform is not officially supported, but I ran the same on Windows 10, and except for Step 7 (Connect) everything else worked fine. There ar

Bay Area Apache Kafka Meetup - Morning of Feb 20

2018-02-07 Thread Vahid S Hashemian
Kafka users and developers, The next *Bay Area Apache Kafka Meetup* is on the *morning of Feb 20* and is hosted by *Index Developer Conference* at Moscone West in San Francisco . Meetup Info: https://www.meetup.com/KafkaBayArea/events/247433783/ Registration Link: https://ibm.co/2n742Jn (required

Re: [VOTE] 1.0.1 RC0

2018-02-07 Thread Vahid S Hashemian
Hi Ewen, +1 Building from source and running the quickstart were successful on Ubuntu and Windows 10. Thanks for running the release. --Vahid From: Ewen Cheslack-Postava To: d...@kafka.apache.org, users@kafka.apache.org, kafka-clie...@googlegroups.com Date: 02/05/2018 07:49 PM Subj

Re: [ANNOUNCE] New Kafka PMC Member: Rajini Sivaram

2018-01-17 Thread Vahid S Hashemian
Great news! Congratulations Rajini! --Vahid From: Gwen Shapira To: "d...@kafka.apache.org" , Users Date: 01/17/2018 10:49 AM Subject:[ANNOUNCE] New Kafka PMC Member: Rajini Sivaram Dear Kafka Developers, Users and Fans, Rajini Sivaram became a committer in April 2017. Si

Re: [ANNOUNCE] New committer: Matthias J. Sax

2018-01-12 Thread Vahid S Hashemian
Congrats Matthias! Well deserved. --Vahid From: Ted Yu To: d...@kafka.apache.org Cc: users@kafka.apache.org Date: 01/12/2018 03:00 PM Subject:Re: [ANNOUNCE] New committer: Matthias J. Sax Congratulations, Matthias. On Fri, Jan 12, 2018 at 2:59 PM, Guozhang Wang wrote:

Re: Questions about kafka-consumer-groups output

2017-11-11 Thread Vahid S Hashemian
Hi Michael, Java based Kafka consumers can consume messages in two ways: - simple: by manually assigning partitions to a consumer to consume from - consumer group: a group of consumers consume messages in a coordinated fashion (by subscribing to topics and having partitions automatically assigne

Re: [ANNOUNCE] New committer: Onur Karaman

2017-11-06 Thread Vahid S Hashemian
Congrats Onur! --Vahid From: Ismael Juma To: d...@kafka.apache.org Cc: "users@kafka.apache.org" Date: 11/06/2017 10:13 AM Subject:Re: [ANNOUNCE] New committer: Onur Karaman Sent by:isma...@gmail.com Congratulations Onur! Ismael On Mon, Nov 6, 2017 at 5:24 PM,

Re: [VOTE] 1.0.0 RC3

2017-10-25 Thread Vahid S Hashemian
Hi Guozhang, +1 for Ubuntu and Mac: successfully built jars and tested quickstarts with this RC (using Java 8). -1 for Windows: because of KAFKA-6075 and KAFKA-6100. To me these two issues (which have simple workarounds) sound like "blockers" - unless Kafka does not officially support Windows.

Re: [VOTE] 1.0.0 RC1

2017-10-17 Thread Vahid S Hashemian
: [VOTE] 1.0.0 RC1 If you don't use the default Scala version, you have to set the SCALA_VERSION environment variable for the bin scripts to work. Ismael On 17 Oct 2017 1:30 am, "Vahid S Hashemian" wrote: Hi Guozhang, I'm not sure if this should be covered by "Java 9 sup

Re: [VOTE] 1.0.0 RC1

2017-10-16 Thread Vahid S Hashemian
Hi Guozhang, I'm not sure if this should be covered by "Java 9 support" in the RC note, but when I try to build jars from source using Java 9 (./gradlew -PscalaVersion=2.12 jar) even though the build reports as succeeded, it doesn't seem to have been successful: $ bin/zookeeper-server-start.sh

Re: [VOTE] 1.0.0 RC0

2017-10-11 Thread Vahid S Hashemian
Hi Guozhang, Thanks for running the release. I tested building from source and the quickstarts on Linux, Mac, and Windows 64 (with Java 8 and Gradle 4.2.1). Everything worked well on Linux and Mac, but I ran into some issues on my Windows 64 VM: I reported one issue in KAFKA-6055, but it's an

Re: kafka-consumer-groups tool with SASL_PLAINTEXT

2017-07-28 Thread Vahid S Hashemian
Hi Vahid, Do you know how to use consumer-group tool with ssl only (without sasl) ? Gabriel. Le 24 juil. 2017 11:15 PM, "Vahid S Hashemian" a écrit : Hi Meghana, I did some experiments with SASL_PLAINTEXT and documented the results here: https://developer.ibm.com/opentech/2017/0

Re: [DISCUSS] KIP-175: Additional '--describe' views for ConsumerGroupCommand

2017-07-25 Thread Vahid S Hashemian
Hi, If there is no further feedback on this KIP, I'll start the vote tomorrow. Thanks. --Vahid From: Vahid S Hashemian/Silicon Valley/IBM To: dev , "Kafka User" Date: 07/03/2017 04:06 PM Subject:[DISCUSS] KIP-175: Additional '--describe' views for

Re: [DISCUSS] KIP-163: Lower the Minimum Required ACL Permission of OffsetFetch

2017-07-24 Thread Vahid S Hashemian
'd be open to doing it, but it'd require some thought about how it would impact users and how we'd migrate them to the updated rule (or just agree that it is a bug and that including upgrade notes would be sufficient). -Ewen On Mon, Jul 10, 2017 at 1:12 PM, Vahid S Hashemian < va

Re: kafka-consumer-groups tool with SASL_PLAINTEXT

2017-07-24 Thread Vahid S Hashemian
Hi Meghana, I did some experiments with SASL_PLAINTEXT and documented the results here: https://developer.ibm.com/opentech/2017/05/31/kafka-acls-in-practice/ I think it covers what you'd like to achieve. If not, please advise. Thanks. --Vahid From: Meghana Narasimhan To: users@kafka.a

Re: [DISCUSS] KIP-175: Additional '--describe' views for ConsumerGroupCommand

2017-07-21 Thread Vahid S Hashemian
Up to you, but I'd probably leave it for future work. -Jason On Thu, Jul 20, 2017 at 4:21 PM, Vahid S Hashemian < vahidhashem...@us.ibm.com> wrote: > Hi Jason, > > Regarding your comment about the current limitation on the information > returned for a consumer group, do

Re: [DISCUSS] KIP-175: Additional '--describe' views for ConsumerGroupCommand

2017-07-20 Thread Vahid S Hashemian
d the --offsets option at all, I don't have a strong opinion, but it seems to make the command semantics a little more consistent. -Jason On Tue, Jul 18, 2017 at 12:56 PM, Vahid S Hashemian < vahidhashem...@us.ibm.com> wrote: > Hi Jason, > > I updated the KIP based on your earlie

Re: [DISCUSS] KIP-175: Additional '--describe' views for ConsumerGroupCommand

2017-07-19 Thread Vahid S Hashemian
think? -Jason On Wed, Jul 19, 2017 at 4:39 PM, Vahid S Hashemian < vahidhashem...@us.ibm.com> wrote: > Hi Jason, > > Thanks for sharing your feedback on the updated KIP. > Your suggestions look good to me. > Do you see a problem with having the `--members` provide member >

Re: [DISCUSS] KIP-175: Additional '--describe' views for ConsumerGroupCommand

2017-07-19 Thread Vahid S Hashemian
her we need the --offsets option at all, I don't have a strong opinion, but it seems to make the command semantics a little more consistent. -Jason On Tue, Jul 18, 2017 at 12:56 PM, Vahid S Hashemian < vahidhashem...@us.ibm.com> wrote: > Hi Jason, > > I updated the KIP based

Re: [DISCUSS] KIP-175: Additional '--describe' views for ConsumerGroupCommand

2017-07-18 Thread Vahid S Hashemian
that behaves exactly like `--describe`. Thanks. --Vahid From: "Vahid S Hashemian" To: d...@kafka.apache.org Cc: Kafka Users Date: 07/17/2017 03:24 PM Subject:Re: [DISCUSS] KIP-175: Additional '--describe' views for ConsumerGroupCommand Hi Jaso

Re: [DISCUSS] KIP-175: Additional '--describe' views for ConsumerGroupCommand

2017-07-17 Thread Vahid S Hashemian
ecessary a --topics option is. The --partitions (or --offsets) option already shows the current assignment. Maybe --topics could be --subscription and just list the topics that the members subscribed to? Thanks, Jason On Mon, Jul 17, 2017 at 11:04 AM, Vahid S Hashemian < vahidhashem...@us.ibm

Re: [DISCUSS] KIP-175: Additional '--describe' views for ConsumerGroupCommand

2017-07-17 Thread Vahid S Hashemian
ts for the group even if the partition is not currently >> assigned. However, I don't think we need to try and fit members without >> any >> assigned partitions into that view. >> >> Thanks, >> Jason >> >> On Fri, Jul 7, 2017 at 10:49 AM, Vahid S H

Re: Kafka authorizer ACLs question

2017-07-11 Thread Vahid S Hashemian
Hi SK, Could you please take a look at this document ( https://developer.ibm.com/opentech/2017/05/31/kafka-acls-in-practice/) and confirm you performed the steps in Broker Setup on all brokers? Thanks. --Vahid From: Sruthi Kumar Annamneedu To: users@kafka.apache.org Date: 07/11/2017

Re: [DISCUSS] KIP-163: Lower the Minimum Required ACL Permission of OffsetFetch

2017-07-10 Thread Vahid S Hashemian
I'm bumping this up again to get some feedback, especially from some of the committers, on the KIP and on the note below. Thanks. --Vahid From: "Vahid S Hashemian" To: d...@kafka.apache.org Cc: "Kafka User" Date: 06/21/2017 12:49 PM Subject:Re:

Re: [DISCUSS] KIP-175: Additional '--describe' views for ConsumerGroupCommand

2017-07-07 Thread Vahid S Hashemian
--- > > Edoardo Comar > > IBM Message Hub > > IBM UK Ltd, Hursley Park, SO21 2JN > > "Vahid S Hashemian" wrote on 04/07/2017 > 17:11:43: > > > From: "Vahid S Hashemian" > > To: d...@kafka.apache.org >

Re: Mirroring multiple clusters into one

2017-07-07 Thread Vahid S Hashemian
at 1:18 AM, Vahid S Hashemian wrote: > > James, > > Thanks for sharing your thoughts and experience. > Could you please also confirm whether > - you do any encryption for the mirrored data? Not at the Kafka level. The data goes over a VPN. > - you have a many-to-one mir

Re: Mirroring multiple clusters into one

2017-07-06 Thread Vahid S Hashemian
to tune settings. Maybe that presentation will give more ideas about what to look at. https://www.slideshare.net/mobile/JiangjieQin/producer-performance-tuning-for-apache-kafka-63147600 -James Sent from my iPhone > On Jul 6, 2017, at 1:00 AM, Vahid S Hashemian wrote: > > The li

Mirroring multiple clusters into one

2017-07-05 Thread Vahid S Hashemian
The literature suggests running the MM on the target cluster when possible (with the exception of when encryption is required for transferred data). I am wondering if this is still the recommended approach when mirroring from multiple clusters to a single cluster (i.e. multiple MM instances). Is

Re: [DISCUSS] KIP-175: Additional '--describe' views for ConsumerGroupCommand

2017-07-04 Thread Vahid S Hashemian
- Edoardo Comar IBM Message Hub IBM UK Ltd, Hursley Park, SO21 2JN From: "Vahid S Hashemian" To: dev , "Kafka User" Date: 04/07/2017 00:06 Subject:[DISCUSS] KIP-175: Additional '--describe' views for Cons

[DISCUSS] KIP-175: Additional '--describe' views for ConsumerGroupCommand

2017-07-03 Thread Vahid S Hashemian
Hi, I created KIP-175 to make some improvements to the ConsumerGroupCommand tool. The KIP can be found here: https://cwiki.apache.org/confluence/display/KAFKA/KIP-175%3A+Additional+%27--describe%27+views+for+ConsumerGroupCommand Your review and feedback is welcome! Thanks. --Vahid

Re: [ANNOUNCE] Apache Kafka 0.11.0.0 Released

2017-06-28 Thread Vahid S Hashemian
+1. Thank you Ismael for your hard work on this release. --Vahid From: Guozhang Wang To: "d...@kafka.apache.org" Cc: "users@kafka.apache.org" , annou...@apache.org, kafka-clients Date: 06/28/2017 07:22 PM Subject:Re: [ANNOUNCE] Apache Kafka 0.11.0.0 Released Ismael,

Re: [VOTE] 0.11.0.0 RC2

2017-06-26 Thread Vahid S Hashemian
anks. --Vahid From: Ismael Juma To: d...@kafka.apache.org Cc: kafka-clients , Kafka Users Date: 06/26/2017 01:54 PM Subject:Re: [VOTE] 0.11.0.0 RC2 Sent by:isma...@gmail.com Hi Vahid, Can you please check which test fails first? The errors you mentioned can

Re: [VOTE] 0.11.0.0 RC2

2017-06-26 Thread Vahid S Hashemian
ter-tick-thread-1) I tested on a VM and a physical machine, and both give me a lot of errors like this. Thanks. --Vahid From: Ismael Juma To: Vahid S Hashemian Cc: d...@kafka.apache.org, kafka-clients , Kafka Users Date: 06/26/2017 03:53 AM Subject:Re: [VOTE] 0.11.0.0 R

Re: [VOTE] 0.11.0.0 RC2

2017-06-23 Thread Vahid S Hashemian
Hi Ismael, Not sure if my response on RC1 was lost or this issue is not a show-stopper: I checked again and with RC2, tests still fail in my Windown 64 bit environment. :clients:checkstyleMain [ant:checkstyle] [ERROR] C:\Users\User\Downloads\kafka-0.11.0.0-src\clients\src\main\java\org\apache

Re: Deleting/Purging data from Kafka topics (Kafka 0.10)

2017-06-23 Thread Vahid S Hashemian
. I was using the GetOffsetShell command to get the count, but seems that is not necessarily the right way .. What command should be used to get the count ? On Thu, Jun 22, 2017 at 8:34 PM, Vahid S Hashemian < vahidhashem...@us.ibm.com> wrote: > Hi Karan, > > Just to clarify, with `--time -1

Re: Deleting/Purging data from Kafka topics (Kafka 0.10)

2017-06-22 Thread Vahid S Hashemian
ld you please advise whether you're seeing a different behavior? Thanks. --Vahid From: "Vahid S Hashemian" To: users@kafka.apache.org Date: 06/22/2017 06:43 PM Subject:Re: Deleting/Purging data from Kafka topics (Kafka 0.10) Hi Karan, I think the issue is in verif

Re: Deleting/Purging data from Kafka topics (Kafka 0.10)

2017-06-22 Thread Vahid S Hashemian
lhost:6095 --topic topicPurge --time -1 --partitions 0,1,2 topicPurge:0:67 topicPurge:1:67 topicPurge:2:66 Any ideas on what the issue might be ? On Thu, Jun 22, 2017 at 1:31 PM, Vahid S Hashemian < vahidhashem...@us.ibm.com> wrote: > Hi Karan, > > The other broker config

Re: Deleting/Purging data from Kafka topics (Kafka 0.10)

2017-06-22 Thread Vahid S Hashemian
Hi Karan, The other broker config that plays a role here is "log.retention.check.interval.ms". For a low log retention time like in your example if this broker config value is much higher, then the broker doesn't delete old logs regular enough. --Vahid From: karan alang To: users@kaf

Re: [DISCUSS] KIP-163: Lower the Minimum Required ACL Permission of OffsetFetch

2017-06-21 Thread Vahid S Hashemian
+OffsetFetch#KIP-163:LowertheMinimumRequiredACLPermissionofOffsetFetch-AdditionalFoodforThought I just added some more details in that section, which I hope further clarifies the suggestion there. Thanks. --Vahid From: Vahid S Hashemian/Silicon Valley/IBM To: d...@kafka.apache.org Cc:

Re: [VOTE] 0.11.0.0 RC1

2017-06-20 Thread Vahid S Hashemian
Hi Ismael, Thanks for running the release. Running tests ('gradlew.bat test') on my Windows 64-bit VM results in these checkstyle errors: :clients:checkstyleMain [ant:checkstyle] [ERROR] C:\Users\User\Downloads\kafka-0.11.0.0-src\clients\src\main\java\org\apache\kafka\common\protocol\Errors.ja

Re: [DISCUSS] KIP-163: Lower the Minimum Required ACL Permission of OffsetFetch

2017-06-19 Thread Vahid S Hashemian
which would intuitively make > > more sense to me. Perhaps any expert could shed some light on this? :) > > > > Viktor > > > > On Tue, Jun 13, 2017 at 2:38 PM, Vahid S Hashemian < > > vahidhashem...@us.ibm.com <mailto:vahidhashem...@us.ibm.com>> wrote: > &g

Re: UNKNOWN_TOPIC_OR_PARTITION with SASL_PLAINTEXT ACL

2017-06-16 Thread Vahid S Hashemian
eeper - 3.4.6 Kafka - 10.2.0 Please let me know if you need any more info. I appreciate your time. Thanks Arunkumar Pichaimuthu, PMP ---- On Fri, 6/16/17, Vahid S Hashemian wrote: Subject: Re: UNKNOWN_TOPIC_OR_PARTITION with SASL_PLAIN

Re: UNKNOWN_TOPIC_OR_PARTITION with SASL_PLAINTEXT ACL

2017-06-16 Thread Vahid S Hashemian
On Thu, 6/15/17, Vahid S Hashemian wrote: Subject: Re: UNKNOWN_TOPIC_OR_PARTITION with SASL_PLAINTEXT ACL To: users@kafka.apache.org Date: Thursday, June 15, 2017, 6:49 PM Hi Arunkumar, Could you please take a look at this article: https://developer.ibm.com/opentech/2017/05/31

Re: UNKNOWN_TOPIC_OR_PARTITION with SASL_PLAINTEXT ACL

2017-06-15 Thread Vahid S Hashemian
run" password="Arun123"; }; Thanks Arunkumar Pichaimuthu, PMP On Thu, 6/15/17, Vahid S Hashemian wrote: Subject: Re: UNKNOWN_TOPIC_OR_PARTITION with SASL_PLAINTEXT ACL To: users@kafka.apache.org Date: Thursday, June 15, 2017, 6:16 PM Hi Arunkuma

Re: UNKNOWN_TOPIC_OR_PARTITION with SASL_PLAINTEXT ACL

2017-06-15 Thread Vahid S Hashemian
Hi Arunkumar, Have you given your Kafka consumer/producer necessary permissions to consume/produce messages? --Vahid From: Arunkumar To: Date: 06/15/2017 04:07 PM Subject:UNKNOWN_TOPIC_OR_PARTITION with SASL_PLAINTEXT ACL Hi I am setting up ACL with SALS_PLAINTEXT. My z

Re: kafka-consumer-groups.sh vs ConsumerOffsetChecker

2017-06-15 Thread Vahid S Hashemian
s not getting earlier) So, How do i interpret this ? is this a caching issue ? On Thu, Jun 15, 2017 at 1:01 PM, Vahid S Hashemian < vahidhashem...@us.ibm.com> wrote: > Hi Karan, > > The message "No topic available for consumer group provided" appears when > ther

Re: kafka-consumer-groups.sh vs ConsumerOffsetChecker

2017-06-15 Thread Vahid S Hashemian
Hi Karan, The message "No topic available for consumer group provided" appears when there is no topic under the consumer group in ZooKeeper (under /consumers/myGroup/owners/). Can you check whether the topic 'newBroker1' exists under this ZK path? Also, do you still get the rows below if you run

Re: [DISCUSS] KIP-163: Lower the Minimum Required ACL Permission of OffsetFetch

2017-06-13 Thread Vahid S Hashemian
Hi Michal, Thanks a lot for your feedback. Your statement about Heartbeat is fair and makes sense. I'll update the KIP accordingly. --Vahid From: Michal Borowiecki To: users@kafka.apache.org, Vahid S Hashemian , d...@kafka.apache.org Date: 06/13/2017 01:35 AM Subject:

Re: [DISCUSS] KIP-163: Lower the Minimum Required ACL Permission of OffsetFetch

2017-06-12 Thread Vahid S Hashemian
ind of weird there. best, Colin On Thu, Jun 8, 2017, at 11:29, Vahid S Hashemian wrote: > Hi all, > > I'm resending my earlier note hoping it would spark some conversation > this > time around :) > > Thanks. > --Vahid > > > > > From: "Vah

Re: [ANNOUNCE] New committer: Damian Guy

2017-06-09 Thread Vahid S Hashemian
Great news. Congrats Damian! --Vahid From: Guozhang Wang To: "d...@kafka.apache.org" , "users@kafka.apache.org" , "priv...@kafka.apache.org" Date: 06/09/2017 01:34 PM Subject:[ANNOUNCE] New committer: Damian Guy Hello all, The PMC of Apache Kafka is pleased to announce

[DISCUSS] KIP-163: Lower the Minimum Required ACL Permission of OffsetFetch

2017-06-08 Thread Vahid S Hashemian
Hi all, I'm resending my earlier note hoping it would spark some conversation this time around :) Thanks. --Vahid From: "Vahid S Hashemian" To: dev , "Kafka User" Date: 05/30/2017 08:33 AM Subject:KIP-163: Lower the Minimum Required ACL Permission

Re: [VOTE] KIP-162: Enable topic deletion by default

2017-06-05 Thread Vahid S Hashemian
+1 (non-binding) Thanks. --Vahid From: Gwen Shapira To: "d...@kafka.apache.org" , Users Date: 06/05/2017 09:38 PM Subject:[VOTE] KIP-162: Enable topic deletion by default Hi, The discussion has been quite positive, so I posted a JIRA, a PR and updated the KIP with the lat

Re: Trouble with querying offsets when using new consumer groups API

2017-05-30 Thread Vahid S Hashemian
Hi Jerry, The behavior you are expecting is implemented in 0.10.2 through KIP-88 ( https://cwiki.apache.org/confluence/display/KAFKA/KIP-88%3A+OffsetFetch+Protocol+Update ) and KAFKA-3853 (https://issues.apache.org/jira/browse/KAFKA-3853). Starting from this release when you query a consumer group

KIP-163: Lower the Minimum Required ACL Permission of OffsetFetch

2017-05-30 Thread Vahid S Hashemian
Hi, I started a new KIP to improve the minimum required ACL permissions of some of the APIs: https://cwiki.apache.org/confluence/display/KAFKA/KIP-163%3A+Lower+the+Minimum+Required+ACL+Permission+of+OffsetFetch The KIP is to address KAFKA-4585. Feedback and suggestions are welcome! Thanks. --V

Re: KIP-162: Enable topic deletion by default

2017-05-27 Thread Vahid S Hashemian
bypass confirmations, etc would be an improvement. However, I've seen no complaints about the current behavior of the command-line and the KIP doesn't change it at all. So I'd rather address things separately. Gwen On Fri, May 26, 2017 at 8:10 PM Vahid S Hashemian wrote: > Gw

Re: KIP-162: Enable topic deletion by default

2017-05-26 Thread Vahid S Hashemian
Gwen, thanks for the KIP. It looks good to me. Just a minor suggestion: It would be great if the command asks for a confirmation (y/n) before deleting the topic (similar to how removing ACLs works). Thanks. --Vahid From: Gwen Shapira To: "d...@kafka.apache.org" , Users Date: 05/26

Re: Kafka 0.10.1.0 - Question about kafka-consumer-groups.sh

2017-05-05 Thread Vahid S Hashemian
On Fri, May 5, 2017 at 5:30 PM, Vahid S Hashemian wrote: > Hi Subhash, > > The broker config that affects group offset retention is > "offsets.retention.minutes" (which defaults to 1 day). > If the group is inactive (i.e. has no consumer consuming from it) for t

Re: Kafka 0.10.1.0 - Question about kafka-consumer-groups.sh

2017-05-05 Thread Vahid S Hashemian
Hi Subhash, The broker config that affects group offset retention is "offsets.retention.minutes" (which defaults to 1 day). If the group is inactive (i.e. has no consumer consuming from it) for this long its offsets will be removed from the internal topic offset and it will not be listed in the

Re: why is it called kafka?

2017-05-02 Thread Vahid S Hashemian
This might help: https://www.quora.com/How-did-Kafka-get-its-name --Vahid From: The Real Plato To: users@kafka.apache.org Date: 05/02/2017 07:20 PM Subject:why is it called kafka? searching google and the docs have not revealed the answer -- -plato

Re: [ANNOUNCE] New committer: Rajini Sivaram

2017-04-24 Thread Vahid S Hashemian
Great news. Congrats Rajini! --Vahid From: Gwen Shapira To: d...@kafka.apache.org, Users , priv...@kafka.apache.org Date: 04/24/2017 02:06 PM Subject:[ANNOUNCE] New committer: Rajini Sivaram The PMC for Apache Kafka has invited Rajini Sivaram as a committer and we are ple

Re: [VOTE] 0.10.2.1 RC1

2017-04-14 Thread Vahid S Hashemian
+1 (non-binding) Built from the source and ran the quickstart successfully on Ubuntu, Mac, Windows (64 bit). Thank you Gwen for running the release. --Vahid From: Gwen Shapira To: d...@kafka.apache.org, Users Cc: Alexander Ayars Date: 04/12/2017 05:25 PM Subject:[VOTE]

Re: Problems with Kafka Quickstart (Windows)

2017-03-04 Thread Vahid S Hashemian
Hi Rajat, It would be great if you could file a JIRA for this in Kafka issue tracking site (https://issues.apache.org/jira/issues) so it can be easily tracked and addressed. Thanks. --Vahid From: "Rajat Yadav (Trainee, HO-IT)" To: users@kafka.apache.org Date: 03/04/2017 08:26 AM Su

Re: Simple data-driven app design using Kafka

2017-02-22 Thread Vahid S Hashemian
Pete, I think this excellent post covers what you are looking for: https://www.confluent.io/blog/tutorial-getting-started-with-the-new-apache-kafka-0-9-consumer-client/ --Vahid From: Peter Figliozzi To: users@kafka.apache.org Date: 02/22/2017 07:29 PM Subject:Simple data-driv

Re: KIP-122: Add a tool to Reset Consumer Group Offsets

2017-02-16 Thread Vahid S Hashemian
Thanks for this useful and the nice KIP write-up. Just a minor suggestion. Would it make sense to avoid repeating the term "reset" in the arguments? We already use the argument "reset-offset", so we may not need to repeat the term in the follow-on argument. For example, instead of kafka-consum

Re: possible bug or inconsistency in kafka-clients

2017-01-28 Thread Vahid S Hashemian
Could this be the same issue as the one reported here? https://issues.apache.org/jira/browse/KAFKA-4547 --Vahid From: Koert Kuipers To: users@kafka.apache.org Date: 01/27/2017 09:34 PM Subject:possible bug or inconsistency in kafka-clients hello all, i just wanted to poi

Re: [ANNOUNCE] New committer: Grant Henke

2017-01-11 Thread Vahid S Hashemian
Congrats Grant! --Vahid From: Sriram Subramanian To: users@kafka.apache.org Cc: d...@kafka.apache.org, priv...@kafka.apache.org Date: 01/11/2017 11:58 AM Subject:Re: [ANNOUNCE] New committer: Grant Henke Congratulations Grant! On Wed, Jan 11, 2017 at 11:51 AM, Gwen Shap

Re: [VOTE] 0.10.1.1 RC1

2016-12-19 Thread Vahid S Hashemian
Hi Guozhang, I also verified the quickstart on Ubuntu and Mac. +1 on those. On Windows OS there are a couple of issues for which the following PRs exist: - https://github.com/apache/kafka/pull/2146 (already merged to trunk) - https://github.com/apache/kafka/pull/2238 (open) These issues are not

Re: Deleting a topic without delete.topic.enable=true?

2016-12-09 Thread Vahid S Hashemian
Any chance the consumer process that consumes from that topic is still running while you are doing all this? --Vahid From: Tim Visher To: users@kafka.apache.org Date: 12/09/2016 08:26 AM Subject:Re: Deleting a topic without delete.topic.enable=true? I did all of that becaus

Re: [VOTE] 0.10.1.1 RC0

2016-12-08 Thread Vahid S Hashemian
+1 Build and quickstart worked fine on Ubuntu, Mac, Windows 32 and 64 bit. Thanks for running the release. Regards, --Vahid From: Guozhang Wang To: "users@kafka.apache.org" , "d...@kafka.apache.org" , kafka-clie...@googlegroups.com Date: 12/07/2016 02:47 PM Subject:[VOTE]

Re: Q about doc of consumer

2016-12-08 Thread Vahid S Hashemian
Ryan, The correct consumer command in the latest doc ( http://kafka.apache.org/quickstart#quickstart_consume) is bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning You used the "--zookeeper" parameter which implies using the old consumer, in which cas

Re: Writing a consumer offset checker

2016-12-02 Thread Vahid S Hashemian
There is a JIRA open that should address this: https://issues.apache.org/jira/browse/KAFKA-3853 Since it requires a change in the protocol, it's awaiting a KIP vote that's happening next week ( https://cwiki.apache.org/pages/viewpage.action?pageId=66849788). Once the vote is passed the code shoul

Re: --group flag for console consumer

2016-11-28 Thread Vahid S Hashemian
lease! Confluent is on the ball. On Wed, Nov 16, 2016 at 5:55 PM Gwen Shapira wrote: Thank you, Vahid! On Wed, Nov 16, 2016 at 1:53 PM, Vahid S Hashemian wrote: > I'll open a JIRA. > > Andrew, let me know if you want to take over the implementation. > Otherwise, I'd be happ

Re: --group flag for console consumer

2016-11-16 Thread Vahid S Hashemian
I'll open a JIRA. Andrew, let me know if you want to take over the implementation. Otherwise, I'd be happy to work on it. Thanks. --Vahid From: Gwen Shapira To: Users Date: 11/16/2016 01:23 PM Subject:Re: --group flag for console consumer Makes sense to me. Do you want t

Re: [ANNOUNCE] New committer: Jiangjie (Becket) Qin

2016-10-31 Thread Vahid S Hashemian
Congrats Becket! --Vahid From: Jason Gustafson To: Kafka Users Cc: d...@kafka.apache.org Date: 10/31/2016 10:56 AM Subject:Re: [ANNOUNCE] New committer: Jiangjie (Becket) Qin Great work, Becket! On Mon, Oct 31, 2016 at 10:54 AM, Onur Karaman < okara...@linkedin.com.inv

Re: [VOTE] 0.10.1.0 RC0

2016-10-07 Thread Vahid S Hashemian
; Change >the consumer side load balancing and distributed co-ordination to use a >consumer co-ordinator > > I thought this was already done in 2015. > > On Thu, Oct 6, 2016 at 4:55 PM, Vahid S Hashemian < > vahidhashem...@us.ibm.com > > wrote: > > >

Re: [VOTE] 0.10.1.0 RC0

2016-10-06 Thread Vahid S Hashemian
Jason, Thanks a lot for managing this release. I ran the quick start (Steps 2-8) with this release candidate on Ubuntu, Windows, and Mac and they mostly look great. These are some, hopefully, minor items and gaps I noticed with respect to the existing quick start documentation (and the updated

  1   2   >