Re: Kafka with Zookeeper behind AWS ELB

The problem with EIP it is a public ip. Another option is to have the secondary interface attached to the instance on start(or a bit later) with the private static ip, but we are investigating the possibility. On Wed 19. Jul 2017 at 23:38, Luigi Tagliamonte < luigi.tagliamont...@gmail.com> wrote:

Re: Consumer blocked when auth failed.

What log level do you have configured? You might bump up to DEBUG or TRACE and see if anything stands out. -Jason On Tue, Jul 18, 2017 at 7:59 PM, 陈江枫 wrote: > Hi, > > I was integrating Kafka with Spark, using DirectStream, when my > authentication fail, the stream just blocked. No log, no exc

Re: [DISCUSS] KIP-175: Additional '--describe' views for ConsumerGroupCommand

It makes sense. Thanks for clarifying. The KIP is updated based on your feedback. Thanks again. --Vahid From: Jason Gustafson To: d...@kafka.apache.org Cc: Kafka Users Date: 07/19/2017 05:06 PM Subject:Re: [DISCUSS] KIP-175: Additional '--describe' views for ConsumerGrou

Re: [DISCUSS] KIP-175: Additional '--describe' views for ConsumerGroupCommand

I was just thinking that the subscription/assignment information can be quite large (especially in MM use cases), so it would be nice to keep the default output concise. I'm also not thrilled about adding more options, but --verbose is sort of a standard one. What do you think? -Jason On Wed, Jul

Re: [DISCUSS] KIP-175: Additional '--describe' views for ConsumerGroupCommand

Hi Jason, Thanks for sharing your feedback on the updated KIP. Your suggestions look good to me. Do you see a problem with having the `--members` provide member subscription and assignment too, so we can avoid an additional `--verbose` option? Thanks. --Vahid From: Jason Gustafson To:

RE: Using JMXMP to access Kafka metrics

Jolokia is not really an option for me although I've looked at it. I just want to know if there's some way to enable using JMXMP to access the metrics pushed to JMX. -Original Message- From: Svante Karlsson [mailto:svante.karls...@csi.se] Sent: Wednesday, July 19, 2017 12:24 PM To: use

Re: [DISCUSS] Streams DSL/StateStore Refactoring

Just to add to Damian's points, I think the optimizations for KAFKA-5581 can be done as a second step after this API refactoring, since it should not require any further public API changes and hence to me a bit orthogonal to this KIP. To illustrate my thinking: ``` someOps.table(); // the library

Re: Kafka with Zookeeper behind AWS ELB

Hello Andrey, I see that the ELB is not going to help directly with the bug, but introduces a nice layer that makes zookeeper DNS management easier. Introducing and ELB I don't have to deal with keep DNS in sync for all the servers in the zk ensemble. For the moment I can use an HAproxy with EIP an

Re: Kafka with Zookeeper behind AWS ELB

Hi, I have just posted almost the same question in dev list. Zookeeper client resolves address only once, on start, introducing ELB won't really help here (ELBs can be replaced, which involved ip change), but I am eager to know if there is a solution for that. On Wed, 19 Jul 2017 at 23:08 Luigi Ta

Kafka with Zookeeper behind AWS ELB

Hello, Users! I'm designing a Kafka deployment on AWS and it's my first time working with Kafka and Zookeeper so I've collected a lot of info so far but also some questions that I would love to submit to a much expert audience like you. I have been experimenting with exhibitor and zookeeper in aut

Re: [DISCUSS] KIP-175: Additional '--describe' views for ConsumerGroupCommand

Hey Vahid, Thanks for the updates. Looks pretty good. A couple comments: 1. For the --state option, should we use the same column-oriented format as we use for the other options? I realize there would only be one row, but the inconsistency is a little vexing. Also, since this tool is working only

Consumers constantly rebalancing

Hi all, I am using kafka* 0.8.2* broker and a *high level consumer API.* *Topic properties:* Number of partitions = 150 Replication factor = 2 Brokers count in cluster = 4 *High level Consumer properties: * There is only one consumer group for this topic. I am running 150 parallel consumer thread

Re: Using JMXMP to access Kafka metrics

I've used jolokia which gets JMX metrics without RMI (actually json over http) https://jolokia.org/ Integrates nicely with telegraf (and influxdb) 2017-07-19 20:47 GMT+02:00 Vijay Prakash < vijay.prak...@microsoft.com.invalid>: > Hey, > > Is there a way to use JMXMP instead of RMI to access Kafk

Re: [DISCUSS] KIP-177 Consumer perf tool should count rebalance time

Ismael, I debated that also, but the main point was to make users aware of the rebalance latency (with KIP-134 in mind). I'm guessing no one would notice if it required another option. Note that the KIP does preserve the existing fields (and in the same order), so if it is parsed as generic csv dat

Using JMXMP to access Kafka metrics

Hey, Is there a way to use JMXMP instead of RMI to access Kafka metrics through JMX? I tried creating a JMXMP JMXConnector but the connect attempt just hangs forever. Thanks, Vijay

Re: KIP-162: Enable topic deletion by default

+1 On Sat, May 27, 2017 at 11:04 AM, Gwen Shapira wrote: > Thanks Vahid, > > Do you mind if we leave the command-line out of scope for this? > > I can see why adding confirmations, options to bypass confirmations, etc > would be an improvement. However, I've seen no complaints about the current

Re: KIP-162: Enable topic deletion by default

+1 On Wed, Jul 19, 2017 at 8:35 PM, Bill Bejeck wrote: > +1 > > On Wed, Jul 19, 2017 at 10:51 AM, Damian Guy wrote: > > > +1 > > > > On Wed, 31 May 2017 at 13:36 Jim Jagielski wrote: > > > > > +1 > > > > On May 27, 2017, at 9:27 PM, Vahid S Hashemian < > > > vahidhashem...@us.ibm.com> wrote: >

Re: Add me to this group

Hi Nikita, You need to send an email to the following address in order to subscribe : users-subscr...@kafka.apache.org <%20users-subscr...@kafka.apache.org> Kind regards, Tariq [image: http://] Tariq, Mohammad about.me/mti [image: http://] On Wed, Jul 19, 2017 at 6:3

Re: KIP-162: Enable topic deletion by default

+1 On Wed, Jul 19, 2017 at 10:51 AM, Damian Guy wrote: > +1 > > On Wed, 31 May 2017 at 13:36 Jim Jagielski wrote: > > > +1 > > > On May 27, 2017, at 9:27 PM, Vahid S Hashemian < > > vahidhashem...@us.ibm.com> wrote: > > > > > > Sure, that sounds good. > > > > > > I suggested that to keep comman

Re: [DISCUSS] KIP-177 Consumer perf tool should count rebalance time

I think this is a good chance although it's unfortunate that it's likely to break code that is parsing the output of the performance tool. Would it make sense to only enable this if an option is provided? Ismael On Mon, Jul 17, 2017 at 3:41 PM, Jason Gustafson wrote: > +Users > > Thanks for the

Re: KIP-162: Enable topic deletion by default

+1 On Wed, 31 May 2017 at 13:36 Jim Jagielski wrote: > +1 > > On May 27, 2017, at 9:27 PM, Vahid S Hashemian < > vahidhashem...@us.ibm.com> wrote: > > > > Sure, that sounds good. > > > > I suggested that to keep command line behavior consistent. > > Plus, removal of ACL access is something that

Want some help about admin client

Hi, My requirement is that i want to know which node of cluster is alive, something like health check, and we will create some topic too with the admin client java api . In health check we use adminClient.describeCluster() to list alive node, at the beginning that works, but after sometimes (allw

Add me to this group

Re: Broker Startup fail on a 2-node cluster setup

Just realised - it should never have worked or at least put some meaningful error message. https://zookeeper.apache.org/doc/trunk/zookeeperStarted.html#sc_InstallingSingleMode At least 3 to sustain a single point failure. On 19 July 2017 at 14:21, mosto...@gmail.com wrote: > To be honest, my a

Re: Kafka compatibility matrix needed

Hi, Are you referring to this topic level config? message.format.version Specify the message format version the broker will use to append messages to the logs. The value should be a valid ApiVersion. Some examples are: 0.8.2, 0.9.0.0, 0.10.0, check ApiVersion for more details. By setting a partic

Re: Broker Startup fail on a 2-node cluster setup

To be honest, my advice would be to consider docker - install docker - create a swarm with both nodes - deploy a zookeeper service with 3 instances - deploy a kafka service with 3 instances On 19/07/17 15:19, M. Manna wrote: Hello, Thanks for the response. So as a workaround, can I do the

Re: Broker Startup fail on a 2-node cluster setup

Hello, Thanks for the response. So as a workaround, can I do the following: 1) Node 1 - zookeeper 1 + 2 +3 (brokers 1,2,3) 2) Node 2 - zookeeper 4 + 5 + 6 (brokers 4,5,6) Do you recommend this way? Regards, On 19 July 2017 at 14:12, mosto...@gmail.com wrote: > AFAIK you cant deploy a 2 nod

Re: Broker Startup fail on a 2-node cluster setup

AFAIK you cant deploy a 2 node cluster, or you'll have split brains issues "out of the box" On 19/07/17 15:11, M. Manna wrote: Hello, After running an emulated 3 node cluster on my local pc, I am now trying to deploy a 2-node cluster on two of my remote machines. The following is the server c

Broker Startup fail on a 2-node cluster setup

Hello, After running an emulated 3 node cluster on my local pc, I am now trying to deploy a 2-node cluster on two of my remote machines. The following is the server configuration: broker.id=1 advertised.listeners=PLAINTEXT://0.0.0.0:9092 listener.security.protocol.map=PLAINTEXT:PLAINTEXT,SSL:SSL

Re: Kafka compatibility matrix needed

Also, make On Wed, Jul 19, 2017 at 5:10 AM, Ismael Juma wrote: > Hi Anish, > > What should happen in that case is that the broker down converts the > messages to a format supported by the older clients. We have tests for this > scenario so it would be good to understand what is different in your

Re: Kafka compatibility matrix needed

Hi Sachin, Yes, that should work as the broker will down convert the messages for the older consumer. The one thing to avoid is decreasing the message format version for an existing topic. That is not supported. As usual, you should test this in a non production environment first. Also, one addit

Re: Kafka compatibility matrix needed

Hi Anish, What should happen in that case is that the broker down converts the messages to a format supported by the older clients. We have tests for this scenario so it would be good to understand what is different in your environment. Please file a JIRA and provide as much information as possibl

Re: Avro Serialization & Schema Registry ..

In short, Avro serializers/deserializers provided by Confluent always integrate with (and thus require) Confluent Schema Registry. That's why you must set the `schema.registry.url` configuration for them. If you want to use Avro but without a schema registry, you'd need to work with the Avro API

Re: [DISCUSS] Streams DSL/StateStore Refactoring

Hi Jan, Thanks for your input. Comments inline On Tue, 18 Jul 2017 at 15:21 Jan Filipiak wrote: > Hi, > > > 1. To many overloads: > Currently, whenever a KTable is the result of an operation it gets and > override with stateStoreName, and StatestoreSupplier in case people want > to query that.

Re: Kafka compatibility matrix needed

Hi, So can anyone confirm if messages to topic produced by java client of 0.10.2 can be consumed by java client 0.8.2. Perhaps brokers support older versions of client, but would need some information on this case as it is not very clear. In our case we have control over the producer but not consu

Re: Kafka compatibility matrix needed

After upgrading Kafka 0.10.0 to 0.11.0, and changing the Message protocol to 0.11 on brokers, the consumers with version 0.8.2.1 started reporting Invalid message error logs. On Tue, Jul 18, 2017 at 6:37 PM Ismael Juma wrote: > Hi all, > > 0.8.x clients should work with 0.9.x, 0.10.x and 0.11.x