Re: Kafka 9 version offset storage mechanism changes

Both are possible, but the 'new' consumer stores the offset in an __offset topic. On Tue, May 10, 2016 at 7:07 AM Snehalata Nagaje < snehalata.nag...@harbingergroup.com> wrote: > > > Hi All, > > > As per kafka 9 version, where does kafka store committed offset? > > is it in zookeeper or kafka bro

Re: Using JMS and Multi-Protocol support - ActiveMQ to Kafka

I think you are confusing wire-based vs. API-based messaging. ActiveMQ is API-based messaging where API is Java Messaging Service (JMS) hence restrictions to both sides (producer/consumer) + broker being Java-based (although it doesn't have to be in theory). Kafka, AMQP and others are wire-base

Kafka 9 version offset storage mechanism changes

Hi All, As per kafka 9 version, where does kafka store committed offset? is it in zookeeper or kafka broker? Also there is option to use offset storage outside kafka, does it mean , kafka will not depend on zookeepr for offset. Thanks, snehalata

[DISCUSS] KIP-59 - Proposal for a kafka broker command - kafka-brokers.sh

Hi All, This is to start off a discussion on the above KIP at  https://cwiki.apache.org/confluence/display/KAFKA/KIP-59%3A+Proposal+for+a+kafka+broker+command The proposal is to fill the void of a command line tool/utility that can provide information on the cluster and brokers in a Kafka clust

Re: Tool to look at what's stored in internal topics on kafka 0.9

Which deserializer class to use? On Mon, May 9, 2016 at 5:14 PM, Guozhang Wang wrote: > Henry, > > I used the ConsoleConsumer to read the __offset__ topic directly to get its > content during my tests. Reminder you need to set the deserializer > correctly and print both key and value. > > Guozha

Using JMS and Multi-Protocol support - ActiveMQ to Kafka

Hi We were wondering if we should move away from Active MQ and adopt Kafka but then I learnt that Kafka is not JMS and it does not support Multi-Protocols. Is that true? And if above is true, how such use cases (JMS and Multiprotocol) are handled in Kafka? Also, people who moved away from Activ

[VOTE] 0.10.0.0 RC4

Hello Kafka users, developers and client-developers, This is the first candidate for release of Apache Kafka 0.10.0.0. This is a major release that includes: (1) New message format including timestamps (2) client interceptor API (3) Kafka Streams. Since this is a major release, we will give people

Re: gobblin error

You are seeing an Hadoop error in Gobblin app, so the Kafka mailing list is probably not your best bet. That said, Gobblin started a MapReduce job which failed. You need to look at the job log and the task logs for MapReduce to find out what happened. Gwen On Sat, May 7, 2016 at 10:51 AM, Mudit

Re: Tool to look at what's stored in internal topics on kafka 0.9

Henry, I used the ConsoleConsumer to read the __offset__ topic directly to get its content during my tests. Reminder you need to set the deserializer correctly and print both key and value. Guozhang On Fri, May 6, 2016 at 6:14 AM, Dustin Cote wrote: > Hi Henry, > > You can also use any other c

Re: ERROR Processor got uncaught exception. (kafka.network.Processor)

Hi Buvana, What is the version of the broker? It looks like the state of connection counts for quotas became inconsistent, that is, a given client ip address has no connections according to the internal state, but a disconnection for that ip happened. Ismael On Tue, May 10, 2016 at 12:14 AM, Ram

Kafka Connect - Kinesis sink or source?

I have a couple of topics in a Kafka cluster that another team would like to process. They have a strong preference for consuming a Kinesis stream instead of a Kafka topic. This seems like a good fit for a Kafka Connect sink. I didn't see a Kinesis connector in the connector hub

ERROR Processor got uncaught exception. (kafka.network.Processor)

Hello, We are trying to understand the following error in one of the brokers in our cluster. It's a 3 broker cluster (all 3 servers are identical - Intel Xeon E5-2670 @2.6GHz, 8cores, 16 threads 64 GB RAM & 1 TB Disk), serving about 8000 topics, single partitioned & replication factor = 3. Ea

Re: reprocessing events from multiple topics

Thanks Ben! Nice to know I was on the right track :-) that Samza doc is very helpful too. On Tue, May 3, 2016 at 8:53 PM Benjamin Manns wrote: > Both of your ideas are doable. Another thing to keep in mind is that > depending on your data source, late arriving data will not be sorted in > front o

Re: [VOTE] 0.10.0.0 RC3

W00t! Thank you! On Fri, May 6, 2016 at 11:13 AM, Mark Grover wrote: > Thanks, Gwen. > > I was testing RC2 with Spark Streaming and found an issue that has a minor > fix. I think this issue exists with RC3 as well: > https://issues.apache.org/jira/browse/KAFKA-3669 > > On Thu, May 5, 2016 at 1:46

Re: Broker/client authentication questions

Giampaolo, You can set up two listeners on the broker with protocols SSL and SASL_SSL. And set *ssl.client.auth=required*, *security.inter.broker.protocol=SSL*. This performs SSL client authentication on the SSL channel and SASL authentication on the SASL_SSL channel. Inter-broker connections are

Make full use of Kafka parallelism

Hello guys, I have a question using the createMessageStreamsByFilter API. The API takes a regex and number of streams. In my use case, I have a regex that acts as filter to match topic names. Each matching topic has 6 partitions. Now I want to achieve max parallelism by having each thread re

Re: Kafka Connect misconfiguration. Need some help

Ok still really struggling with this. We have sped up the flush time quite a bit but still failing. We are seeing all three group members are assigned partitions and are assigned tasks. Then tasks start dropping off. The log line I think that indicates what is wrong is this: >[Thread-9] INFO Ma

Re: Invalid TimeStamp Error while running WordCountDemo - kafka-0.10.0

Thanks for the confirmation! Glad to hear it. On Mon, May 9, 2016 at 7:57 AM, Ramanan, Buvana (Nokia - US) < buvana.rama...@nokia.com> wrote: > Guozhang, > > I am able to confirm that with the 0.10.0 console producer (without your > one-line fix) producing to a new topic, there is no issue. > > S

New broker as follower for a topic replication

Hello, in order to try running Kafka in Kubernetes I'm facing the following problem ... Imagine that I start a cluster with a zookeeper instance and two kafka broker. I create a topic with one partition but replication factor 2 : I have broker1 as leader and broker2 as follower. All works great

RE: Invalid TimeStamp Error while running WordCountDemo - kafka-0.10.0

Guozhang, I am able to confirm that with the 0.10.0 console producer (without your one-line fix) producing to a new topic, there is no issue. Sorry for the delay in getting back to you on this. Regards, Buvana -Original Message- From: EXT Guozhang Wang [mailto:wangg...@gmail.com] Sen

Broker/client authentication questions

Hi to all, I successfully put my simple Kafka cluster under TSL following official documentation. So in current configuration, inter-broker communication is done through SSL (security.inter.broker.protocol=SSL) and client authentication is required ( ssl.client.auth=required). I would like to kno

Re: unknown (kafka) offsets after restart

I dug further into this and log.cleaner.enable is false by default in 0.8.2.x which pretty much explains all this. Fixing it now. Thanks! On 09.05.2016 08:17, Jörg Wagner wrote: Thank you James. The log cleaner log is empty and we didn't notice any issue there. The Topic however is ~174GB pe