Re: KafkaStreams - java.lang.ClassCastException received when utilizing KStream.through()

Hi Fred, You probably need to cast the configs to Map before adding the specific.avro.reader config. Thanks, Liquan On Tue, May 17, 2016 at 5:29 PM, Guozhang Wang wrote: > Hello Fred, > > Thanks for reporting this. I looked through your code and found this is due > to a bug in io.confluent.exa

Re: Kafka for event sourcing architecture

A curious question: How does database + Kafka broker solution work? Would the consumer(s) load the database and then overlay the events? Or is it that the events are overlaid on the database and the database is queried? I am curious on how a complete solution would look like with "database + Kafka

Re: Kafka for event sourcing architecture

Oli, Kafka is only a partial solution. As I describe here ( http://www.slideshare.net/chris.e.richardson/hacksummit-2016-eventdriven-microservices-events-on-the-outside-on-the-inside-and-at-the-core/57) an event store is a hybrid of a database and a message broker. It is a database because it prov

[VOTE] 0.10.0.0 RC6

Hello Kafka users, developers and client-developers, This is the seventh (!) candidate for release of Apache Kafka 0.10.0.0. This is a major release that includes: (1) New message format including timestamps (2) client interceptor API (3) Kafka Streams. This RC was rolled out to fix an issue with

Re: kafka broker report "kafka.common.NotLeaderForPartitionException"

i find the same situation in stackoverflow: http://stackoverflow.com/questions/35857130/kafka-broker-removed-from-zookeeper-while-leader-election-occurs-error but there are no explaination 2016-05-18 12:02 GMT+08:00 Fredo Lee : > hi all > > we have 20 kafka brokers in on cluster. and this morning

Re: Increase number of topic in Kafka leads zookeeper fail

Hi there, Thank you for your valuable support now I changed the method as one topic with several partition for all devices and its solved my problem. *Thanks & Regards,* Anas A DBA, Trinity Mobility [image: facebook] [image: twitter]

Re: Kafka for event sourcing architecture

You might also look at the Apache bookkeeper project which is a distributed log, but a more low-level interface. > On May 17, 2016, at 19:57, Olivier Lalonde wrote: > > Thanks all for the detailed answer Tom and everyone else who replied. I > will look into Pintrest's secor. > > The use case I'm

kafka broker report "kafka.common.NotLeaderForPartitionException"

hi all we have 20 kafka brokers in on cluster. and this morning, we cannot find one kafka broker when using "kafka-topic.sh" to get topic information. we find lots of error: "[2016-05-18 09:59:59,998] ERROR [ReplicaFetcherThread-0-1005], Error for partition [33,58] to broker 1005:class k

kafka streams internal topic

Hi, I was reading about Kafka streams and trying to understand its programming model. Some observations that I wanted to get some clarity on.. 1) Joins & aggregations use an internal topic for shuffle. Source processors will write to this topic with the key used for join. Then it is free to commi

Kafka Windows Support

Hello, Have a question in installing Kafka on windows. Our server farm is totally windows based. I have a C++ application and an asp.net mvc application that need to exchange huge volume of fast data. We are contemplating on using Kafka for the data passing layer. We are not sure if Kafka is re

Kafka Consumer error handling

Hi Everyone, I wanted to what is the best and secure way of error handling for KafkaConsumer. I am using confluent's recommended consumer implementation. my delivery semantics is at least once. I am switching off the auto commit as well. Or I should just switch on the auto commit. The thing is I a

Re: Kafka for event sourcing architecture

Thanks all for the detailed answer Tom and everyone else who replied. I will look into Pintrest's secor. The use case I'm trying to achieve here is probably different than what most people seem to use Kafka for and it might be the wrong tool for the job. I'm actually don't really need high scalabi

Monitoring active threads of 0.8 consumer

Hi all, As one partition only allows one consumer thread to claim, there would be a case that some consumer threads are idle when we over assign consumer threads. Is there anyway I can monitor the number of active threads(not active threads in ExecutorService, but threads are actually consuming me

Re: KafkaStreams - java.lang.ClassCastException received when utilizing KStream.through()

Hello Fred, Thanks for reporting this. I looked through your code and found this is due to a bug in io.confluent.examples.streams.utils.SpecificAvroDeserializer. I will fix this bug in Confluent's repo, as for you to work around it, you can create your own SpecificAvroDeserializer with the correc

Re: Kafka for event sourcing architecture

Will do as soon as electricity is back. On edge at the moment. Get Outlook for iOS On Tue, May 17, 2016 at 5:01 PM -0700, "Christian Posta" wrote: Please create a JIRA with your thoughts. I'd be happy to help out with something like that. On Tue, May 17, 2016 at 4:57 PM, Radoslaw

Re: Kafka for event sourcing architecture

Please create a JIRA with your thoughts. I'd be happy to help out with something like that. On Tue, May 17, 2016 at 4:57 PM, Radoslaw Gruchalski wrote: > Not as far as I'm aware. I'd be happy to contribute if there is a desire > to have such feature. From experience with other projects, I know t

Re: Kafka for event sourcing architecture

Not as far as I'm aware. I'd be happy to contribute if there is a desire to have such feature. From experience with other projects, I know that without the initial pitch / discussion, it could be difficult to get such feature in. I can create a jira in the morning, no electricity again tonight :

Re: Kafka for event sourcing architecture

I have described a cold storage solution for Kafka: https://medium.com/@rad_g/the-case-for-kafka-cold-storage-32929d0a57b2#.kf0jf8cwv. Also described it here a couple of times. Thd potential solution seems rather straightforward. Get Outlook for iOS _ From: Luke

Re: Kafka for event sourcing architecture

+1 to your solution of log.cleanup.policy. Other brokers (ie, ActiveMQ) have a feature like that. Is there a JIRA for this? On Tue, May 17, 2016 at 4:48 PM, Radoslaw Gruchalski wrote: > I have described a cold storage solution for Kafka: > https://medium.com/@rad_g/the-case-for-kafka-cold-storag

Re: Kafka for event sourcing architecture

It's harder in Kafka because the unit of replication is an entire partition, not a single key/value pair. Partitions are large and constantly growing, where key/value pairs are typically much smaller and don't change in size. There would theoretically be no difference if you had one partition per k

JMX metrics count towards replication

Hi , Do MsgsInPerSecond , BytesInPerSecond and BytesOutPerSecond JMX metrics count replicated data coming in and out of the broker as well? Thanks Tushar

Slides and Video from Kafka Summit are now available

Thanks Gwen and the editing team. On Tuesday, 17 May 2016, Gwen Shapira > wrote: > Turns out that due to the live demo, that video required extra editing. > > We just uploaded the edited version, so you can go watch it :) > > On Fri, May 13, 2016 at 10:05 PM, Ben Davison > wrote: > > Hi Gwen, >

Re: [VOTE] 0.10.0.0 RC5

FYI, there's a blocker issue with this RC due to Apache licensing restrictions. One of Connect's dependencies transitively includes the findbugs annotations jar, which is used for static analysis. Luckily it doesn't affect functionality and looks like it can easily be filtered out. We also discover

Re: Slides and Video from Kafka Summit are now available

Turns out that due to the live demo, that video required extra editing. We just uploaded the edited version, so you can go watch it :) On Fri, May 13, 2016 at 10:05 PM, Ben Davison wrote: > Hi Gwen, > > Awesome stuff, can't wait to go through these -- I noticed that one link > didn't go anywhere

java.lang.ClassNotFoundException: example.producer.SimplePartitioner? For kafka 0.8.2

Hello, This is my first time trying kafka in Java. I was following this tutorial: https://cwiki.apache.org/confluence/display/KAFKA/0.8.0+Producer+Example for the sample code & maven dependencies. I tweaked the code a little for my needs and run the code in a Thread-based class with the main cla

KafkaStreams - java.lang.ClassCastException received when utilizing KStream.through()

Before, switching the example from to() to through() things seemed to work fine. Afterward, I get ClassCastExceptions from GenericRecord to SpecificRecord. Hopefully, someone can point out something quick and dumb so I can get my demo wrapped up. I never get past the through() method. I am using th

newbie: kafka 0.9.0.0 producer does not terminate after producer.close()

I wrote a little test client that reads from a file an publishes using the 0.9.0.0 API. I am contacting to an older 0.8.x sever. I am able to send messages how ever I noticed that once I am done reading the input file my test program hangs Any idea what I am doing wrong? Kind regards Andy publ

Re: Kafka for event sourcing architecture

I'm not sure why Kafka at least in theory cannot be used for infinite retention – any replicated database system would need to have a new node ingest all the data from failed node from its replicas. Surely this is no different in S3 itself. Why is this harder to do in Kafka than in other systems? T

Re: Mirrormaker schema exception

Thanks for the reply Gwen. I have verified that the source and destination clusters are both running the same version , 0.9.0.1-cp1. So the brokers and MirrorMaker are all on the same version. Can the mirrormaker not be of the same version ? MirrorMaker has been running fine since we upgraded to t

Re: Kafka Protocol API versions

Hi Dana, One comment below. On Tue, May 17, 2016 at 2:39 PM, Dana Powers wrote: > (myself). Asish or others can correct me if I'm wrong here, but I > believe the java client is not attempting to do this yet outside of a > single request flow (SASL authentication). > The Java client is not usin

Re: Kafka Protocol API versions

I'd also add that there is some discussion of the mixed-broker issue on the KIP-35 wiki page as well: https://cwiki.apache.org/confluence/display/KAFKA/KIP-35+-+Retrieving+protocol+version#KIP-35-Retrievingprotocolversion-Aclientdeveloperwantstoaddsupportforanewfeature Of course, since 0.9 (and e

Re: Kafka Protocol API versions

Hi Oleksiy, Ashish has just created a PR to add some information about how to use KIP-35: https://github.com/apache/kafka/pull/1395 Maybe you can have a look and leave a comment with areas that are not covered by the first draft. Ismael On Tue, May 17, 2016 at 1:34 PM, Oleksiy Krivoshey wrote

Kafka Protocol API versions

Given lots of protocol changes in 0.10 compared to 0.9, with many API methods now having up to 3 possible versions, what is the suggested way for the client library to coordinate with Kafka brokers? I can see there is ApiVersions request which is great, but what happens if Kafka cluster consists of

Re: Increase number of topic in Kafka leads zookeeper fail

Hi, On Monday, 16 May 2016, Abhaya P wrote: > I was reading a nice summary article by Jun Rao on the implications of # of > topics/partitions. > > http://www.confluent.io/blog/how-to-choose-the-number-of-topicspartitions-in-a-kafka-cluster/ > > There are many trade-offs to be considered, as it l

Re: Kafka for event sourcing architecture

Hi Oli, Inline. On Tuesday, 17 May 2016, Olivier Lalonde wrote: > Hi all, > > I am considering adopting an "event sourcing" architecture for a system I > am developing and Kafka seems like a good choice of store for events. > > For those who aren't aware, this architecture style consists in sto

Kafka for event sourcing architecture

Hi all, I am considering adopting an "event sourcing" architecture for a system I am developing and Kafka seems like a good choice of store for events. For those who aren't aware, this architecture style consists in storing all state changes of the system as an ordered log of events and building