Re: Kafka for event sourcing architecture

Dave: Thanks for the suggestion, Bookkeeper looks interesting. I assume you meant "more high-level interface"? Chris: Thanks Chris, I actually came across a few of your talks and articles when researching this style of architecture (along with stuff by Greg Young and Martin Kleppmann) and really

Configuring client-side timeouts in the Java Producer so that send() doesn't block

Hello Kafka Users, We've been using the old Scala Producer for quite some time now. In the Scala Producer, the send() call throws an exception when Kafka is down. We catch this exception, and perform error handling by spooling the message locally, which another thread consumes and retries with an

Leader not available problem

Hello, I'm trying to run single Kafka broker, with few topics. Basically 1 broker, 1 partition per topic, 1 replica, few topics. I've been using spotify/kafka dockerhub image which apparently just downloads Kafka release (0.8.2.1 in my case) and start it with default config + advertised host setti

OffSet checker.

The command line tool that comes with Kafka one seems to check offsets in zookeeper. This does not seem to be related to this issue - As I am sure the offsets are stored in Kafka for burrow seem to retrieve it properly. Is there any easier way to

Re: Kafka for event sourcing architecture

So Kafka is a fine solution as part of an event-sourced story. It's not a simple solution, but it fits. Kakfa can store data for a long time and you shouldn't discount this, however, using it as the primary long-term data store might not be a good fit if we're talking storing raw events for years

Re: Configuring client-side timeouts in the Java Producer so that send() doesn't block

Hi Samuel, Your second example should work as well. Can you please include the code you are using to test the scenario and what Kafka version you are using (0.9.0.0 or 0.9.0.1, I guess)? Ismael On Wed, May 18, 2016 at 11:07 AM, Samuel Chase wrote: > Hello Kafka Users, > > We've been using the

Re: OffSet checker.

Maybe give it a try with the kafka-consumer-groups.sh tool and the --new-consumer flag: On Wed, May 18, 2016 at 3:40 AM, Sathyakumar Seshachalam < sathyakumar_seshacha...@trimble.com> wrote: > The command line tool that comes with Kafka one seems to check offsets in > zookeeper. This does not s

Re: Kafka for event sourcing architecture

The issue I have with that log.cleanup.policy approach is how it impacts with replication. Different replicas often have different sets of segment files. There are effectively two options then: 1. All replicas upload all their segments. This has issues with storage space, but is easy to implement.

Re: Leader not available problem

Hi, Did you check the critical Kafka metrics? http://docs.confluent.io/2.0.1/kafka/monitoring.html has a good set of them. We've seen a few issues with 0.8.X where the Controller gets stuck in an infinite loop (even on single broker clusters), which would possibly result in the case you see. Look

Re: OffSet checker.

I was just trying to do the same thing, but it does not seem to support ssl (or at least not in combination with the acl). I get similar errors as in https://issues.apache.org/jira/browse/KAFKA-3151 but when I used --command-config /home/kafka/consumer.properties to give the ssl properties it worke

Re: OffSet checker.

On Wed, May 18, 2016 at 7:12 AM, Gerard Klijs wrote: > I was just trying to do the same thing, but it does not seem to support ssl > (or at least not in combination with the acl). I get similar errors as in > https://issues.apache.org/jira/browse/KAFKA-3151 but when I used > --command-config /hom

Re: Kafka for event sourcing architecture

Hi Tom, There is, indeed, the problem with replication in case of the leader change for the partition. Hence, I think the best, best approach would to have Kafka emit events in case of: - partition leader change - offset file to be cleaned up This still leaves a lot of work for the ops people

Re: Kafka for event sourcing architecture

I have created the JIRA: https://issues.apache.org/jira/browse/KAFKA-3726 – Best regards,
 Radek Gruchalski 
ra...@gruchalski.com de.linkedin.com/in/radgruchalski Confidentiality: This communication is intended for the above-named person and may be confidential and/or legally privileged. If it

Re: Leader not available problem

Hi Tom, Thanks for your answer. No, I haven't checked the Kafka metrics yet, weren't actually aware of them. I will turn them on and will monitor to see what will happen upon next failure. In meantime I've also upgraded to 0.9, so according to what you said there is also a chance, that it wont ha

Kafka Node.js Integration Questions/Advice

We recently released kafka-java-bridge - new nodejs high level consumer. npm ,github Its tested stable and -- Reem Diab Software Engineer T: +972-74-700-4580

Re: Kafka Node.js Integration Questions/Advice

We recently released kafka-java-bridge - new nodejs high level consumer. npm ,github Its tested stable and -- Reem Diab Software Engineer T: +972-74-700-4580

download - 0.10.0.0 RC6

Hello, A naïve question. How do I download 0.10.0 RC6 (or RC5) version? I clicked on "Download zip" under 0.10.0 trunk @ github, downloaded kafka-0.10.0.zip today, compiled & started the server. Mbeans shows the version to be 0.10.0-SNAPSHOT and am not sure where / how to download RC6. Please

Re: download - 0.10.0.0 RC6

Hi From Gwen’s announcement of RC6 last night: > * Release artifacts to be voted upon (source and binary): > http://home.apache.org/~gwenshap/0.10.0.0-rc6/ Regards Ian. --- Ian Wrigley Director, Education Services Confluent, Inc Cell: (323) 819-4075 > On May 18, 2016, at 1:04 PM, Ramanan, Buv

SchemaException error on 0.10.0.0-snapshot

Setting up a basic word count job, similar to the one from confluent-2.1.0-alpha1 (WordCountJob). Mine does the same thing essentially, just within our namespace. Coming across this error though, which is strange because the code for the moment is identical with the exception of the topic name

RE: [COMMERCIAL] Re: download - 0.10.0.0 RC6

Ian, Thanks a lot for the prompt response. What is the difference between the following? 1) kafka-0.10.0.0-src.tgz 2) kafka_2.10-0.10.0.0.tgz 3) kafka_2.11-0.10.0.0.tgz I suppose the 1st one is the source and the 2nd & 3rd are binaries... I am not able to figure out the difference between 2nd

Re: [COMMERCIAL] Re: download - 0.10.0.0 RC6

[Removing dev@ to avoid cross-posting] The second and third are binaries build against two different versions of Scala. Ian. > On May 18, 2016, at 1:28 PM, Ramanan, Buvana (Nokia - US) > wrote: > > Ian, > > Thanks a lot for the prompt response. > > What is the difference between the followi

RE: [COMMERCIAL] Re: [COMMERCIAL] Re: download - 0.10.0.0 RC6

OK. Went ahead with source installation. Now I see the version to be 0.10.0.0 in MBeans, but the commitID is still unknown. Is that expected? Also, I expected the RC to be available in the version information. -Original Message- From: Ian Wrigley [mailto:i...@confluent.io] Sent

Re: Configuring client-side timeouts in the Java Producer so that send() doesn't block

Hello Ismael, On Wed, May 18, 2016 at 5:54 PM, Ismael Juma wrote: > Your second example should work as well. Can you please include the code > you are using to test the scenario and what Kafka version you are using > (0.9.0.0 or 0.9.0.1, I guess)? This is the code I'm using: ``` (with-open [p (

Re: KafkaStreams - java.lang.ClassCastException received when utilizing KStream.through()

Thanks, so much, Guozhang. It didn't work immediately when I tried end of day, so I deferred til this morning. It seems I also needed to invoke configure manually, or pass the props to the deserializer constructor. I am providing the revised example in case it helps any one. Thank you Liquan for po

Re: KafkaStreams - java.lang.ClassCastException received when utilizing KStream.through()

Fred, You are right, only default serdes provided through configs are auto-configured today. But we could auto-configure other serdes passed along side the topology builder as well. Do you want to file a JIRA to keep track of this? Guozhang On Wed, May 18, 2016 at 11:36 AM, Fred Patton wrote:

Re: [COMMERCIAL] Re: [COMMERCIAL] Re: download - 0.10.0.0 RC6

RC is never available in the version information, because the RC we vote on is identical to the version we release. This is Apache rules, not mine :) I am not sure about the MBeans - what is the commitID in previous versions? Gwen On Wed, May 18, 2016 at 10:41 AM, Ramanan, Buvana (Nokia - US) w

Re: KafkaStreams - java.lang.ClassCastException received when utilizing KStream.through()

Thanks, Guozhang, I'll file a JIRA this afternoon. --Fred On Wed, May 18, 2016 at 11:51 AM, Guozhang Wang wrote: > Fred, > > You are right, only default serdes provided through configs are > auto-configured today. But we could auto-configure other serdes passed > along side the topology builder

Re: KafkaStreams - java.lang.ClassCastException received when utilizing KStream.through()

Thanks! On Wed, May 18, 2016 at 12:38 PM, Fred Patton wrote: > Thanks, Guozhang, I'll file a JIRA this afternoon. --Fred > > On Wed, May 18, 2016 at 11:51 AM, Guozhang Wang > wrote: > > > Fred, > > > > You are right, only default serdes provided through configs are > > auto-configured today. Bu

Consumer group ACL limited to new consumer API?

I have been playing around with ACLs and was hoping to limit access to a topic and consumer group by IP, but was unable to get it working. Basically, I was able to Read from a topic as a consumer group that was not allowed. KIP-11 has the following line about consumer groups: In order to consume

Kafka and Custom Serializer Not being Called Issue.

Hello All, I am trying to consume an AVRO formatted message using the confluent.schemaregistry.client based serializer. The code is in Python. The message was posted to Kafka topic using the confluent AVRO encoder. But, when I try to consume the message using KafkaConsumer, I get the following

Question regarding enhancement for Apache Kafka

Hello, I Janagan Sivagnanasundaram from Sri Lanka, and pursuing final year undergraduate studies under software engineering at University of Colombo School of Computing, Sri Lanka. As a part of the fulfillment of the degree we have to do a final year software engineering project with 3 members. So

RE: Question regarding enhancement for Apache Kafka

Can you give some examples of what "content based" means? Thanks, Dave -Original Message- From: Janagan Sivagnanasundaram [mailto:janagan1...@gmail.com] Sent: Wednesday, May 18, 2016 1:06 PM To: d...@kafka.apache.org; users@kafka.apache.org Subject: Question regarding enhancement for