Re: Using Kafka Stream in the cluster of kafka on one or multiple docker-machine/s

If you want to deploy a Kafka Streams application, then essentially you only need the (fat) jar of your application and a JRE in your container. In other words, it's the same setup you'd use to deploy *any* kind of Java application. You do not need to containerize "Kafka", which I assume you meant

Re: ThoughWorks Tech Radar: Assess Kafka Streams

Aye! Thanks for sharing, Jan. :-) On Wed, Mar 29, 2017 at 8:56 PM, Eno Thereska wrote: > Thanks for the heads up Jan! > > Eno > > > On 29 Mar 2017, at 19:08, Jan Filipiak wrote: > > > > Regardless of how usefull you find the tech radar. > > > > Well deserved! even though we all here agree that

Re: weird SerializationException when consumer is fetching and parsing record in streams application

Could this be a corrupted message ("poison pill") in your topic? If so, take a look at http://docs.confluent.io/current/streams/faq.html#handling-corrupted-records-and-deserialization-errors-poison-pill-messages FYI: We're currently investigating a more elegant way to address such poison pill pro

Re: Understanding ReadOnlyWindowStore.fetch

Jon, perhaps you could share the full integration test (or whatever code you're using to experiment)? We had a similar "how does X work?" question on StackOverflow recently [1], and it was much easier to help once we e.g. also understood how the test data was exactly being generated. -Michael

Re: weird SerializationException when consumer is fetching and parsing record in streams application

I think I am following the third option. My pipeline is: serde= Serdes.serdeFrom(new VSerializer(), new VDeserializer ()); builder.stream(Serdes.String(), serde, "advice-stream") .filter(new Predicate() { ...}) .groupByKey() .aggregate(new Initializer() {...}, new Aggregator() {...}, windo

Re: weird SerializationException when consumer is fetching and parsing record in streams application

Hi Sachin, Have you tried firing up a consumer (non-streams), seeking to that offset on the topic and seeing what the message is? Might be easier to debug? Like you say, it is failing in the consumer. Thanks, Damian On Thu, 30 Mar 2017 at 10:35 Sachin Mittal wrote: > I think I am following the

Re: weird SerializationException when consumer is fetching and parsing record in streams application

Well I try to read that offset via kafka-console-consumer.sh too and it fails with same error. So was wondering if I can apply any of the suggestion as per http://docs.confluent.io/3.2.0/streams/faq.html#handling-corrupted-records-and-deserialization-errors-poison-pill-messages If there is any ot

Re: weird SerializationException when consumer is fetching and parsing record in streams application

The suggestions in that FAQ won't help as it is too late, i.e., the message has already been received into Streams. You could create a simple app that uses the Consumer, seeks to the offset, and tries to read the message. If you did this in debug mode you might find out some more information. On

Re: weird SerializationException when consumer is fetching and parsing record in streams application

My streams application does run in debug mode only. Also I have checked the code around these lines at org.apache.kafka.common.utils.Utils.sizeDelimited(Utils.java:791) ~[kafka-clients-0.10.2.0.jar:na] at org.apache.kafka.common.record.Record.value(Record.java:268) ~[kafka-clients-0.10.2.0.jar

Re: weird SerializationException when consumer is fetching and parsing record in streams application

Sachin, you have this line: > builder.stream(Serdes.String(), serde, "advice-stream") Could the problem be that not the record values are causing the problem -- because your value deserializer does try-catch any such errors -- but that the record *keys* are malformed? The built-in `Serdes.Strin

Re: weird SerializationException when consumer is fetching and parsing record in streams application

Hmm, I re-read the stacktrace again. It does look like the value-side being the culprit (as Sachin suggested earlier). -Michael On Thu, Mar 30, 2017 at 3:18 PM, Michael Noll wrote: > Sachin, > > you have this line: > > > builder.stream(Serdes.String(), serde, "advice-stream") > > Could the pro

Re: weird SerializationException when consumer is fetching and parsing record in streams application

Sachin, there's a JIRA that seems related to what you're seeing: https://issues.apache.org/jira/browse/KAFKA-4740 Perhaps you could check the above and report back? -Michael On Thu, Mar 30, 2017 at 3:23 PM, Michael Noll wrote: > Hmm, I re-read the stacktrace again. It does look like the va

Re: Using Kafka Stream in the cluster of kafka on one or multiple docker-machine/s

Thank you Michael for the prompt response, really appreciate it! Best regards, Mina On Thu, Mar 30, 2017 at 4:50 AM, Michael Noll wrote: > If you want to deploy a Kafka Streams application, then essentially you > only need the (fat) jar of your application and a JRE in your container. > In othe

Re: weird SerializationException when consumer is fetching and parsing record in streams application

Sachin, Not sure if this will help, but you might want to try running https://cwiki.apache.org/confluence/display/KAFKA/System+Tools#SystemTools-DumpLogSegment on the partition that is causing you problems. Thanks Damian On Thu, 30 Mar 2017 at 14:29 Michael Noll wrote: > Sachin, > > there's a

kafka not throwing any exception not any response in call back

Hi , I am using kafka 10.2 and sometime my producer does not sends me any ACK and in that case data is also not pushed. Whenever I get ACK I am able to consume data. But this is happening quiet often with me and i have no clue why data is not being pushed inside and my request ends gracefully a

Re: weird SerializationException when consumer is fetching and parsing record in streams application

I am not sure if https://issues.apache.org/jira/browse/KAFKA-4740 is same issue as mine. What I suspect may be happening is that: at java.nio.Buffer.limit(Buffer.java:275) ~[na:1.8.0_122-ea] at org.apache.kafka.common.utils.Utils.sizeDelimited(Utils.java:791) ~[kafka-clients-0.10.2.0.jar:na]

Re: weird SerializationException when consumer is fetching and parsing record in streams application

Damian, Is there any way where I can just dump out the contents at a given offset from a given log segment file. I am not sure how DumpLogSegment helps. I already know the log segment file where that messag

Understanding producer connections

Hi, I am a new user of kafka. My client library is on 0.8.2 and I want to understand how one producer instance running on a multi threaded environment in a single jvm creating 300+ connections to the leader broker. The producer instance is closed only when the jvm exits. This producer is writi

Re: weird SerializationException when consumer is fetching and parsing record in streams application

Not that i'm aware of On Thu, 30 Mar 2017 at 16:00 Sachin Mittal wrote: > Damian, > Is there any way where I can just dump out the contents at a given offset > from a given log segment file. > > I am not sure how DumpLogSegment > < > https://cwiki.apache.org/confluence/display/KAFKA/System+Tools

Re: kafka not throwing any exception not any response in call back

In your producer have you set acks to be “all” and retries to be something higher than “0”? props.put("acks", "all"); props.put("retries", 2147483647); Also if the order of the messages matter you should also set the max inflight messages to 1 so retries happen and succeed before any other mes

Re: Custom stream processor not triggering #punctuate()

Does this fix the problem though? The docs indicate that new data is required for each *partition*, not topic. Overall I think the "stream time" notion is a good thing for a lot of use-cases, but some others definitely require wall-clock based windowing. Is something planned for this? -Tommy O

Authentication using SASL/Kerberos and znode permissions

Hello, I've implemented Authentication using SASL/Kerberos, on kafka_2.12-0.10.2.0. Everything is working fine, however, I've noticed that the kafka-acl znode is world readable / writable. So, couldn't anyone just bypass security by modifying the znode via zookeeper, granting their principle acc

Edit Permission on Wiki

I'd like to update https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example with a note that it's for the 0.8 consumer version. Can you give me commenting or editing access via my user https://cwiki.apache.org/confluence/display/~rehevkor5 Thanks! Shannon

Fwd: Authentication using SASL/Kerberos and znode permissions

Hello, I've implemented Authentication using SASL/Kerberos, on kafka_2.12-0.10.2.0. Everything is working fine, however, I've noticed that the kafka-acl znode is world readable / writable. So, couldn't anyone just bypass security by modifying the znode via zookeeper, granting their principle acc

Re: Custom stream processor not triggering #punctuate()

We plan to do a KIP for this. Should come up soon. Please follow dev list for details and participate in the discussion! -Matthias On 3/30/17 11:02 AM, Thomas Becker wrote: > Does this fix the problem though? The docs indicate that new data is > required for each *partition*, not topic. Overall

Re: Authentication using SASL/Kerberos and znode permissions

Never mind The issue was that I need to add authentication for kafka-acls.sh, it then create the znode with the correct permissions. Thanks, Rob kafka-acls.sh AUTH_JVM_OPTS=" -Djava.security.auth.login.config=/opt/kafka/security/kafka_client_jaas.conf " exec $(dirname $0)/kafka-run-class.

Kafka streams - Large time windows ?

Hi, I have a question about kafka streams : when doing a groupByKey on a kstream and then aggregate, is it possible to use a very large time window ? (For example, a 7 days long time window ? 30 days ?) What is the impact of this ? Is it something we should avoid ? Thanks, Walid. Envoyé de mon

Re: Kafka streams - Large time windows ?

I don't see any problem with this. You might want to increase window retention time though. It's configures for each window individually (default is 1 day IIRC). You set this via `.until()` when you define a window in your code. -Matthias On 3/30/17 2:27 PM, Walid Lezzar wrote: > Hi, > I have

Can multiple Kafka consumers read from the same partition of same topic by default?

Hi All, Can multiple Kafka consumers read from the same partition of same topic by default? By default, I mean since group.id is not mandatory I am wondering if I spawn multiple kafka consumers without specifying any group.id and give them the same topic and partition name will they be able to rea

Re: Can multiple Kafka consumers read from the same partition of same topic by default?

Yes, you can do that. -Matthias On 3/30/17 6:09 PM, kant kodali wrote: > Hi All, > > Can multiple Kafka consumers read from the same partition of same topic by > default? By default, I mean since group.id is not mandatory I am wondering > if I spawn multiple kafka consumers without specifying

Re: Can multiple Kafka consumers read from the same partition of same topic by default?

Hi, Thanks Can you explain little bit whats happening underneath? Does Kafka creates different group.id's by default when group.id's are not set ? When specified a group only one consumer can consumer from the group the for a particular partition right? Thanks! On Thu, Mar 30, 2017 at 9:00 PM, M

Re: Can multiple Kafka consumers read from the same partition of same topic by default?

It's explained in the JavaDocs: https://kafka.apache.org/0102/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html Also here: http://docs.confluent.io/current/clients/consumer.html If you have any more up question, just follow up :) -Matthias On 3/30/17 9:07 PM, kant koda

how to listen each partition with separate threads ?

Hi , I was thinking to listen each partition with separate thread in Kafka. But i get error saying : *org.apache.kafka.clients.consumer.KafkaConsumer@383ad023KafkaConsumer is not safe for multi-threaded accessSubscription to topics, partitions and pattern are mutually exclusive.*​ Is there a