Re: Need help in understanding bunch of rocksdb errors on kafka_2.10-0.10.1.1

Hi, Everything is understood and I will try out 0.10.2.0-rc0 shortly. However one this is not clear: Firstly i'd recommend you have different state directory configs for each application instance. Well I am running three separate instance of 4 threads each on three different machines. So each mac

Re: Format Kafka raw message to avro json format programatically + KafkaUtils

You say the data is Avro, but your Spark example specifies the key and value types as String and uses String decoders. If the Spark example works fine, you can just run the console consumer without the formatter and read the strings... On Mon, Feb 6, 2017 at 7:41 PM, kush batra wrote: > Hi Team,

Format Kafka raw message to avro json format programatically + KafkaUtils

Hi Team, I am trying to implement below kafka-console-consumer command(works well & output the intended json data) functionality in the form of program using spark streaming. kafka-console-consumer.sh --zookeeper host..com:2181,host..com:2181 --topic mytopic --formatter CustomAvroMessageF

Re: KIP-121 [Discuss]: Add KStream peek method

I've read the wiki and am unclear about the proposal. Can you provide something like a Javadoc for peek()? What would this method do? Also, forgive me if I'm missing an important point here, but can't I put the println statement in a map()? On Mon, Feb 6, 2017 at 5:48 PM, Matthias J. Sax wrote:

Re: [DISCUSS] KIP-120: Cleanup Kafka Streams builder API

Sounds good :) On Mon, Feb 6, 2017 at 5:40 PM, Matthias J. Sax wrote: > Gwen, > > thanks for your feedback. > > I completely agree that KStreamBuilder#merge() is miss placed and should > belong to KStream. However, I wanted to keep this KIP focus on one thing. > > As mentioned in a previous reply

users@kafka.apache.org

join in mailing list.

Re: KIP-121 [Discuss]: Add KStream peek method

Steven, Thanks for your KIP. I move this discussion to dev mailing list -- KIPs need to be discussed there (and can be cc'ed to user list). Can you also add the KIP to the table "KIPs under discussion": https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals#KafkaImprovemen

Re: [DISCUSS] KIP-120: Cleanup Kafka Streams builder API

Gwen, thanks for your feedback. I completely agree that KStreamBuilder#merge() is miss placed and should belong to KStream. However, I wanted to keep this KIP focus on one thing. As mentioned in a previous reply, we plan to have at least one more KIP to clean up DSL -- this future KIP should inc

Re: [DISCUSS] KIP-120: Cleanup Kafka Streams builder API

I like the cleanup a lot :) The cleaner lines between PAPI and DSL are very helpful to beginners who try to make sense of a long list of methods. I noticed that the "merge" method is still part of StreamBuilder. I thought it belongs inside KStream. Merge seems a lot like the SQL "union" operator,

KIP-121 [Discuss]: Add KStream peek method

Hello users@kafka, I would like to propose a small KIP on the Streams framework that simply adds a KStream#peek implementation. https://cwiki.apache.org/confluence/display/KAFKA/KIP-121%3A+Add+KStream+peek+method https://issues.apache.org/jira/browse/KAFKA-4720 https://github.com/apache/kafka/pul

SASL Security Roadmap in Kafka Connect

Hi, As written here: http://docs.confluent.io/3.1.2/connect/security.html#acl-considerations "Note that if you are using SASL for authentication, you must use the same principal for workers and connectors as only a single JAAS is currently supported on the client side at this time as described her

Re: [DISCUSS] KIP-120: Cleanup Kafka Streams builder API

Mathieu, Damian, thanks a lot for your feedback. It's very valuable to see what, how and why people are using certain methods right now. We don't want to lock people out (that's why we put this KIP on users list, too) and we want to keep the ability to mix-and-match DSL and Processor API. Furthe

Re: Kafka Streams delivery semantics and state store

Ok, so now I understand it. Thank you for all the clarifications :) Kind Regards Krzysztof Lesniewski On 06.02.2017 18:36, Matthias J. Sax wrote: Completely agree. Very good and deep analysis! -Matthias On 2/6/17 3:15 AM, Krzysztof Lesniewski, Nexiot AG wrote: Thank you Matthias for your ans

Next week: Streams Processing meetup with speakers from LinkedIn, Uber, and Optimizely

Hi all, LinkedIn will be hosting a Streams Processing meetup next Thursday, February 16 with speakers from LinkedIn, Uber, and Optimizely. Abstracts and event information is available at: https://www.meetup.com/Stream-Processing-Meetup-LinkedIn/events/237171557/ Hope to see you there! Ed Yakabo

Re: KIP-119: Drop Support for Scala 2.10 in Kafka 0.11

because: kafka will always be plagued by synchronisation issues between scala and java codebases: Scala consistently fails converting Java generics (necessitating manual conversion from scala to Java) +1 to drop Scala 2.10 (non-binding) Martin _

Re: KIP-119: Drop Support for Scala 2.10 in Kafka 0.11

+1 (non-binding) for KIP-119 I agree that supporting the 2 most recently release versions of Java and Scala is a reasonable guideline. However, I think it makes sense to have a brief discussion when dropping support for an old Java or Scala version. It's a fairly infrequent event, and it's worth

Re: log.retention.hours

Rebooting is not required for this config. After changing the configuration wait for the time as set for log.retention.check.interval.ms and then messages older than 1 hour should get deleted. Thanks, Akhilesh On Mon, Feb 6, 2017 at 12:16 PM, Алексей Федосов wrote: > Hello, > I've just started

Re: Kafka Streams delivery semantics and state store

Completely agree. Very good and deep analysis! -Matthias On 2/6/17 3:15 AM, Krzysztof Lesniewski, Nexiot AG wrote: > Thank you Matthias for your answer. > > In general I agree with what you are saying. I was however thinking > about this one case, in which I think the at-least-once delivery > se

Re: "auto offset commit failed"

Next time you successfully auto commit it should be fine. Michael > On 6 Feb 2017, at 12:38, Jon Yeargers wrote: > > This message seems to come and go for various consumers: > > WARN o.a.k.c.c.i.ConsumerCoordinator - Auto offset commit failed for > group : Commit offsets failed with retriabl

Re: Need help in understanding bunch of rocksdb errors on kafka_2.10-0.10.1.1

Hi Sachin, Firstly i'd recommend you have different state directory configs for each application instance. I suspect you are potentially hitting an issue where the partition assignment has changed, the state directory locks get released, and i directory gets removed just before the lock is taken o

Re: [DISCUSS] KIP-120: Cleanup Kafka Streams builder API

Hi Matthias, Thanks for the KIP. Should TopologyBuilder#build() and KStreamBuilder#build accept a StreamsConfig as an argument? Should we add a KStreamBuilder#topologyBuilder() for cases where people want to mix and match DSL & PAPI? Or do you think we already provide enough support for that via

log.retention.hours

Hello, I've just started working with the product apache kafka and from time to time I have different questions. Could you please clarify one moment - in the configuration file servers.properties I change value of the parameter log.retention.hours by 1. The new value of this parameter is applied

Re: [DISCUSS] KIP-120: Cleanup Kafka Streams builder API

Hi Matthias, I use a few of the methods that you're pointing out that will be deprecated and don't have an apparent alternative, so I wanted to just let you know what they are and what my use-cases are for them. First of all, I use a combination of DSL and PAPI in the same application very happil

Broker becoming unresponsive and bringing down the entire cluster down with it

Hi all, We're running Kafka and have several Samza jobs consuming and producing messages, as well as a KConnect job consuming messages and storing them in a database table. >From time to time, at random intervals (sometimes the cluster runs for days >without issues, sometimes only a few hours)

"auto offset commit failed"

This message seems to come and go for various consumers: WARN o.a.k.c.c.i.ConsumerCoordinator - Auto offset commit failed for group : Commit offsets failed with retriable exception. You should retry committing offsets. Since Im not tracking offsets - how would I go about retrying them?

Re: Kafka Streams delivery semantics and state store

Thank you Matthias for your answer. In general I agree with what you are saying. I was however thinking about this one case, in which I think the at-least-once delivery semantics may still be broken. However, I came to conclusion it does not really matter in that application. Here is the case:

Re: Need help in understanding bunch of rocksdb errors on kafka_2.10-0.10.1.1

Hi, Yes on first we have three machines with same data directory setting. So the state dir config is same in for each. If it helps this is the sequence of logs just before the thread shutting down stream-thread [StreamThread-3] Committing all tasks because the commit interval 3ms has el

Re: Need help in understanding bunch of rocksdb errors on kafka_2.10-0.10.1.1

Hi Sachin, The first exception - Is each instance of your streams app on a single machine running with the same state directory config? The second exception - i believe is a bug in 0.10.1 that has been fixed in 0.10.2. There has been a number of issues fixed in this area. Thanks, Damian On Mon,

Re: trying to understand CommitFailedException

Hi, I have debugged the problem further and looks like something is not right. Here are the logs from one of the thread stream-thread [StreamThread-3] Committing all tasks because the commit interval 3ms has elapsed stream-thread [StreamThread-3] Committing task 0_0 stream-thread [StreamThrea

Re: At Least Once semantics for Kafka Streams

Ah OK! Thanks! On Mon, Feb 6, 2017, 3:09 PM Eno Thereska wrote: > Oh, by "other" I meant the original one you started discussing: > COMMIT_INTERVAL_MS_CONFIG. > > Eno > > On 6 Feb 2017, at 09:28, Mahendra Kariya > wrote: > > > > Thanks Eno! > > > > I am just wondering what is this other commit

Re: At Least Once semantics for Kafka Streams

Oh, by "other" I meant the original one you started discussing: COMMIT_INTERVAL_MS_CONFIG. Eno > On 6 Feb 2017, at 09:28, Mahendra Kariya wrote: > > Thanks Eno! > > I am just wondering what is this other commit parameter? > > On Mon, Feb 6, 2017, 12:52 PM Eno Thereska wrote: > >> Hi Mahendr

Re: At Least Once semantics for Kafka Streams

Thanks Eno! I am just wondering what is this other commit parameter? On Mon, Feb 6, 2017, 12:52 PM Eno Thereska wrote: > Hi Mahendra, > > That is a good question. Streams uses consumers and that config applies to > consumers. However, in streams we always set enable.auto.commit to false, > and