Re: kafka 1.0.1?

thanks for info maciek On 18/12/2017 22:49, Ismael Juma wrote: Hi Maciek, I expect that 1.0.1 will be released some time in January. Ismael On Mon, Dec 18, 2017 at 10:42 AM, Maciek Próchniak wrote: Hello, are there plans to release version 1.0.1? We are affected by https://issues.apach

Re: Using Custom Partitioner in Streams

Could you please clarify, if i just choose to use low level processor api, what directs it to do re partitioning. I am not using them in conjunction with DSLs, I plan to use only them. Apart from sink processors, are there any conditions for re partitioning to occur. -Sameer. On Tue, Dec 19, 2017

In Kafka how to get the exact offset according producing time

I want to get offsets according to producing time by offsetsForTimes of KafkaConsumer. if current time is 2017/12/19 12:00:00, how to get offsets by time 2017/12/18 12:00:00. I got a problem when using offsetsForTimes, for example, topic:topic_test partition:0, the message produced before 2017/1

Kafka Streams 0.11 consumers losing offsets for all group.ids

I am running two kafka 0.11 clusters. Cluster A has two 0.11.0.0 brokers with 3 zookeepers. Cluster B has 4 0.11.0.1 brokers with 5 zookeepers. We have recently updated from running 0.8.2 client and brokers to 0.11. In addition, we added two kafka streams group.id that process data from one of the

Kafka Producer BenchMarking

Hello, I came across this really good article on kafka benchmarking but it has been with an older version( https://engineering.linkedin.com/kafka/benchmarking-apache-kafka-2-million-writes-second-three-cheap-machines) . We are planning to enable idempotence for exactly once guarantee . Has any o

Re: Using Custom Partitioner in Streams

Processor API does not do any automatic repartitioning. If you need to repartition data, you always need to do it "manually" by writeing to and reading back from a topic. -Matthias On 12/19/17 3:49 AM, Sameer Kumar wrote: > Could you please clarify, if i just choose to use low level processor a

Re: Kafka Producer BenchMarking

This should help (Section "The good news: Kafka is still fast!") https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/ -Matthias On 12/19/17 8:59 AM, Nishanth S wrote: > Hello, > I came across this really good article on kafka benchmarking but it has

Kafka streams for golang

Hi, Is anybody aware of any kafka streams work for golang? Thanks, -- V.

Re: Kafka streams for golang

Hi, Watching this space as well. Here is the current shortlist: https://github.com/lovoo/goka https://github.com/movio/kasper https://github.com/chrislusf/gleam Maybe Apache Beam will eventua

Re: Kafka Producer BenchMarking

Thank you ! On Tue, Dec 19, 2017 at 10:49 AM, Matthias J. Sax wrote: > This should help (Section "The good news: Kafka is still fast!") > > https://www.confluent.io/blog/exactly-once-semantics-are- > possible-heres-how-apache-kafka-does-it/ > > -Matthias > > On 12/19/17 8:59 AM, Nishanth S wrote

MirrorMaker optimization - num.streams vs. MM processes

Hello - i've a question on optimizing MirrorMaker setup... I've 2 topics, each with 12 partitions and i'm setting up MirrorMaker for the two topics. Is it preferable to have 3 Mirror Maker processes (on different m/c), with num.streams = 4 (or 8) OR Have a singe Mirror Maker process, with num.strea

Re: Running SSL and PLAINTEXT mode together (Kafka 10.2.1)

Anyone ? On Mon, Dec 18, 2017 at 7:25 AM, Darshan wrote: > Hi > > I am wondering if there is a way to run the SSL and PLAINTEXT mode > together ? I am running Kafka 10.2.1. We want our internal clients to use > the PLAINTEXT mode to write to certain topics, but any external clients > should use

Re: Kafka streams for golang

You can call the REST endpoints in KSQL from any programming language. I wrote some stuff in node.js to call KSQL this way and it works great. The results don't even have to go to a Kafka topic as the results of and POST to /query all stream using HTTP. -hans /** * Hans Jespersen, Principal Syst

Re: Running SSL and PLAINTEXT mode together (Kafka 10.2.1)

What exact issue are you running into with thta configs? -Jaikiran On 20/12/17 7:24 AM, Darshan wrote: Anyone ? On Mon, Dec 18, 2017 at 7:25 AM, Darshan wrote: Hi I am wondering if there is a way to run the SSL and PLAINTEXT mode together ? I am running Kafka 10.2.1. We want our internal

Re: MirrorMaker optimization - num.streams vs. MM processes

To achive high-availability, we need to run multiple instances of mirror maker process. On Wed, Dec 20, 2017 at 5:37 AM, sham singh wrote: > Hello - i've a question on optimizing MirrorMaker setup... > I've 2 topics, each with 12 partitions and i'm setting up MirrorMaker for > the two topics. >

Re: deduplication strategy for Kafka Streams DSL

Thanks a lot Matthias. Adding serde and state store as arguments in the left join solves the problem as described in jira. On Tue, Dec 19, 2017 at 12:18 AM, Matthias J. Sax wrote: > The Jira priority just increase by your report! > > Of course, we are always happy about pull request :D > > > -

Re: Faced this rebalancing error | Kafka 1.0.0 | exactly once

logs shows "failed due to org.apache.kafka.common.errors.RecordTooLargeException, returning UNKNOWN error code to the client (kafka.coordinator.group. GroupMetadataManager)" Try increasing max.message.bytes config on broker and/or enable compression on the offsets topic via the broker config `off

Re: Using Custom Partitioner in Streams

ok..Thanks Matthius for all the help. -Sameer. On Tue, Dec 19, 2017 at 11:14 PM, Matthias J. Sax wrote: > Processor API does not do any automatic repartitioning. If you need to > repartition data, you always need to do it "manually" by writeing to and > reading back from a topic. > > > -Matthia