Hi all,
Here at Dropbox we're still (off and on) trying to get to the bottom of
the data loss that's been hitting our largest cluster during preferred
replica elections. It unfortunately has repeated a few times, so now we
have a question.
To make sure we're understanding, message commit status (
Hi,
reading about the Kafka protocol specification here
(https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol)
I see that the producer and fetch requests share a MessageSet (for messages
sent and received).
As it is defined, the MessageSet is [Offset MessageSize Mes
Hello,
I'd like to know the number of connections that Kafka clients establish. I mean
...
The producer establishes a TCP connection for each "leader" broker which has
the partition where the producer itself wants to send messages.
Is there any other connection to consider ?
The consumer e
Hi,
as far as I understand, calling `KStream.process` prevents the developer
from adding further operations to a `KStreamBuilder` [1], because its
return type is `void`. Good.
But it also prevents the developer from adding operations to its superclass
`TopologyBuilder`. In my case I wanted to add
Hi Nicolas,
Good question! I'm not sure why it is a terminal operation, maybe one of
the original authors can chip in. However, you could probably work around
it by using TopologyBuilder.addProcessor(...) rather then KStream.process
Thanks,
Damian
On Wed, 18 Jan 2017 at 13:48 Nicolas Fouché wro
You could possibly also use KStream.transform(...)
On Wed, 18 Jan 2017 at 14:22 Damian Guy wrote:
> Hi Nicolas,
>
> Good question! I'm not sure why it is a terminal operation, maybe one of
> the original authors can chip in. However, you could probably work around
> it by using TopologyBuilder.a
Hi,
We have a 3 node cluster with 0.9.0.1 version. The controller is reporting
that one of the brokers has a leader imbalance.
One of the topics with 120 partitions has Broker 0 and 2 acting as leaders
for all its partitions.None of the partitions have Broker 1 as their leader.
So the controller
Nicolas,
if I understand your question correctly you'd like to add further
operations after having called `KStream#process()`, which -- as you report
-- doesn't work because `process()` returns void.
If that's indeed the case, +1 to Damian's suggest to use
`KStream.transform()` instead of `KStrea
Anybody ever seen this before? Anybody have any ideas as to where I can start
troubleshooting?
From: Michael Noll
Sent: Wednesday, January 18, 2017 9:56:35 AM
To: users@kafka.apache.org
Subject: Re: Kafka Streams: how can I get the name of the Processor when
cal
Anybody ever seen this before? Anybody have any ideas as to where I can start
troubleshooting?
From: Zac Harvey
Sent: Tuesday, January 17, 2017 4:40:35 PM
To: users@kafka.apache.org
Subject: Re: Correlation Id errors for both console producer and consumer
Hi J
Sorry that last response was for a different thread - please ignore (and sorry!)
From: Michael Noll
Sent: Wednesday, January 18, 2017 9:56:35 AM
To: users@kafka.apache.org
Subject: Re: Kafka Streams: how can I get the name of the Processor when
calling `KStream.p
The reason I would not use `KStream.transform()` is that I want to call
`ProcessorContext.forward()` several times, to different children. These
children are sinks.
My use case: I need to route my beacons to different topics. Right now, I
use a series of `KStream.branch()` calls [1]. But would it b
Ho my, I'm dumb. One can give multiple predicates to `KStream.branch()`.
2017-01-18 17:18 GMT+01:00 Nicolas Fouché :
> The reason I would not use `KStream.transform()` is that I want to call
> `ProcessorContext.forward()` several times, to different children. These
> children are sinks.
> My use
Hello,
I'm attempting to upgrade an application from 0.8 to 0.10 broker / client
libs, and integrate kafka streams. I am currently using the following
producer / consumer configs:
Producer:
Properties props = new Properties();
props.put("bootstrap.servers", brokerList);
Hello, fellow Apache enthusiast. Thanks for your participation, and
interest in, the projects of the Apache Software Foundation.
I wanted to remind you that the Call For Papers (CFP) for ApacheCon
North America, and Apache: Big Data North America, closes in less than a
month. If you've been puttin
Nicolas,
here's some information I shared on StackOverflow (perhaps a bit outdated
by now, was back in Aug 2016) about how you can add a state store when
using KStreamBuilder: http://stackoverflow.com/a/39086805/1743580
-Michael
On Wed, Jan 18, 2017 at 5:18 PM, Nicolas Fouché wrote:
> The r
Hi, Mark,
Your understanding about HWM transition is correct. When the leader changes
due to preferred leader election, the new leader will have all the
committed messages, but a potentially stale HWM. The new leader won't do
any truncation to its local log though. Instead, it will try to commit a
Producer will not know the offset of the message(s) at the time they send to
the broker but they can receive that information back as confirmation of
successful publish.
-hans
> On Jan 18, 2017, at 1:25 AM, Paolo Patierno wrote:
>
> Hi,
>
>
> reading about the Kafka protocol specificati
Hi there,
Can anyone please answer to my below follow-up questions for Ewen's
responses.
Thanks
On Tue, 17 Jan 2017 at 00:28 Greenhorn Techie
wrote:
> Thanks Ewen for the detailed response. This is quite helpful and cleared
> some of my doubts. However, I do have some follow-up queries. Can yo
Hi Team
Could you please suggest us on the below Kafka servers issue whom can we reach
out to fix this issue and we already reached out to kafka-support (mailer
alias) and they don’t support the below mentioned servers and we would like to
explore who can help us on the below mentioned issue on
Yes I know so ... what's the value of the Offset field in the MessageSet when
producer sends messages ?
From: Hans Jespersen
Sent: Wednesday, January 18, 2017 11:38:16 PM
To: users@kafka.apache.org
Subject: Re: Kafka Protocol : about "Offset" in the MessageSet
P
21 matches
Mail list logo
