Hey everyone,
I wanted to share a small tool I developed last weekend named Kafka Hawk.
Kafka Hawk monitors the __consumer_offsets topic in Kafka and reports on
the number of commits it sees from each consumer group and topic. It can
also optionally report information on the deltas between offset
I'm a +1 (non-binding) — This looks like it would have saved us a lot of
pain in an issue we had to debug recently. I can't go into details, but
figuring out how to achieve this effect gave me quite a headache. :)
On Mon, Nov 12, 2018 at 1:00 PM xiongqi wu wrote:
> Hi all,
>
> Can I have one mor
Thanks for the KIP.
Will this cap be a global cap across the entire cluster or per broker?
Either way the default value seems a bit high to me, but that could just be
from my own usage patterns. I’d have probably started with 500 or 1k but
could be easily convinced that’s wrong.
Thanks,
Matt
On
Hi there,
Thanks for the KIP.
We’ve run into issues with this at Mailchimp so something to address
consuming behavior would save us from having to always ensure we’re running
enough consumers that each consumer has only one partition (which is our
usual MO).
I wonder though if it would be simple
Hi there,
Thanks for this KIP.
What’s the thinking behind doing this in ProductionExceptionHandler versus
handling these cases in your serializer implementation?
On Mon, Dec 3, 2018 at 1:09 AM Kamal Chandraprakash <
kamal.chandraprak...@gmail.com> wrote:
> Hello dev,
>
> I hope to initiate th
rializer, then
> one cannot continue the stream processing by skipping the record.
> To continue, you may have to send a empty record serialized key/value (new
> byte[0]) to the downstream on hitting the error which may cause un-intended
> results.
>
>
>
>
>
> On Wed, Dec 5,
; - `AlwaysProductionExceptionHandler` ->
> > `AlwaysContinueProductionExceptionHandler`
> >
> > - `DefaultProductionExceptionHandler` is not mentioned
> >
> > - Why do you distinguish between `ClassCastException` and "any other
> > unchecked
so we can get this in Kafka 2.0 :)
On Tue, Apr 17, 2018 at 7:11 PM, Matt Farmer wrote:
> Hello all, I've updated a KIP again to add a few sentences about the
> general problem we were facing in the motivation section. Please let me
> know if there is any further feedback.
>
>
Hi Arjun,
I'm following this very closely as better error handling in Connect is a
high priority
for MailChimp's Data Systems team.
A few thoughts (in no particular order):
For the dead letter queue configuration, could we use deadLetterQueue
instead of
dlq? Acronyms are notoriously hard to keep
b.com/apache/kafka/pull/5002
Thanks!
On Wed, Apr 25, 2018 at 1:04 PM, Matt Farmer wrote:
> Bump!
>
> We're currently at 1 non-binding +1.
>
> Still soliciting votes here. =)
>
> On Wed, Apr 18, 2018 at 3:41 PM, Ted Yu wrote:
>
>> +1
>>
>> On Wed, Apr 18
into the external
> system partially, and then fail with a ConnectException. Since the
> framework has no way of knowing what was written and what was not, a retry
> here might cause the same data to written again into the sink.
>
> Best,
>
>
> On Mon, May 14, 2018 at 12
+1 (non-binding)
On Tue, May 15, 2018 at 4:26 AM, Edoardo Comar wrote:
> Hi,
> bumping the thread as the current vote count for this KIP is
> 2 binding +1
> 5 non-binding +1
>
> thanks, Edo
>
> On 8 May 2018 at 16:14, Edoardo Comar wrote:
> > Hi,
> > bumping the thread as the current vote count
Bumping this again as it's been languishing for a few weeks. Would love to
get further feedback (or know for sure that this won't happen).
On Mon, May 14, 2018 at 3:48 PM, Matt Farmer wrote:
> Bumping this thread.
>
> For anyone who needs a refresher the discussion thread is
That config setting will only work if there are no offsets stored in the
consumer offsets target.
Something I’ve done in the past is to make the application.id config setting
have a random string component to it. So have “my-app-name-[randomchars]” or
some such. This ensures that there are neve
Someone will need to correct me if I’m wrong, but my understanding is that a
topic log on disk is divided into segments. Compaction will occur when a
segment “rolls off” - so when a new active segment is created and the previous
segment becomes inactive.
Segments can be bounded by size and time
ion
> <https://kafka.apache.org/documentation/#compaction> of kafka topics.
>
> On Fri, Jan 19, 2018 at 12:36 PM, Matt Farmer wrote:
>
>> Someone will need to correct me if I’m wrong, but my understanding is that
>> a topic log on disk is divided into segments. Compacti
+1 from me (non-binding) =)
> On Jan 22, 2018, at 7:35 PM, Sönke Liebau
> wrote:
>
> All,
>
> this KIP has been discussed for quite some time now and I believe we
> addressed all major concerns in the current revision, so I'd like to
> start a vote.
>
> KIP can be found here:
> https://cwiki.
What’s the status of this? This is a pretty hard blocker for us to meet
requirements internally to deploy connect in a distributed fashion.
@Ewen - Regarding the concern of accessing information securely - has there
been any consideration of adding authentication to the connect api?
> On Jan 17
can have a
human investigate.
---
I would love thoughts on all of the above from anyone on this list.
Thanks,
Matt Farmer
open to?
> On Mar 19, 2018, at 10:00 PM, Ewen Cheslack-Postava wrote:
>
> Responses inline.
>
> On Mon, Mar 19, 2018 at 3:02 PM, Matt Farmer <mailto:m...@frmr.me>> wrote:
>
>> Hi everyone,
>>
>> We’ve been experimenting recently with some limited use
Hello all,
I am proposing KIP-275 to improve Connect's SinkTaskContext so that Sinks
can be informed
in their preCommit hook if the hook is being invoked as a part of a
rebalance or Connect
shutdown.
The KIP is here:
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=75977607
Pleas
cking that
enum
without breaking compatibility.
However, if we think multiple states here are imminent for some reason, I
would
be pretty easy to convince adding that would be worth the extra complexity!
:)
Matt
—
Matt Farmer | Blog <http://farmdawgnation.com/> | Twitter
<http://twitter
Thanks for this KIP. I can think of some ways we would apply this.
I, too, am ~ on the compatibility story though, however I'm not sure
which way I'd prefer we go at this moment.
On Thu, Mar 29, 2018 at 4:36 PM, Ismael Juma wrote:
> Thanks for the KIP. I think this is going in the right directio
JavaDoc to add a paragraph that explains this
> scenario. Thoughts?
>
> Randall
>
> On Wed, Mar 28, 2018 at 9:29 PM, Ted Yu wrote:
>
> > I looked at WorkerSinkTask and it seems using a boolean for KIP-275
> should
> > suffice for now.
> >
> > Thanks
> &g
Ugh
* I can update
I need more caffeine...
On Mon, Apr 2, 2018 at 11:01 AM, Matt Farmer wrote:
> I'm can update the rejected alternatives section as you describe.
>
> Also, adding a paragraph to the preCommit javadoc also seems like a
> Very Very Good Idea™ so I'll make
; will be pushed to the next release - all the more reason to work on the KIP
> and the implementation well before deadline.
>
> Randall
>
> On Tue, Mar 20, 2018 at 9:49 AM, Matt Farmer wrote:
>
> > Hi Ewen,
> >
> > Thanks for the thoughtful response. I’m happy to take
I have made the requested updates to the KIP! :)
On Mon, Apr 2, 2018 at 11:02 AM, Matt Farmer wrote:
> Ugh
>
> * I can update
>
> I need more caffeine...
>
> On Mon, Apr 2, 2018 at 11:01 AM, Matt Farmer wrote:
>
>> I'm can update the rejected alternatives
or retries? (We did this in the
> Elasticsearch connector using backoff with jitter; see
> https://github.com/confluentinc/kafka-connect-elasticsearch/pull/116 for
> details.)
>
> Best regards,
>
> Randall
>
> On Tue, Apr 3, 2018 at 8:39 AM, Matt Farmer wrote:
>
>
Hello all, I've updated a KIP again to add a few sentences about the
general problem we were facing in the motivation section. Please let me
know if there is any further feedback.
On Tue, Apr 3, 2018 at 1:46 PM, Matt Farmer wrote:
> Hey Randall,
>
> Devil's advocate sparr
Good afternoon/evening/morning all:
I'd like to start voting on KIP-275: Indicate "isClosing" in the
SinkTaskContext
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=75977607
I'm going to start preparing the patch we've been using internally for PR
and get it up for review later t
+1 (non-binding). TY!
On Thu, Apr 19, 2018 at 11:56 AM, tao xiao wrote:
> +1 non-binding. thx Ismael
>
> On Thu, 19 Apr 2018 at 23:14 Vahid S Hashemian
> wrote:
>
> > +1 (non-binding).
> >
> > Thanks Ismael.
> >
> > --Vahid
> >
> >
> >
> > From: Jorge Esteban Quilcate Otoya
> > To: dev@k
Bump!
We're currently at 1 non-binding +1.
Still soliciting votes here. =)
On Wed, Apr 18, 2018 at 3:41 PM, Ted Yu wrote:
> +1
>
> On Wed, Apr 18, 2018 at 12:40 PM, Matt Farmer wrote:
>
> > Good afternoon/evening/morning all:
> >
> > I'd like to star
Is it worth spelling out explicitly what the behavior is when two topics
have the same priority? I'm a bit fuzzy on how we choose what topics to
consume from right now, if I'm being honest, so it could be useful to
outline the current behavior in the background and to spell out how that
would chang
Oh, almost forgot, thanks for the KIP - I can see this being a very useful
addition. :)
On Wed, Aug 8, 2018 at 9:09 PM Matt Farmer wrote:
> Is it worth spelling out explicitly what the behavior is when two topics
> have the same priority? I'm a bit fuzzy on how we choose wha
PM Gwen Shapira wrote:
> Can you guys spell it out for me? I just don't really see when I want to
> subscribe to two topics but not get events from both at the same time.
> Is this a work-queue type pattern?
>
> On Wed, Aug 8, 2018 at 6:10 PM, Matt Farmer wrote:
>
> > O
In thinking on it, another solution for this is another consumer external
to the stream - but then we run into timing issues and complexity with
using a state store as the storage of record. :/
On Sun, Aug 12, 2018 at 9:34 AM Matt Farmer wrote:
> The work-queue use case is mostly how I see t
Ah, sorry, yes it does.
On Sun, Aug 12, 2018 at 2:58 PM wrote:
> Does this clarify ?
> --
> Nick
>
> On Aug 9, 2018, at 7:44 PM, n...@afshartous.com wrote:
>
> Since there are questions I changed the heading from VOTE to DISCUSS
>
> On Aug 8, 2018, at 9:09 PM, M
Given that voting and discussion have stalled out it seems like this is a
thing that folks aren't particularly interested in. I'll be moving the KIP
status to abandoned unless I hear an objection in the next day or so. :)
On Thu, May 31, 2018 at 12:39 PM Matt Farmer wrote:
> Bumpi
Congrats Dong!
On Sat, Sep 15, 2018 at 4:40 PM Bill Bejeck wrote:
> Congrats!
>
> On Sat, Sep 15, 2018 at 1:27 PM Jorge Esteban Quilcate Otoya <
> quilcate.jo...@gmail.com> wrote:
>
> > Congratulations!!
> >
> > El sáb., 15 sept. 2018 a las 15:18, Dongjin Lee ()
> > escribió:
> >
> > > Congratul
Hey everyone,
I'm a software engineer at MailChimp working on our Data Systems team. I'm
looking to file a KIP to improve the error handling hooks that Kafka
Streams exposes when producing messages. We've got a use case internally
that requires us to be able to ignore certain classes of errors (in
toward a solution we can contribute back. :)
Cheers,
Matt Farmer
; ? This way it is clear that class name should be specified.
>
> Cheers
>
> On Wed, Oct 18, 2017 at 2:40 PM, Matt Farmer wrote:
>
> > Hello everyone,
> >
> > This is the discussion thread for the KIP that I just filed here:
> > https://cwiki.apache.org/conf
perience.
>
>
>
> -Matthias
>
>
> On 10/18/17 4:20 PM, Matt Farmer wrote:
> > I’ll create the JIRA ticket.
> >
> > I think that config name will work. I’ll update the KIP accordingly.
> > On Wed, Oct 18, 2017 at 6:09 PM Ted Yu wrote:
> >
> >&
changed by the time this particular handler
runs. So I think the signature would change to something like:
ProductionExceptionHandlerResponse handle(final ProducerRecord<..> record,
final Exception exception)
Would this be acceptable?
On Thu, Oct 19, 2017 at 7:33 PM Matt Farmer wrote:
&g
I did some more digging tonight.
@Ted: It looks like the deserialization handler uses
"default.deserialization.exception.handler" for the config name. No
".class" on the end. I'm inclined to think this should use
"default.production.exception.handler".
On Fri,
a specific "error" Kafka topic, or
> record it as an app-level metrics (
> https://kafka.apache.org/documentation/#kafka_streams_monitoring) for
> monitoring.
>
> Guozhang
>
>
>
> On Fri, Oct 20, 2017 at 5:47 PM, Matt Farmer wrote:
t; We may probably try to write down at least the following handling logic
and
> see if the given API is sufficient for it
I've added some examples to the KIP. Let me know what you think.
Cheers,
Matt
On Mon, Oct 23, 2017 at 9:00 PM Matt Farmer wrote:
> Thanks for this feedback. I
; not passing the context in the handle function, we may not be implement the
> logic to send the fail records into another queue for future processing.
> Would people think that would be a big issue?
>
>
> Guozhang
>
>
> On Thu, Oct 26, 2017 at 12:14 PM, Matt Farmer wrote:
>
x27;t hear anything by tomorrow afternoon I'll probably start a
[VOTE] thread.
Thanks,
Matt
On Fri, Oct 27, 2017 at 7:33 PM Matt Farmer wrote:
> I can’t think of a reason that would be problematic.
>
> Most of the time I would write a handler like this, I either want to
> ignore
gt; compatible with your changes.
>
>
>
> Guozhang
>
>
> On Tue, Oct 31, 2017 at 10:59 AM, Matt Farmer wrote:
>
> > I've opened this pull request to implement the KIP as currently written:
> > https://github.com/apache/kafka/pull/4165. It still needs some
Hello all,
It seems like discussion around KIP-210 has gone to a lull. I've got some
candidate work underway for it already, so I'd like to go ahead and call it
to a vote.
For reference, the KIP can be found here:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-210+-+Provide+for+custom+erro
This seems like an A+ improvement to me.
On Fri, Nov 3, 2017 at 7:49 PM Guozhang Wang wrote:
> Hello folks,
>
> I have filed a new KIP on adding AdminClient into Streams for internal
> topic management.
>
> Looking for feedback on
>
> *
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-220
> wrote:
> >
> > > The vote should stay open for at least 72 hours. The bylaws can be
> found
> > > here https://cwiki.apache.org/confluence/display/KAFKA/Bylaws
> > >
> > > On Wed, Nov 1, 2017 at 8:09 AM, Matt Farmer wrote:
> > >
> >
n
> strategy, as both are the two most common cases. This class should also
> be part of public API (so users can just set in the config) with a
> proper name.
>
>
> Otherwise, I like the KIP a lot! Thanks.
>
>
> -Matthias
>
>
> On 11/1/17 12:23 AM, Matt Farmer wr
Hello, a bit later than I'd anticipated, but I've updated this KIP as
outlined above. The updated KIP is now ready for review again!
On Sat, Nov 4, 2017 at 1:03 PM Matt Farmer wrote:
> Ah. I actually created both of those in the PR and forgot to mention them
> by name in the
value is FAIL to
> indicate that this thread is going to shutdown and trigger the exception
> handler.
>
>
> Guozhang
>
>
> On Sun, Nov 5, 2017 at 6:09 AM, Matt Farmer wrote:
>
> > Hello, a bit later than I'd anticipated, but I've updated this KIP as
> &
M Guozhang Wang wrote:
> That makes sense.
>
>
> Guozhang
>
> On Sun, Nov 5, 2017 at 12:33 PM, Matt Farmer wrote:
>
> > Interesting. I'm not sure I agree. I've been bitten many times by
> > unintentionally shipping code that fails to properly impleme
IP should
> list all exceptions for which the handler is not called.
>
> WDYT?
>
>
> -Matthias
>
>
> On 11/10/17 2:56 PM, Matthias J. Sax wrote:
> > Just catching up on this KIP.
> >
> > One tiny comment: I would prefer to remove the "Always" from the
t if the fenced
exceptions are, indeed, self healing that we still invoke the handler but
ignore its result for those exceptions.
On Tue, Nov 14, 2017 at 9:37 AM Matt Farmer wrote:
> Hi there,
>
> Following up here...
>
> > One tiny comment: I would prefer to remove the "Always&qu
pt with this handler and exceptions you cannot would be
really error-prone and hard to keep correct.
- I'm happy to file a KIP for the creation of this new Exception type
if there is interest.
@ Matthias — What do you think about the above?
On Tue, Nov 14, 2017 at 9:44 AM Matt Farme
> for fatal exception.
>
>
>
> About the "always continue" case. Sounds good to me to remove it (not
> sure why we need it in test package?) and to rename the "failing"
> handler to "Default" (even if "default" is less descriptive and I wo
Hi Matthias,
I certainly have found the auto-generated names unwieldy while doing
cluster administration.
I will point out that your KIP doesn't outline what would happen if you
picked a name that resulted in a non unique topic name? What would be the
error handling behavior there?
On Wed, Nov 2
Bump! It's been three days here and I haven't seen any further feedback.
Eager to get this resolved, approved, and merged. =)
On Tue, Nov 28, 2017 at 9:53 AM Matt Farmer wrote:
> Hi there, sorry for the delay in responding. Last week had a holiday and
> several busy work days i
understanding of the code paths. If this is incorrect we can revisit.
Understood. That makes sense. We should explain this clearly in the KIP
and maybe log all other following exceptions at DEBUG level?
-Matthias
On 12/1/17 11:43 AM, Matt Farmer wrote:
> Bump! It's been three days
PM, Matt Farmer (m...@frmr.me) wrote:
Hey Matthias, thanks for getting back to me.
That's fine. But if we add it to `test` package, we don't need to talk
about it in the KIP. `test` is not public API.
Yes, that makes sense. It was in the KIP originally because I was, at one
point, p
p! Very clearly explained what
> the change will look like!
>
> Looks good to me. No further comments from my side.
>
>
> -Matthias
>
>
> On 12/5/17 9:14 AM, Matt Farmer wrote:
> > I have updated this KIP accordingly.
> >
> > Can you please take a look and
:
> Yes. A KIP needs 3 binding "+1" to be accepted.
>
> You can still work on the PR and get it ready to get merged -- I am
> quite confident that this KIP will be accepted :)
>
>
> -Matthias
>
> On 11/4/17 3:56 PM, Matt Farmer wrote:
> > Bump! I beli
!
On December 6, 2017 at 2:07:08 PM, Matthias J. Sax (matth...@confluent.io)
wrote:
+1
On 12/6/17 7:54 AM, Bill Bejeck wrote:
> +1
>
> On Wed, Dec 6, 2017 at 9:54 AM, Matt Farmer wrote:
>
>> Bumping this thread so it’s visible given that the conversation on
KIP-210
>
10:42 AM Matt Farmer wrote:
> Current tally here is 2 binding +1s, 4 non-binding +1s.
>
> The remaining remarks on the PR seem to mostly be nits, so I feel like
> we’ve converged a bit. If a committer could take a look and either leave
me
> some feedback on the discussion thread
Would it be possible to get official images that aren't Confluent branded
and only include the things that are maintained by under the scope of the
Apache PMC? IIRC, the Confluent images typically include extras that are
maintained solely by Confluent. (e.g. implementation of code to work with
Conf
n the onPartitionsAssigned handler?
Thanks,
Matt Farmer
Do we have an ETA on when y'all think 2.3.1 will land?
On Sat, Sep 28, 2019 at 1:55 PM Matthias J. Sax
wrote:
> There was a recent report about vulnerabilities of some dependent
> libraries: https://issues.apache.org/jira/browse/KAFKA-8952
>
> I think we should fix this for 2.3.1.
>
> Furthermor
Matt Farmer created KAFKA-6725:
--
Summary: Indicate "isClosing" in the SinkTaskContext
Key: KAFKA-6725
URL: https://issues.apache.org/jira/browse/KAFKA-6725
Project: Kafka
Issue
Matt Farmer created KAFKA-6086:
--
Summary: KIP-210 Provide for custom error handling when Kafka
Streams fails to produce
Key: KAFKA-6086
URL: https://issues.apache.org/jira/browse/KAFKA-6086
Project
Matt Farmer created KAFKA-6214:
--
Summary: Using standby replicas with an in memory state store
causes Streams to crash
Key: KAFKA-6214
URL: https://issues.apache.org/jira/browse/KAFKA-6214
Project
Matt Farmer created KAFKA-5207:
--
Summary: Addition of a way to manually revoke individual
partitions from a consumer
Key: KAFKA-5207
URL: https://issues.apache.org/jira/browse/KAFKA-5207
Project: Kafka
[
https://issues.apache.org/jira/browse/KAFKA-5207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Farmer resolved KAFKA-5207.
Resolution: Fixed
Heh, it was pointed out to me that assignment is an overwrite operation, not a
77 matches
Mail list logo
