Is there a summary of which blockers were fixed in RC0 somewhere?
On Mon, Jun 19, 2017 at 1:41 PM, Eno Thereska
wrote:
> +1 (non-binding) passes Kafka Streams tests.
>
> Thanks,
> Eno
> > On 19 Jun 2017, at 06:49, Magnus Edenhill wrote:
> >
> > +1 (non-binding)
> >
> > Passes librdkafka integra
dn't expect this feature to be perfect in this
release. We expect to test this this week though.
Given that the blockers fixed between RC0 and RC1 haven't changed much in
the areas we tested, I think the positive results here still apply.
Thanks
Tom Crayford
Heroku Kafka
On Thu, Jun 8, 2017
this:
https://github.com/apache/kafka/pull/3398 (the change is very small). Happy
to make a jira as well, if that makes sense.
Thanks
Tom Crayford
Heroku Kafka
On Tue, Jun 20, 2017 at 8:32 PM, Vahid S Hashemian <
vahidhashem...@us.ibm.com> wrote:
> Hi Ismael,
>
> Thanks fo
oesn't really affect the usability of the feature any way.
>
> Thanks,
> Apurva
>
> On Wed, Jun 21, 2017 at 11:12 AM, Tom Crayford
> wrote:
>
> > Hi there,
> >
> > I'm -1 (non-binding) on shipping this RC.
> >
> > Heroku has carried on perfo
afka setups and workloads,
but it does change significantly with what hardware you're using.
Because our performance tests run across cloud instances where there might
be contention and variability, we typically do a multiple runs on different
clusters with each setup before reporting results.
e on what should and shouldn't delay a release documented
somewhere?
Thanks
Tom Crayford
Heroku Kafka
On Thu, Jun 22, 2017 at 4:45 PM, Ismael Juma wrote:
> Hi Tom,
>
> We are going to do another RC to include Apurva's significant performance
> improvement when transactions
ough the logs etc.
All in all, this is shaping up to be a great release. We're going to
continue some further testing, but right now are heading towards a +1.
Thanks
Tom Crayford
Heroku Kafka
On Fri, Jun 23, 2017 at 2:36 AM, Ismael Juma wrote:
> A quick note on notable changes since
Kafka promises one thing and one thing only for backwards compatability,
which is that brokers with newer versions will always support older
clients. The inverse: old brokers with new clients is not true.
On Wed, Jun 28, 2017 at 6:18 AM, saurabh mimani
wrote:
> I am trying to use partitionsFor m
ore we could or should do there?
We're happy writing the code here, and want to continue contributing back,
I'd just love a hand thinking about what perf tests for compacted topics
should look like.
Thanks
Tom Crayford
Heroku Kafka
ent
> kafka.log:type=LogCleaner,name=max-buffer-utilization-percent
> kafka.log:type=LogCleaner,name=max-clean-time-secs
> kafka.log:type=LogCleanerManager,name=max-dirty-percent
>
> Manikumar
>
> On Tue, May 17, 2016 at 8:45 PM, Tom Crayford
> wrote:
>
> > Hi there,
cases, which leave tuning just to the 1% of folk who really really care.
The current defaults seem to be doing well here (barring the above note
about log compaction size), and any future changes here should keep this up.
Thanks
Tom Crayford
Heroku Kafka
On Fri, May 20, 2016 at 4:48 AM, Jay Krep
it, and/or
write a pull request for this change.
Thanks
Tom Crayford
Heroku Kafka
Nowak, James Cheng, Jason Gustafson,
> > > Jay Kreps, Jeff Klukas, Jeremy Custenborder, Jesse Anderson, jholoman,
> > > Jiangjie Qin, Jin Xing, jinxing, Jonathan Bond, Jun Rao, Ján Koščo,
> > > Kaufman Ng, kenji yoshida, Kim Christensen, Kishore Senji, Konrad,
> > &
Which consumer are you using? Can you see it connecting to the broker in
the broker logs? I'd recommend putting your configs for producer, consumer
and broker in a reply to assist debugging. Also please attach any relevant
code or log files.
Thanks
Tom Crayford
Heroku Kafka
On Wednesday, 2
t;just add more
tuning". I can't however, come up with any better mechanism (that doesn't
require tuning at all) without gross interactions with consumer offset
storage, so I remain a +1 here.
Thanks
Tom Crayford
Heroku Kafka
On Wednesday, 25 May 2016, Ewen Cheslack-Postav
+1 (non-binding)
On Wed, Jun 8, 2016 at 8:56 PM, Rajini Sivaram wrote:
> I would like to initiate the vote for KIP-55.
>
> The KIP details are here: KIP-55: Secure quotas for authenticated users
> <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-55%3A+Secure+Quotas+for+Authenticated+Use
+1 (non-binding)
On Mon, Jun 20, 2016 at 8:07 PM, Harsha wrote:
> +1 (binding)
> -Harsha
>
> On Mon, Jun 20, 2016, at 11:33 AM, Ismael Juma wrote:
> > +1 (binding)
> >
> > On Mon, Jun 20, 2016 at 8:27 PM, Dana Powers
> > wrote:
> >
> > > +1 -- thanks for the update
> > >
> > > On Mon, Jun 20, 2
+1 (non-binding)
On Thursday, 7 July 2016, Rajini Sivaram
wrote:
> I would like to initiate voting for KIP-55 (
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-55%3A+Secure+Quotas+for+Authenticated+Users
> ).
> Since the KIP has changed quite a lot since the last vote, we will discard
gment, then to
FetchResponse and FetchResponseSend (in case you want some pointers to some
code).
I may be missing something here, but there seems to be a deeper issue here,
Tom Crayford
Heroku Kafka
On Thu, Jul 21, 2016 at 10:49 AM, Andrey L. Neporada <
anepor...@yandex-team.ru> wrote:
> Hi a
This makes good sense to me, and seems to have very low amounts of downside
with large amounts of upside. +1
On Thursday, 4 August 2016, radai wrote:
> Hello,
>
> I'd like to initiate a discussion about
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 72%3A+Allow+Sizing+Incoming+Reques
Heroku has tested this using the same performance testing setup we used to
evaluate the impact of 0.9 -> 0.10 (see https://engineering.
heroku.com/blogs/2016-05-27-apache-kafka-010-evaluating-
performance-in-distributed-systems/).
We see no issues at all with them, so +1 (non-binding) from here.
Seeing as voting passed on this, can somebody with access update the wiki?
Is there code for this KIP in a PR somewhere that needs merging?
Thanks
Tom Crayford
Heroku Kafka
On Friday, 1 July 2016, Rajini Sivaram wrote:
> Thank you, Jun.
>
> Hi all,
>
> Please let me know
+1 (non-binding)
On Thu, Dec 1, 2016 at 12:11 AM, Apurva Mehta wrote:
> +1 (non-binding)
>
> On Wed, Nov 30, 2016 at 2:00 PM, Jason Gustafson
> wrote:
>
> > +1. Thanks for the KIP!
> >
> > On Wed, Nov 30, 2016 at 1:47 PM, Gwen Shapira wrote:
> >
> > > +1 (binding)
> > >
> > > On Wed, Nov 30, 2
+1. We've been running it in production for a long time and it's the right
default.
On Tue, Jan 3, 2017 at 7:17 PM, Ismael Juma wrote:
> Thanks for the KIP, +1 from me.
>
> Ismael
>
> On 3 Jan 2017 6:54 pm, "Ben Stopford" wrote:
>
> > Hi All
> >
> > Please find the below KIP which proposes chan
+1 (non-binding)
On Tue, 28 Feb 2017 at 11:42, Molnár Bálint wrote:
> +1
>
> 2017-02-28 12:17 GMT+01:00 Dongjin Lee :
>
> >
> >
> > +1.
> >
> >
> >
> > Best,
> >
> > Dongjin
> >
> >
> > --
> >
> >
> >
> >
> > Dongjin Lee
> >
> >
> >
> >
> >
> > Software developer in Line+.
> >
> > So interested
+1 (non-binding)
On Tue, Feb 28, 2017 at 6:56 PM, Apurva Mehta wrote:
> +1 (non-binding) for 0.11.0
>
> I do agree with Ismael's point that exactly-once should go through one
> release of stabilization before bumping the version to 1.0.
>
> Thanks,
> Apurva
>
> On Mon, Feb 27, 2017 at 7:47 PM, I
Colin,
Reminder that ACLs don't just apply to topics, but also to consumer groups
and cluster operations. It seems like having two sets of APIs, one of which
is (topics + acls) and one of which is just acls is more complex than just
having acls.
On Fri, Apr 14, 2017 at 12:17 AM, Colin McCabe wro
Thanks for running the release and being so transparent around plans, it's
a great boon to the community.
Looking at the plan (
https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+0.10.2.0), I
see KIP-4 and KIP-35 under "Open Issues", but not under "Planned KIP
Content". Are those KIPs
> 0.10.2.0 is released. We haven't made much progress on that front during
> this release cycle, but I hope we'll do better for the next release cycle.
>
> Ismael
>
> On Wed, Jan 4, 2017 at 4:08 PM, Tom Crayford wrote:
>
> > Thanks for running the release and
+1
On Wed, Jan 4, 2017 at 5:28 PM, Gwen Shapira wrote:
> +1 - thanks for tackling those old and painful bugs!
>
> On Wed, Jan 4, 2017 at 9:24 AM, Ben Stopford wrote:
> > Hi All
> >
> > We’re having some problems with this thread being subsumed by the
> [Discuss] thread. Hopefully this one will
It's not time for voting, but a huge +1 from me.
On Thu, Jan 5, 2017 at 8:25 PM, Vahid S Hashemian wrote:
> Thanks Ismael. I added that to the KIP.
>
>
>
>
> From: Ismael Juma
> To: dev@kafka.apache.org
> Date: 01/05/2017 12:16 PM
> Subject:Re: [DISCUSS] KIP-109: Old Consumer De
+1 (non-binding)
On Fri, Jan 6, 2017 at 6:58 PM, Colin McCabe wrote:
> Looks good. +1 (non-binding).
>
> What do you think about changing "protocol label" to "listener key"?
>
> best,
> Colin
>
>
> On Fri, Jan 6, 2017, at 09:23, Neha Narkhede wrote:
> > +1
> >
> > On Fri, Jan 6, 2017 at 9:21 AM
+1 (non-binding)
On Wed, Jan 11, 2017 at 11:12 PM, Stevo Slavić wrote:
> +1 (non-binding)
>
> On Thu, Jan 12, 2017 at 12:11 AM, Guozhang Wang
> wrote:
>
> > +1
> >
> > On Wed, Jan 11, 2017 at 12:09 PM, Jeff Widman wrote:
> >
> > > +1 nonbinding. We were bit by this in a production environment.
-1 (non-binding)
I've been slow at keeping up with the KIP and the discussion thread. This
is an exciting and quite complex new feature, which provides great new
functionality.
There's a thing I noticed missing from the KIP that's present in the google
doc - the doc talks about ACLs for Transacti
I said this in the voting thread, but can the authors include a section
about new ACLs if there are going to be ACLs for TransactionalId. It's
mentioned in the google doc, but I think new ACLs should be in a KIP
directly.
On Thu, Feb 2, 2017 at 2:42 PM, Ismael Juma wrote:
> Thanks for the respon
.
This bug is still present in 0.10.2.0 RC0
To quote KIP-13 on this matter:
> Note that in case of producers, we should be careful about retaining
references to large Message bodies because we could easily exhaust broker
memory if parking hundreds of requests.
Tom Crayford
On Thu, Feb 2, 2
This is great, thanks.
On Sat, Feb 4, 2017 at 12:22 PM, Dongjin Lee wrote:
> It also makes sense to me. Thanks for your great work, Ismael.
>
> Add to this, it seems like there should be some following work on utilizing
> some parts of the code by applying Java 8's features.
>
> Best,
> Dongjin
I think the updated wiki page makes sense with respect to ACLs, there seems
to be little potential for abuse there (other than the noted and known
issues).
I am going to note that this is a major complexity increase for Kafka, and
that I'm concerned about performance impact (the JVM is quite… peda
+1 (non-binding)
On Thu, Feb 9, 2017 at 6:37 PM, Apurva Mehta wrote:
> +1 (non-binding)
>
> On Thu, Feb 9, 2017 at 10:33 AM, Colin McCabe wrote:
>
> > +1 (non-binding)
> >
> >
> >
> > On Thu, Feb 9, 2017, at 07:31, Ismael Juma wrote:
> > > Hi everyone,
> > >
> > > Since everyone in the discuss
; > On Thu, Feb 2, 2017 at 6:49 AM, Ismael Juma
> wrote:
> > > >
> > > > > Hi Tom,
> > > > >
> > > > > That is a good point. During the discussion, it was agreed that
> > changes
> > > > to
> > > > > public in
Heroku tested this with our usual round of performance benchmarks, and
there seem to be no notable regressions in this RC that we can see (for a
sample on earlier regressions we found using these benchmarks during the
0.10.0.0 release,
https://engineering.heroku.com/blogs/2016-05-27-apache-kafka-01
+1 (non-binding)
I didn't explicitly state my voting status in my previous comment, sorry.
On Thu, Feb 16, 2017 at 1:59 PM, Rajini Sivaram
wrote:
> +1 (non-binding)
>
> Ran quick start and some security tests on binary, checked source build and
> tests.
>
> Thank you,
>
> Rajini
>
> On Thu, Feb
+1 (non binding)
On Friday, 22 April 2016, Jason Gustafson wrote:
> +1 (non-binding)
>
> On Thu, Apr 21, 2016 at 5:26 PM, Ewen Cheslack-Postava >
> wrote:
>
> > +1
> >
> > On Thu, Apr 21, 2016 at 5:25 PM, Guozhang Wang > wrote:
> >
> > > +1
> > >
> > > On Thu, Apr 21, 2016 at 5:07 PM, Gwen Sha
We've started running our usual suite of performance tests against Kafka
0.10.0.0 RC. These tests orchestrate multiple consumer/producer machines to
run a fairly normal mixed workload of producers and consumers (each
producer/consumer are just instances of kafka's inbuilt consumer/producer
perf tes
Yep, confirm.
On Thu, May 12, 2016 at 9:37 PM, Gwen Shapira wrote:
> Just to confirm:
> You tested both versions with plain text and saw no performance drop?
>
>
> On Thu, May 12, 2016 at 1:26 PM, Tom Crayford
> wrote:
> > We've started running our usual suite
more of them, could you create a jira and attach your findings
> there?
Yep. I only wrote the email because JIRA was in lockdown mode and I
couldn't create new issues.
>
> Thanks,
>
> Jun
>
>
>
> On Thu, May 12, 2016 at 1:26 PM, Tom Crayford > wrote:
>
>
Ok, I don't seem to be able to file a new Jira issue at all. Can somebody
check my permissions on Jira? My user is `tcrayford-heroku`
Tom Crayford
Heroku Kafka
On Fri, May 13, 2016 at 12:24 AM, Jun Rao wrote:
> Tom,
>
> We don't have a CSV metrics reporter in the prod
and you should be able to file a ticket now.
>
> Thanks,
> Ismael
>
> On Fri, May 13, 2016 at 12:17 PM, Tom Crayford
> wrote:
>
> > Ok, I don't seem to be able to file a new Jira issue at all. Can somebody
> > check my permissions on Jira? My user is `tcrayford-
anks,
>
> Jiangjie (Becket) Qin
>
> On Fri, May 13, 2016 at 6:19 AM, Tom Crayford
> wrote:
>
> > Ismael,
> >
> > Thanks. I'm writing up an issue with some new findings since yesterday
> > right now.
> >
> > Thanks
> >
> >
...
>
> On Fri, May 13, 2016 at 5:31 PM, Ismael Juma wrote:
> > Thanks Tom. I just wanted to share that I have been unable to reproduce
> > this so far. Please feel free to share whatever you information you have
> so
> > far when you have a chance, don't feel th
:
>
> https://github.com/apache/kafka/blob/0.10.0/docs/upgrade.html#L67
>
> Please feel free to suggest improvements.
>
> Thanks,
> Ismael
>
> On Sun, May 15, 2016 at 6:39 PM, Tom Crayford
> wrote:
>
> > I've been digging into this some more. It seems like
can skip the JIRA, just prefix the PR title with
> `MINOR:`.
>
> Thanks,
> Ismael
>
> On Sun, May 15, 2016 at 9:17 PM, Tom Crayford
> wrote:
>
> > How about this?
> >
> > Note: Due to the additional timestamp introduced in each
> message
&g
Are you running with unclean leader election on? Are you setting min in
sync replicas at all?
Can you attach controller and any other logs from the brokers you have?
They would be crucial in debugging this kind of issue.
Thanks
Tom Crayford
Heroku Kafka
On Thursday, 11 August 2016, Mazhar
imply the number of client objects you have running. It's under
your control.
Note that "did not result in lost or duplicate message" is not entirely
possible in the presence of failure. You have to pick one and optimize for
that (Kafka's docs often refer to this as "at
+1 (non binding)
On Fri, Aug 19, 2016 at 6:20 AM, Manikumar Reddy
wrote:
> +1 (non-binding)
>
> This feature help us control memory footprint and allows consumer to
> progress on fetching large messages.
>
> On Fri, Aug 19, 2016 at 10:32 AM, Gwen Shapira wrote:
>
> > +1 (binding)
> >
> > On Th
Hi,
I don't think I understand *why* you need this. Kafka is by default a
distributed HA system that balances data and leadership over nodes. Why do
you need to change this?
You could accomplish something like this with mirror maker, that may make
more sense.
Thanks
Tom Crayford
Heroku
that isn't
very typical for most Kafka usage.
That's for both produce and consume traffic. We do see increase amounts of
garbage collection under high consumer load with SSL enabled, but nothing
too horrible (brokers are well below the SLA of GC to keep their zookeeper
sessions alive).
Th
e that this is one of the reasons for the slowness.
>>
>>
>> Thanks for all your inputs. I was trying to avoid coming up with a
>> reproducer which closely matches our application usage scenario (given our
>> tight delivery timelines for the product), but from what y
On Tue, Sep 6, 2016 at 12:07 PM, Ismael Juma wrote:
> On Tue, Sep 6, 2016 at 11:35 AM, Tom Crayford
> wrote:
>
> > Our performance tests were run with the JVM producer and consumer, and we
> > didn't notice any real slowdown on that side either when enabling SSL.
>
I did some stuff like this recently with simple calls to `tc` (samples that
I used were in the README for https://github.com/tylertreat/comcast). The
only notable bug I found so far is that if you cut all the kafka nodes
entirely off from zookeeper for say, 60 seconds, then reconnect them, the
node
Heroku has tested this using the same performance testing setup we used to
evaluate the impact of 0.9 -> 0.10 (see https://
engineering.heroku.com/blogs/2016-05-27-apache-kafka-010-
evaluating-performance-in-distributed-systems/).
We see no issues at all with them on RC2.
On Thu, Oct 13, 2016 at
alent
problems ("how long does it spend processing any individual message from
the queue, broken down by message type").
These are minor improvements though, the addition of more metrics to the
controller is already going to be very helpful.
Thanks
Tom Crayford
Heroku Kafka
On Thu, Apr
it clear.
>
> Regarding locks, they are removed by KAFKA-5028, as you say. So, if I
> understand correctly, you are suggesting an event processing rate metric
> with event type as a tag? Onur and Jun, what do you think?
>
> Ismael
>
> On Thu, Apr 27, 2017 at 3:47 PM, Tom Cra
+1 (non-binding)
On Fri, 5 May 2017 at 00:28, Michael André Pearce <
michael.andre.pea...@me.com> wrote:
> +1 , really good to see better operational visibility being added to the
> broker.
>
> Sent from my iPhone
>
> > On 5 May 2017, at 03:34, Ismael Juma wrote:
> >
> > Hi everyone,
> >
> > It
+1 (non-binding)
On Fri, May 26, 2017 at 3:38 PM, Damian Guy wrote:
> +1
> Also agree with what Ismael said.
>
> On Fri, 26 May 2017 at 15:26 Ismael Juma wrote:
>
> > Thanks for the KIP, sounds good to me. One comment: not sure we need to
> add
> > the config to server.properties. Do we expect
Tom Crayford created KAFKA-5973:
---
Summary: ShutdownableThread catching errors can lead to partial
hard to diagnose broker failure
Key: KAFKA-5973
URL: https://issues.apache.org/jira/browse/KAFKA-5973
[
https://issues.apache.org/jira/browse/KAFKA-3894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15346463#comment-15346463
]
Tom Crayford commented on KAFKA-3894:
-
(disclaimer: I work with Tim)
It feels
Tom Crayford created KAFKA-3933:
---
Summary: Kafka OOM During Log Recovery Due to Leaked Native Memory
Key: KAFKA-3933
URL: https://issues.apache.org/jira/browse/KAFKA-3933
Project: Kafka
Issue
[
https://issues.apache.org/jira/browse/KAFKA-3933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366329#comment-15366329
]
Tom Crayford commented on KAFKA-3933:
-
That makes sense to me. Thanks for the poi
[
https://issues.apache.org/jira/browse/KAFKA-3933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tom Crayford reassigned KAFKA-3933:
---
Assignee: Tom Crayford
> Kafka OOM During Log Recovery Due to Leaked Native Mem
[
https://issues.apache.org/jira/browse/KAFKA-3933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15366721#comment-15366721
]
Tom Crayford commented on KAFKA-3933:
-
Done. I hope to work on a fix tomo
[
https://issues.apache.org/jira/browse/KAFKA-3933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15367625#comment-15367625
]
Tom Crayford commented on KAFKA-3933:
-
Ismael: I've pushed the PR he
Tom Crayford created KAFKA-3937:
---
Summary: Kafka Clients Leak Native Memory For Longer Than Needed
With Compressed Messages
Key: KAFKA-3937
URL: https://issues.apache.org/jira/browse/KAFKA-3937
Project
Tom Crayford created KAFKA-3955:
---
Summary: Kafka log recovery doesn't truncate logs on non-monotonic
offsets, leading to failed broker boot
Key: KAFKA-3955
URL: https://issues.apache.org/jira/browse/KAFKA
[
https://issues.apache.org/jira/browse/KAFKA-3894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15376858#comment-15376858
]
Tom Crayford commented on KAFKA-3894:
-
Jun:
#4 seems potentially very complex t
Tom Crayford created KAFKA-3976:
---
Summary: Kafka Controller gets stuck, potentially due to zookeeper
session id reuse
Key: KAFKA-3976
URL: https://issues.apache.org/jira/browse/KAFKA-3976
Project
Tom Crayford created KAFKA-4441:
---
Summary: Kafka Monitoring is incorrect during rapid topic creation
and deletion
Key: KAFKA-4441
URL: https://issues.apache.org/jira/browse/KAFKA-4441
Project: Kafka
Tom Crayford created KAFKA-4596:
---
Summary: KIP-73 rebalance throttling breaks on plans for specific
partitions
Key: KAFKA-4596
URL: https://issues.apache.org/jira/browse/KAFKA-4596
Project: Kafka
[
https://issues.apache.org/jira/browse/KAFKA-3894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15417675#comment-15417675
]
Tom Crayford commented on KAFKA-3894:
-
Hi Jun,
We're probably going to s
Tom Crayford created KAFKA-4084:
---
Summary: automated leader rebalance causes replication downtime
for clusters with too many partitions
Key: KAFKA-4084
URL: https://issues.apache.org/jira/browse/KAFKA-4084
80 matches
Mail list logo
