[jira] [Created] (KAFKA-12667) Incorrect error log on StateDirectory close

[Jan Justesen (Jira)]

Jan Justesen created KAFKA-12667:


 Summary: Incorrect error log on StateDirectory close
 Key: KAFKA-12667
 URL: https://issues.apache.org/jira/browse/KAFKA-12667
 Project: Kafka
  Issue Type: Bug
  Components: streams
Affects Versions: 2.6.1, 2.7.0
Reporter: Jan Justesen


{{In StateDirectory.close() an error is logged about unclean shutdown if all 
locks are in fact released, and nothing is logged in case of an unclean 
shutdown.}}

 
{code:java}
// all threads should be stopped and cleaned up by now, so none should remain 
holding a lock
if (locks.isEmpty()) {
 log.error("Some task directories still locked while closing state, this 
indicates unclean shutdown: {}", locks);
}

{code}
 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] KIP-334 Include partitions in exceptions raised during consumer record deserialization/validation

[Stanislav Kozlovski]

Hey all,

To revive this old KIP, Sarwar Bhuiyan has volunteered to take over
ownership.
He will continue to drive this KIP to approval and completion - I
understand that he will re-start the discussion with a new [DISCUSS] or
[VOTE] thread.

Thank you Sarwar!

Best,
Stanislav

On Fri, Jan 10, 2020 at 5:55 PM Gwen Shapira  wrote:

> Sorry for the super late reply, but since the vote thread showed up,
> I've read the KIP and have a concern.
> The concern was raised by Matthias Sax earlier, but I didn't see it
> addressed.
>
> Basically the current iteration of the KIP unifies deserialization
> errors with corruption. As was pointed out, these are not the same
> thing. Corruption means that the message itself (including envelope,
> not just the payload) is broken. De-serialization errors mean that
> either key or value serializers have a problem. It can even be a
> temporary problem of connecting to schema registry, I believe. Corrupt
> messages can only be skipped. De-serialization errors can (and
> arguably should) be retried with a different serializer. Something
> like Connect will need to skip corrupt messages, but messages with
> SerDe issues should probably go into a dead-letter queue.
>
> Anyway, IMO we need exceptions that will let us tell the difference.
>
> Gwen
>
> On Fri, Oct 11, 2019 at 10:05 AM Stanislav Kozlovski
>  wrote:
> >
> > Thanks Jason. I've edited the KIP with the latest proposal.
> >
> > On Thu, Oct 10, 2019 at 2:00 AM Jason Gustafson 
> wrote:
> >
> > > Hi Stanislav,
> > >
> > > Sorry for the late comment. I'm happy with the current proposal. Just
> one
> > > small request is to include an example which shows how a user could use
> > > this to skip over a record.
> > >
> > > I'd suggest pushing this to a vote to see if anyone else has feedback.
> > >
> > > Thanks,
> > > Jason
> > >
> > > On Sat, Jul 13, 2019 at 2:27 PM Stanislav Kozlovski <
> > > stanis...@confluent.io>
> > > wrote:
> > >
> > > > Hello,
> > > >
> > > > Could we restart the discussion here again?
> > > >
> > > > My last message was sent on the 3rd of June but I haven't received
> > > replies
> > > > since then.
> > > >
> > > > I'd like to get this KIP to a finished state, regardless of whether
> that
> > > is
> > > > merged-in or discarded. It has been almost one year since the
> publication
> > > > of the KIP.
> > > >
> > > > Thanks,
> > > > Stanislav
> > > >
> > > > On Mon, Jun 3, 2019 at 11:19 AM Stanislav Kozlovski <
> > > > stanis...@confluent.io>
> > > > wrote:
> > > >
> > > > > Do people agree with the approach I outlined in my last reply?
> > > > >
> > > > > On Mon, May 6, 2019 at 2:12 PM Stanislav Kozlovski <
> > > > stanis...@confluent.io>
> > > > > wrote:
> > > > >
> > > > >> Hey there Kamal,
> > > > >>
> > > > >> I'm sincerely sorry for missing your earlier message. As I open
> this
> > > > >> thread up, I see I have an unsent draft message about resuming
> > > > discussion
> > > > >> from some time ago.
> > > > >>
> > > > >> In retrospect, I think I may have been too pedantic with the
> exception
> > > > >> naming and hierarchy.
> > > > >> I now believe a single exception type of
> > > > `RecordDeserializationException`
> > > > >> is enough. Let's go with that.
> > > > >>
> > > > >> On Mon, May 6, 2019 at 6:40 AM Kamal Chandraprakash <
> > > > >> kamal.chandraprak...@gmail.com> wrote:
> > > > >>
> > > > >>> Matthias,
> > > > >>>
> > > > >>> We already have CorruptRecordException which doesn't extend the
> > > > >>> SerializationException. So, we need an alternate
> > > > >>> name suggestion for the corrupted record error if we decide to
> extend
> > > > the
> > > > >>> FaultyRecordException class.
> > > > >>>
> > > > >>> Stanislav,
> > > > >>>
> > > > >>> Our users are also facing this error. Could we bump up this
> > > discussion?
> > > > >>>
> > > > >>> I think we can have a single exception type
> > > > >>> FaultyRecordException/RecordDeserialization exception to capture
> both
> > > > >>> the errors. We can add an additional enum field to differentiate
> the
> > > > >>> errors
> > > > >>> if required.
> > > > >>>
> > > > >>> Thanks,
> > > > >>> Kamal Chandraprakash
> > > > >>>
> > > > >>> On Wed, Apr 24, 2019 at 1:49 PM Kamal Chandraprakash <
> > > > >>> kamal.chandraprak...@gmail.com> wrote:
> > > > >>>
> > > > >>> > Stanislav,
> > > > >>> >
> > > > >>> > Any updates on this KIP? We have internal users who want to
> skip
> > > the
> > > > >>> > corrupted message while consuming the records.
> > > > >>> >
> > > > >>> >
> > > > >>> > On Fri, Oct 19, 2018 at 11:34 PM Matthias J. Sax <
> > > > >>> matth...@confluent.io>
> > > > >>> > wrote:
> > > > >>> >
> > > > >>> >> I am not 100% familiar with the details of the consumer code,
> > > > however
> > > > >>> I
> > > > >>> >> tend to disagree with:
> > > > >>> >>
> > > > >>> >> > There's no difference between the two cases -- if (and only
> if)
> > > > the
> > > > >>> >> message is corrupt, it can't be deserialized.  If (and only
> if) it
> > > > >>> can

Re: [VOTE] KIP-334 Include partitions in exceptions raised during consumer record deserialization/validation

[Sarwar Bhuiyan]

Forgive me while I find my feet, but since this reached a VOTE stage, is it
really required to go through a DISCUSS thread again or can we continue to
address the issues on this thread since the issues were raised on this
thread? If I do have to create a new DISCUSS thread, do I just copy the
messages from the old thread to it to address them?

Thank you.

Sarwar

On Wed, Apr 14, 2021 at 9:18 AM Stanislav Kozlovski 
wrote:

> Hey all,
>
> To revive this old KIP, Sarwar Bhuiyan has volunteered to take over
> ownership.
> He will continue to drive this KIP to approval and completion - I
> understand that he will re-start the discussion with a new [DISCUSS] or
> [VOTE] thread.
>
> Thank you Sarwar!
>
> Best,
> Stanislav
>
> On Fri, Jan 10, 2020 at 5:55 PM Gwen Shapira  wrote:
>
>> Sorry for the super late reply, but since the vote thread showed up,
>> I've read the KIP and have a concern.
>> The concern was raised by Matthias Sax earlier, but I didn't see it
>> addressed.
>>
>> Basically the current iteration of the KIP unifies deserialization
>> errors with corruption. As was pointed out, these are not the same
>> thing. Corruption means that the message itself (including envelope,
>> not just the payload) is broken. De-serialization errors mean that
>> either key or value serializers have a problem. It can even be a
>> temporary problem of connecting to schema registry, I believe. Corrupt
>> messages can only be skipped. De-serialization errors can (and
>> arguably should) be retried with a different serializer. Something
>> like Connect will need to skip corrupt messages, but messages with
>> SerDe issues should probably go into a dead-letter queue.
>>
>> Anyway, IMO we need exceptions that will let us tell the difference.
>>
>> Gwen
>>
>> On Fri, Oct 11, 2019 at 10:05 AM Stanislav Kozlovski
>>  wrote:
>> >
>> > Thanks Jason. I've edited the KIP with the latest proposal.
>> >
>> > On Thu, Oct 10, 2019 at 2:00 AM Jason Gustafson 
>> wrote:
>> >
>> > > Hi Stanislav,
>> > >
>> > > Sorry for the late comment. I'm happy with the current proposal. Just
>> one
>> > > small request is to include an example which shows how a user could
>> use
>> > > this to skip over a record.
>> > >
>> > > I'd suggest pushing this to a vote to see if anyone else has feedback.
>> > >
>> > > Thanks,
>> > > Jason
>> > >
>> > > On Sat, Jul 13, 2019 at 2:27 PM Stanislav Kozlovski <
>> > > stanis...@confluent.io>
>> > > wrote:
>> > >
>> > > > Hello,
>> > > >
>> > > > Could we restart the discussion here again?
>> > > >
>> > > > My last message was sent on the 3rd of June but I haven't received
>> > > replies
>> > > > since then.
>> > > >
>> > > > I'd like to get this KIP to a finished state, regardless of whether
>> that
>> > > is
>> > > > merged-in or discarded. It has been almost one year since the
>> publication
>> > > > of the KIP.
>> > > >
>> > > > Thanks,
>> > > > Stanislav
>> > > >
>> > > > On Mon, Jun 3, 2019 at 11:19 AM Stanislav Kozlovski <
>> > > > stanis...@confluent.io>
>> > > > wrote:
>> > > >
>> > > > > Do people agree with the approach I outlined in my last reply?
>> > > > >
>> > > > > On Mon, May 6, 2019 at 2:12 PM Stanislav Kozlovski <
>> > > > stanis...@confluent.io>
>> > > > > wrote:
>> > > > >
>> > > > >> Hey there Kamal,
>> > > > >>
>> > > > >> I'm sincerely sorry for missing your earlier message. As I open
>> this
>> > > > >> thread up, I see I have an unsent draft message about resuming
>> > > > discussion
>> > > > >> from some time ago.
>> > > > >>
>> > > > >> In retrospect, I think I may have been too pedantic with the
>> exception
>> > > > >> naming and hierarchy.
>> > > > >> I now believe a single exception type of
>> > > > `RecordDeserializationException`
>> > > > >> is enough. Let's go with that.
>> > > > >>
>> > > > >> On Mon, May 6, 2019 at 6:40 AM Kamal Chandraprakash <
>> > > > >> kamal.chandraprak...@gmail.com> wrote:
>> > > > >>
>> > > > >>> Matthias,
>> > > > >>>
>> > > > >>> We already have CorruptRecordException which doesn't extend the
>> > > > >>> SerializationException. So, we need an alternate
>> > > > >>> name suggestion for the corrupted record error if we decide to
>> extend
>> > > > the
>> > > > >>> FaultyRecordException class.
>> > > > >>>
>> > > > >>> Stanislav,
>> > > > >>>
>> > > > >>> Our users are also facing this error. Could we bump up this
>> > > discussion?
>> > > > >>>
>> > > > >>> I think we can have a single exception type
>> > > > >>> FaultyRecordException/RecordDeserialization exception to
>> capture both
>> > > > >>> the errors. We can add an additional enum field to
>> differentiate the
>> > > > >>> errors
>> > > > >>> if required.
>> > > > >>>
>> > > > >>> Thanks,
>> > > > >>> Kamal Chandraprakash
>> > > > >>>
>> > > > >>> On Wed, Apr 24, 2019 at 1:49 PM Kamal Chandraprakash <
>> > > > >>> kamal.chandraprak...@gmail.com> wrote:
>> > > > >>>
>> > > > >>> > Stanislav,
>> > > > >>> >
>> > > > >>> > Any updates on this KIP? We have internal users who want to
>> skip
>> > > the

[VOTE] KIP-718: Make KTable Join on Foreign key unopinionated

[Marco Aurélio Lotz]

Hi all,

I would like to start a vote on KIP-718, which proposes to make KTable join
on foreign key unopinionated in terms of persistence method (currently it
forces RocksDB usage for subscription store and no other option is
available).

https://cwiki.apache.org/confluence/display/KAFKA/KIP-718%3A+Make+KTable+Join+on+Foreign+key+unopinionated

Many thanks,
Marco Lotz

Re: [DISCUSS] KIP-494 Connect REST Endpoint for Transformations (SMTs) and other Plugins

[Tom Bentley]

Hi Cyrus,

Thanks for the KIP. A few questions:

1. What status code does the /plugins/{plugin-type} endpoint return when an
unknown type is given?
2. The result of /connector-plugins is a list of objects with 'class',
'type' and 'version' properties. Presumably /plugins/connector is the same,
but what is the schema for the other plugin types?
3. You're not proposing an equivalent of the
/connector-plugins/{connectorType}/config/validate endpoint for
non-connector types, which I think makes sense, because you would validate
an SMT's config via its used by a connector, so the existing endpoint
suffices, right?
4. Would /connector-plugins eventually be deprecated in favour of
/plugins/connector, or do we expect it to remain in the API indefinitely
and accept that /connector-plugins and /plugins/connector provide identical
responses? If we had the intention to deprecate in the future maybe we
should add a /plugins/connector/config/validate endpoint now?

Many thanks,

Tom

On Tue, Apr 13, 2021 at 6:46 PM Cyrus Vafadari  wrote:

> Hello all,
>
> As the number of connect plugins, SMT's, etc have grown, I wanted to bump
> this thread to see if there is more interest in adding a Connect REST
> endpoint to get the current plugins in the worker.
>
> Thanks all, and thanks Chris for the initial feedback on this.
>
> On Mon, Feb 17, 2020 at 12:56 PM Cyrus Vafadari 
> wrote:
>
> > Thanks for the feedback, Chris.
> >
> > WRT Worker-plugins, I see your point that they aren't very useful for
> > people trying to create connectors to see what exists already, but I
> would
> > find them useful for things like making assertions against a docker image
> > to confirm the pluginpaths/classpaths are configured correctly. It is
> more
> > useful for "sanity check" kinds of things than exploring/browsing. That
> > said, I don't feel too strongly and if more feedback from the community
> > thinks they are better excluded then we can remove them.
> >
> > WRT camelCase vs hyphens, I see your point here, will update the KIP.
> >
> > WRT compatibility -- Good question -- I would expect that yes, new
> plugins
> > would match this format. I will add this explicitly in the KIP for
> clarity.
> > I do think it's a valuable, simple feature to have the worker able to
> > report new plugins.
> >
> > Thanks again!
> >
> > On Wed, Feb 12, 2020 at 4:44 PM Christopher Egerton  >
> > wrote:
> >
> >> Hi Cyrus,
> >>
> >> Thanks for the KIP!
> >>
> >> One quick question--I see the use case for being able to list
> >> per-connector
> >> plugins (SMTs, converters, and header converters) via REST API, but I'm
> >> not
> >> so sure about worker plugins (REST extensions and config providers).
> Since
> >> those are configured at startup for the worker, is there any advantage
> to
> >> being able to see the worker plugins available on a running worker? It's
> >> not like they can be added during the lifetime of that worker, and that
> >> information wouldn't be useful to anyone trying to create a connector as
> >> they wouldn't be able to include worker plugins in a connector config
> >> anyways.
> >>
> >> And a small nit--it seems like the precedent set ever-so-slightly by the
> >> /connector-plugins resource is to use a hyphen to separate a series of
> >> lowercase words in an endpoint path as opposed to camel case with
> >> capitalization. What do you think about changing the possible plugin
> types
> >> to "connector", "converter", "header-converter", etc.?
> >>
> >> One final question about forwards compatibility--the description for the
> >> /plugins endpoint indicates that it "Returns plugin descriptions of all
> >> types". If another type of pluggable component is added to the
> framework,
> >> should we expect this endpoint to be updated to include that type of
> >> plugin
> >> as well? And if so, should we also expect a matching
> /plugins/{pluginType}
> >> endpoint to be added?
> >>
> >> Cheers,
> >>
> >> Chris
> >>
> >> On Fri, Sep 6, 2019 at 11:48 AM Cyrus Vafadari 
> >> wrote:
> >>
> >> > I've updated the KIP here to simplify and clarify the goal -- it's a
> >> pretty
> >> > simple KIP to add a REST endpoint for SMTs and other plugins. This
> will
> >> > make it easier to see what SMTs you've loaded.
> >> >
> >> > Hoping for some good discussion!
> >> >
> >> > Thanks, all
> >> >
> >> > On Sat, Jul 20, 2019 at 9:07 PM Cyrus Vafadari 
> >> wrote:
> >> >
> >> > > Hello, all,
> >> > >
> >> > > I'd like to start discussion on a new KIP I'm proposing to add HTTP
> >> > > endpoints to Kafka Connect to give us some more insight into other
> >> > > non-connector plugins, like Simple Message Transformations (SMTs).
> The
> >> > KIP
> >> > > would not change how Connect works in any meaningful way except to
> >> expose
> >> > > some more data by the endpoint, analogous to existing
> >> > > "ConnectorPluginsResource" class.
> >> > >
> >> > > Looking forward to getting some feedback.
> >> > >
> >> > > Cyrus
> >> > >
> >> > >
> >> > >
> >> >
> >>
> https

Re: [VOTE] KIP-718: Make KTable Join on Foreign key unopinionated

[John Roesler]

I’m +1 (binding)

Thanks, Marco!
-John

On Wed, Apr 14, 2021, at 07:59, Marco Aurélio Lotz wrote:
> Hi all,
> 
> I would like to start a vote on KIP-718, which proposes to make KTable join
> on foreign key unopinionated in terms of persistence method (currently it
> forces RocksDB usage for subscription store and no other option is
> available).
> 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-718%3A+Make+KTable+Join+on+Foreign+key+unopinionated
> 
> Many thanks,
> Marco Lotz
> 

Re: [DISCUSS] KIP-732: Deprecate eos-alpha and replace eos-beta with eos-v2

[John Roesler]

Thanks for the KIP, Sophie. 

I think this is a great approach, and it makes perfect sense in the 3.0 release 
as well.

-John

On Wed, Apr 14, 2021, at 01:17, Bruno Cadonna wrote:
> Hi Sophie,
> 
> the KIP makes sense to me! Thank you for writing it!
> 
> Best,
> Bruno
> 
> On 14.04.21 04:52, Sophie Blee-Goldman wrote:
> > Hey all,
> > 
> > I'd like to kick off discussion on a small KIP to move towards a unified
> > EOS and clean up the current options. Please give it a pass and let me know
> > what you think.
> > 
> > KIP-732: Deprecate eos-alpha and replace eos-beta with eos-v2
> > 
> > KAFKA-12574: Deprecate eos-alpha
> > 
> > 
> > Thanks,
> > Sophie
> > 
> 

[jira] [Resolved] (KAFKA-12664) Mirrormaker 2.0 infinite rebalance loop when dealing with more than 2 clusters in standalone mode

[Daniel Urban (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-12664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Daniel Urban resolved KAFKA-12664.
--
Resolution: Not A Bug

> Mirrormaker 2.0 infinite rebalance loop when dealing with more than 2 
> clusters in standalone mode
> -
>
> Key: KAFKA-12664
> URL: https://issues.apache.org/jira/browse/KAFKA-12664
> Project: Kafka
>  Issue Type: Bug
>  Components: mirrormaker
>Affects Versions: 2.5.0, 2.4.1, 2.6.0, 2.7.0
>Reporter: Edward Vaisman
>Assignee: Daniel Urban
>Priority: Major
> Attachments: connect.log.tar.gz, docker-compose-multi.yml, 
> mm2.properties
>
>
> Hi Folks, I came across this issue when trying to aggregate data from two 
> separate data centres into one data centre.
> In the configuration below, you can see I am trying to replicate a topic from 
> dc1 (named test_topic_dc1) to dc3 as well as replicate a topic from dc2 
> (test_topic_dc2) to dc3.
> However, when I try to replicate both topics from those datacenters at the 
> same time I notice that connect gets stuck in a rebalance loop (see 
> attachment for logs)
>  [^connect.log.tar.gz]
> excerpt of connect.log
> {code:java}
> 2021-04-13 17:03:06,360] INFO [Worker clientId=connect-3, groupId=mm2-dc2] 
> Successfully synced group in generation Generation{generationId=347, 
> memberId='connect-3-c59342c3-ca62-41cc-964c-41a0f98351c0', 
> protocol='sessioned'} 
> (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:756)[2021-04-13
>  17:03:06,360] INFO [Worker clientId=connect-4, groupId=mm2-dc2] Rebalance 
> started 
> (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator:225)[2021-04-13
>  17:03:06,362] INFO [Worker clientId=connect-4, groupId=mm2-dc2] (Re-)joining 
> group 
> (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540)[2021-04-13
>  17:03:06,368] INFO [Worker clientId=connect-2, groupId=mm2-dc3] Rebalance 
> started 
> (org.apache.kafka.connect.runtime.distributed.WorkerCoordinator:225)[2021-04-13
>  17:03:06,369] INFO [Worker clientId=connect-2, groupId=mm2-dc3] (Re-)joining 
> group 
> (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:540)[2021-04-13
>  17:03:06,370] INFO [Worker clientId=connect-3, groupId=mm2-dc2] Joined group 
> at generation 347 with protocol version 2 and got assignment: 
> Assignment{error=1, leader='connect-3-c59342c3-ca62-41cc-964c-41a0f98351c0', 
> leaderUrl='NOTUSED/dc1', offset=13, connectorIds=[MirrorSourceConnector], 
> taskIds=[], revokedConnectorIds=[], revokedTaskIds=[], delay=0} with 
> rebalance delay: 0 
> (org.apache.kafka.connect.runtime.distributed.DistributedHerder:1688)
> {code}
> To replicate the issue here is what I used:
> [^mm2.properties]
> {code:java}
> clusters = dc1, dc2, dc3
> dc1.bootstrap.servers = kafka-dc1:19092
> dc2.bootstrap.servers = kafka-dc2:19093
> dc3.bootstrap.servers = kafka-dc3:19094
> dc1.group.id=mm2-dc1
> dc2.group.id=mm2-dc2
> dc3.group.id=mm2-dc3
> replication.factor=1
> checkpoints.topic.replication.factor=1
> heartbeats.topic.replication.factor=1
> offset-syncs.topic.replication.factor=1
> offset.storage.replication.factor=1
> status.storage.replication.factor=1
> config.storage.replication.factor=1
> dc1->dc3.enabled = true
> dc1->dc3.topics = test_topic_dc1
> dc2->dc3.enabled = true
> dc2->dc3.topics = test_topic_dc2
> dc3->dc2 = falsedc3->dc1 = false
> {code}
> This [^docker-compose-multi.yml] file to create local kafka clusters 
> (dc1,dc2,dc3)
>  (I set docker to use 6 cpus, 8gb mem, swap 2gb)
> I then ran an interactive shell to run mirror maker within the same 
> docker-compose network (change network to match yours)
> {code:java}
> docker run --network kafka-examples_default -it wurstmeister/kafka:latest bash
> # Upload mm2 properties on server
> /opt/kafka/bin/connect-mirror-maker.sh mm2.properties{code}
> Kafkacat commands to produce to dc1, dc2
> {code:java}
> kafkacat -b localhost:9092 -t test_topic_dc1 -P
> Hello World from DC1!{code}
> {code:java}
> kafkacat -b localhost:9093 -t test_topic_dc2 -P
> Hello World from DC2{code}
> I then tried to remove one of the datacenters to confirm if it was a 
> configuration problem, however mirror maker ran successfully with the below 
> configuration
> mm2.properties
> {code:java}
> clusters = dc2, dc3
> dc2.bootstrap.servers = kafka-dc2:19093
> dc3.bootstrap.servers = kafka-dc3:19094
> dc2.group.id=mm2-dc2
> dc3.group.id=mm2-dc3
> replication.factor=1
> checkpoints.topic.replication.factor=1
> heartbeats.topic.replication.factor=1
> offset-syncs.topic.replication.factor=1
> offset.storage.replication.factor=1
> status.storage.replication.factor=1
> config.storage.replication.factor=1
> dc2->dc3.enabled = true
> dc2->dc3.topics = test_topic_dc2
> {code

Re: [DISCUSS] KIP-732: Deprecate eos-alpha and replace eos-beta with eos-v2

[Ismael Juma]

Thanks Sophie. This makes sense to me. One question: do we want to be a bit
clearer about the removal plans? That is, can we say that the deprecated
configs will be removed in 4.0 (instead of likely to be removed)? The
implication would be that exactly-once would only work with 2.5+ while at
least once would work with all versions. 4.0 is probably 1.5-2 years away,
so this seems reasonable to me.

Ismael

On Tue, Apr 13, 2021 at 7:53 PM Sophie Blee-Goldman
 wrote:

> Hey all,
>
> I'd like to kick off discussion on a small KIP to move towards a unified
> EOS and clean up the current options. Please give it a pass and let me know
> what you think.
>
> KIP-732: Deprecate eos-alpha and replace eos-beta with eos-v2
> <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-732%3A+Deprecate+eos-alpha+and+replace+eos-beta+with+eos-v2
> >
> KAFKA-12574: Deprecate eos-alpha
> 
>
> Thanks,
> Sophie
>

[jira] [Created] (KAFKA-12668) MockScheduler is not safe to use in concurrent code.

[Jose Armando Garcia Sancio (Jira)]

Jose Armando Garcia Sancio created KAFKA-12668:
--

 Summary: MockScheduler is not safe to use in concurrent code.
 Key: KAFKA-12668
 URL: https://issues.apache.org/jira/browse/KAFKA-12668
 Project: Kafka
  Issue Type: Improvement
  Components: unit tests
Reporter: Jose Armando Garcia Sancio


The current implementation of MockScheduler executes tasks in the same stack 
when schedule is called. This violates Log's assumption since Log calls 
schedule while holding a lock. This can cause deadlock in tests.

One solution is to change MockSchedule schedule method so that tick is not 
called. tick should be called by a stack (thread) that doesn't hold any locks.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] KIP-732: Deprecate eos-alpha and replace eos-beta with eos-v2

[Guozhang Wang]

Thanks Sophie for writing the KIP! I'm +1 on the proposal.

On Wed, Apr 14, 2021 at 8:56 AM Ismael Juma  wrote:

> Thanks Sophie. This makes sense to me. One question: do we want to be a bit
> clearer about the removal plans? That is, can we say that the deprecated
> configs will be removed in 4.0 (instead of likely to be removed)? The
> implication would be that exactly-once would only work with 2.5+ while at
> least once would work with all versions. 4.0 is probably 1.5-2 years away,
> so this seems reasonable to me.
>
> Ismael
>
> On Tue, Apr 13, 2021 at 7:53 PM Sophie Blee-Goldman
>  wrote:
>
> > Hey all,
> >
> > I'd like to kick off discussion on a small KIP to move towards a unified
> > EOS and clean up the current options. Please give it a pass and let me
> know
> > what you think.
> >
> > KIP-732: Deprecate eos-alpha and replace eos-beta with eos-v2
> > <
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-732%3A+Deprecate+eos-alpha+and+replace+eos-beta+with+eos-v2
> > >
> > KAFKA-12574: Deprecate eos-alpha
> > 
> >
> > Thanks,
> > Sophie
> >
>


-- 
-- Guozhang

Re: [VOTE] KIP-718: Make KTable Join on Foreign key unopinionated

[Guozhang Wang]

Thanks Marco, +1 from me too.


Guozhang

On Wed, Apr 14, 2021 at 8:15 AM John Roesler  wrote:

> I’m +1 (binding)
>
> Thanks, Marco!
> -John
>
> On Wed, Apr 14, 2021, at 07:59, Marco Aurélio Lotz wrote:
> > Hi all,
> >
> > I would like to start a vote on KIP-718, which proposes to make KTable
> join
> > on foreign key unopinionated in terms of persistence method (currently it
> > forces RocksDB usage for subscription store and no other option is
> > available).
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-718%3A+Make+KTable+Join+on+Foreign+key+unopinionated
> >
> > Many thanks,
> > Marco Lotz
> >
>


-- 
-- Guozhang

Re: [DISCUSS] KIP-730: Producer ID generation in KRaft mode

[Guozhang Wang]

Hi David,

Just putting my paranoid hat here :) Could we name the req/resp name as
"AllocateProducerIds" instead of "AllocateProducerId"? Otherwise, LGTM!

Guozhang

On Thu, Apr 8, 2021 at 2:23 PM Ron Dagostino  wrote:

> Hi David.  I'm wondering if it might be a good idea to have the broker
> send information about the last block it successfully received when it
> requests a new block.  As the RPC stands right now it can't be
> idempotent -- it just tells the controller "provide me a new block,
> please".  One case where it might be useful for the RPC to be
> idempotent is if the broker never receives the response from the
> controller such that it asks again.  That would result in the burning
> of the block that the controller provided but that the broker never
> received.  Now, granted, the ID space is 64 bits, so we would have to
> make ~2^54 requests to burn the entire space, and that isn't going to
> happen.  So whether this is actually needed is questionable.  And it
> might not be worth it to write the controller side code to make it act
> idempotently even if we added the request field to make it possible.
> But I figured this is worth mentioning even if we explicitly decide to
> reject it.
>
> Ron
>
> On Thu, Apr 8, 2021 at 3:16 PM Ron Dagostino  wrote:
> >
> > Oh, I see.  Yes, my mistake -- I read it wrong.  You are right that
> > all we need in the metadata log is the latest value allocated.
> >
> > Ron
> >
> > On Thu, Apr 8, 2021 at 11:21 AM David Arthur  wrote:
> > >
> > > Ron -- I considered making the RPC response and record use the same (or
> > > very similar) fields, but the use case is slightly different. A broker
> > > handling the RPC needs to know the bounds of the block since it has no
> idea
> > > what the block size is. Also, the brokers will normally see
> non-contiguous
> > > blocks.
> > >
> > > For the metadata log, we can just keep track of the latest producer Id
> that
> > > was allocated. It's kind of like a high watermark for producer IDs.
> This
> > > actually saves us from needing an extra field in the record (the KIP
> has
> > > just ProducerIdEnd => int64 in the record).
> > >
> > > Does that make sense?
> > >
> > > On Wed, Apr 7, 2021 at 8:44 AM Ron Dagostino 
> wrote:
> > >
> > > > Thanks for the KIP, David.
> > > >
> > > > With the RPC returning a start and length, should the record in the
> > > > metadata log do the same thing for consistency and to save the byte
> > > > per record?
> > > >
> > > > Ron
> > > >
> > > >
> > > > On Tue, Apr 6, 2021 at 11:06 PM Ismael Juma 
> wrote:
> > > > >
> > > > > Great, thanks. Instead of calling it "bridge release", can we say
> 3.0?
> > > > >
> > > > > Ismael
> > > > >
> > > > > On Tue, Apr 6, 2021 at 7:48 PM David Arthur 
> wrote:
> > > > >
> > > > > > Thanks for the feedback, Ismael. Renaming the RPC and using
> start+len
> > > > > > instead of start+end sounds fine.
> > > > > >
> > > > > > And yes, the controller will allocate the IDs in ZK mode for the
> bridge
> > > > > > release.
> > > > > >
> > > > > > I'll update the KIP to reflect these points.
> > > > > >
> > > > > > Thanks!
> > > > > >
> > > > > > On Tue, Apr 6, 2021 at 7:30 PM Ismael Juma 
> wrote:
> > > > > >
> > > > > > > Sorry, one more question: the allocation of ids will be done
> by the
> > > > > > > controller even in ZK mode, right?
> > > > > > >
> > > > > > > Ismael
> > > > > > >
> > > > > > > On Tue, Apr 6, 2021 at 4:26 PM Ismael Juma 
> > > > wrote:
> > > > > > >
> > > > > > > > One additional comment: if you return the number of ids
> instead of
> > > > the
> > > > > > > end
> > > > > > > > range, you can use an int32.
> > > > > > > >
> > > > > > > > Ismael
> > > > > > > >
> > > > > > > > On Tue, Apr 6, 2021 at 4:25 PM Ismael Juma <
> ism...@juma.me.uk>
> > > > wrote:
> > > > > > > >
> > > > > > > >> Thanks for the KIP, David. Any reason not to rename
> > > > > > > >> AllocateProducerIdBlockRequest to
> AllocateProducerIdsRequest?
> > > > > > > >>
> > > > > > > >> Ismael
> > > > > > > >>
> > > > > > > >> On Tue, Apr 6, 2021 at 3:51 PM David Arthur <
> mum...@gmail.com>
> > > > wrote:
> > > > > > > >>
> > > > > > > >>> Hello everyone,
> > > > > > > >>>
> > > > > > > >>> I'd like to start the discussion for KIP-730
> > > > > > > >>>
> > > > > > > >>>
> > > > > > > >>>
> > > > > > >
> > > > > >
> > > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-730%3A+Producer+ID+generation+in+KRaft+mode
> > > > > > > >>>
> > > > > > > >>> This KIP proposes a new RPC for generating blocks of IDs
> for
> > > > > > > >>> transactional
> > > > > > > >>> and idempotent producers.
> > > > > > > >>>
> > > > > > > >>> Cheers,
> > > > > > > >>> David Arthur
> > > > > > > >>>
> > > > > > > >>
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > David Arthur
> > > > > >
> > > >
> > >
> > >
> > > --
> > > David Arthur
>


-- 
-- Guozhang

[jira] [Created] (KAFKA-12669) Add deleteRange to WindowStore / KeyValueStore interfaces

[Guozhang Wang (Jira)]

Guozhang Wang created KAFKA-12669:
-

 Summary: Add deleteRange to WindowStore / KeyValueStore interfaces
 Key: KAFKA-12669
 URL: https://issues.apache.org/jira/browse/KAFKA-12669
 Project: Kafka
  Issue Type: Bug
  Components: streams
Reporter: Guozhang Wang


We can consider adding such APIs where the underlying implementation classes 
have better optimizations than deleting the keys as get-and-delete one by one.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] KIP-718: Make KTable Join on Foreign key unopinionated

[Matthias J. Sax]

I just re-read the KIP, and I am wondering why we would *only add* new
methods.

Is it an expected use case to only change the main stores but not the
subscription stores?

Wondering if we should deprecate the existing methods that only accept a
single `Materialized` parameter?


-Matthias

On 4/9/21 1:32 PM, Marco Aurélio Lotz wrote:
> Hi everyone,
> 
> I am fine with sticking with Materialized and adding the info to the
> javadoc then - so we keep the footprint smaller.
> I will then update the KIP to match what we discussed here and send it for
> a vote next week.
> 
> I will raise a new bug-fix ticket and change KAFKA-10383
>  to become a feature -
> if that's ok.
> 
> Cheers,
> Marco
> 
> On Wed, Apr 7, 2021 at 4:15 AM Matthias J. Sax  wrote:
> 
>> Just catching up here.
>>
>> I agree that we have two issue, and the first (align subscription store
>> to main store) can be done as a bug-fix.
>>
>> For the KIP (that addressed the second), I tend to agree that reusing
>> `Materialized` might be better as it would keep the API surface area
>> smaller.
>>
>>
>> -Matthias
>>
>> On 4/6/21 8:48 AM, John Roesler wrote:
>>> Hi Marco,
>>>
>>> Just a quick clarification: I just reviewed the Materialized class. It
>> looks like the only undesirable members are:
>>> 1. Retention
>>> 2. Key/Value serdes
>>>
>>> The underlying store type would be “KeyValueStore” , for
>> which case the withRetention javadoc already says it’s ignored.
>>>
>>> Perhaps we could just stick with Materialized by adding a note to the
>> Key/Value serdes setters that they are ignored for FKJoin subscription
>> stores?
>>>
>>> Not as elegant as a new config class, but these config classes actually
>> bring a fair amount of complexity, so it might be nice to avoid a new one.
>>>
>>> Thanks,
>>> John
>>>
>>> On Tue, Apr 6, 2021, at 10:28, Marco Aurélio Lotz wrote:
 Hi John / Guozhang,

 If I correctly understood John's message, he agrees on having the two
 scenarios (piggy-back and api extension). In my view, these two
>> scenarios
 are separate tasks - the first one is a bug-fix and the second is an
 improvement on the current API.

 - bug-fix: On the current API, we change its implementation to piggy
>> back
 on the materialization method provided to the materialized parameter.
>> This
 way it will not be opinionated anymore and will not force RocksDb
 persistence for subscription store. Thus an in-memory materialized
 parameter would imply an in-memory subscription store, for example.
>> From my
 understanding, the original implementation tried to be as unopionated
 towards storage methods as possible - and the current implementation is
>> not
 allowing that. Was that the case? We would still need to add this
 modification to the update notes, since it may affect some deployments.

 - improvement: We extend the API to allow a user to fine tune different
 materialization methods for subscription and join store. This is done by
 adding a new parameter to the associated methods.

 Does it sound reasonable to you Guozhang?
 On your question, does it make sense for an user to decide retention
 policies (withRetention method) or caching for subscription stores? I
>> can
 see why to finetune Logging for example, but in a first moment not these
 other behaviours. That's why I am unsure about using Materialized class.

 @John, I will update the KIP with your points as soon as we clarify
>> this.

 Cheers,
 Marco

 On Tue, Apr 6, 2021 at 1:17 AM Guozhang Wang 
>> wrote:

> Thanks Marco / John,
>
> I think the arguments for not piggy-backing on the existing
>> Materialized
> makes sense; on the other hand, if we go this route should we just use
>> a
> separate Materialized than using an extended /
> narrowed-scoped MaterializedSubscription since it seems we want to
>> allow
> users to fully customize this store?
>
> Guozhang
>
> On Thu, Apr 1, 2021 at 5:28 PM John Roesler 
>> wrote:
>
>> Thanks Marco,
>>
>> Sorry if I caused any trouble!
>>
>> I don’t remember what I was thinking before, but reasoning about it
>> now,
>> you might need the fine-grained choice if:
>>
>> 1. The number or size of records in each partition of both tables is
>> small(ish), but the cardinality of the join is very high. Then you
>> might
>> want an in-memory table store, but an on-disk subscription store.
>>
>> 2. The number or size of records is very large, but the join
>> cardinality
>> is low. Then you might need an on-disk table store, but an in-memory
>> subscription store.
>>
>> 3. You might want a different kind (or differently configured) store
>> for
>> the subscription store, since it’s access pattern is so different.
>>
>> If you buy these, it might be good to

Re: [DISCUSS] KIP-732: Deprecate eos-alpha and replace eos-beta with eos-v2

[John Roesler]

Hello again, Sophie,

I've just had a late-breaking thought about your KIP. It
might be worthwhile also printing a WARN log when the
deprecated configs are used. People may not always be
referencing the StreamsConfig fields in their source code
when they set these configs, so it would be good to make an
extra effort to raise the visibility of this deprecation.

Thanks,
-John

On Wed, 2021-04-14 at 10:47 -0700, Guozhang Wang wrote:
> Thanks Sophie for writing the KIP! I'm +1 on the proposal.
> 
> On Wed, Apr 14, 2021 at 8:56 AM Ismael Juma  wrote:
> 
> > Thanks Sophie. This makes sense to me. One question: do we want to be a bit
> > clearer about the removal plans? That is, can we say that the deprecated
> > configs will be removed in 4.0 (instead of likely to be removed)? The
> > implication would be that exactly-once would only work with 2.5+ while at
> > least once would work with all versions. 4.0 is probably 1.5-2 years away,
> > so this seems reasonable to me.
> > 
> > Ismael
> > 
> > On Tue, Apr 13, 2021 at 7:53 PM Sophie Blee-Goldman
> >  wrote:
> > 
> > > Hey all,
> > > 
> > > I'd like to kick off discussion on a small KIP to move towards a unified
> > > EOS and clean up the current options. Please give it a pass and let me
> > know
> > > what you think.
> > > 
> > > KIP-732: Deprecate eos-alpha and replace eos-beta with eos-v2
> > > <
> > > 
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-732%3A+Deprecate+eos-alpha+and+replace+eos-beta+with+eos-v2
> > > > 
> > > KAFKA-12574: Deprecate eos-alpha
> > > 
> > > 
> > > Thanks,
> > > Sophie
> > > 
> > 
> 
> 

Re: [DISCUSS] KIP-726: Make the CooperativeStickyAssignor as the default assignor

[Guozhang Wang]

Hello Sophie,

Thanks for the detailed explanation, a few clarifying questions:

1) when the short-circuit is triggered, what would happen next? Would the
consumers switch back to EAGER, and try to re-join the group, and then upon
succeeding the next rebalance reset the flag to allow committing? Or would
we just fail the consumer immediately.

2) at the overview you mentioned "New applications on 3.0 will enable
cooperative rebalancing by default", but in the detailed description as
"With ["cooperative-sticky", "range”], the initial protocol will be EAGER
when the member first joins the group." which seems contradictory? If we
want to have cooperative behavior be the default, then with the
default ["cooperative-sticky", "range”] the member would start with
COOPERATIVE protocol right away.


Guozhang



On Mon, Apr 12, 2021 at 5:19 AM Chris Egerton 
wrote:

> Whoops, small correction--meant to say
> ConsumerRebalanceListener::onPartitionsLost, not Consumer::onPartitionsLost
>
> On Mon, Apr 12, 2021 at 8:17 AM Chris Egerton  wrote:
>
> > Hi Sophie,
> >
> > This sounds fantastic. I've made a note on KAFKA-12487 about being sure
> to
> > implement Consumer::onPartitionsLost to avoid unnecessary task failures
> on
> > consumer protocol downgrade, but besides that, I don't think things could
> > get any smoother for Connect users or developers. The automatic protocol
> > upgrade/downgrade behavior appears safe, intuitive, and pain-free.
> >
> > Really excited for this development and hoping we can see it come to
> > fruition in time for the 3.0 release!
> >
> > Cheers,
> >
> > Chris
> >
> > On Fri, Apr 9, 2021 at 2:43 PM Sophie Blee-Goldman
> >  wrote:
> >
> >> 1) Yes, all of the above will be part of KAFKA-12477 (not KIP-726)
> >>
> >> 2) No, KAFKA-12638 would be nice to have but I don't think it's
> >> appropriate
> >> to remove
> >> the default implementation of #onPartitionsLost in 3.0 since we never
> gave
> >> any indication
> >> yet that we intend to remove it
> >>
> >> 3) Yes, this would be similar to when a Consumer drops out of the group.
> >> It's always been
> >> possible for a member to miss a rebalance and have its partition be
> >> reassigned to another
> >> member, during which time both members would claim to own said
> partition.
> >> But this is safe
> >> because the member who dropped out is blocked from committing offsets on
> >> that partition.
> >>
> >> On Fri, Apr 9, 2021 at 2:46 AM Luke Chen  wrote:
> >>
> >> > Hi Sophie,
> >> > That sounds great to take care of each case I can think of.
> >> > Questions:
> >> > 1. Do you mean the short-Circuit will also be implemented in
> >> KAFKA-12477?
> >> > 2. I don't think KAFKA-12638 is the blocker of this KIP-726, Am I
> right?
> >> > 3. So, does that mean we still have possibility to have multiple
> >> consumer
> >> > owned the same topic partition? And in this situation, we avoid them
> >> doing
> >> > committing, and waiting for next rebalance (should be soon). Is my
> >> > understanding correct?
> >> >
> >> > Thank you very much for finding this great solution.
> >> >
> >> > Luke
> >> >
> >> > On Fri, Apr 9, 2021 at 11:37 AM Sophie Blee-Goldman
> >> >  wrote:
> >> >
> >> > > Alright, here's the detailed proposal for KAFKA-12477. This assumes
> we
> >> > will
> >> > > change the default assignor to ["cooperative-sticky", "range"] in
> >> > KIP-726.
> >> > > It also acknowledges that users may attempt any kind of upgrade
> >> without
> >> > > reading the docs, and so we need to put in safeguards against data
> >> > > corruption rather than assume everyone will follow the safe upgrade
> >> path.
> >> > >
> >> > > With this proposal,
> >> > > 1) New applications on 3.0 will enable cooperative rebalancing by
> >> default
> >> > > 2) Existing applications which don’t set an assignor can safely
> >> upgrade
> >> > to
> >> > > 3.0 using a single rolling bounce with no extra steps, and will
> >> > > automatically transition to cooperative rebalancing
> >> > > 3) Existing applications which do set an assignor that uses EAGER
> can
> >> > > likewise upgrade their applications to COOPERATIVE with a single
> >> rolling
> >> > > bounce
> >> > > 4) Once on 3.0, applications can safely go back and forth between
> >> EAGER
> >> > and
> >> > > COOPERATIVE
> >> > > 5) Applications can safely downgrade away from 3.0
> >> > >
> >> > > The high-level idea for dynamic protocol upgrades is that the group
> >> will
> >> > > leverage the assignor selected by the group coordinator to determine
> >> when
> >> > > it’s safe to upgrade to COOPERATIVE, and trigger a fail-safe to
> >> protect
> >> > the
> >> > > group in case of rare events or user misconfiguration. The group
> >> > > coordinator selects the most preferred assignor that’s supported by
> >> all
> >> > > members of the group, so we know that all members will support
> >> > COOPERATIVE
> >> > > once we receive the “cooperative-sticky” assignor after a rebalance.
> >> At
> >> > > this point, each member can up

Re: [DISCUSS] KIP-732: Deprecate eos-alpha and replace eos-beta with eos-v2

[Bill Bejeck]

Hi Sophie,

Thanks for the KIP, it makes sense to me.  I'm a +1 on the proposal.
I'm also a +1 to Ismael's suggestion for stating the removal of the
deprecated configs in 4.0

-Bill

On Wed, Apr 14, 2021 at 3:02 PM John Roesler  wrote:

> Hello again, Sophie,
>
> I've just had a late-breaking thought about your KIP. It
> might be worthwhile also printing a WARN log when the
> deprecated configs are used. People may not always be
> referencing the StreamsConfig fields in their source code
> when they set these configs, so it would be good to make an
> extra effort to raise the visibility of this deprecation.
>
> Thanks,
> -John
>
> On Wed, 2021-04-14 at 10:47 -0700, Guozhang Wang wrote:
> > Thanks Sophie for writing the KIP! I'm +1 on the proposal.
> >
> > On Wed, Apr 14, 2021 at 8:56 AM Ismael Juma  wrote:
> >
> > > Thanks Sophie. This makes sense to me. One question: do we want to be
> a bit
> > > clearer about the removal plans? That is, can we say that the
> deprecated
> > > configs will be removed in 4.0 (instead of likely to be removed)? The
> > > implication would be that exactly-once would only work with 2.5+ while
> at
> > > least once would work with all versions. 4.0 is probably 1.5-2 years
> away,
> > > so this seems reasonable to me.
> > >
> > > Ismael
> > >
> > > On Tue, Apr 13, 2021 at 7:53 PM Sophie Blee-Goldman
> > >  wrote:
> > >
> > > > Hey all,
> > > >
> > > > I'd like to kick off discussion on a small KIP to move towards a
> unified
> > > > EOS and clean up the current options. Please give it a pass and let
> me
> > > know
> > > > what you think.
> > > >
> > > > KIP-732: Deprecate eos-alpha and replace eos-beta with eos-v2
> > > > <
> > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-732%3A+Deprecate+eos-alpha+and+replace+eos-beta+with+eos-v2
> > > > >
> > > > KAFKA-12574: Deprecate eos-alpha
> > > > 
> > > >
> > > > Thanks,
> > > > Sophie
> > > >
> > >
> >
> >
>
>
>

[VOTE] 2.8.0 RC2

[John Roesler]

Hello Kafka users, developers and client-developers,

This is the third candidate for release of Apache Kafka
2.8.0.This is a major release that includes many new
features, including:

 * Early-access release of replacing Zookeeper with a
   self-managed quorum
 * Add Describe Cluster API
 * Support mutual TLS authentication on SASL_SSL listeners
 * Ergonomic improvements to Streams TopologyTestDriver
 * Logger API improvement to respect the hierarchy
 * Request/response trace logs are now JSON-formatted
 * New API to add and remove Streams threads while running
 * New REST API to expose Connect task configurations
 * Fixed the TimeWindowDeserializer to be able to
   deserialize
   keys outside of Streams (such as in the console consumer)
 * Streams resilient improvement: new uncaught exception
   handler
 * Streams resilience improvement: automatically recover
   from transient timeout exceptions

Release notes for the 2.8.0 release:
https://home.apache.org/~vvcephei/kafka-2.8.0-rc2/RELEASE_NOTES.html

*** Please download, test and vote by 19 April 2021 ***

Kafka's KEYS file containing PGP keys we use to sign the
release:
https://kafka.apache.org/KEYS

* Release artifacts to be voted upon (source and binary):
https://home.apache.org/~vvcephei/kafka-2.8.0-rc2/

* Maven artifacts to be voted upon:
https://repository.apache.org/content/groups/staging/org/apache/kafka/

* Javadoc:
https://home.apache.org/~vvcephei/kafka-2.8.0-rc2/javadoc/

* Tag to be voted upon (off 2.8 branch) is the 2.8.0 tag:
https://github.com/apache/kafka/releases/tag/2.8.0-rc2

* Documentation:
https://kafka.apache.org/28/documentation.html

* Protocol:
https://kafka.apache.org/28/protocol.html

* Successful Jenkins builds for the 2.8 branch:
Unit/integration tests:
https://ci-builds.apache.org/job/Kafka/job/kafka/job/2.8/
(still flaky)

System tests:
 
https://jenkins.confluent.io/job/system-test-kafka/job/2.8/6
0/ 
 
http://confluent-kafka-2-8-system-test-results.s3-us-west-2.amazonaws.com/2021-04-14--001.1618407001--confluentinc--2.8--1b61272d45/report.html

/**

Thanks,
John

Re: [kafka-clients] Re: Subject: [VOTE] 2.8.0 RC1

[John Roesler]

Thanks for the feedback, all.

I am now closing this vote thread in favor of the vote on
2.8.0 RC2.

Thanks,
John

On Mon, 2021-04-12 at 21:24 -0400, Israel Ekpo wrote:
> No problem, I will assign in to you shortly.
> 
> https://issues.apache.org/jira/browse/KAFKA-12658
> 
> On Mon, Apr 12, 2021 at 8:47 PM John Roesler
>  wrote:
> > Good catch, Israel!
> > 
> > I’ll make sure that gets fixed. 
> > 
> > Thanks,
> > John
> > 
> > On Mon, Apr 12, 2021, at 19:30, Israel Ekpo wrote:
> > > I just noticed that with the latest release candidate,
> > the binaries from
> > > the Scala 2.13 and 2.12 tarballs are not finding the
> > class for the meta
> > > data shell
> > > 
> > > https://home.apache.org/~vvcephei/kafka-2.8.0-rc1/
> > > 
> > > It looks like kafka-run-class.sh is not able to load
> > it.
> > > 
> > > Is this a known issue? Should I open an issue to track
> > it?
> > > 
> > > isekpo@MININT-5RPA920:/mnt/c/Users/isekpo/kafka_2.12-
> > 2.8.0$
> > > bin/kafka-metadata-shell.sh --help
> > > Error: Could not find or load main class
> > > org.apache.kafka.shell.MetadataShell
> > > Caused by: java.lang.ClassNotFoundException:
> > > org.apache.kafka.shell.MetadataShell
> > > 
> > > isekpo@MININT-5RPA920:/mnt/c/Users/isekpo/kafka_2.12-
> > 2.8.0$ cd
> > > ../kafka_2.13-2.8.0/
> > > 
> > > isekpo@MININT-5RPA920:/mnt/c/Users/isekpo/kafka_2.13-
> > 2.8.0$
> > > bin/kafka-metadata-shell.sh --help
> > > Error: Could not find or load main class
> > > org.apache.kafka.shell.MetadataShell
> > > Caused by: java.lang.ClassNotFoundException:
> > > org.apache.kafka.shell.MetadataShell
> > > 
> > > 
> > > 
> > > On Fri, Apr 9, 2021 at 4:52 PM Bill Bejeck
> >  wrote:
> > > 
> > > > Hi John,
> > > >
> > > > Thanks for running the 2.8.0 release!
> > > >
> > > > I've started to validate it and noticed the site-
> > docs haven't been
> > > > installed to
> > https://kafka.apache.org/28/documentation.html yet.
> > > >
> > > > Thanks again!
> > > >
> > > > -Bill
> > > >
> > > > On Tue, Apr 6, 2021 at 5:37 PM John Roesler
> >  wrote:
> > > >
> > > >> Hello Kafka users, developers and client-
> > developers,
> > > >>
> > > >> This is the second candidate for release of Apache
> > Kafka
> > > >> 2.8.0. This is a major release that includes many
> > new
> > > >> features, including:
> > > >>
> > > >> * Early-access release of replacing Zookeeper with
> > a self-
> > > >> managed quorum
> > > >> * Add Describe Cluster API
> > > >> * Support mutual TLS authentication on SASL_SSL
> > listeners
> > > >> * Ergonomic improvements to Streams
> > TopologyTestDriver
> > > >> * Logger API improvement to respect the hierarchy
> > > >> * Request/response trace logs are now JSON-
> > formatted
> > > >> * New API to add and remove Streams threads while
> > running
> > > >> * New REST API to expose Connect task
> > configurations
> > > >> * Fixed the TimeWindowDeserializer to be able to
> > deserialize
> > > >> keys outside of Streams (such as in the console
> > consumer)
> > > >> * Streams resilient improvement: new uncaught
> > exception
> > > >> handler
> > > >> * Streams resilience improvement: automatically
> > recover from
> > > >> transient timeout exceptions
> > > >>
> > > >>
> > > >>
> > > >>
> > > >> Release notes for the 2.8.0 release:
> > > >>
> >
> https://home.apache.org/~vvcephei/kafka-2.8.0-rc1/RELEASE_NOTES.html
> > > >>
> > > >>
> > > >> *** Please download, test and vote by 6 April 2021
> > ***
> > > >>
> > > >> Kafka's KEYS file containing PGP keys we use to
> > sign the
> > > >> release:
> > > >> https://kafka.apache.org/KEYS
> > > >>
> > > >> * Release artifacts to be voted upon (source and
> > binary):
> > > >>
> > > >> https://home.apache.org/~vvcephei/kafka-2.8.0-rc1/
> > > >>
> > > >> * Maven artifacts to be voted upon:
> > > >>
> >
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > > >>
> > > >> * Javadoc:
> > > >>
> > > >>
> >
> https://home.apache.org/~vvcephei/kafka-2.8.0-rc1/javadoc/
> > > >>
> > > >> * Tag to be voted upon (off 2.8 branch) is the
> > 2.8.0 tag:
> > > >>
> > > >>
> > https://github.com/apache/kafka/releases/tag/2.8.0-rc1
> > > >>
> > > >> * Documentation:
> > > >> https://kafka.apache.org/28/documentation.html
> > > >>
> > > >> * Protocol:
> > > >> https://kafka.apache.org/28/protocol.html
> > > >>
> > > >>
> > > >> /**
> > > >>
> > > >> Thanks,
> > > >> John
> > > >>
> > > >>
> > > >>
> > > >> --
> > > > You received this message because you are subscribed
> > to the Google Groups
> > > > "kafka-clients" group.
> > > > To unsubscribe from this group and stop receiving
> > emails from it, send an
> > > > email to kafka-clients+unsubscr...@googlegroups.com.
> > > > To view this discussion on the web visit
> > > >
> >
> https://groups.google.com/d/msgid/kafka-clients/CAF7WS%2BrK%3DWMyM3bamNoxa9L-onZbw6UnJFASx0ZO5ywzj38WvA%40mail.gmail.com
> > > >
> >
> 

[jira] [Resolved] (KAFKA-12667) Incorrect error log on StateDirectory close

[A. Sophie Blee-Goldman (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-12667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

A. Sophie Blee-Goldman resolved KAFKA-12667.

Resolution: Fixed

> Incorrect error log on StateDirectory close
> ---
>
> Key: KAFKA-12667
> URL: https://issues.apache.org/jira/browse/KAFKA-12667
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 2.7.0, 2.6.1, 2.8.0
>Reporter: Jan Justesen
>Assignee: A. Sophie Blee-Goldman
>Priority: Major
> Fix For: 3.0.0, 2.6.3, 2.7.2, 2.8.1
>
>
> {{In StateDirectory.close() an error is logged about unclean shutdown if all 
> locks are in fact released, and nothing is logged in case of an unclean 
> shutdown.}}
>  
> {code:java}
> // all threads should be stopped and cleaned up by now, so none should remain 
> holding a lock
> if (locks.isEmpty()) {
>  log.error("Some task directories still locked while closing state, this 
> indicates unclean shutdown: {}", locks);
> }
> {code}
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] KIP-718: Make KTable Join on Foreign key unopinionated

[Marco Aurélio Lotz]

Hi Matthias,

Thanks for reviewing it. That's a nice question. I can provide you the
answer as a user of the API - but of course it very much depends on what
the original intention towards the user is.
The only reason I found out it was opinionated was when my tests using
in-memory materialization started failing due to the state-store clean-up
bug

in Windows systems.

To be honest, I think I wouldn't have investigated how subscription stores
are used without seeing that my system couldn't handle the expected data.
Must the user know about the subscription-stores beforehand? If yes, then
indeed we should deprecate.

Having said that, it seems to me that deprecating would not be consistent
with other parts of the API that have parameterized methods hiding
complexity.
E.g. from the top of my head, join() will make a rocksDB materialization if
no-materialization method is provided.

public  KTable join(final KTable other, final
ValueJoiner joiner0


I see this as something equivalent. If the user didn't provide all the
methods, shouldn't it be fine to fallback to a default materialization
method as the other sections of the API already does?
In that scenario, there's no need to deprecate and we just need to make a
reasonable guess towards it. Using the same method as the table-store
sounds like a reasonable guess in this scenario.

In my perspective as a user, I would prefer to have simpler methods knowing
I would be losing some fine-grain configuration. Does it sound reasonable
to you?

Cheers,
Marco Lotz

On Wed, Apr 14, 2021 at 8:47 PM Matthias J. Sax  wrote:

> I just re-read the KIP, and I am wondering why we would *only add* new
> methods.
>
> Is it an expected use case to only change the main stores but not the
> subscription stores?
>
> Wondering if we should deprecate the existing methods that only accept a
> single `Materialized` parameter?
>
>
> -Matthias
>
> On 4/9/21 1:32 PM, Marco Aurélio Lotz wrote:
> > Hi everyone,
> >
> > I am fine with sticking with Materialized and adding the info to the
> > javadoc then - so we keep the footprint smaller.
> > I will then update the KIP to match what we discussed here and send it
> for
> > a vote next week.
> >
> > I will raise a new bug-fix ticket and change KAFKA-10383
> >  to become a feature
> -
> > if that's ok.
> >
> > Cheers,
> > Marco
> >
> > On Wed, Apr 7, 2021 at 4:15 AM Matthias J. Sax  wrote:
> >
> >> Just catching up here.
> >>
> >> I agree that we have two issue, and the first (align subscription store
> >> to main store) can be done as a bug-fix.
> >>
> >> For the KIP (that addressed the second), I tend to agree that reusing
> >> `Materialized` might be better as it would keep the API surface area
> >> smaller.
> >>
> >>
> >> -Matthias
> >>
> >> On 4/6/21 8:48 AM, John Roesler wrote:
> >>> Hi Marco,
> >>>
> >>> Just a quick clarification: I just reviewed the Materialized class. It
> >> looks like the only undesirable members are:
> >>> 1. Retention
> >>> 2. Key/Value serdes
> >>>
> >>> The underlying store type would be “KeyValueStore” , for
> >> which case the withRetention javadoc already says it’s ignored.
> >>>
> >>> Perhaps we could just stick with Materialized by adding a note to the
> >> Key/Value serdes setters that they are ignored for FKJoin subscription
> >> stores?
> >>>
> >>> Not as elegant as a new config class, but these config classes actually
> >> bring a fair amount of complexity, so it might be nice to avoid a new
> one.
> >>>
> >>> Thanks,
> >>> John
> >>>
> >>> On Tue, Apr 6, 2021, at 10:28, Marco Aurélio Lotz wrote:
>  Hi John / Guozhang,
> 
>  If I correctly understood John's message, he agrees on having the two
>  scenarios (piggy-back and api extension). In my view, these two
> >> scenarios
>  are separate tasks - the first one is a bug-fix and the second is an
>  improvement on the current API.
> 
>  - bug-fix: On the current API, we change its implementation to piggy
> >> back
>  on the materialization method provided to the materialized parameter.
> >> This
>  way it will not be opinionated anymore and will not force RocksDb
>  persistence for subscription store. Thus an in-memory materialized
>  parameter would imply an in-memory subscription store, for example.
> >> From my
>  understanding, the original implementation tried to be as unopionated
>  towards storage methods as possible - and the current implementation
> is
> >> not
>  allowing that. Was that the case? We would still need to add this
>  modification to the update notes, since it may affect some
> deployments.
> 
>  - improvement: We extend the API to allow a user to fine tune
> different
>  materialization methods for subscription and join store. This is done
> by
>  adding a new parameter to the associ

Re: [DISCUSS] KIP-732: Deprecate eos-alpha and replace eos-beta with eos-v2

[Ismael Juma]

Hey John,

Yes, we should definitely do that. It's meant to be the standard procedure
for configs that are deprecated.

Ismael

On Wed, Apr 14, 2021 at 12:02 PM John Roesler  wrote:

> Hello again, Sophie,
>
> I've just had a late-breaking thought about your KIP. It
> might be worthwhile also printing a WARN log when the
> deprecated configs are used. People may not always be
> referencing the StreamsConfig fields in their source code
> when they set these configs, so it would be good to make an
> extra effort to raise the visibility of this deprecation.
>
> Thanks,
> -John
>
> On Wed, 2021-04-14 at 10:47 -0700, Guozhang Wang wrote:
> > Thanks Sophie for writing the KIP! I'm +1 on the proposal.
> >
> > On Wed, Apr 14, 2021 at 8:56 AM Ismael Juma  wrote:
> >
> > > Thanks Sophie. This makes sense to me. One question: do we want to be
> a bit
> > > clearer about the removal plans? That is, can we say that the
> deprecated
> > > configs will be removed in 4.0 (instead of likely to be removed)? The
> > > implication would be that exactly-once would only work with 2.5+ while
> at
> > > least once would work with all versions. 4.0 is probably 1.5-2 years
> away,
> > > so this seems reasonable to me.
> > >
> > > Ismael
> > >
> > > On Tue, Apr 13, 2021 at 7:53 PM Sophie Blee-Goldman
> > >  wrote:
> > >
> > > > Hey all,
> > > >
> > > > I'd like to kick off discussion on a small KIP to move towards a
> unified
> > > > EOS and clean up the current options. Please give it a pass and let
> me
> > > know
> > > > what you think.
> > > >
> > > > KIP-732: Deprecate eos-alpha and replace eos-beta with eos-v2
> > > > <
> > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-732%3A+Deprecate+eos-alpha+and+replace+eos-beta+with+eos-v2
> > > > >
> > > > KAFKA-12574: Deprecate eos-alpha
> > > > 
> > > >
> > > > Thanks,
> > > > Sophie
> > > >
> > >
> >
> >
>
>
>

Re: [DISCUSS] KIP-730: Producer ID generation in KRaft mode

[Ismael Juma]

Hi Guozhang,

That was my original suggestion, so I am naturally +1 :)

Ismael

On Wed, Apr 14, 2021 at 11:44 AM Guozhang Wang  wrote:

> Hi David,
>
> Just putting my paranoid hat here :) Could we name the req/resp name as
> "AllocateProducerIds" instead of "AllocateProducerId"? Otherwise, LGTM!
>
> Guozhang
>
> On Thu, Apr 8, 2021 at 2:23 PM Ron Dagostino  wrote:
>
> > Hi David.  I'm wondering if it might be a good idea to have the broker
> > send information about the last block it successfully received when it
> > requests a new block.  As the RPC stands right now it can't be
> > idempotent -- it just tells the controller "provide me a new block,
> > please".  One case where it might be useful for the RPC to be
> > idempotent is if the broker never receives the response from the
> > controller such that it asks again.  That would result in the burning
> > of the block that the controller provided but that the broker never
> > received.  Now, granted, the ID space is 64 bits, so we would have to
> > make ~2^54 requests to burn the entire space, and that isn't going to
> > happen.  So whether this is actually needed is questionable.  And it
> > might not be worth it to write the controller side code to make it act
> > idempotently even if we added the request field to make it possible.
> > But I figured this is worth mentioning even if we explicitly decide to
> > reject it.
> >
> > Ron
> >
> > On Thu, Apr 8, 2021 at 3:16 PM Ron Dagostino  wrote:
> > >
> > > Oh, I see.  Yes, my mistake -- I read it wrong.  You are right that
> > > all we need in the metadata log is the latest value allocated.
> > >
> > > Ron
> > >
> > > On Thu, Apr 8, 2021 at 11:21 AM David Arthur  wrote:
> > > >
> > > > Ron -- I considered making the RPC response and record use the same
> (or
> > > > very similar) fields, but the use case is slightly different. A
> broker
> > > > handling the RPC needs to know the bounds of the block since it has
> no
> > idea
> > > > what the block size is. Also, the brokers will normally see
> > non-contiguous
> > > > blocks.
> > > >
> > > > For the metadata log, we can just keep track of the latest producer
> Id
> > that
> > > > was allocated. It's kind of like a high watermark for producer IDs.
> > This
> > > > actually saves us from needing an extra field in the record (the KIP
> > has
> > > > just ProducerIdEnd => int64 in the record).
> > > >
> > > > Does that make sense?
> > > >
> > > > On Wed, Apr 7, 2021 at 8:44 AM Ron Dagostino 
> > wrote:
> > > >
> > > > > Thanks for the KIP, David.
> > > > >
> > > > > With the RPC returning a start and length, should the record in the
> > > > > metadata log do the same thing for consistency and to save the byte
> > > > > per record?
> > > > >
> > > > > Ron
> > > > >
> > > > >
> > > > > On Tue, Apr 6, 2021 at 11:06 PM Ismael Juma 
> > wrote:
> > > > > >
> > > > > > Great, thanks. Instead of calling it "bridge release", can we say
> > 3.0?
> > > > > >
> > > > > > Ismael
> > > > > >
> > > > > > On Tue, Apr 6, 2021 at 7:48 PM David Arthur 
> > wrote:
> > > > > >
> > > > > > > Thanks for the feedback, Ismael. Renaming the RPC and using
> > start+len
> > > > > > > instead of start+end sounds fine.
> > > > > > >
> > > > > > > And yes, the controller will allocate the IDs in ZK mode for
> the
> > bridge
> > > > > > > release.
> > > > > > >
> > > > > > > I'll update the KIP to reflect these points.
> > > > > > >
> > > > > > > Thanks!
> > > > > > >
> > > > > > > On Tue, Apr 6, 2021 at 7:30 PM Ismael Juma 
> > wrote:
> > > > > > >
> > > > > > > > Sorry, one more question: the allocation of ids will be done
> > by the
> > > > > > > > controller even in ZK mode, right?
> > > > > > > >
> > > > > > > > Ismael
> > > > > > > >
> > > > > > > > On Tue, Apr 6, 2021 at 4:26 PM Ismael Juma <
> ism...@juma.me.uk>
> > > > > wrote:
> > > > > > > >
> > > > > > > > > One additional comment: if you return the number of ids
> > instead of
> > > > > the
> > > > > > > > end
> > > > > > > > > range, you can use an int32.
> > > > > > > > >
> > > > > > > > > Ismael
> > > > > > > > >
> > > > > > > > > On Tue, Apr 6, 2021 at 4:25 PM Ismael Juma <
> > ism...@juma.me.uk>
> > > > > wrote:
> > > > > > > > >
> > > > > > > > >> Thanks for the KIP, David. Any reason not to rename
> > > > > > > > >> AllocateProducerIdBlockRequest to
> > AllocateProducerIdsRequest?
> > > > > > > > >>
> > > > > > > > >> Ismael
> > > > > > > > >>
> > > > > > > > >> On Tue, Apr 6, 2021 at 3:51 PM David Arthur <
> > mum...@gmail.com>
> > > > > wrote:
> > > > > > > > >>
> > > > > > > > >>> Hello everyone,
> > > > > > > > >>>
> > > > > > > > >>> I'd like to start the discussion for KIP-730
> > > > > > > > >>>
> > > > > > > > >>>
> > > > > > > > >>>
> > > > > > > >
> > > > > > >
> > > > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-730%3A+Producer+ID+generation+in+KRaft+mode
> > > > > > > > >>>
> > > > > > > > >>> This KIP proposes a new RPC for generating blocks of IDs
> > for
> > > > > > > > >>> transactiona

Re: [DISCUSS] KIP-732: Deprecate eos-alpha and replace eos-beta with eos-v2

[Sophie Blee-Goldman]

Thanks all. I've updated the KIP to clarify these will be removed in 4.0,
and added a note about
logging a warning as well. I agree that some users may set the config value
directly and could
miss the deprecation notice.

I'll call for a vote today since there seems to be consensus, but please
let me know if you have
any more late-breaking ideas

-Sophie

On Wed, Apr 14, 2021 at 1:36 PM Ismael Juma  wrote:

> Hey John,
>
> Yes, we should definitely do that. It's meant to be the standard procedure
> for configs that are deprecated.
>
> Ismael
>
> On Wed, Apr 14, 2021 at 12:02 PM John Roesler  wrote:
>
> > Hello again, Sophie,
> >
> > I've just had a late-breaking thought about your KIP. It
> > might be worthwhile also printing a WARN log when the
> > deprecated configs are used. People may not always be
> > referencing the StreamsConfig fields in their source code
> > when they set these configs, so it would be good to make an
> > extra effort to raise the visibility of this deprecation.
> >
> > Thanks,
> > -John
> >
> > On Wed, 2021-04-14 at 10:47 -0700, Guozhang Wang wrote:
> > > Thanks Sophie for writing the KIP! I'm +1 on the proposal.
> > >
> > > On Wed, Apr 14, 2021 at 8:56 AM Ismael Juma  wrote:
> > >
> > > > Thanks Sophie. This makes sense to me. One question: do we want to be
> > a bit
> > > > clearer about the removal plans? That is, can we say that the
> > deprecated
> > > > configs will be removed in 4.0 (instead of likely to be removed)? The
> > > > implication would be that exactly-once would only work with 2.5+
> while
> > at
> > > > least once would work with all versions. 4.0 is probably 1.5-2 years
> > away,
> > > > so this seems reasonable to me.
> > > >
> > > > Ismael
> > > >
> > > > On Tue, Apr 13, 2021 at 7:53 PM Sophie Blee-Goldman
> > > >  wrote:
> > > >
> > > > > Hey all,
> > > > >
> > > > > I'd like to kick off discussion on a small KIP to move towards a
> > unified
> > > > > EOS and clean up the current options. Please give it a pass and let
> > me
> > > > know
> > > > > what you think.
> > > > >
> > > > > KIP-732: Deprecate eos-alpha and replace eos-beta with eos-v2
> > > > > <
> > > > >
> > > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-732%3A+Deprecate+eos-alpha+and+replace+eos-beta+with+eos-v2
> > > > > >
> > > > > KAFKA-12574: Deprecate eos-alpha
> > > > > 
> > > > >
> > > > > Thanks,
> > > > > Sophie
> > > > >
> > > >
> > >
> > >
> >
> >
> >
>

Re: [DISCUSS] KIP-732: Deprecate eos-alpha and replace eos-beta with eos-v2

[Gary Russell]

Sophie,

Why not also deprecate Producer.sendOffsetsToTransaction​ method variant with 
the consumerGroupId​ parameter?

This would encourage non-Streams EOS users to move to beta/v2 implementations, 
and the fallback to alpha/v1 mode code can also be removed in 4.0.

From: Sophie Blee-Goldman 
Sent: Wednesday, April 14, 2021 4:38 PM
To: dev 
Subject: Re: [DISCUSS] KIP-732: Deprecate eos-alpha and replace eos-beta with 
eos-v2

Thanks all. I've updated the KIP to clarify these will be removed in 4.0,
and added a note about
logging a warning as well. I agree that some users may set the config value
directly and could
miss the deprecation notice.

I'll call for a vote today since there seems to be consensus, but please
let me know if you have
any more late-breaking ideas

-Sophie

On Wed, Apr 14, 2021 at 1:36 PM Ismael Juma  wrote:

> Hey John,
>
> Yes, we should definitely do that. It's meant to be the standard procedure
> for configs that are deprecated.
>
> Ismael
>
> On Wed, Apr 14, 2021 at 12:02 PM John Roesler  wrote:
>
> > Hello again, Sophie,
> >
> > I've just had a late-breaking thought about your KIP. It
> > might be worthwhile also printing a WARN log when the
> > deprecated configs are used. People may not always be
> > referencing the StreamsConfig fields in their source code
> > when they set these configs, so it would be good to make an
> > extra effort to raise the visibility of this deprecation.
> >
> > Thanks,
> > -John
> >
> > On Wed, 2021-04-14 at 10:47 -0700, Guozhang Wang wrote:
> > > Thanks Sophie for writing the KIP! I'm +1 on the proposal.
> > >
> > > On Wed, Apr 14, 2021 at 8:56 AM Ismael Juma  wrote:
> > >
> > > > Thanks Sophie. This makes sense to me. One question: do we want to be
> > a bit
> > > > clearer about the removal plans? That is, can we say that the
> > deprecated
> > > > configs will be removed in 4.0 (instead of likely to be removed)? The
> > > > implication would be that exactly-once would only work with 2.5+
> while
> > at
> > > > least once would work with all versions. 4.0 is probably 1.5-2 years
> > away,
> > > > so this seems reasonable to me.
> > > >
> > > > Ismael
> > > >
> > > > On Tue, Apr 13, 2021 at 7:53 PM Sophie Blee-Goldman
> > > >  wrote:
> > > >
> > > > > Hey all,
> > > > >
> > > > > I'd like to kick off discussion on a small KIP to move towards a
> > unified
> > > > > EOS and clean up the current options. Please give it a pass and let
> > me
> > > > know
> > > > > what you think.
> > > > >
> > > > > KIP-732: Deprecate eos-alpha and replace eos-beta with eos-v2
> > > > > <
> > > > >
> > > >
> >
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-732%253A%2BDeprecate%2Beos-alpha%2Band%2Breplace%2Beos-beta%2Bwith%2Beos-v2&data=04%7C01%7Cgrussell%40vmware.com%7Ce6b018cadb2a430f68fc08d8ff853e59%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C63754029503538%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=h5heHaemAR70PbdKJu1NzCLchMfHLUDel%2BDSnR80S%2Bs%3D&reserved=0
> > > > > >
> > > > > KAFKA-12574: Deprecate eos-alpha
> > > > > 
> > > > >
> > > > > Thanks,
> > > > > Sophie
> > > > >
> > > >
> > >
> > >
> >
> >
> >
>

Re: [DISCUSS] KIP-732: Deprecate eos-alpha and replace eos-beta with eos-v2

[Sophie Blee-Goldman]

Hey Gary, thanks for the suggestion. My impression is that the tech debt
burden of maintaining
the eos-alpha style code in the Producer is significantly less than what we
have to do in Streams,
since in the Producers eyes eos-alpha reduces to a subset of eos-beta/v2 in
which only a single
Producer/transactionalId is active across the entire consumer group.

That said, we may as well encourage non-Streams users to migrate towards
eos-v2 anyways and at
least reduce the surface area of the API if nothing else.

I don't have a strong opinion either way, so unless anyone objects I'll add
this deprecation to the KIP
as well:

@Deprecated
public void sendOffsetsToTransaction(Map
offsets, String consumerGroupId) ;


On Wed, Apr 14, 2021 at 1:54 PM Gary Russell  wrote:

> Sophie,
>
> Why not also deprecate Producer.sendOffsetsToTransaction​ method variant
> with the consumerGroupId​ parameter?
>
> This would encourage non-Streams EOS users to move to beta/v2
> implementations, and the fallback to alpha/v1 mode code can also be removed
> in 4.0.
> 
> From: Sophie Blee-Goldman 
> Sent: Wednesday, April 14, 2021 4:38 PM
> To: dev 
> Subject: Re: [DISCUSS] KIP-732: Deprecate eos-alpha and replace eos-beta
> with eos-v2
>
> Thanks all. I've updated the KIP to clarify these will be removed in 4.0,
> and added a note about
> logging a warning as well. I agree that some users may set the config value
> directly and could
> miss the deprecation notice.
>
> I'll call for a vote today since there seems to be consensus, but please
> let me know if you have
> any more late-breaking ideas
>
> -Sophie
>
> On Wed, Apr 14, 2021 at 1:36 PM Ismael Juma  wrote:
>
> > Hey John,
> >
> > Yes, we should definitely do that. It's meant to be the standard
> procedure
> > for configs that are deprecated.
> >
> > Ismael
> >
> > On Wed, Apr 14, 2021 at 12:02 PM John Roesler 
> wrote:
> >
> > > Hello again, Sophie,
> > >
> > > I've just had a late-breaking thought about your KIP. It
> > > might be worthwhile also printing a WARN log when the
> > > deprecated configs are used. People may not always be
> > > referencing the StreamsConfig fields in their source code
> > > when they set these configs, so it would be good to make an
> > > extra effort to raise the visibility of this deprecation.
> > >
> > > Thanks,
> > > -John
> > >
> > > On Wed, 2021-04-14 at 10:47 -0700, Guozhang Wang wrote:
> > > > Thanks Sophie for writing the KIP! I'm +1 on the proposal.
> > > >
> > > > On Wed, Apr 14, 2021 at 8:56 AM Ismael Juma 
> wrote:
> > > >
> > > > > Thanks Sophie. This makes sense to me. One question: do we want to
> be
> > > a bit
> > > > > clearer about the removal plans? That is, can we say that the
> > > deprecated
> > > > > configs will be removed in 4.0 (instead of likely to be removed)?
> The
> > > > > implication would be that exactly-once would only work with 2.5+
> > while
> > > at
> > > > > least once would work with all versions. 4.0 is probably 1.5-2
> years
> > > away,
> > > > > so this seems reasonable to me.
> > > > >
> > > > > Ismael
> > > > >
> > > > > On Tue, Apr 13, 2021 at 7:53 PM Sophie Blee-Goldman
> > > > >  wrote:
> > > > >
> > > > > > Hey all,
> > > > > >
> > > > > > I'd like to kick off discussion on a small KIP to move towards a
> > > unified
> > > > > > EOS and clean up the current options. Please give it a pass and
> let
> > > me
> > > > > know
> > > > > > what you think.
> > > > > >
> > > > > > KIP-732: Deprecate eos-alpha and replace eos-beta with eos-v2
> > > > > > <
> > > > > >
> > > > >
> > >
> >
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-732%253A%2BDeprecate%2Beos-alpha%2Band%2Breplace%2Beos-beta%2Bwith%2Beos-v2&data=04%7C01%7Cgrussell%40vmware.com%7Ce6b018cadb2a430f68fc08d8ff853e59%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C63754029503538%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=h5heHaemAR70PbdKJu1NzCLchMfHLUDel%2BDSnR80S%2Bs%3D&reserved=0
> > > > > > >
> > > > > > KAFKA-12574: Deprecate eos-alpha
> > > > > > <
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FKAFKA-12574&data=04%7C01%7Cgrussell%40vmware.com%7Ce6b018cadb2a430f68fc08d8ff853e59%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C63754029503538%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=mNS1c%2BackYwumSJUZlU5R92HokWB6vUs%2FNkR%2BeZ0GSY%3D&reserved=0
> >
> > > > > >
> > > > > > Thanks,
> > > > > > Sophie
> > > > > >
> > > > >
> > > >
> > > >
> > >
> > >
> > >
> >
>

Re: [DISCUSS] KIP-732: Deprecate eos-alpha and replace eos-beta with eos-v2

[Matthias J. Sax]

I don't think we should deprecate the producer method, because there are
also "producer only" application that may use this method.

For example, source connectors.


-Matthias

On 4/14/21 2:12 PM, Sophie Blee-Goldman wrote:
> Hey Gary, thanks for the suggestion. My impression is that the tech debt
> burden of maintaining
> the eos-alpha style code in the Producer is significantly less than what we
> have to do in Streams,
> since in the Producers eyes eos-alpha reduces to a subset of eos-beta/v2 in
> which only a single
> Producer/transactionalId is active across the entire consumer group.
> 
> That said, we may as well encourage non-Streams users to migrate towards
> eos-v2 anyways and at
> least reduce the surface area of the API if nothing else.
> 
> I don't have a strong opinion either way, so unless anyone objects I'll add
> this deprecation to the KIP
> as well:
> 
> @Deprecated
> public void sendOffsetsToTransaction(Map
> offsets, String consumerGroupId) ;
> 
> 
> On Wed, Apr 14, 2021 at 1:54 PM Gary Russell  wrote:
> 
>> Sophie,
>>
>> Why not also deprecate Producer.sendOffsetsToTransaction​ method variant
>> with the consumerGroupId​ parameter?
>>
>> This would encourage non-Streams EOS users to move to beta/v2
>> implementations, and the fallback to alpha/v1 mode code can also be removed
>> in 4.0.
>> 
>> From: Sophie Blee-Goldman 
>> Sent: Wednesday, April 14, 2021 4:38 PM
>> To: dev 
>> Subject: Re: [DISCUSS] KIP-732: Deprecate eos-alpha and replace eos-beta
>> with eos-v2
>>
>> Thanks all. I've updated the KIP to clarify these will be removed in 4.0,
>> and added a note about
>> logging a warning as well. I agree that some users may set the config value
>> directly and could
>> miss the deprecation notice.
>>
>> I'll call for a vote today since there seems to be consensus, but please
>> let me know if you have
>> any more late-breaking ideas
>>
>> -Sophie
>>
>> On Wed, Apr 14, 2021 at 1:36 PM Ismael Juma  wrote:
>>
>>> Hey John,
>>>
>>> Yes, we should definitely do that. It's meant to be the standard
>> procedure
>>> for configs that are deprecated.
>>>
>>> Ismael
>>>
>>> On Wed, Apr 14, 2021 at 12:02 PM John Roesler 
>> wrote:
>>>
 Hello again, Sophie,

 I've just had a late-breaking thought about your KIP. It
 might be worthwhile also printing a WARN log when the
 deprecated configs are used. People may not always be
 referencing the StreamsConfig fields in their source code
 when they set these configs, so it would be good to make an
 extra effort to raise the visibility of this deprecation.

 Thanks,
 -John

 On Wed, 2021-04-14 at 10:47 -0700, Guozhang Wang wrote:
> Thanks Sophie for writing the KIP! I'm +1 on the proposal.
>
> On Wed, Apr 14, 2021 at 8:56 AM Ismael Juma 
>> wrote:
>
>> Thanks Sophie. This makes sense to me. One question: do we want to
>> be
 a bit
>> clearer about the removal plans? That is, can we say that the
 deprecated
>> configs will be removed in 4.0 (instead of likely to be removed)?
>> The
>> implication would be that exactly-once would only work with 2.5+
>>> while
 at
>> least once would work with all versions. 4.0 is probably 1.5-2
>> years
 away,
>> so this seems reasonable to me.
>>
>> Ismael
>>
>> On Tue, Apr 13, 2021 at 7:53 PM Sophie Blee-Goldman
>>  wrote:
>>
>>> Hey all,
>>>
>>> I'd like to kick off discussion on a small KIP to move towards a
 unified
>>> EOS and clean up the current options. Please give it a pass and
>> let
 me
>> know
>>> what you think.
>>>
>>> KIP-732: Deprecate eos-alpha and replace eos-beta with eos-v2
>>> <
>>>
>>

>>>
>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-732%253A%2BDeprecate%2Beos-alpha%2Band%2Breplace%2Beos-beta%2Bwith%2Beos-v2&data=04%7C01%7Cgrussell%40vmware.com%7Ce6b018cadb2a430f68fc08d8ff853e59%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C63754029503538%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=h5heHaemAR70PbdKJu1NzCLchMfHLUDel%2BDSnR80S%2Bs%3D&reserved=0

>>> KAFKA-12574: Deprecate eos-alpha
>>> <
>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FKAFKA-12574&data=04%7C01%7Cgrussell%40vmware.com%7Ce6b018cadb2a430f68fc08d8ff853e59%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C63754029503538%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=mNS1c%2BackYwumSJUZlU5R92HokWB6vUs%2FNkR%2BeZ0GSY%3D&reserved=0
>>>
>>>
>>> Thanks,
>>> Sophie
>>>
>>
>
>



>>>
>>
> 

Re: [DISCUSS] KIP-732: Deprecate eos-alpha and replace eos-beta with eos-v2

[Sophie Blee-Goldman]

Well, the method being deprecated has a consumerGroupId parameter... if
it's a "producer only"
application then do they just set this as null, or empty string?

Btw: this method actually just invokes the other overload, which accepts a
ConsumerGroupMetadata
instead of a String consumerGroupId. So even a "producer only" application
could use this, and just
do the same:

public void sendOffsetsToTransaction(Map offsets,
 String consumerGroupId) throws
ProducerFencedException {
sendOffsetsToTransaction(offsets, new
ConsumerGroupMetadata(consumerGroupId));
}


On Wed, Apr 14, 2021 at 2:19 PM Matthias J. Sax  wrote:

> I don't think we should deprecate the producer method, because there are
> also "producer only" application that may use this method.
>
> For example, source connectors.
>
>
> -Matthias
>
> On 4/14/21 2:12 PM, Sophie Blee-Goldman wrote:
> > Hey Gary, thanks for the suggestion. My impression is that the tech debt
> > burden of maintaining
> > the eos-alpha style code in the Producer is significantly less than what
> we
> > have to do in Streams,
> > since in the Producers eyes eos-alpha reduces to a subset of eos-beta/v2
> in
> > which only a single
> > Producer/transactionalId is active across the entire consumer group.
> >
> > That said, we may as well encourage non-Streams users to migrate towards
> > eos-v2 anyways and at
> > least reduce the surface area of the API if nothing else.
> >
> > I don't have a strong opinion either way, so unless anyone objects I'll
> add
> > this deprecation to the KIP
> > as well:
> >
> > @Deprecated
> > public void sendOffsetsToTransaction(Map OffsetAndMetadata>
> > offsets, String consumerGroupId) ;
> >
> >
> > On Wed, Apr 14, 2021 at 1:54 PM Gary Russell 
> wrote:
> >
> >> Sophie,
> >>
> >> Why not also deprecate Producer.sendOffsetsToTransaction​ method variant
> >> with the consumerGroupId​ parameter?
> >>
> >> This would encourage non-Streams EOS users to move to beta/v2
> >> implementations, and the fallback to alpha/v1 mode code can also be
> removed
> >> in 4.0.
> >> 
> >> From: Sophie Blee-Goldman 
> >> Sent: Wednesday, April 14, 2021 4:38 PM
> >> To: dev 
> >> Subject: Re: [DISCUSS] KIP-732: Deprecate eos-alpha and replace eos-beta
> >> with eos-v2
> >>
> >> Thanks all. I've updated the KIP to clarify these will be removed in
> 4.0,
> >> and added a note about
> >> logging a warning as well. I agree that some users may set the config
> value
> >> directly and could
> >> miss the deprecation notice.
> >>
> >> I'll call for a vote today since there seems to be consensus, but please
> >> let me know if you have
> >> any more late-breaking ideas
> >>
> >> -Sophie
> >>
> >> On Wed, Apr 14, 2021 at 1:36 PM Ismael Juma  wrote:
> >>
> >>> Hey John,
> >>>
> >>> Yes, we should definitely do that. It's meant to be the standard
> >> procedure
> >>> for configs that are deprecated.
> >>>
> >>> Ismael
> >>>
> >>> On Wed, Apr 14, 2021 at 12:02 PM John Roesler 
> >> wrote:
> >>>
>  Hello again, Sophie,
> 
>  I've just had a late-breaking thought about your KIP. It
>  might be worthwhile also printing a WARN log when the
>  deprecated configs are used. People may not always be
>  referencing the StreamsConfig fields in their source code
>  when they set these configs, so it would be good to make an
>  extra effort to raise the visibility of this deprecation.
> 
>  Thanks,
>  -John
> 
>  On Wed, 2021-04-14 at 10:47 -0700, Guozhang Wang wrote:
> > Thanks Sophie for writing the KIP! I'm +1 on the proposal.
> >
> > On Wed, Apr 14, 2021 at 8:56 AM Ismael Juma 
> >> wrote:
> >
> >> Thanks Sophie. This makes sense to me. One question: do we want to
> >> be
>  a bit
> >> clearer about the removal plans? That is, can we say that the
>  deprecated
> >> configs will be removed in 4.0 (instead of likely to be removed)?
> >> The
> >> implication would be that exactly-once would only work with 2.5+
> >>> while
>  at
> >> least once would work with all versions. 4.0 is probably 1.5-2
> >> years
>  away,
> >> so this seems reasonable to me.
> >>
> >> Ismael
> >>
> >> On Tue, Apr 13, 2021 at 7:53 PM Sophie Blee-Goldman
> >>  wrote:
> >>
> >>> Hey all,
> >>>
> >>> I'd like to kick off discussion on a small KIP to move towards a
>  unified
> >>> EOS and clean up the current options. Please give it a pass and
> >> let
>  me
> >> know
> >>> what you think.
> >>>
> >>> KIP-732: Deprecate eos-alpha and replace eos-beta with eos-v2
> >>> <
> >>>
> >>
> 
> >>>
> >>
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FKAFKA%2FKIP-732%253A%2BDeprecate%2Beos-alpha%2Band%2Breplace%2Beos-beta%2Bwith%2Beos-v2&data=04%7C01%7Cgrussell%40vmware.com%7Ce6b018cadb2a430f68fc08d8ff853e59%7Cb39138

Re: [DISCUSS] KIP-718: Make KTable Join on Foreign key unopinionated

[Matthias J. Sax]

If you argue this way, would the consequence not be that we would need
to add even more overloads that allow to only use Materialized for the
subscription store?

-> I only want to switch to in-memory for the subscription store: Why do
I need to set Materialized for the main store?

Btw: the above is not something I would really argue for.


The surface area of our public API is quite big and tend to overwhelm
users, and thus I would like to keep it small is possible.


> Must the user know about the subscription-stores beforehand? If yes, then
> indeed we should deprecate.

I tend to argue "yes".


Keeping the old methods seems to be a rare use case IMHO. It seems most
likely that user want to overwrite all stores. And thus, it seems to be
error prone: I want to overwrite the stores, but I forget to overwrite
the subscription store.

And even if one wants to keep RocksDB for the subscription stores only,
they can still pass in both Materialized object using the new methods.



-Matthias


On 4/14/21 1:23 PM, Marco Aurélio Lotz wrote:
> Hi Matthias,
> 
> Thanks for reviewing it. That's a nice question. I can provide you the
> answer as a user of the API - but of course it very much depends on what
> the original intention towards the user is.
> The only reason I found out it was opinionated was when my tests using
> in-memory materialization started failing due to the state-store clean-up
> bug
> 
> in Windows systems.
> 
> To be honest, I think I wouldn't have investigated how subscription stores
> are used without seeing that my system couldn't handle the expected data.
> Must the user know about the subscription-stores beforehand? If yes, then
> indeed we should deprecate.
> 
> Having said that, it seems to me that deprecating would not be consistent
> with other parts of the API that have parameterized methods hiding
> complexity.
> E.g. from the top of my head, join() will make a rocksDB materialization if
> no-materialization method is provided.
> 
> public  KTable join(final KTable other, final
> ValueJoiner joiner0
> 
> 
> I see this as something equivalent. If the user didn't provide all the
> methods, shouldn't it be fine to fallback to a default materialization
> method as the other sections of the API already does?
> In that scenario, there's no need to deprecate and we just need to make a
> reasonable guess towards it. Using the same method as the table-store
> sounds like a reasonable guess in this scenario.
> 
> In my perspective as a user, I would prefer to have simpler methods knowing
> I would be losing some fine-grain configuration. Does it sound reasonable
> to you?
> 
> Cheers,
> Marco Lotz
> 
> On Wed, Apr 14, 2021 at 8:47 PM Matthias J. Sax  wrote:
> 
>> I just re-read the KIP, and I am wondering why we would *only add* new
>> methods.
>>
>> Is it an expected use case to only change the main stores but not the
>> subscription stores?
>>
>> Wondering if we should deprecate the existing methods that only accept a
>> single `Materialized` parameter?
>>
>>
>> -Matthias
>>
>> On 4/9/21 1:32 PM, Marco Aurélio Lotz wrote:
>>> Hi everyone,
>>>
>>> I am fine with sticking with Materialized and adding the info to the
>>> javadoc then - so we keep the footprint smaller.
>>> I will then update the KIP to match what we discussed here and send it
>> for
>>> a vote next week.
>>>
>>> I will raise a new bug-fix ticket and change KAFKA-10383
>>>  to become a feature
>> -
>>> if that's ok.
>>>
>>> Cheers,
>>> Marco
>>>
>>> On Wed, Apr 7, 2021 at 4:15 AM Matthias J. Sax  wrote:
>>>
 Just catching up here.

 I agree that we have two issue, and the first (align subscription store
 to main store) can be done as a bug-fix.

 For the KIP (that addressed the second), I tend to agree that reusing
 `Materialized` might be better as it would keep the API surface area
 smaller.


 -Matthias

 On 4/6/21 8:48 AM, John Roesler wrote:
> Hi Marco,
>
> Just a quick clarification: I just reviewed the Materialized class. It
 looks like the only undesirable members are:
> 1. Retention
> 2. Key/Value serdes
>
> The underlying store type would be “KeyValueStore” , for
 which case the withRetention javadoc already says it’s ignored.
>
> Perhaps we could just stick with Materialized by adding a note to the
 Key/Value serdes setters that they are ignored for FKJoin subscription
 stores?
>
> Not as elegant as a new config class, but these config classes actually
 bring a fair amount of complexity, so it might be nice to avoid a new
>> one.
>
> Thanks,
> John
>
> On Tue, Apr 6, 2021, at 10:28, Marco Aurélio Lotz wrote:
>> Hi John / Guozhang,
>>
>> If I correctly understood John's message, he agrees on having the two

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #39

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-732: Deprecate eos-alpha and replace eos-beta with eos-v2

[Matthias J. Sax]

Ah. My bad.

For a "producer only" app, you won't call either of both... As you don't
have any associated consumer, you won't need to add offsets to a TX to
begin with.

Calling `sendOffsetsToTransaction` to optional after all.

I withdraw my objection :)


-Matthias

On 4/14/21 2:22 PM, Sophie Blee-Goldman wrote:
> Well, the method being deprecated has a consumerGroupId parameter... if
> it's a "producer only"
> application then do they just set this as null, or empty string?
> 
> Btw: this method actually just invokes the other overload, which accepts a
> ConsumerGroupMetadata
> instead of a String consumerGroupId. So even a "producer only" application
> could use this, and just
> do the same:
> 
> public void sendOffsetsToTransaction(Map OffsetAndMetadata> offsets,
>  String consumerGroupId) throws
> ProducerFencedException {
> sendOffsetsToTransaction(offsets, new
> ConsumerGroupMetadata(consumerGroupId));
> }
> 
> 
> On Wed, Apr 14, 2021 at 2:19 PM Matthias J. Sax  wrote:
> 
>> I don't think we should deprecate the producer method, because there are
>> also "producer only" application that may use this method.
>>
>> For example, source connectors.
>>
>>
>> -Matthias
>>
>> On 4/14/21 2:12 PM, Sophie Blee-Goldman wrote:
>>> Hey Gary, thanks for the suggestion. My impression is that the tech debt
>>> burden of maintaining
>>> the eos-alpha style code in the Producer is significantly less than what
>> we
>>> have to do in Streams,
>>> since in the Producers eyes eos-alpha reduces to a subset of eos-beta/v2
>> in
>>> which only a single
>>> Producer/transactionalId is active across the entire consumer group.
>>>
>>> That said, we may as well encourage non-Streams users to migrate towards
>>> eos-v2 anyways and at
>>> least reduce the surface area of the API if nothing else.
>>>
>>> I don't have a strong opinion either way, so unless anyone objects I'll
>> add
>>> this deprecation to the KIP
>>> as well:
>>>
>>> @Deprecated
>>> public void sendOffsetsToTransaction(Map> OffsetAndMetadata>
>>> offsets, String consumerGroupId) ;
>>>
>>>
>>> On Wed, Apr 14, 2021 at 1:54 PM Gary Russell 
>> wrote:
>>>
 Sophie,

 Why not also deprecate Producer.sendOffsetsToTransaction​ method variant
 with the consumerGroupId​ parameter?

 This would encourage non-Streams EOS users to move to beta/v2
 implementations, and the fallback to alpha/v1 mode code can also be
>> removed
 in 4.0.
 
 From: Sophie Blee-Goldman 
 Sent: Wednesday, April 14, 2021 4:38 PM
 To: dev 
 Subject: Re: [DISCUSS] KIP-732: Deprecate eos-alpha and replace eos-beta
 with eos-v2

 Thanks all. I've updated the KIP to clarify these will be removed in
>> 4.0,
 and added a note about
 logging a warning as well. I agree that some users may set the config
>> value
 directly and could
 miss the deprecation notice.

 I'll call for a vote today since there seems to be consensus, but please
 let me know if you have
 any more late-breaking ideas

 -Sophie

 On Wed, Apr 14, 2021 at 1:36 PM Ismael Juma  wrote:

> Hey John,
>
> Yes, we should definitely do that. It's meant to be the standard
 procedure
> for configs that are deprecated.
>
> Ismael
>
> On Wed, Apr 14, 2021 at 12:02 PM John Roesler 
 wrote:
>
>> Hello again, Sophie,
>>
>> I've just had a late-breaking thought about your KIP. It
>> might be worthwhile also printing a WARN log when the
>> deprecated configs are used. People may not always be
>> referencing the StreamsConfig fields in their source code
>> when they set these configs, so it would be good to make an
>> extra effort to raise the visibility of this deprecation.
>>
>> Thanks,
>> -John
>>
>> On Wed, 2021-04-14 at 10:47 -0700, Guozhang Wang wrote:
>>> Thanks Sophie for writing the KIP! I'm +1 on the proposal.
>>>
>>> On Wed, Apr 14, 2021 at 8:56 AM Ismael Juma 
 wrote:
>>>
 Thanks Sophie. This makes sense to me. One question: do we want to
 be
>> a bit
 clearer about the removal plans? That is, can we say that the
>> deprecated
 configs will be removed in 4.0 (instead of likely to be removed)?
 The
 implication would be that exactly-once would only work with 2.5+
> while
>> at
 least once would work with all versions. 4.0 is probably 1.5-2
 years
>> away,
 so this seems reasonable to me.

 Ismael

 On Tue, Apr 13, 2021 at 7:53 PM Sophie Blee-Goldman
  wrote:

> Hey all,
>
> I'd like to kick off discussion on a small KIP to move towards a
>> unified
> EOS and clean up the current options. Please give it a pass and
 let
>> me
 know
> what you think.
>
>>>

Re: [DISCUSS] KIP-732: Deprecate eos-alpha and replace eos-beta with eos-v2

[Gary Russell]

There is also all that downgrade stuff in the TxnOffsetCommitRequest​ that will 
downgrade to "alpha" if the broker is < 2.5 (if 
ProducerConfig.AUTO_DOWNGRADE_TXN_COMMIT​ is set).

If the intention is to eventually pull support for alpha/v1 altogether then why 
not remove all that too?

From: Sophie Blee-Goldman 
Sent: Wednesday, April 14, 2021 5:22 PM
To: dev 
Subject: Re: [DISCUSS] KIP-732: Deprecate eos-alpha and replace eos-beta with 
eos-v2

Well, the method being deprecated has a consumerGroupId parameter... if
it's a "producer only"
application then do they just set this as null, or empty string?

Btw: this method actually just invokes the other overload, which accepts a
ConsumerGroupMetadata
instead of a String consumerGroupId. So even a "producer only" application
could use this, and just
do the same:

public void sendOffsetsToTransaction(Map offsets,
 String consumerGroupId) throws
ProducerFencedException {
sendOffsetsToTransaction(offsets, new
ConsumerGroupMetadata(consumerGroupId));
}


On Wed, Apr 14, 2021 at 2:19 PM Matthias J. Sax  wrote:

> I don't think we should deprecate the producer method, because there are
> also "producer only" application that may use this method.
>
> For example, source connectors.
>
>
> -Matthias
>
> On 4/14/21 2:12 PM, Sophie Blee-Goldman wrote:
> > Hey Gary, thanks for the suggestion. My impression is that the tech debt
> > burden of maintaining
> > the eos-alpha style code in the Producer is significantly less than what
> we
> > have to do in Streams,
> > since in the Producers eyes eos-alpha reduces to a subset of eos-beta/v2
> in
> > which only a single
> > Producer/transactionalId is active across the entire consumer group.
> >
> > That said, we may as well encourage non-Streams users to migrate towards
> > eos-v2 anyways and at
> > least reduce the surface area of the API if nothing else.
> >
> > I don't have a strong opinion either way, so unless anyone objects I'll
> add
> > this deprecation to the KIP
> > as well:
> >
> > @Deprecated
> > public void sendOffsetsToTransaction(Map OffsetAndMetadata>
> > offsets, String consumerGroupId) ;
> >
> >
> > On Wed, Apr 14, 2021 at 1:54 PM Gary Russell 
> wrote:
> >
> >> Sophie,
> >>
> >> Why not also deprecate Producer.sendOffsetsToTransaction​ method variant
> >> with the consumerGroupId​ parameter?
> >>
> >> This would encourage non-Streams EOS users to move to beta/v2
> >> implementations, and the fallback to alpha/v1 mode code can also be
> removed
> >> in 4.0.
> >> 
> >> From: Sophie Blee-Goldman 
> >> Sent: Wednesday, April 14, 2021 4:38 PM
> >> To: dev 
> >> Subject: Re: [DISCUSS] KIP-732: Deprecate eos-alpha and replace eos-beta
> >> with eos-v2
> >>
> >> Thanks all. I've updated the KIP to clarify these will be removed in
> 4.0,
> >> and added a note about
> >> logging a warning as well. I agree that some users may set the config
> value
> >> directly and could
> >> miss the deprecation notice.
> >>
> >> I'll call for a vote today since there seems to be consensus, but please
> >> let me know if you have
> >> any more late-breaking ideas
> >>
> >> -Sophie
> >>
> >> On Wed, Apr 14, 2021 at 1:36 PM Ismael Juma  wrote:
> >>
> >>> Hey John,
> >>>
> >>> Yes, we should definitely do that. It's meant to be the standard
> >> procedure
> >>> for configs that are deprecated.
> >>>
> >>> Ismael
> >>>
> >>> On Wed, Apr 14, 2021 at 12:02 PM John Roesler 
> >> wrote:
> >>>
>  Hello again, Sophie,
> 
>  I've just had a late-breaking thought about your KIP. It
>  might be worthwhile also printing a WARN log when the
>  deprecated configs are used. People may not always be
>  referencing the StreamsConfig fields in their source code
>  when they set these configs, so it would be good to make an
>  extra effort to raise the visibility of this deprecation.
> 
>  Thanks,
>  -John
> 
>  On Wed, 2021-04-14 at 10:47 -0700, Guozhang Wang wrote:
> > Thanks Sophie for writing the KIP! I'm +1 on the proposal.
> >
> > On Wed, Apr 14, 2021 at 8:56 AM Ismael Juma 
> >> wrote:
> >
> >> Thanks Sophie. This makes sense to me. One question: do we want to
> >> be
>  a bit
> >> clearer about the removal plans? That is, can we say that the
>  deprecated
> >> configs will be removed in 4.0 (instead of likely to be removed)?
> >> The
> >> implication would be that exactly-once would only work with 2.5+
> >>> while
>  at
> >> least once would work with all versions. 4.0 is probably 1.5-2
> >> years
>  away,
> >> so this seems reasonable to me.
> >>
> >> Ismael
> >>
> >> On Tue, Apr 13, 2021 at 7:53 PM Sophie Blee-Goldman
> >>  wrote:
> >>
> >>> Hey all,
> >>>
> >>> I'd like to kick off discussion on a small KIP to move towards a
>  unified
> >>> EOS and clean up the current options. Please giv

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 2.8 #11

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-732: Deprecate eos-alpha and replace eos-beta with eos-v2

[Ismael Juma]

Kafka protocol support has a different compatibility policy. For example,
we still support 0.8.x protocol versions to this day.

Ismael

On Wed, Apr 14, 2021 at 2:43 PM Gary Russell  wrote:

> There is also all that downgrade stuff in the TxnOffsetCommitRequest​ that
> will downgrade to "alpha" if the broker is < 2.5 (if
> ProducerConfig.AUTO_DOWNGRADE_TXN_COMMIT​ is set).
>
> If the intention is to eventually pull support for alpha/v1 altogether
> then why not remove all that too?
> 
> From: Sophie Blee-Goldman 
> Sent: Wednesday, April 14, 2021 5:22 PM
> To: dev 
> Subject: Re: [DISCUSS] KIP-732: Deprecate eos-alpha and replace eos-beta
> with eos-v2
>
> Well, the method being deprecated has a consumerGroupId parameter... if
> it's a "producer only"
> application then do they just set this as null, or empty string?
>
> Btw: this method actually just invokes the other overload, which accepts a
> ConsumerGroupMetadata
> instead of a String consumerGroupId. So even a "producer only" application
> could use this, and just
> do the same:
>
> public void sendOffsetsToTransaction(Map OffsetAndMetadata> offsets,
>  String consumerGroupId) throws
> ProducerFencedException {
> sendOffsetsToTransaction(offsets, new
> ConsumerGroupMetadata(consumerGroupId));
> }
>
>
> On Wed, Apr 14, 2021 at 2:19 PM Matthias J. Sax  wrote:
>
> > I don't think we should deprecate the producer method, because there are
> > also "producer only" application that may use this method.
> >
> > For example, source connectors.
> >
> >
> > -Matthias
> >
> > On 4/14/21 2:12 PM, Sophie Blee-Goldman wrote:
> > > Hey Gary, thanks for the suggestion. My impression is that the tech
> debt
> > > burden of maintaining
> > > the eos-alpha style code in the Producer is significantly less than
> what
> > we
> > > have to do in Streams,
> > > since in the Producers eyes eos-alpha reduces to a subset of
> eos-beta/v2
> > in
> > > which only a single
> > > Producer/transactionalId is active across the entire consumer group.
> > >
> > > That said, we may as well encourage non-Streams users to migrate
> towards
> > > eos-v2 anyways and at
> > > least reduce the surface area of the API if nothing else.
> > >
> > > I don't have a strong opinion either way, so unless anyone objects I'll
> > add
> > > this deprecation to the KIP
> > > as well:
> > >
> > > @Deprecated
> > > public void sendOffsetsToTransaction(Map > OffsetAndMetadata>
> > > offsets, String consumerGroupId) ;
> > >
> > >
> > > On Wed, Apr 14, 2021 at 1:54 PM Gary Russell 
> > wrote:
> > >
> > >> Sophie,
> > >>
> > >> Why not also deprecate Producer.sendOffsetsToTransaction​ method
> variant
> > >> with the consumerGroupId​ parameter?
> > >>
> > >> This would encourage non-Streams EOS users to move to beta/v2
> > >> implementations, and the fallback to alpha/v1 mode code can also be
> > removed
> > >> in 4.0.
> > >> 
> > >> From: Sophie Blee-Goldman 
> > >> Sent: Wednesday, April 14, 2021 4:38 PM
> > >> To: dev 
> > >> Subject: Re: [DISCUSS] KIP-732: Deprecate eos-alpha and replace
> eos-beta
> > >> with eos-v2
> > >>
> > >> Thanks all. I've updated the KIP to clarify these will be removed in
> > 4.0,
> > >> and added a note about
> > >> logging a warning as well. I agree that some users may set the config
> > value
> > >> directly and could
> > >> miss the deprecation notice.
> > >>
> > >> I'll call for a vote today since there seems to be consensus, but
> please
> > >> let me know if you have
> > >> any more late-breaking ideas
> > >>
> > >> -Sophie
> > >>
> > >> On Wed, Apr 14, 2021 at 1:36 PM Ismael Juma 
> wrote:
> > >>
> > >>> Hey John,
> > >>>
> > >>> Yes, we should definitely do that. It's meant to be the standard
> > >> procedure
> > >>> for configs that are deprecated.
> > >>>
> > >>> Ismael
> > >>>
> > >>> On Wed, Apr 14, 2021 at 12:02 PM John Roesler 
> > >> wrote:
> > >>>
> >  Hello again, Sophie,
> > 
> >  I've just had a late-breaking thought about your KIP. It
> >  might be worthwhile also printing a WARN log when the
> >  deprecated configs are used. People may not always be
> >  referencing the StreamsConfig fields in their source code
> >  when they set these configs, so it would be good to make an
> >  extra effort to raise the visibility of this deprecation.
> > 
> >  Thanks,
> >  -John
> > 
> >  On Wed, 2021-04-14 at 10:47 -0700, Guozhang Wang wrote:
> > > Thanks Sophie for writing the KIP! I'm +1 on the proposal.
> > >
> > > On Wed, Apr 14, 2021 at 8:56 AM Ismael Juma 
> > >> wrote:
> > >
> > >> Thanks Sophie. This makes sense to me. One question: do we want to
> > >> be
> >  a bit
> > >> clearer about the removal plans? That is, can we say that the
> >  deprecated
> > >> configs will be removed in 4.0 (instead of likely to be removed)?
> > >> The
> > >> implication would be that exa

[VOTE} KIP-733: change Kafka Streams default replication factor config

[Matthias J. Sax]

Hi,

Because this KIP is rather small, I would like to skip a dedicated
discussion thread and call for a vote right way. If there are any
concerns, we can just discuss on this vote thread:

https://cwiki.apache.org/confluence/display/KAFKA/KIP-733%3A+change+Kafka+Streams+default+replication+factor+config

Note, that we actually backed this change via
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=113708722
already.

However, I felt it might be worth do make this change more explicit as
KIP-464 is rather old now.

Quote from KIP-464:

> To exploit this new feature in KafkaStreams, we update the default value of 
> Streams configuration parameter `replication.factor` from `1` to `-1`.




-Matthias

Re: [VOTE} KIP-733: change Kafka Streams default replication factor config

[Sophie Blee-Goldman]

Thanks Matthias. I'm +1 (binding)

-Sophie

On Wed, Apr 14, 2021 at 3:36 PM Matthias J. Sax  wrote:

> Hi,
>
> Because this KIP is rather small, I would like to skip a dedicated
> discussion thread and call for a vote right way. If there are any
> concerns, we can just discuss on this vote thread:
>
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-733%3A+change+Kafka+Streams+default+replication+factor+config
>
> Note, that we actually backed this change via
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=113708722
> already.
>
> However, I felt it might be worth do make this change more explicit as
> KIP-464 is rather old now.
>
> Quote from KIP-464:
>
> > To exploit this new feature in KafkaStreams, we update the default value
> of Streams configuration parameter `replication.factor` from `1` to `-1`.
>
>
>
>
> -Matthias
>

Re: [DISCUSS] KIP-732: Deprecate eos-alpha and replace eos-beta with eos-v2

[Sophie Blee-Goldman]

Thanks all, I added a note to the KIP about deprecating the old
Producer#sendOffsetsToTransaction method.
I think this is sufficient for now, and keeps this KIP well scoped. If we
have reason to do a more aggressive
cleanup that doesn't violate any compatibility policy, someone can pick
that up in another KIP.

Cheers,
Sophie

On Wed, Apr 14, 2021 at 3:05 PM Ismael Juma  wrote:

> Kafka protocol support has a different compatibility policy. For example,
> we still support 0.8.x protocol versions to this day.
>
> Ismael
>
> On Wed, Apr 14, 2021 at 2:43 PM Gary Russell  wrote:
>
> > There is also all that downgrade stuff in the TxnOffsetCommitRequest​
> that
> > will downgrade to "alpha" if the broker is < 2.5 (if
> > ProducerConfig.AUTO_DOWNGRADE_TXN_COMMIT​ is set).
> >
> > If the intention is to eventually pull support for alpha/v1 altogether
> > then why not remove all that too?
> > 
> > From: Sophie Blee-Goldman 
> > Sent: Wednesday, April 14, 2021 5:22 PM
> > To: dev 
> > Subject: Re: [DISCUSS] KIP-732: Deprecate eos-alpha and replace eos-beta
> > with eos-v2
> >
> > Well, the method being deprecated has a consumerGroupId parameter... if
> > it's a "producer only"
> > application then do they just set this as null, or empty string?
> >
> > Btw: this method actually just invokes the other overload, which accepts
> a
> > ConsumerGroupMetadata
> > instead of a String consumerGroupId. So even a "producer only"
> application
> > could use this, and just
> > do the same:
> >
> > public void sendOffsetsToTransaction(Map > OffsetAndMetadata> offsets,
> >  String consumerGroupId) throws
> > ProducerFencedException {
> > sendOffsetsToTransaction(offsets, new
> > ConsumerGroupMetadata(consumerGroupId));
> > }
> >
> >
> > On Wed, Apr 14, 2021 at 2:19 PM Matthias J. Sax 
> wrote:
> >
> > > I don't think we should deprecate the producer method, because there
> are
> > > also "producer only" application that may use this method.
> > >
> > > For example, source connectors.
> > >
> > >
> > > -Matthias
> > >
> > > On 4/14/21 2:12 PM, Sophie Blee-Goldman wrote:
> > > > Hey Gary, thanks for the suggestion. My impression is that the tech
> > debt
> > > > burden of maintaining
> > > > the eos-alpha style code in the Producer is significantly less than
> > what
> > > we
> > > > have to do in Streams,
> > > > since in the Producers eyes eos-alpha reduces to a subset of
> > eos-beta/v2
> > > in
> > > > which only a single
> > > > Producer/transactionalId is active across the entire consumer group.
> > > >
> > > > That said, we may as well encourage non-Streams users to migrate
> > towards
> > > > eos-v2 anyways and at
> > > > least reduce the surface area of the API if nothing else.
> > > >
> > > > I don't have a strong opinion either way, so unless anyone objects
> I'll
> > > add
> > > > this deprecation to the KIP
> > > > as well:
> > > >
> > > > @Deprecated
> > > > public void sendOffsetsToTransaction(Map > > OffsetAndMetadata>
> > > > offsets, String consumerGroupId) ;
> > > >
> > > >
> > > > On Wed, Apr 14, 2021 at 1:54 PM Gary Russell 
> > > wrote:
> > > >
> > > >> Sophie,
> > > >>
> > > >> Why not also deprecate Producer.sendOffsetsToTransaction​ method
> > variant
> > > >> with the consumerGroupId​ parameter?
> > > >>
> > > >> This would encourage non-Streams EOS users to move to beta/v2
> > > >> implementations, and the fallback to alpha/v1 mode code can also be
> > > removed
> > > >> in 4.0.
> > > >> 
> > > >> From: Sophie Blee-Goldman 
> > > >> Sent: Wednesday, April 14, 2021 4:38 PM
> > > >> To: dev 
> > > >> Subject: Re: [DISCUSS] KIP-732: Deprecate eos-alpha and replace
> > eos-beta
> > > >> with eos-v2
> > > >>
> > > >> Thanks all. I've updated the KIP to clarify these will be removed in
> > > 4.0,
> > > >> and added a note about
> > > >> logging a warning as well. I agree that some users may set the
> config
> > > value
> > > >> directly and could
> > > >> miss the deprecation notice.
> > > >>
> > > >> I'll call for a vote today since there seems to be consensus, but
> > please
> > > >> let me know if you have
> > > >> any more late-breaking ideas
> > > >>
> > > >> -Sophie
> > > >>
> > > >> On Wed, Apr 14, 2021 at 1:36 PM Ismael Juma 
> > wrote:
> > > >>
> > > >>> Hey John,
> > > >>>
> > > >>> Yes, we should definitely do that. It's meant to be the standard
> > > >> procedure
> > > >>> for configs that are deprecated.
> > > >>>
> > > >>> Ismael
> > > >>>
> > > >>> On Wed, Apr 14, 2021 at 12:02 PM John Roesler  >
> > > >> wrote:
> > > >>>
> > >  Hello again, Sophie,
> > > 
> > >  I've just had a late-breaking thought about your KIP. It
> > >  might be worthwhile also printing a WARN log when the
> > >  deprecated configs are used. People may not always be
> > >  referencing the StreamsConfig fields in their source code
> > >  when they set these configs, so it would be good to make 

Re: [VOTE] 2.6.2 RC1

[John Roesler]

I verified the signatures and checksums, ran the tests, and
spot-checked the license.

I'm +1 (binding).

Thanks for driving this, Sophie!
-John

On Thu, 2021-04-08 at 16:19 -0700, Guozhang Wang wrote:
> Looked over the javadocs and web docs again. Download the jars and check
> the license files are all updated now (thanks to John again!). +1
> 
> On Thu, Apr 8, 2021 at 1:48 PM Sophie Blee-Goldman
>  wrote:
> 
> > Hello Kafka users, developers and client-developers,
> > 
> > This is the second candidate for release of Apache Kafka 2.6.2.
> > 
> > Apache Kafka 2.6.2 is a bugfix release and fixes 35 issues since the 2.6.1
> > release. Please see the release notes for more information.
> > 
> > Release notes for the 2.6.2 release:
> > https://home.apache.org/~ableegoldman/kafka-2.6.2-rc1/RELEASE_NOTES.html
> > 
> > *** Please download, test and vote by Tuesday, April 13th, 9am PST
> > 
> > Kafka's KEYS file containing PGP keys we use to sign the release:
> > https://kafka.apache.org/KEYS
> > 
> > * Release artifacts to be voted upon (source and binary):
> > https://home.apache.org/~ableegoldman/kafka-2.6.2-rc1/
> > 
> > * Maven artifacts to be voted upon:
> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > 
> > * Javadoc:
> > https://home.apache.org/~ableegoldman/kafka-2.6.2-rc1/javadoc/
> > 
> > * Tag to be voted upon (off 2.6 branch) is the 2.6.2 tag:
> > https://github.com/apache/kafka/releases/tag/2.6.2-rc1
> > 
> > * Documentation:
> > https://kafka.apache.org/26/documentation.html
> > 
> > * Protocol:
> > https://kafka.apache.org/26/protocol.html
> > 
> > * Successful Jenkins builds for the 2.6 branch:
> > Unit/integration tests:
> > https://ci-builds.apache.org/job/Kafka/job/kafka-2.6-jdk8/114/
> > System tests:
> > https://jenkins.confluent.io/job/system-test-kafka-branch-builder/4458/
> > 
> > /**
> > 
> > Thanks,
> > Sophie
> > 
> 
> 

Re: [VOTE} KIP-733: change Kafka Streams default replication factor config

[Bill Bejeck]

Thanks for the KIP Matthias.

+1 (binding)

-Bill

On Wed, Apr 14, 2021 at 7:06 PM Sophie Blee-Goldman
 wrote:

> Thanks Matthias. I'm +1 (binding)
>
> -Sophie
>
> On Wed, Apr 14, 2021 at 3:36 PM Matthias J. Sax  wrote:
>
> > Hi,
> >
> > Because this KIP is rather small, I would like to skip a dedicated
> > discussion thread and call for a vote right way. If there are any
> > concerns, we can just discuss on this vote thread:
> >
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-733%3A+change+Kafka+Streams+default+replication+factor+config
> >
> > Note, that we actually backed this change via
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=113708722
> > already.
> >
> > However, I felt it might be worth do make this change more explicit as
> > KIP-464 is rather old now.
> >
> > Quote from KIP-464:
> >
> > > To exploit this new feature in KafkaStreams, we update the default
> value
> > of Streams configuration parameter `replication.factor` from `1` to `-1`.
> >
> >
> >
> >
> > -Matthias
> >
>

Re: [VOTE] 2.7.1 RC2

[John Roesler]

Hi Mickael,

I verified the signatures and checksums, ran the tests, and
spot-checked the license.

I'm +1 (binding).

Thanks for driving this release!
-John

On Fri, 2021-04-09 at 11:07 +0100, Mickael Maison wrote:
> Hi,
> 
> Here is a successful build of 2.7.1 RC2:
> https://ci-builds.apache.org/job/Kafka/job/kafka-2.7-jdk8/144/
> 
> Thanks
> 
> On Thu, Apr 8, 2021 at 6:27 PM Mickael Maison  wrote:
> > 
> > Hello Kafka users, developers and client-developers,
> > 
> > This is the third candidate for release of Apache Kafka 2.7.1.
> > 
> > Since 2.7.1 RC1, the following JIRAs have been fixed: KAFKA-12593,
> > KAFKA-12474, KAFKA-12602.
> > 
> > Release notes for the 2.7.1 release:
> > https://home.apache.org/~mimaison/kafka-2.7.1-rc2/RELEASE_NOTES.html
> > 
> > *** Please download, test and vote by Friday, April 16, 5pm BST
> > 
> > Kafka's KEYS file containing PGP keys we use to sign the release:
> > https://kafka.apache.org/KEYS
> > 
> > * Release artifacts to be voted upon (source and binary):
> > https://home.apache.org/~mimaison/kafka-2.7.1-rc2/
> > 
> > * Maven artifacts to be voted upon:
> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > 
> > * Javadoc:
> > https://home.apache.org/~mimaison/kafka-2.7.1-rc2/javadoc/
> > 
> > * Tag to be voted upon (off 2.7 branch) is the 2.7.1 tag:
> > https://github.com/apache/kafka/releases/tag/2.7.1-rc2
> > 
> > * Documentation:
> > https://kafka.apache.org/27/documentation.html
> > 
> > * Protocol:
> > https://kafka.apache.org/27/protocol.html
> > 
> > * Successful Jenkins builds for the 2.7 branch:
> > The build is still running, I'll update the thread once it's complete
> > 
> > /**
> > 
> > Thanks,
> > Mickael

[jira] [Created] (KAFKA-12670) KRaft support for unclean.leader.election.enable

[Colin McCabe (Jira)]

Colin McCabe created KAFKA-12670:


 Summary: KRaft support for unclean.leader.election.enable
 Key: KAFKA-12670
 URL: https://issues.apache.org/jira/browse/KAFKA-12670
 Project: Kafka
  Issue Type: Task
Reporter: Colin McCabe
Assignee: Colin McCabe


Implement KRaft support for the unclean.leader.election.enable configurations.  
These configurations can be set at the topic, broker, or cluster level.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[VOTE] KIP-732: Deprecate eos-alpha and replace eos-beta with eos-v2

[Sophie Blee-Goldman]

Hey all,

I'd like to kick off the vote on KIP-732, to deprecate eos-alpha in Kafka
Streams and migrate away from the "eos-beta" terminology by replacing it
with "eos-v2" to shore up user confidence in this feature.

https://cwiki.apache.org/confluence/display/KAFKA/KIP-732%3A+Deprecate+eos-alpha+and+replace+eos-beta+with+eos-v2

Please respond on the discussion thread if you have any late-breaking
concerns or questions.

Thanks!
Sophie

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 2.8 #12

[Apache Jenkins Server]

See 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #40

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-726: Make the CooperativeStickyAssignor as the default assignor

[Sophie Blee-Goldman]

1) Once the short-circuit is triggered, the member will downgrade to the
EAGER protocol, but
won't necessarily try to rejoin the group right away.

In the "happy path", the user has implemented #onPartitionsLost correctly
and will not attempt
to commit partitions that are lost. And since these partitions have indeed
been revoked, the user
application should not attempt to commit those partitions after this point.
In this case, there's no
reason for the consumer to immediately rejoin the group. Since a
non-cooperative assignor was
selected, we know that all partitions have been assigned. This member can
continue on as usual,
processing the remaining un-revoked partitions and will follow the EAGER
protocol in the next
rebalance. There's no user-facing impact or handling required; all that
happens is that the work
since the last commit on those revoked partitions has been lost.

In the less-happy path, the user has implemented #onPartitionsLost
incorrectly or not implemented
it at all, falling back on the default which invokes #onPartitionsRevoked
which in turn will attempt to
commit those partitions during the rebalance callback. In this case we rely
on the flag to prevent
this commit request from being sent to the broker.

Originally I was thinking we should throw a CommitFailedException up
through the #onPartitionsLost
callback, and eventually up through poll(), then rejoin the group. But now
I'm wondering if this is really
necessary -- the important point in all cases is just to prevent the
commit, but there's no reason the
consumer should not be allowed to continue processing its other partitions,
and it hasn't dropped out
of the group. What do you think about this slight amendment to my original
proposal: if a user does end up
calling commit for whatever reason when invoking #onPartitionsLost, we'll
just swallow the resulting
CommitFailedException. So the user application wouldn't see anything, and
the only impact would be
that these partitions were not able to commit those last set of offsets on
the revoked partitions.

WDYT? My only concern there is that the user might have some implicit
assumption that unless a
CommitFailedException was thrown, the offsets of revoked partitions were
successfully committed
and they may have some downstream logic that should trigger only in this
case. If that's a concern,
then I would keep the original proposal which says a CommitFailedException
will be thrown up through
poll(), and leave it up to the user to decide if they want to trigger a new
rebalance/rejoin the group or not.

Regarding the flag which prevents committing the revoked partitions, this
will need to continue
blocking such commit attempts until the next time the consumer rejoins the
group, ie until the end
of the next successful rebalance. Technically this shouldn't matter, since
the consumer no longer
owns those partitions this member shouldn't attempt to commit them anyways.
Usually we can
rely on the broker rejecting commit attempts on partitions that are not
owned, in which case the
consumer will throw a CommitFailedException. This is similar, except that
we can't rely on the
broker having been informed of the change in ownership before this consumer
might attempt to
commit. So to avoid this race condition, we'll keep the "blockCommit" flag
until the next rebalance
when we can be certain that the broker is clear on this
partition's ownership.

2) I guess maybe the wording here is unclear -- what I meant is that all
3.0 applications will *eventually*
enable cooperative rebalancing in the stable state. This doesn't mean that
it will select COOPERATIVE
when it first starts up, and in order for this dynamic protocol upgrade to
be safe we do indeed need to
start off with EAGER and only upgrade once the selected assignor indicates
that it's safe to do so.
(This only applies if multiple assignors are used, if the assignors are
"cooperative-sticky" only then it
will just start out and forever remain on COOPERATIVE, like in Streams)

Since it's just the first rebalance, the choice of COOPERATIVE vs EAGER
actually doesn't matter at
all since the consumer won't own any partitions until it's joined the
group. So we may as well continue
the initial protocol selection strategy of "highest commonly supported
protocol", but the point is that
3.0 applications will upgrade to COOPERATIVE as soon as they have any
partitions. If you can think
of a better way to phrase "New applications on 3.0 will enable cooperative
rebalancing by default" then
please let me know.


Thanks for the response -- hope this makes sense so far, but I'm happy to
elaborate any aspects of the
proposal which aren't clear. I'll also update the ticket description
for KAFKA-12477 with the latest.


On Wed, Apr 14, 2021 at 12:03 PM Guozhang Wang  wrote:

> Hello Sophie,
>
> Thanks for the detailed explanation, a few clarifying questions:
>
> 1) when the short-circuit is triggered, what would happen next? Would the
> consumers switch back to EAGER, a

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 2.8 #13

[Apache Jenkins Server]

See