Rolling Upgrade failed after bumping version to 2.1 from 1.1

[M. Manna]

Hello,

I have recently upgraded to 2.12-2.1.0 from 2.11-1.1.0. After bumping
inter.broker.protocol.version to 2.1 (and doing 2 bleed-and-restart from
1.1), my services are all failing to start. Slightly clueless at this stage
regarding what to do other than reverting.

I followed the steps in main site, and didn't use log message version
change (since I am doing the uprade from 1.1.0 to 2.1).

Could anyone please help ?

Thanks,

Re: Rolling Upgrade failed after bumping version to 2.1 from 1.1

[M. Manna]

By investigating journalctl logs - I see that it says version 2.1 is not a
valid version? So what is a valid version, does anyone know?

Thanks,

On Tue, 29 Jan 2019 at 11:37, M. Manna  wrote:

> Hello,
>
> I have recently upgraded to 2.12-2.1.0 from 2.11-1.1.0. After bumping
> inter.broker.protocol.version to 2.1 (and doing 2 bleed-and-restart from
> 1.1), my services are all failing to start. Slightly clueless at this stage
> regarding what to do other than reverting.
>
> I followed the steps in main site, and didn't use log message version
> change (since I am doing the uprade from 1.1.0 to 2.1).
>
> Could anyone please help ?
>
> Thanks,
>

[jira] [Created] (KAFKA-7881) Inter.broker.procol.version is incorrect for Rolling Upgrade

[M. Manna (JIRA)]

M. Manna created KAFKA-7881:
---

 Summary: Inter.broker.procol.version is incorrect for Rolling 
Upgrade
 Key: KAFKA-7881
 URL: https://issues.apache.org/jira/browse/KAFKA-7881
 Project: Kafka
  Issue Type: Bug
  Components: admin
Affects Versions: 2.1.0
Reporter: M. Manna


We are getting the following error when upgrading from 1.1.0 to 2.1.0. We have 
not changed the log message format version.

 

Jan 29 06:06:14 one-drive-loc-01 systemd[1]: Started Zookeeper unit for this 
machine.
Jan 29 06:06:15 one-drive-loc-01 kafka-server-start.sh[15881]: [2019-01-29 
06:06:15,272] INFO Registered kafka:type=kafka.Log4jController MBean 
(kafka.utils.Log4jControl
Jan 29 06:06:15 one-drive-loc-01 kafka-server-start.sh[15881]: [2019-01-29 
06:06:15,599] ERROR Exiting Kafka due to fatal exception (kafka.Kafka$)
Jan 29 06:06:15 one-drive-loc-01 kafka-server-start.sh[15881]: 
java.lang.IllegalArgumentException: Version `2.1` is not a valid version
Jan 29 06:06:15 one-drive-loc-01 kafka-server-start.sh[15881]: at 
kafka.api.ApiVersion$$anonfun$apply$1.apply(ApiVersion.scala:88)
Jan 29 06:06:15 one-drive-loc-01 kafka-server-start.sh[15881]: at 
kafka.api.ApiVersion$$anonfun$apply$1.apply(ApiVersion.scala:88)
Jan 29 06:06:15 one-drive-loc-01 kafka-server-start.sh[15881]: at 
scala.collection.MapLike$class.getOrElse(MapLike.scala:128)
Jan 29 06:06:15 one-drive-loc-01 kafka-server-start.sh[15881]: at 
scala.collection.AbstractMap.getOrElse(Map.scala:59)
Jan 29 06:06:15 one-drive-loc-01 kafka-server-start.sh[15881]: at 
kafka.api.ApiVersion$.apply(ApiVersion.scala:88)
Jan 29 06:06:15 one-drive-loc-01 kafka-server-start.sh[15881]: at 
kafka.server.KafkaConfig.(KafkaConfig.scala:1127)
Jan 29 06:06:15 one-drive-loc-01 kafka-server-start.sh[15881]: at 
kafka.server.KafkaConfig.(KafkaConfig.scala:989)
Jan 29 06:06:15 one-drive-loc-01 kafka-server-start.sh[15881]: at 
kafka.server.KafkaConfig$.fromProps(KafkaConfig.scala:969)
Jan 29 06:06:15 one-drive-loc-01 kafka-server-start.sh[15881]: at 
kafka.server.KafkaServerStartable$.fromProps(KafkaServerStartable.scala:28)
Jan 29 06:06:15 one-drive-loc-01 kafka-server-start.sh[15881]: at 
kafka.Kafka$.main(Kafka.scala:82)
Jan 29 06:06:15 one-drive-loc-01 kafka-server-start.sh[15881]: at 
kafka.Kafka.main(Kafka.scala)

 

we have also tried to make it to 2.1 but no change. If the version map is being 
keyed using shortVersion, shouldn't it match? This is the first time we are 
upgrading (from 1.1.0) and we have never had to change log message format. 

Please advise.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-7881) Inter.broker.procol.version is incorrect for Rolling Upgrade

[M. Manna (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-7881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

M. Manna resolved KAFKA-7881.
-
Resolution: Cannot Reproduce
  Assignee: M. Manna

Duplicate jars were present - a delete and copy of new jars resolved the issue. 
closing this.

> Inter.broker.procol.version is incorrect for Rolling Upgrade
> 
>
> Key: KAFKA-7881
> URL: https://issues.apache.org/jira/browse/KAFKA-7881
> Project: Kafka
>  Issue Type: Bug
>  Components: admin
>Affects Versions: 2.1.0
>Reporter: M. Manna
>Assignee: M. Manna
>Priority: Critical
>   Original Estimate: 2m
>  Remaining Estimate: 2m
>
> We are getting the following error when upgrading from 1.1.0 to 2.1.0. We 
> have not changed the log message format version.
>  
> [https://kafka.apache.org/21/documentation.html]
>  
> Jan 29 06:06:14 one-drive-loc-01 systemd[1]: Started Zookeeper unit for this 
> machine.
>  Jan 29 06:06:15 one-drive-loc-01 kafka-server-start.sh[15881]: [2019-01-29 
> 06:06:15,272] INFO Registered kafka:type=kafka.Log4jController MBean 
> (kafka.utils.Log4jControl
>  Jan 29 06:06:15 one-drive-loc-01 kafka-server-start.sh[15881]: [2019-01-29 
> 06:06:15,599] ERROR Exiting Kafka due to fatal exception (kafka.Kafka$)
>  Jan 29 06:06:15 one-drive-loc-01 kafka-server-start.sh[15881]: 
> java.lang.IllegalArgumentException: Version `2.1` is not a valid version
>  Jan 29 06:06:15 one-drive-loc-01 kafka-server-start.sh[15881]: at 
> kafka.api.ApiVersion$$anonfun$apply$1.apply(ApiVersion.scala:88)
>  Jan 29 06:06:15 one-drive-loc-01 kafka-server-start.sh[15881]: at 
> kafka.api.ApiVersion$$anonfun$apply$1.apply(ApiVersion.scala:88)
>  Jan 29 06:06:15 one-drive-loc-01 kafka-server-start.sh[15881]: at 
> scala.collection.MapLike$class.getOrElse(MapLike.scala:128)
>  Jan 29 06:06:15 one-drive-loc-01 kafka-server-start.sh[15881]: at 
> scala.collection.AbstractMap.getOrElse(Map.scala:59)
>  Jan 29 06:06:15 one-drive-loc-01 kafka-server-start.sh[15881]: at 
> kafka.api.ApiVersion$.apply(ApiVersion.scala:88)
>  Jan 29 06:06:15 one-drive-loc-01 kafka-server-start.sh[15881]: at 
> kafka.server.KafkaConfig.(KafkaConfig.scala:1127)
>  Jan 29 06:06:15 one-drive-loc-01 kafka-server-start.sh[15881]: at 
> kafka.server.KafkaConfig.(KafkaConfig.scala:989)
>  Jan 29 06:06:15 one-drive-loc-01 kafka-server-start.sh[15881]: at 
> kafka.server.KafkaConfig$.fromProps(KafkaConfig.scala:969)
>  Jan 29 06:06:15 one-drive-loc-01 kafka-server-start.sh[15881]: at 
> kafka.server.KafkaServerStartable$.fromProps(KafkaServerStartable.scala:28)
>  Jan 29 06:06:15 one-drive-loc-01 kafka-server-start.sh[15881]: at 
> kafka.Kafka$.main(Kafka.scala:82)
>  Jan 29 06:06:15 one-drive-loc-01 kafka-server-start.sh[15881]: at 
> kafka.Kafka.main(Kafka.scala)
>  
> we have also tried to make it to 2.1 but no change. If the version map is 
> being keyed using shortVersion, shouldn't it match? This is the first time we 
> are upgrading (from 1.1.0) and we have never had to change log message format.
> Please advise.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KAFKA-7882) StateStores are frequently closed during the 'transform' method

[Mateusz Owczarek (JIRA)]

Mateusz Owczarek created KAFKA-7882:
---

 Summary: StateStores are frequently closed during the 'transform' 
method
 Key: KAFKA-7882
 URL: https://issues.apache.org/jira/browse/KAFKA-7882
 Project: Kafka
  Issue Type: Bug
  Components: streams
Affects Versions: 2.0.0
Reporter: Mateusz Owczarek


Hello, I have a problem with the state store being closed frequently while 
transforming upcoming records. To ensure only one record of the same key and 
the window comes to an aggregate I have created a custom Transformer (I know 
something similar is going to be introduced with suppress method on KTable in 
the future, but my implementation is quite simple and imo should work 
correctly) with the following implementation:

```
override def transform(key: Windowed[K], value: V): (Windowed[K], V) = {

{ 
val partition = context.partition() 
if (partition != -1) store.put(key.key(), (value, partition), 
key.window().start()) 
else logger.warn(s"-1 partition") 
}

null //Ensuring no 1:1 forwarding, context.forward and commit logic is in the 
punctuator callback
}
```

What I go get is the following error:
```
Store MyStore is currently closed
```

This problem appears only when the number of streaming threads (or input topic 
partitions) is greater than 1 even if I'm just saving to the store and turn off 
the punctuation.

If punctuation is present, however, I sometimes get -1 as a partition value in 
the transform method. I'm familiar with the basic docs, however I haven't found 
anything that could help me here.

INB4: I don't close any state stores manually, I gave them retention time as 
long as possible for the debugging stage, I tried to hotfix that with the retry 
in the transform method and the state stores reopen at the end and the `put` 
method works, but this approach is questionable and I am concerned if it 
actually works.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KAFKA-7883) Add schema.namespace support to SetSchemaMetadata SMT in Kafka Connect

[JIRA]

Jérémy Thulliez created KAFKA-7883:
--

 Summary: Add schema.namespace support to SetSchemaMetadata SMT in 
Kafka Connect
 Key: KAFKA-7883
 URL: https://issues.apache.org/jira/browse/KAFKA-7883
 Project: Kafka
  Issue Type: New Feature
  Components: KafkaConnect
Affects Versions: 2.1.0
Reporter: Jérémy Thulliez


When using a connector with AvroConverter & SchemaRegistry, users should be 
able to specify the namespace in the SMT.

Currently, only "schema.version" and "schema.name" can be specified.

This is needed because if not specified, generated classes (from avro schema)  
are in the default package and not accessible.

Currently, the workaround is to add a Transformation implementation to the 
connect classpath.

It should be native.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KAFKA-7884) Docs for message.format.version and log.message.format.version show invalid (corrupt?) "valid values"

[James Cheng (JIRA)]

James Cheng created KAFKA-7884:
--

 Summary: Docs for message.format.version and 
log.message.format.version show invalid (corrupt?) "valid values"
 Key: KAFKA-7884
 URL: https://issues.apache.org/jira/browse/KAFKA-7884
 Project: Kafka
  Issue Type: Bug
  Components: documentation
Reporter: James Cheng


In the docs for message.format.version and log.message.format.version, the list 
of valid values is

 
{code:java}
kafka.api.ApiVersionValidator$@56aac163 
{code}
 

It appears it's simply doing a .toString on the class/instance.

At a minimum, we should remove this java-y-ness.

Even better is, it should show all the valid values.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [VOTE] KIP-183 - Change PreferredReplicaLeaderElectionCommand to use AdminClient

[Colin McCabe]

Thanks, Tom!  Great work.

best,
Colin

On Mon, Jan 28, 2019, at 04:33, Tom Bentley wrote:
> Hi Folks,
> 
> It took a while, but the work for KIP-183 has now been merged. My thanks to
> everyone involved.
> 
> A few details changed between what was voted on and what ultimately got
> merged. I've updated the KIP to reflect what was actually merged. If 
> anyone
> is interested in the gory details they can look at
> https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=73632065&selectedPageVersions=20&selectedPageVersions=18
> and
> https://github.com/apache/kafka/commit/269b65279c746bc54c611141a5a6509f9b310f11
> 
> Kind regards,
> 
> Tom
> 
> On Fri, 8 Sep 2017 at 16:30, Tom Bentley  wrote:
> 
> > Since no one has objected, I conclude that this KIP is again accepted.
> >
> > Thanks,
> >
> > Tom
> >
> > On 7 September 2017 at 22:31, Guozhang Wang  wrote:
> >
> >> Hi Tom,
> >>
> >> The updated part in "AdminClient:electPreferredLeaders()" looks reasonable
> >> to me. If there is no objections from the voted committer by end of the
> >> day, I think you can mark it as accepted.
> >>
> >>
> >> Guozhang
> >>
> >>
> >> On Wed, Sep 6, 2017 at 7:42 AM, Tom Bentley 
> >> wrote:
> >>
> >> > Unfortunately I've had to make a small change to the
> >> > ElectPreferredLeadersResult, because exposing a Map >> > KafkaFuture> was incompatible with the case where
> >> > electPreferredLeaders() was called with a null partitions argument. The
> >> > change exposes methods to access the map which return futures, rather
> >> than
> >> > exposing the map (and crucially its keys) directly.
> >> >
> >> > This is described in more detail in the [DISCUSS] thread.
> >> >
> >> > Please take a look and recast your votes:
> >> >
> >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-183+-+Change+
> >> > PreferredReplicaLeaderElectionCommand+to+use+AdminClient#KIP-183-
> >> > ChangePreferredReplicaLeaderElectionCommandtouseAdminClient-AdminClient:
> >> > electPreferredLeaders()
> >> >
> >> > Thanks,
> >> >
> >> > Tom
> >> >
> >> > On 4 September 2017 at 10:52, Ismael Juma  wrote:
> >> >
> >> > > Hi Tom,
> >> > >
> >> > > You can update the KIP for minor things like that. Worth updating the
> >> > > thread if it's something that is done during the PR review.
> >> > >
> >> > > With regards to exceptions, yes, that's definitely desired. I filed a
> >> > JIRA
> >> > > a while back for this:
> >> > >
> >> > > https://issues.apache.org/jira/browse/KAFKA-5445
> >> > >
> >> > > Ideally, new methods that we add would have this so that we don't
> >> > increase
> >> > > the tech debt that already exists.
> >> > >
> >> > > Ismael
> >> > >
> >> > > On Mon, Sep 4, 2017 at 10:11 AM, Tom Bentley 
> >> > > wrote:
> >> > >
> >> > > > Hi Jun,
> >> > > >
> >> > > > You're correct about those other expected errors. If it's OK to
> >> update
> >> > > the
> >> > > > KIP after the vote I'll add those.
> >> > > >
> >> > > > But this makes me wonder about the value of documenting expected
> >> errors
> >> > > in
> >> > > > the Javadocs for the AdminClient (on the Results class, to be
> >> > specific).
> >> > > > Currently we don't do this, but it would be helpful for people using
> >> > the
> >> > > > AdminClient to know the kinds of errors they should expect, for
> >> testing
> >> > > > purposes for example. On the other hand it's a maintenance burden.
> >> > Should
> >> > > > we start documenting likely errors like this?
> >> > > >
> >> > > > Cheers,
> >> > > >
> >> > > > Tom
> >> > > >
> >> > > > On 4 September 2017 at 10:10, Tom Bentley 
> >> > wrote:
> >> > > >
> >> > > > > I see three +1s, no +0s and no -1, so the vote passes.
> >> > > > >
> >> > > > > Thanks to those who voted and/or commented on the discussion
> >> thread.
> >> > > > >
> >> > > > > On 1 September 2017 at 07:36, Gwen Shapira 
> >> > wrote:
> >> > > > >
> >> > > > >> Thank you! +1 (binding).
> >> > > > >>
> >> > > > >> On Thu, Aug 31, 2017 at 9:48 AM Jun Rao 
> >> wrote:
> >> > > > >>
> >> > > > >> > Hi, Tom,
> >> > > > >> >
> >> > > > >> > Thanks for the KIP. +1. Just one more minor comment. It seems
> >> that
> >> > > the
> >> > > > >> > ElectPreferredLeadersResponse
> >> > > > >> > should expect at least 3 other types of errors : (1) request
> >> > timeout
> >> > > > >> > exception, (2) leader rebalance in-progress exception, (3)
> >> can't
> >> > > move
> >> > > > to
> >> > > > >> > the preferred replica exception (i.e., preferred replica not in
> >> > sync
> >> > > > >> yet).
> >> > > > >> >
> >> > > > >> > Jun
> >> > > > >> >
> >> > > > >> > On Tue, Aug 29, 2017 at 8:56 AM, Tom Bentley <
> >> > t.j.bent...@gmail.com
> >> > > >
> >> > > > >> > wrote:
> >> > > > >> >
> >> > > > >> > > Hi all,
> >> > > > >> > >
> >> > > > >> > > I would like to start the vote on KIP-183 which will provide
> >> an
> >> > > > >> > AdminClient
> >> > > > >> > > interface for electing the preferred replica, and refactor
> >> the
> >> > > > >> > > kafka-preferred-replica-election.sh t

Why is enable.auto.commit=true the default value for consumer?

[Adam Bellemare]

As the question indicates.

Should this not be default false? I think this is a bit nefarious to
someone launching their application into production without testing it
extensively around failure modes. I can see a scenario where a consumer
polls for events, processes them, produces to output topic, and commits the
offsets. Say it takes 30 seconds for a batch. If it fails halfway through,
upon restarting it will skip everything that was unprocessed/unpublished up
to the committed offset.

Is there a historic reason why it's set to default true? Is it because to
change it to default false it could affect the upgrade path of previous
implementations?

Adam

Re: [EXTERNAL] - Re: [DISCUSS] KIP-387: Fair Message Consumption Across Partitions in KafkaConsumer

[Colin McCabe]

On Mon, Jan 28, 2019, at 06:57, ChienHsing Wu wrote:
> So... Does this non-response mean I should drop this topic after almost 
> one month, folks?

Hi ChienHsing,

I would recommend dropping it, since I don't see a lot of uses for it.  Maybe 
there is something I missed, though.  See my responses below:

> 
> -Original Message-
> From: ChienHsing Wu  
> Sent: Monday, January 21, 2019 12:47 PM
> To: dev@kafka.apache.org
> Subject: RE: [EXTERNAL] - Re: [DISCUSS] KIP-387: Fair Message 
> Consumption Across Partitions in KafkaConsumer
> 
> Hi all, not sure what to do next as weeks have gone by, guys. --CH
> 
> -Original Message-
> From: ChienHsing Wu 
> Sent: Monday, January 14, 2019 9:13 AM
> To: dev@kafka.apache.org
> Subject: RE: [EXTERNAL] - Re: [DISCUSS] KIP-387: Fair Message 
> Consumption Across Partitions in KafkaConsumer
> 
> Hi,
> 
> I know everyone is busy. But I would appreciate someone letting me know 
> what to do next. I started this effort back in last year early 
> November...
> 
> Thanks, CH
> 
> -Original Message-
> From: ChienHsing Wu 
> Sent: Monday, January 07, 2019 9:24 AM
> To: dev@kafka.apache.org
> Subject: RE: [EXTERNAL] - Re: [DISCUSS] KIP-387: Fair Message 
> Consumption Across Partitions in KafkaConsumer
> 
> Hi guys,
> 
> I am not sure what to do next in this KIP process. Could anyone please 
> help/advise me on what to do next? 
> 
> Thanks, CH
> 
> -Original Message-
> From: ChienHsing Wu 
> Sent: Wednesday, January 02, 2019 12:55 PM
> To: dev@kafka.apache.org
> Subject: RE: [EXTERNAL] - Re: [DISCUSS] KIP-387: Fair Message 
> Consumption Across Partitions in KafkaConsumer
> 
> Hi Colin,
> 
> Setting max.partition.fetch.bytes was discussed in the ticket. It's not 
> as desirable if the message size is highly variable. Also this decrease 
> the efficiency of network communication. 

Even if the message size is highly variable, you can still set the 
max.partition.fetch.bytes to a very small value., perhaps even 1.  Since we 
always return at least one record, regardless of size limits, this will have 
the effect you want.  I agree that this may be less efficient than fetching a 
larger number of records.  But it will get you true round-robin behavior.

> 
> In the case you mentioned below where a consumer can get messages from 
> A, B, C and D but the consumer currently only has messages from A, B 
> and C, the proposed change will NOT wait until some messages from D 
> arrives to start returning messages; it will just serve those from A, B 
> and  It will include those from D when they are available. That IS 
> the current behavior. The proposed change does not impose a strict 
> round robin pattern.

I guess this is debatable, but the behavior you're proposing doesn't seem 
"fairer" than what we do now.  Imagine you are subscribed to 100 partitions and 
each fetch request only gives you back records from 3 of them.  Then the order 
the consumer sees records might be something like:

ABCABCABCDEFDEFDEFDEFGHIGHIGHI... etc.

Is this fairer than AAABBBCCCDDDEEEFFFGGGHHHIII?  Seems questionable.

It feels like maybe what you really want is a work queue for A, B, C, D, etc. 
so that messages from different partitions can be processed in parallel.  And 
then perhaps pause fetching a partition when the work queue for that partition 
grows too long.

> 
> The original KIP 41 discussed "Ensuring Fair Consumption", that means 
> it originally intended to take that into account in the Consumer code, 
> the proposed change takes the current algorithm closer to that goal, 
> IMHO. I could implement that logic at the caller side but, that would 
> mean each library user need to know the inner working of the consumer 
> code and to implement the logic on their own. Though as a first timer 
> here, I do appreciate the complexity and functionalities in the client 
> library and feel that we'd be better off as a community to implement 
> the logic in the library so the complexity is hidden from library users.

The discussion in KIP-41 in the "ensuring fair consumption" section is about 
making sure that no partitions get starved forever.  This would happen if, for 
example, we just constantly fetched from a single partition and never fetched 
from some other partition.  The discussion in that KIP isn't about interleaving 
the partition order of the buffered records we return to the consumer.

best,
Colin


> 
> Thanks, CH
> 
> -Original Message-
> From: Colin McCabe 
> Sent: Saturday, December 22, 2018 3:53 AM
> To: dev@kafka.apache.org
> Subject: Re: [EXTERNAL] - Re: [DISCUSS] KIP-387: Fair Message 
> Consumption Across Partitions in KafkaConsumer
> 
> Hi ChienHsing Wu,
> 
> Maybe I'm misunderstanding something, but I'm not sure I see the need 
> for a KIP here.  You can just set max.partition.fetch.bytes to a very 
> small value.  That will cause Kafka to fetch only one message from each 
> partition.  This will give you the round robin behavior you want

Re: Why is enable.auto.commit=true the default value for consumer?

[Colin McCabe]

This is a very fair question.  I suspect the answer is that people feel that 
automatic offset commit is easier to get started with, even though it has the 
error handling shortcomings you mention.  It would be very hard to change the 
default now, since it would break such a huge amount of code.  Perhaps if we 
create a whole new consumer interface.

best.
Colin


On Tue, Jan 29, 2019, at 12:53, Adam Bellemare wrote:
> As the question indicates.
> 
> Should this not be default false? I think this is a bit nefarious to
> someone launching their application into production without testing it
> extensively around failure modes. I can see a scenario where a consumer
> polls for events, processes them, produces to output topic, and commits the
> offsets. Say it takes 30 seconds for a batch. If it fails halfway through,
> upon restarting it will skip everything that was unprocessed/unpublished up
> to the committed offset.
> 
> Is there a historic reason why it's set to default true? Is it because to
> change it to default false it could affect the upgrade path of previous
> implementations?
> 
> Adam
>

RE: [EXTERNAL] - Re: [DISCUSS] KIP-387: Fair Message Consumption Across Partitions in KafkaConsumer

[ChienHsing Wu]

I beg to differ... The original KIP intended to consume messages fairly and the 
current implementation can be improved with the patch that can benefit the 
community. I believe that fair consumption is an important characteristic at 
the consumer side. But anyhow, looks like this is not received well enough.

--CH

-Original Message-
From: Colin McCabe  
Sent: Tuesday, January 29, 2019 4:00 PM
To: dev@kafka.apache.org
Subject: Re: [EXTERNAL] - Re: [DISCUSS] KIP-387: Fair Message Consumption 
Across Partitions in KafkaConsumer

On Mon, Jan 28, 2019, at 06:57, ChienHsing Wu wrote:
> So... Does this non-response mean I should drop this topic after 
> almost one month, folks?

Hi ChienHsing,

I would recommend dropping it, since I don't see a lot of uses for it.  Maybe 
there is something I missed, though.  See my responses below:

> 
> -Original Message-
> From: ChienHsing Wu 
> Sent: Monday, January 21, 2019 12:47 PM
> To: dev@kafka.apache.org
> Subject: RE: [EXTERNAL] - Re: [DISCUSS] KIP-387: Fair Message 
> Consumption Across Partitions in KafkaConsumer
> 
> Hi all, not sure what to do next as weeks have gone by, guys. --CH
> 
> -Original Message-
> From: ChienHsing Wu 
> Sent: Monday, January 14, 2019 9:13 AM
> To: dev@kafka.apache.org
> Subject: RE: [EXTERNAL] - Re: [DISCUSS] KIP-387: Fair Message 
> Consumption Across Partitions in KafkaConsumer
> 
> Hi,
> 
> I know everyone is busy. But I would appreciate someone letting me 
> know what to do next. I started this effort back in last year early 
> November...
> 
> Thanks, CH
> 
> -Original Message-
> From: ChienHsing Wu 
> Sent: Monday, January 07, 2019 9:24 AM
> To: dev@kafka.apache.org
> Subject: RE: [EXTERNAL] - Re: [DISCUSS] KIP-387: Fair Message 
> Consumption Across Partitions in KafkaConsumer
> 
> Hi guys,
> 
> I am not sure what to do next in this KIP process. Could anyone please 
> help/advise me on what to do next?
> 
> Thanks, CH
> 
> -Original Message-
> From: ChienHsing Wu 
> Sent: Wednesday, January 02, 2019 12:55 PM
> To: dev@kafka.apache.org
> Subject: RE: [EXTERNAL] - Re: [DISCUSS] KIP-387: Fair Message 
> Consumption Across Partitions in KafkaConsumer
> 
> Hi Colin,
> 
> Setting max.partition.fetch.bytes was discussed in the ticket. It's 
> not as desirable if the message size is highly variable. Also this 
> decrease the efficiency of network communication.

Even if the message size is highly variable, you can still set the 
max.partition.fetch.bytes to a very small value., perhaps even 1.  Since we 
always return at least one record, regardless of size limits, this will have 
the effect you want.  I agree that this may be less efficient than fetching a 
larger number of records.  But it will get you true round-robin behavior.

> 
> In the case you mentioned below where a consumer can get messages from 
> A, B, C and D but the consumer currently only has messages from A, B 
> and C, the proposed change will NOT wait until some messages from D 
> arrives to start returning messages; it will just serve those from A, 
> B and  It will include those from D when they are available. That IS 
> the current behavior. The proposed change does not impose a strict 
> round robin pattern.

I guess this is debatable, but the behavior you're proposing doesn't seem 
"fairer" than what we do now.  Imagine you are subscribed to 100 partitions and 
each fetch request only gives you back records from 3 of them.  Then the order 
the consumer sees records might be something like:

ABCABCABCDEFDEFDEFDEFGHIGHIGHI... etc.

Is this fairer than AAABBBCCCDDDEEEFFFGGGHHHIII?  Seems questionable.

It feels like maybe what you really want is a work queue for A, B, C, D, etc. 
so that messages from different partitions can be processed in parallel.  And 
then perhaps pause fetching a partition when the work queue for that partition 
grows too long.

> 
> The original KIP 41 discussed "Ensuring Fair Consumption", that means 
> it originally intended to take that into account in the Consumer code, 
> the proposed change takes the current algorithm closer to that goal, 
> IMHO. I could implement that logic at the caller side but, that would 
> mean each library user need to know the inner working of the consumer 
> code and to implement the logic on their own. Though as a first timer 
> here, I do appreciate the complexity and functionalities in the client 
> library and feel that we'd be better off as a community to implement 
> the logic in the library so the complexity is hidden from library users.

The discussion in KIP-41 in the "ensuring fair consumption" section is about 
making sure that no partitions get starved forever.  This would happen if, for 
example, we just constantly fetched from a single partition and never fetched 
from some other partition.  The discussion in that KIP isn't about interleaving 
the partition order of the buffered records we return to the consumer.

best,
Colin


> 
> Thanks, CH
> 
> -Ori

Build failed in Jenkins: kafka-trunk-jdk11 #251

[Apache Jenkins Server]

See 


Changes:

[github] MINOR: Update usage of deprecated API (#6146)

--
[...truncated 2.28 MB...]

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualWithNullForCompareValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualWithNullForCompareValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentForCompareKeyValueWithProducerRecord STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentForCompareKeyValueWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfTimestampIsDifferentForCompareValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfTimestampIsDifferentForCompareValueTimestampWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualForCompareKeyValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualForCompareKeyValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualWithNullForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualWithNullForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualForCompareValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualForCompareValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullExpectedRecordForCompareKeyValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullExpectedRecordForCompareKeyValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareValueTimestampWithProducerRecord STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareValueTimestampWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullExpectedRecordForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullExpectedRecordForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareValueWithProducerRecord STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareValueWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueIsEqualForCompareValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueIsEqualForCompareValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordWithExpectedRecordForCompareKeyValueTimestamp 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordWithExpectedRecordForCompareKeyValueTimestamp 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentForCompareKeyValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentForCompareKeyValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordWithExpectedRecordForCompareKeyValue STARTED

org.apache.kaf

Build failed in Jenkins: kafka-trunk-jdk8 #3354

[Apache Jenkins Server]

See 


Changes:

[github] MINOR: Update usage of deprecated API (#6146)

--
[...truncated 2.28 MB...]

org.apache.kafka.connect.json.JsonConverterTest > stringHeaderToConnect STARTED

org.apache.kafka.connect.json.JsonConverterTest > stringHeaderToConnect PASSED

org.apache.kafka.connect.json.JsonConverterTest > mapToJsonNonStringKeys STARTED

org.apache.kafka.connect.json.JsonConverterTest > mapToJsonNonStringKeys PASSED

org.apache.kafka.connect.json.JsonConverterTest > longToJson STARTED

org.apache.kafka.connect.json.JsonConverterTest > longToJson PASSED

org.apache.kafka.connect.json.JsonConverterTest > mismatchSchemaJson STARTED

org.apache.kafka.connect.json.JsonConverterTest > mismatchSchemaJson PASSED

org.apache.kafka.connect.json.JsonConverterTest > mapToConnectNonStringKeys 
STARTED

org.apache.kafka.connect.json.JsonConverterTest > mapToConnectNonStringKeys 
PASSED

org.apache.kafka.connect.json.JsonConverterTest > 
testJsonSchemaMetadataTranslation STARTED

org.apache.kafka.connect.json.JsonConverterTest > 
testJsonSchemaMetadataTranslation PASSED

org.apache.kafka.connect.json.JsonConverterTest > bytesToJson STARTED

org.apache.kafka.connect.json.JsonConverterTest > bytesToJson PASSED

org.apache.kafka.connect.json.JsonConverterTest > shortToConnect STARTED

org.apache.kafka.connect.json.JsonConverterTest > shortToConnect PASSED

org.apache.kafka.connect.json.JsonConverterTest > intToJson STARTED

org.apache.kafka.connect.json.JsonConverterTest > intToJson PASSED

org.apache.kafka.connect.json.JsonConverterTest > nullSchemaAndNullValueToJson 
STARTED

org.apache.kafka.connect.json.JsonConverterTest > nullSchemaAndNullValueToJson 
PASSED

org.apache.kafka.connect.json.JsonConverterTest > timestampToConnectOptional 
STARTED

org.apache.kafka.connect.json.JsonConverterTest > timestampToConnectOptional 
PASSED

org.apache.kafka.connect.json.JsonConverterTest > structToConnect STARTED

org.apache.kafka.connect.json.JsonConverterTest > structToConnect PASSED

org.apache.kafka.connect.json.JsonConverterTest > stringToJson STARTED

org.apache.kafka.connect.json.JsonConverterTest > stringToJson PASSED

org.apache.kafka.connect.json.JsonConverterTest > nullSchemaAndArrayToJson 
STARTED

org.apache.kafka.connect.json.JsonConverterTest > nullSchemaAndArrayToJson 
PASSED

org.apache.kafka.connect.json.JsonConverterTest > byteToConnect STARTED

org.apache.kafka.connect.json.JsonConverterTest > byteToConnect PASSED

org.apache.kafka.connect.json.JsonConverterTest > nullSchemaPrimitiveToConnect 
STARTED

org.apache.kafka.connect.json.JsonConverterTest > nullSchemaPrimitiveToConnect 
PASSED

org.apache.kafka.connect.json.JsonConverterTest > byteToJson STARTED

org.apache.kafka.connect.json.JsonConverterTest > byteToJson PASSED

org.apache.kafka.connect.json.JsonConverterTest > intToConnect STARTED

org.apache.kafka.connect.json.JsonConverterTest > intToConnect PASSED

org.apache.kafka.connect.json.JsonConverterTest > dateToJson STARTED

org.apache.kafka.connect.json.JsonConverterTest > dateToJson PASSED

org.apache.kafka.connect.json.JsonConverterTest > noSchemaToConnect STARTED

org.apache.kafka.connect.json.JsonConverterTest > noSchemaToConnect PASSED

org.apache.kafka.connect.json.JsonConverterTest > noSchemaToJson STARTED

org.apache.kafka.connect.json.JsonConverterTest > noSchemaToJson PASSED

org.apache.kafka.connect.json.JsonConverterTest > nullSchemaAndPrimitiveToJson 
STARTED

org.apache.kafka.connect.json.JsonConverterTest > nullSchemaAndPrimitiveToJson 
PASSED

org.apache.kafka.connect.json.JsonConverterTest > 
decimalToConnectOptionalWithDefaultValue STARTED

org.apache.kafka.connect.json.JsonConverterTest > 
decimalToConnectOptionalWithDefaultValue PASSED

org.apache.kafka.connect.json.JsonConverterTest > mapToJsonStringKeys STARTED

org.apache.kafka.connect.json.JsonConverterTest > mapToJsonStringKeys PASSED

org.apache.kafka.connect.json.JsonConverterTest > arrayToJson STARTED

org.apache.kafka.connect.json.JsonConverterTest > arrayToJson PASSED

org.apache.kafka.connect.json.JsonConverterTest > nullToConnect STARTED

org.apache.kafka.connect.json.JsonConverterTest > nullToConnect PASSED

org.apache.kafka.connect.json.JsonConverterTest > timeToJson STARTED

org.apache.kafka.connect.json.JsonConverterTest > timeToJson PASSED

org.apache.kafka.connect.json.JsonConverterTest > structToJson STARTED

org.apache.kafka.connect.json.JsonConverterTest > structToJson PASSED

org.apache.kafka.connect.json.JsonConverterTest > 
testConnectSchemaMetadataTranslation STARTED

org.apache.kafka.connect.json.JsonConverterTest > 
testConnectSchemaMetadataTranslation PASSED

org.apache.kafka.connect.json.JsonConverterTest > shortToJson STARTED

org.apache.kafka.connect.json.JsonConverterTest > shortToJson PASSED

org.apache.kafka.connect.json.JsonConverterTest > dateToConnect STARTED

org.apac

RE: Why is enable.auto.commit=true the default value for consumer?

[Pellerin, Clement]

I had the same impression at some point but this is not how auto-commit works.
Auto-commit can only commit when the application comes back to poll and
if it decides to commit at that time, it will only commit the previous batch.
In your example, the app might come back and have to re-execute all
the records in the uncommitted batch but it will never skip over
unprocessed records.

-Original Message-
From: Adam Bellemare [mailto:adam.bellem...@gmail.com] 
Sent: Tuesday, January 29, 2019 3:54 PM
To: dev@kafka.apache.org
Subject: Why is enable.auto.commit=true the default value for consumer?

As the question indicates.

Should this not be default false? I think this is a bit nefarious to
someone launching their application into production without testing it
extensively around failure modes. I can see a scenario where a consumer
polls for events, processes them, produces to output topic, and commits the
offsets. Say it takes 30 seconds for a batch. If it fails halfway through,
upon restarting it will skip everything that was unprocessed/unpublished up
to the committed offset.

Is there a historic reason why it's set to default true? Is it because to
change it to default false it could affect the upgrade path of previous
implementations?

Adam

Re: Why is enable.auto.commit=true the default value for consumer?

[Colin McCabe]

Hi Clement,

You are assuming that the client application is single-threaded-- or at least 
processes all the records before polling for more.  This may or may not be the 
case.  But that is a fair point-- in this case, auto-commit would be safe.

best,
Colin

On Tue, Jan 29, 2019, at 16:23, Pellerin, Clement wrote:
> I had the same impression at some point but this is not how auto-commit works.
> Auto-commit can only commit when the application comes back to poll and
> if it decides to commit at that time, it will only commit the previous batch.
> In your example, the app might come back and have to re-execute all
> the records in the uncommitted batch but it will never skip over
> unprocessed records.
> 
> -Original Message-
> From: Adam Bellemare [mailto:adam.bellem...@gmail.com] 
> Sent: Tuesday, January 29, 2019 3:54 PM
> To: dev@kafka.apache.org
> Subject: Why is enable.auto.commit=true the default value for consumer?
> 
> As the question indicates.
> 
> Should this not be default false? I think this is a bit nefarious to
> someone launching their application into production without testing it
> extensively around failure modes. I can see a scenario where a consumer
> polls for events, processes them, produces to output topic, and commits the
> offsets. Say it takes 30 seconds for a batch. If it fails halfway through,
> upon restarting it will skip everything that was unprocessed/unpublished up
> to the committed offset.
> 
> Is there a historic reason why it's set to default true? Is it because to
> change it to default false it could affect the upgrade path of previous
> implementations?
> 
> Adam
>

RE: Why is enable.auto.commit=true the default value for consumer?

[Pellerin, Clement]

Indeed, but this is what the documentation says you must do with auto-commit.
I say it is a user error if you don't.

Regardless, I think this is a fairly common misconception, so it would not hurt
to debunk it explicitly in the documentation.

-Original Message-
From: Colin McCabe [mailto:cmcc...@apache.org] 
Sent: Tuesday, January 29, 2019 7:26 PM
To: dev@kafka.apache.org
Subject: Re: Why is enable.auto.commit=true the default value for consumer?

Hi Clement,

You are assuming that the client application is single-threaded-- or at least 
processes all the records before polling for more.  This may or may not be the 
case.  But that is a fair point-- in this case, auto-commit would be safe.

best,
Colin

On Tue, Jan 29, 2019, at 16:23, Pellerin, Clement wrote:
> I had the same impression at some point but this is not how auto-commit works.
> Auto-commit can only commit when the application comes back to poll and
> if it decides to commit at that time, it will only commit the previous batch.
> In your example, the app might come back and have to re-execute all
> the records in the uncommitted batch but it will never skip over
> unprocessed records.
> 
> -Original Message-
> From: Adam Bellemare [mailto:adam.bellem...@gmail.com] 
> Sent: Tuesday, January 29, 2019 3:54 PM
> To: dev@kafka.apache.org
> Subject: Why is enable.auto.commit=true the default value for consumer?
> 
> As the question indicates.
> 
> Should this not be default false? I think this is a bit nefarious to
> someone launching their application into production without testing it
> extensively around failure modes. I can see a scenario where a consumer
> polls for events, processes them, produces to output topic, and commits the
> offsets. Say it takes 30 seconds for a batch. If it fails halfway through,
> upon restarting it will skip everything that was unprocessed/unpublished up
> to the committed offset.
> 
> Is there a historic reason why it's set to default true? Is it because to
> change it to default false it could affect the upgrade path of previous
> implementations?
> 
> Adam
>

Re: [DISCUSS] KIP-390: Add producer option to adjust compression level

[Dongjin Lee]

Hello.

Do you have any idea on Becket's Idea of new config format (example below)?

```
compression.config="gzip.compression.level=5, lz4.compression.level=17,
zstd.compression.level=22"
```

It requires some additional KIP for supporting new config format (map), but
it can significantly simplify the configuration with flexibility and
extensibility. If you prefer this way, I hope to carry the ball.

If not, please give me an opinion here or the voting thread.

Thanks,
Dongjin


On Fri, Jan 25, 2019 at 1:25 AM Dongjin Lee  wrote:

> Hi Becket,
>
> Thank you for your opinion. Frankly, I have no strong opinion on
> configuration name. In this problem, I will follow the community's choice.
> (I like your idea in that it has a notion of 'scope' per compression codec.
> However, it should be implemented on top of new config type like Map; It
> will require another KIP as a prerequisite, but if the community prefer
> this way, I will take the task.)
>
> (One minor correction: the one who thought 'producer' compression config
> would cause a problem at broker was me, not Ismael - and Ismael reassured
> me there will be no problem with it.)
>
> To All,
>
> How about Becket's idea of 'compression.config' option?
>
> Best,
> Dongjin
>
> On Wed, Jan 23, 2019 at 1:16 PM Becket Qin  wrote:
>
>> Hi Dongjin,
>>
>> Thanks for the KIP and sorry for being a bit late on the discussion.
>>
>> It makes sense to expose the configuration for compression types. But I am
>> wondering if there is a better way to do that than what proposed in the
>> KIP. What I feel confusing is that we are effectively sharing the
>> configuration across different compression types, the meaning of the
>> configuration are actually kind of different depending on the compression
>> type. This will end up with issues like what Ismael has brought up
>> earlier.
>> Say if the broker has compression type of producer (this may result in
>> mixed compression type in the same topic), and for some reason the broker
>> needs to re-compress the topic (e.g. due to log compaction), a single
>> topic
>> level compression config may not work, because a valid compression level
>> for lz4 maybe invalid for gzip.
>>
>> One alternative I am thinking is to provide a "compression.config"
>> configuration, inside which it specifies configuration used by each
>> specific compression type as k-v pairs. The format could use some name
>> space as well. For example,
>>
>>
>> compression.config="gzip.compression.level=5,lz4.compression.level=17,zstd.compression.level=22".
>>
>> Each compression type will just pick whatever configuration they need from
>> the k-v pairs defined in this config.
>>
>> Besides clarity, some other benefits are:
>> 1. Extensibility, we don't have to add more configuration when we add new
>> compression types, or expose new config for a particular compression type.
>> 2. Even for the same config, different compression type may have different
>> terminologies. With the approach we can honor those terminologies instead
>> of shoehorning them into the same configuration name.
>>
>> What do you think?
>>
>> Thanks,
>>
>> Jiangjie (Becket) Qin
>>
>>
>>
>>
>>
>> On Wed, Jan 23, 2019 at 12:07 AM Dongjin Lee  wrote:
>>
>> > Hello. I just fixed the draft implementation, with rebasing onto the
>> latest
>> > trunk. The KIP was also restored.
>> >
>> > Please have a look, and if there is no major problem, please vote to the
>> > voting thread. You know, KIP freeze for 2.2.0 is almost imminent.
>> >
>> > Thanks,
>> > Dongjin
>> >
>> > On Tue, Jan 22, 2019 at 1:04 AM Ismael Juma  wrote:
>> >
>> > > Thanks!
>> > >
>> > > Ismael
>> > >
>> > > On Mon, Jan 21, 2019 at 6:02 AM Dongjin Lee 
>> wrote:
>> > >
>> > > > Hi Ismael,
>> > > >
>> > > > After reviewing
>> > > `LogValidator#validateMessagesAndAssignOffsetsCompressed`,
>> > > > yes, you are right. If source codec and target codec is identical
>> and
>> > the
>> > > > magic is above 0, the broker can do an in-place assignment, without
>> > > > recompressing. Sorry for my misunderstanding.
>> > > >
>> > > > Since we don't need `compression.[gzip,lz4,zstd].level` and
>> > > > `compression.[gzip,snappy,lz4].buffer.size` anymore, I will revert
>> > those
>> > > > changes and update the KIP with the recent discussions. I will
>> complete
>> > > it
>> > > > in 48 hours from now.
>> > > >
>> > > > Thanks,
>> > > > Dongjin
>> > > >
>> > > > On Mon, Jan 21, 2019 at 4:18 PM Dongjin Lee 
>> > wrote:
>> > > >
>> > > > > I see. Let me have a check. If not needed, of course, we don't
>> have
>> > to
>> > > > > waste on configuration options.
>> > > > >
>> > > > > Since the KIP deadline is imminent, I just opened the voting
>> thread.
>> > > > Let's
>> > > > > continue the discussion here.
>> > > > >
>> > > > > Best,
>> > > > > Dongjin
>> > > > >
>> > > > > On Mon, Jan 21, 2019 at 1:30 AM Ismael Juma 
>> > wrote:
>> > > > >
>> > > > >> Hi Dongjin,
>> > > > >>
>> > > > >> When the compression type is "producer", then the broker doe

[DISCUSS] KIP-424: Allow suppression of intermediate events based on wall clock time

[jonathangordon]

Hi all,

I just published KIP-424: Allow suppression of intermediate events based on 
wall clock time

https://cwiki.apache.org/confluence/display/KAFKA/KIP-424%3A+Allow+suppression+of+intermediate+events+based+on+wall+clock+time

I am eager to hear your feedback and concerns. Thanks John Roesler for your 
guidance in shaping my first KIP!

I look forward to working with the Kafka community to see this through,

Jonathan