[DISCUSS] KIP-822: Optimize the semantics of KafkaConsumer#pause to be consistent between the two RebalanceProtocols

[Riven Sun]

Hi team, I opened a KIP to add the option to make the semantics of the
KafkaConsuemr#Pause method consistent under different RebalanceProtocols:
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=199534763

Let me know if you have any feedback. Thanks, RivenSun

Re: [DISCUSS] KIP-822: Optimize the semantics of KafkaConsumer#pause to be consistent between the two RebalanceProtocols

[Riven Sun]

>
>
> Sorry, I sent this email via GMail. Refer to the contents of other
> people's DISSCUSS emails. Mistakenly introduced someone else's KIP.
>
> The KIP related to this DISCUSS is
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=199534763
>
> Thank you for your kindness
> RivenSun
>

Re: [DISCUSS] KIP-822: Optimize the semantics of KafkaConsumer#pause to be consistent between the two RebalanceProtocols

[Riven Sun]

Sorry, I sent this email via GMail. Refer to the contents of other people's
DISSCUSS emails. Mistakenly introduced someone else's KIP.

The KIP related to this DISCUSS is
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=199534763

Thank you for your kindness
RivenSun

On Sat, Feb 12, 2022 at 5:32 PM Riven Sun  wrote:

>
>> Sorry, I sent this email via GMail. Refer to the contents of other
>> people's DISSCUSS emails. Mistakenly introduced someone else's KIP.
>>
>> The KIP related to this DISCUSS is
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=199534763
>>
>> Thank you for your kindness
>> RivenSun
>>
>

Re: [VOTE] KIP-770: Replace "buffered.records.per.partition" with "input.buffer.max.bytes"

[Sagar]

@Guozhang,

I have sent an update to this KIP. I have a question though.. Should this
new metric be defined in TaskMetrics level or NamedCacheMetrics? I think
the latter makes sense as that holds the cache size at a task level and
exposes some other cache related metrics as well like hit-ratio.

Thanks!
Sagar.


On Sat, Feb 12, 2022 at 1:14 PM Sagar  wrote:

> Hi All,
>
> There's another amendment proposed for this KIP. We are adding a new
> metric type called *cache-size-bytes-total  *to capture the cache size in
> bytes accumulated by a task.
>
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=186878390
>
> Thanks!
> Sagar.
>
> On Mon, Jan 24, 2022 at 7:55 AM Guozhang Wang  wrote:
>
>> Thanks Sagar, I'm +1 on the renamed metric.
>>
>> On Sat, Jan 22, 2022 at 6:56 PM Sagar  wrote:
>>
>> > Hi All,
>> >
>> > There is a small update to the KIP whereby the newly introduced metric
>> > *total-bytes
>> > *has been renamed to *input-buffer-bytes-total.*
>> >
>> > Thanks!
>> > Sagar.
>> >
>> > On Wed, Sep 29, 2021 at 9:57 AM Sagar 
>> wrote:
>> >
>> > > We have 3 binding votes: Sophie/Guozhang/Mathias
>> > > and 2 non-binding votes: Josep/Luke and no -1 votes.
>> > >
>> > > Marking this KIP as accepted.
>> > >
>> > > Thanks everyone!
>> > >
>> > > Thanks!
>> > > Sagar.
>> > >
>> > >
>> > >
>> > > On Wed, Sep 29, 2021 at 7:18 AM Matthias J. Sax 
>> > wrote:
>> > >
>> > >> +1 (binding)
>> > >>
>> > >> On 9/28/21 10:40 AM, Sagar wrote:
>> > >> > Hi All,
>> > >> >
>> > >> > Bumping this vote thread again!
>> > >> >
>> > >> > Thanks!
>> > >> > Sagar.
>> > >> >
>> > >> > On Wed, Sep 8, 2021 at 1:19 PM Luke Chen 
>> wrote:
>> > >> >
>> > >> >> Thanks for the KIP.
>> > >> >>
>> > >> >> + 1 (non-binding)
>> > >> >>
>> > >> >> Thanks.
>> > >> >> Luke
>> > >> >>
>> > >> >> On Wed, Sep 8, 2021 at 2:48 PM Josep Prat
>> > > > >> >
>> > >> >> wrote:
>> > >> >>
>> > >> >>> +1 (non binding).
>> > >> >>>
>> > >> >>> Thanks for the KIP Sagar!
>> > >> >>> ———
>> > >> >>> Josep Prat
>> > >> >>>
>> > >> >>> Aiven Deutschland GmbH
>> > >> >>>
>> > >> >>> Immanuelkirchstraße 26, 10405 Berlin
>> > >> >>>
>> > >> >>> Amtsgericht Charlottenburg, HRB 209739 B
>> > >> >>>
>> > >> >>> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
>> > >> >>>
>> > >> >>> m: +491715557497
>> > >> >>>
>> > >> >>> w: aiven.io
>> > >> >>>
>> > >> >>> e: josep.p...@aiven.io
>> > >> >>>
>> > >> >>> On Wed, Sep 8, 2021, 03:29 Sophie Blee-Goldman
>> > >> >> > > >> 
>> > >> >>> wrote:
>> > >> >>>
>> > >>  +1 (binding)
>> > >> 
>> > >>  Thanks for the KIP!
>> > >> 
>> > >>  -Sophie
>> > >> 
>> > >>  On Tue, Sep 7, 2021 at 1:59 PM Guozhang Wang <
>> wangg...@gmail.com>
>> > >> >> wrote:
>> > >> 
>> > >> > Thanks Sagar, +1 from me.
>> > >> >
>> > >> >
>> > >> > Guozhang
>> > >> >
>> > >> > On Sat, Sep 4, 2021 at 10:29 AM Sagar <
>> sagarmeansoc...@gmail.com>
>> > >> >>> wrote:
>> > >> >
>> > >> >> Hi All,
>> > >> >>
>> > >> >> I would like to start a vote on the following KIP:
>> > >> >>
>> > >> >>
>> > >> >
>> > >> 
>> > >> >>>
>> > >> >>
>> > >>
>> >
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=186878390
>> > >> >>
>> > >> >> Thanks!
>> > >> >> Sagar.
>> > >> >>
>> > >> >
>> > >> >
>> > >> > --
>> > >> > -- Guozhang
>> > >> >
>> > >> 
>> > >> >>>
>> > >> >>
>> > >> >
>> > >>
>> > >
>> >
>>
>>
>> --
>> -- Guozhang
>>
>

Re: [VOTE] KIP-808: Add support for unix epoch precision in TimestampConverter SMT

[Julien Chanaud]

Hi all,

With 3 binding votes (Tom, Mickael, John) and 1 non-binding vote (Donjin),
the KIP passes.

Thanks,

Julien

Le ven. 11 févr. 2022 à 20:41, John Roesler  a écrit :

> Thanks, Julien!
>
> I’m +1 (binding)
>
> -John
>
> On Fri, Feb 11, 2022, at 08:42, Dongjin Lee wrote:
> > +1 (non-binding)
> >
> > Thanks for the KIP!
> >
> > Thanks,
> > Dongjin
> >
> > On Fri, Feb 11, 2022, 10:16 PM Julien Chanaud 
> > wrote:
> >
> >> Bumping this vote.
> >>
> >> We have 2 binding votes so far.
> >>
> >> The associated PR is already implemented:
> >> https://github.com/apache/kafka/pull/11575
> >> Let me know if you have any feedback.
> >>
> >> Thanks,
> >> Julien
> >>
> >> PS: Have a look at the KIP-769 voting thread as well :)
> >>
> >> Le mer. 26 janv. 2022 à 12:46, Mickael Maison 
> a
> >> écrit :
> >>
> >> > +1 (binding)
> >> >
> >> > Thanks Julien for the KIP!
> >> >
> >> > On Wed, Jan 26, 2022 at 12:05 PM Tom Bentley 
> >> wrote:
> >> > >
> >> > > Hi Julien,
> >> > >
> >> > > Thanks again for this KIP. +1 (binding).
> >> > >
> >> > > Kind regards,
> >> > >
> >> > > Tom
> >> > >
> >> > > On Tue, 18 Jan 2022 at 08:15, Julien Chanaud <
> chanaud.jul...@gmail.com
> >> >
> >> > > wrote:
> >> > >
> >> > > > Hi everyone,
> >> > > >
> >> > > > I'd like to start a vote for KIP-808: Add support for unix epoch
> >> > precision
> >> > > > in TimestampConverter SMT
> >> > > >
> >> > > > https://cwiki.apache.org/confluence/x/GJuqCw
> >> > > >
> >> > > > Thanks for your help,
> >> > > >
> >> > > > Julien
> >> > > >
> >> >
> >>
>

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #687

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-13666) Tests should not ignore exceptions for supported OS

[Rob Leland (Jira)]

Rob Leland created KAFKA-13666:
--

 Summary: Tests should not ignore exceptions for supported OS
 Key: KAFKA-13666
 URL: https://issues.apache.org/jira/browse/KAFKA-13666
 Project: Kafka
  Issue Type: Test
Affects Versions: 3.0.0
Reporter: Rob Leland


A few of the tests are swallowing exceptions for all operations systems when 
using the testDrivers becasue it fails on windows.
Change this so exceptions are only ignored for Windows OS.

I am preparing a PR for this small fix.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Gigantic list of Avro spec issues

[Askar Safin]

Hi. I'm writing my own Avro implementation in Rust for personal use. During 
this work I found a lot of issues in Avro 
spec (the list follows).

I send this mail not only to user and dev mailing lists of Avro, but also to 
Apache community list, Kafka list and 3 
semi-randomly chosen Materialize employees. Because I want to draw attention to 
this problems. I hope this wider 
community helps Avro fix their issues and possible will give necessary 
resources.

As well as I understand Avro is used in Kafka. And Kafka, according to their 
site, is used in "80% of all Fortune 100 
companies". So Avro is critical piece of infrastructure of humanity. It should 
be absolutely perfect (and so I list 
even very small bugs). But it is not perfect.

Some of items in this list are (small and big) bugs, some are typos, some are 
my objections to the design. Some are 
fixable while keeping compatibility, some are not. I don't want to spend my 
time to report them as separate bugs, but 
you can try to convince me to do so.

I created this list simply by reading the spec from end to end (I skipped 
sections on RPC and logical types). And I 
even didn't look at implementations!

I write this is hope to help Avro.

I think big audit of spec and its implementations should be done.

All line numbers apply to spec.xml from tag release-1.11.0 (i. e. 
https://github.com/apache/avro/blob/release-1.11.0/doc/src/content/xdocs/spec.xml
 ). As well as I understand this tag 
corresponds to currently published version at 
https://avro.apache.org/docs/current/spec.html .

So, here we go.

* [Opinion] [No line]. In Avro one have to define named records inside each 
other like so:

{ "type": "record", "name": "a", "fields": 
[{"name":"b","type":{"type":"record","name":"c",...}}] }

This is very unnatural. In popular programming languages one usually define 
named record next to each other, not one 
inside other. Such representation is not handy to deal programmatically. In my 
implementation I have to convert this 
representation to usual form "root type + list of named types" right after 
reading JSON and convert back just before 
writing.

* [Opinion] [No line]. In this list you will see a lot of questions on Avro 
schema (encoded as JSON). Good JSON schema 
( https://json-schema.org/ ) would resolve many of them

* [Seems to be bug] [Line 49]. "derived type name" is vague term. In fact, in 
whole spec phrase "type name" is used 
very vaguely. Sometimes it means strings like "record" and sometimes it means 
names of named types. I propose to define 
in very beginning of the spec terms for primitive types, terms for strings like 
"record" and terms for names of defined 
types. Here is one possible way to do this: name strings like "record" and 
"fixed" "type kinds" and never name them 
type names, thus reserving term "type name" to named types only (and possibly 
to primitive types).

This issue already caused problems: look, for example, to this problems with 
{"type":"record","name":"record",...}: 
https://lists.apache.org/thread/0wmgyx6z69gy07lvj9ndko75752b8cn2 .

* [Opinion] [Line 58]. There is no primitive type for unsigned 64 bit integers. 
Such type is present in languages such 
as C and Rust

* [Very theoretical bug, possible even security-related] [Line 435]. "The float 
is converted into a 32-bit integer 
using a method equivalent to Java's floatToIntBits and then encoded in 
little-endian format". If we click at provided 
link, we will see that this Java function does NaN normalization. I think NaN 
normalization is good thing. But I think 
this quite possible spec implementers overlooked this NaN normalization 
requirement. So I propose: write explicitly 
directly in Avro spec that NaN are normalized. Audit all Avro implementations: 
whether they actually implemented this 
requirement. Create tests, which will actually test this requirement.

Also I don't know whether bit pattern provided in that Java doc (0x7fc0) is 
quiet NaN or signaling. If it is 
signaling, this is very bad.

As well as I understand if you will configure your FPU particular way than 
merely copying signaling NaN from one place 
to another will abort your program. So, if your FPU is configured certain way 
then feeding particular binary Avro data 
to a program can crash it. I. e. this is security problem. So a reader should 
be careful to check whether input data is 
signaling NaN *before* storing it in floating point registers.

I checked whether manipulating signaling NaN can actually crash a program in 
default settings in Windows and Linux. And 
it turned out that a program will not crash. Still I think signaling NaN should 
be handled carefully.

Write to spec that writers should normalize NaNs, that readers should reject 
non-normalized NaNs and that readers 
should be careful not to store incoming floating number to floating-point 
variable before its sanitizing. Write that 
this is security issue.

* [Opinion] [Line 68]. "unicode character seq

[GitHub] [kafka-site] showuon merged pull request #396: MINOR: Add CVE-2022-23302 and CVE-2022-23305 to cve-list

[GitBox]

showuon merged pull request #396:
URL: https://github.com/apache/kafka-site/pull/396


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org