[jira] [Created] (KAFKA-14168) Constant memory usage increase

[zhangdong7 (Jira)]

zhangdong7 created KAFKA-14168:
--

 Summary: Constant memory usage increase
 Key: KAFKA-14168
 URL: https://issues.apache.org/jira/browse/KAFKA-14168
 Project: Kafka
  Issue Type: Bug
  Components: KafkaConnect
Affects Versions: 2.8.0
Reporter: zhangdong7
 Attachments: image-2022-08-16-17-16-53-039.png

 the producer threads  grows on demand and is not reduced

!image-2022-08-16-17-16-53-039.png!



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Re: KIP 678: New Kafka Connect SMT for plainText => Struct(or Map) with Regex

[gyejun choi]

Hello Chris,

Long time no feed back about this,

do I have any work left to do to proceed with discuss and vote?

I will waiting for your guide,

thanks, always

whsoul

2022년 6월 14일 (화) 오후 9:29, gyejun choi 님이 작성:

> Hi Chris
>
> I updated KIP document
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP+678%3A+New+Kafka+Connect+SMT+for+plainText+%3D%3E+Struct%28or+Map%29+with+Regex
> )
> about discussion and changes before,
>
> and commited some code according to your review.
>
> https://github.com/apache/kafka/pull/12219/commits/1541a538eaff676441f125c400a7807d17f2e138
>
> I understand you mean the simplicity is more important with SMT ( because
> it is Simple Message Trasform.. )
> so I removed about "message" structured format input support
>
> and also move GroupRegexValidator to inner private class.
>
> thanks, always
>
> whsoul
>
> 2022년 6월 14일 (화) 오전 11:30, Chris Egerton 님이 작성:
>
>> Hi whsoul,
>>
>> Would you mind updating the KIP document (
>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP+678%3A+New+Kafka+Connect+SMT+for+plainText+%3D%3E+Struct%28or+Map%29+with+Regex
>> )
>> with all of these changes? We tend to discuss and vote on what's included
>> in the Wiki as our source of truth, as opposed to pull requests.
>>
>> RE 4:
>> > it seems the chained transform with extractField SMT + plainText value
>> ParseStructByRegex make the same result with struct value
>> ParseSturctByRegex, but it will drop collector meta data during
>> extractField ( I think.. )
>> This is exactly right--using ExtractField + ParseStructByRegex will
>> replace
>> the record key/value with the parsed struct, and drop everything else. Do
>> you think it'd increase the implementation complexity significantly to add
>> support for users to specify a field to operate on, which would preserve
>> the rest of the record key/value as-is? I'm not so sure that everyone is
>> going to want to drop all other metadata from their messages, but if
>> there's something that makes this particularly difficult to implement (as
>> compared to the other SMTs that are already included out of the box with
>> Kafka Connect), then we could probably leave it out for now.
>>
>> RE 6:
>> It's a tricky distinction, but what I mean is that, although we might add
>> a
>> class named GroupRegexValidator and use that class in our SMT library,
>> unless it's part of the public interface we're trying to change, it's an
>> implementation detail and we don't have to call it out in the KIP. The
>> advantage to leaving it out is that it makes the KIP more concise and
>> therefore easier to review, you can choose to rename, remove, decompose,
>> etc. the class in your PR without having to worry about sticking to the
>> plan in the KIP that everyone reviewed and voted on. It's a minor detail
>> though, I'm noting it here more because it may be useful when writing
>> future KIPs than because it's necessary to adhere to strictly in this one.
>>
>> Cheers,
>>
>> Chris
>>
>> On Fri, Jun 10, 2022 at 8:13 AM gyejun choi  wrote:
>>
>> > Hi Chris,
>> >
>> > I applied some code fix according your second reviews.
>> >
>> >
>> https://github.com/apache/kafka/pull/12219/commits/f673ea2eae0d907502e44c0ecd53b616386627bf
>> >
>> >
>> > 1. [applied] changed name as ParseStructByRegex
>> >
>> > 2. [applied] throw DataException, when a line that the SMT sees doesn't
>> > match the regex...
>> > originally, it will be skipped if no data match with regex,
>> > but change code to throw DataException according to your review
>> >
>> > 3. [already applied]
>> > I already delete code about desc ":{TYPE}" with commit below
>> >
>> >
>> https://github.com/apache/kafka/pull/12219/commits/534d995b3e6371c37443eb72eee03884cb23c85d
>> >
>> > 4. [need discuss]
>> > In my use case about log data collection,
>> > I configured the pipeline below
>> > nginx => filebeat => kafka => kafka connect es connector => es
>> >
>> > filebeat ( or most other log collector ) usually send value as struct (
>> not
>> > plaintext ) with collector meta data,
>> > and the key name as "message" ( in case filebeat )
>> >
>> > I think there are more use cases log message wrapped struct value than
>> > plain text value,
>> > and it seems the chained transform with extractField SMT + plainText
>> value
>> > ParseStructByRegex make the same result with struct value
>> > ParseSturctByRegex,
>> > but it will drop collector meta data during extractField ( I think.. )
>> > and also in almost case, users will use ParseStructByRegex SMT with
>> > extractField SMT
>> >
>> > 5. [applied] throw DataException, when there are difference between
>> group
>> > size and mapping names size
>> >
>> >
>> > 6. [ question ]
>> > you mean, use RegexValidator class already exist?
>> > without group "(.*)" pattern check, it will not provide early detection
>> > about regex config mistake,
>> > If you think it is enough as runtime DataException detection?
>> >
>> > always thanks.
>> >
>> > whsoul
>> >
>> > 2022년 6월 8일 (수) 오전 

Re: [DISCISS] KIP-860: Add client-provided option to guard against unintentional replication factor change during partition reassignments

[David Jacot]

Thanks Stan. Overall, the KIP looks good to me. I have two minor comments:

* Could we document the different versions in the
AlterPartitionReassignmentsRequest/Request schema? You can look at
other requests/responses to see how we have done this so far.

* I wonder if --disallow-replication-factor-change would be a better
name. I don't feel strong about this so I am happy to go with the
quorum here.

Best,
David

On Tue, Aug 16, 2022 at 12:31 AM Stanislav Kozlovski
 wrote:
>
> Thanks for the discussion all,
>
> I have updated the KIP to mention throwing an UnsupportedVersionException
> if the server is running an old version that would not honor the configured
> allowReplicationFactor option.
>
> Please take a look:
> - KIP:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-860%3A+Add+client-provided+option+to+guard+against+replication+factor+change+during+partition+reassignments
> - changes:
> https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=217392873&selectedPageVersions=4&selectedPageVersions=3
>
> If there aren't extra comments, I plan on starting a vote thread by the end
> of this week.
>
> Best,
> Stanislav
>
> On Tue, Aug 9, 2022 at 5:06 AM David Jacot 
> wrote:
>
> > Throwing an UnsupportedVersionException with an appropriate message
> > seems to be the best option when the new API is not supported and
> > AllowReplicationFactorChange is not set to the default value.
> >
> > Cheers,
> > David
> >
> > On Mon, Aug 8, 2022 at 6:25 PM Vikas Singh 
> > wrote:
> > >
> > > I personally like the UVE option. It provides options for clients to go
> > > either way, retry or abort. If we do it in AdminClient, then users have
> > to
> > > live with what we have chosen.
> > >
> > > > Note this can happen during an RF change too. e.g [1,2,3] => [4,5,6,7]
> > (RF
> > > > change, intermediate set is [1,2,3,4,5,6,7]), and we try to do a
> > > > reassignment to [9,10,11], the logic will compare [4,5,6,7] to
> > [9,10,11].
> > > > In such a situation where one wants to cancel the RF increase and
> > reassign
> > > > again, one first needs to cancel the existing reassignment via the API
> > (no
> > > > special action required despite RF change)
> > >
> > > Thanks for the explanation. I did realize this nuance and thus requested
> > to
> > > put that in KIP as it's not mentioned why the choice was made. I am fine
> > if
> > > you choose to not do it in the interest of brevity.
> > >
> > > Vikas
> > >
> > > On Sun, Aug 7, 2022 at 9:02 AM Stanislav Kozlovski
> > >  wrote:
> > >
> > > > Thank you for the reviews.
> > > >
> > > > Vikas,
> > > > > > In the case of an already-reassigning partition being reassigned
> > again,
> > > > the validation compares the targetReplicaSet size of the reassignment
> > to
> > > > the targetReplicaSet size of the new reassignment and throws if those
> > > > differ.
> > > > > Can you add more detail to this, or clarify what is targetReplicaSet
> > (for
> > > > e.g. why not sourceReplicaSet?) and how the target replica set will be
> > > > calculated?
> > > > If a reassignment is ongoing, such that [1,2,3] => [4,5,6] (the
> > replica set
> > > > in Kafka will be [1,2,3,4,5,6] during the reassignment), and you try to
> > > > issue a new reassignment (e.g [7,8,9], Kafka should NOT think that the
> > RF
> > > > of the partition is 6 just because a reassignment is ongoing. Hence, we
> > > > compare [4,5,6]'s length to [7,8,9]
> > > > The targetReplicaSet is a term we use in KIP-455
> > > > <
> > > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-455%3A+Create+an+Administrative+API+for+Replica+Reassignment
> > > > >.
> > > > It means the desired replica set that a given reassignment is trying to
> > > > achieve. Here we compare said set of the on-going reassignment to the
> > new
> > > > reassignment.
> > > >
> > > > Note this can happen during an RF change too. e.g [1,2,3] => [4,5,6,7]
> > (RF
> > > > change, intermediate set is [1,2,3,4,5,6,7]), and we try to do a
> > > > reassignment to [9,10,11], the logic will compare [4,5,6,7] to
> > [9,10,11].
> > > > In such a situation where one wants to cancel the RF increase and
> > reassign
> > > > again, one first needs to cancel the existing reassignment via the API
> > (no
> > > > special action required despite RF change)
> > > >
> > > >
> > > > > And what about the reassign partitions CLI? Do we want to expose the
> > > > option there too?
> > > > Yes, this is already present in the KIP if I'm not mistaken. We
> > describe it
> > > > in "Accordingly, the kafka-reassign-partitions.sh tool will be updated
> > to
> > > > allow supplying the new option:"
> > > > I have edited the KIP to contain two clear paragraphs called Admin API
> > and
> > > > CLI now.
> > > >
> > > > Colin,
> > > >
> > > > >  it would be nice for the first paragraph to be a bit more explicit
> > about
> > > > this goal.
> > > > sounds good, updated it with that suggestion.
> > > >
> > > > > client-side forward compatibility
> > > > I was

[jira] [Resolved] (KAFKA-14148) Outdated doc for reset-offsets option

[David Jacot (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Jacot resolved KAFKA-14148.
-
Fix Version/s: 3.3.0
   Resolution: Fixed

> Outdated doc for reset-offsets option
> -
>
> Key: KAFKA-14148
> URL: https://issues.apache.org/jira/browse/KAFKA-14148
> Project: Kafka
>  Issue Type: Improvement
>  Components: admin
>Reporter: K8sCat
>Priority: Minor
> Fix For: 3.3.0
>
> Attachments: image-2022-08-08-19-19-34-873.png
>
>
> !image-2022-08-08-19-19-34-873.png!
> --by-period should be --by-duration, and --to-offset show be added



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

RE: [DISCUSS] KIP-793: Sink Connectors: Support topic-mutating SMTs for async connectors (preCommit users)

[Yash Mayya]

Hi Diego,

Thanks for writing this KIP. Are you still planning to work on this? If
not, I'd be happy to try and take this to completion!

Hi Chris,

Thanks for your valuable inputs as always!

> It looks like this KIP does not take the SinkTask::open and
SinkTask::close methods into account

> I think we may want to add this type of support to the KIP so that we can
solve the mutating SMT/asynchronous sink connector problem once and for all

Could you please clarify what you mean here? Do you mean to say that the
partitions passed to the SinkTask::open / SinkTask::close methods should
not be the original topic partitions? If so, how would we be able to change
that while maintaining backward compatibility? Would we want to add new
methods to SinkTask's public API which connectors could choose to implement
instead? Also, currently SinkTask::open is called when partitions are
assigned to the task's consumer in a consumer rebalance; at this point
there is no way to know of any "transformed" topic partitions - that can
only be determined after the consumer polls and records are converted +
transformed. Similarly, SinkTask::close is called when partitions are
revoked from the task's consumer in a consumer rebalance. Are you
suggesting we change when SinkTask::open and SinkTask::close are called
(maybe by book-keeping all currently known post-transformation topic
partitions)?

>  I agree with the rationale for not exposing more of the original
consumer record for the most part, but what about the record's offset?

Should we pollute the public API with a method that in all likelihood will
never be used? Maybe I'm lacking imagination here, but why would an SMT
ever want to modify the record's offset? Is there any such SMT currently?
It seems to me that adding such a method may just be unnecessary complexity
and a potential source of confusion to Connect developers, WDYT?

> do you think it'd make sense to separate out the newly-proposed
SinkTask::originalTopicPartition method into separate
SinkTask::originalTopic and SinkTask::originalKafkaPartition methods, to
stay in line with the convention that's been loosely set by the existing,
separate SinkTask::topic and SinkTask::kafkaPartition methods?

I would tend to agree with this rationale.

Thanks,
Yash

On 2021/11/03 22:17:06 Diego Erdody wrote:
> Hello,
>
> I'd like to propose a small KIP to add a new field to SinkRecord in order
> to add support for topic-mutating SMTs (e.g. RegexRouter) to asynchronous
> Sink Connectors (the ones that override preCommit for internal offset
> tracking, like S3
> <
https://github.com/confluentinc/kafka-connect-storage-cloud/blob/master/kafka-connect-s3/src/main/java/io/confluent/connect/s3/S3SinkTask.java#L274
>
> ).
>
> Links:
>
> - KIP-793: Sink Connectors: Support topic-mutating SMTs for async
> connectors (preCommit users)
> 
> - PR #11464 
>
> Thanks,
>
> Diego
>

[jira] [Created] (KAFKA-14169) Memory leak (?) with no activity

[Ricardo Giacomin (Jira)]

Ricardo Giacomin created KAFKA-14169:


 Summary: Memory leak (?) with no activity
 Key: KAFKA-14169
 URL: https://issues.apache.org/jira/browse/KAFKA-14169
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 3.2.1
 Environment: AKS, Linux-based Kafka cluster setup
Reporter: Ricardo Giacomin
 Attachments: image-2022-08-16-11-05-32-227.png, 
image-2022-08-16-11-07-51-565.png

I have deployed a 3-node Kafka cluster (see setup below).

!image-2022-08-16-11-05-32-227.png|width=918,height=643!

I didn't touch the cluster since installation (did not create any topic or run 
any consumer/producer) and yet the memory usage is increasing, as displayed by 
the last 48 hours chart below.

!image-2022-08-16-11-07-51-565.png|width=916,height=440!

 

Sounds like a memory leak??? The strange (increasing in steps) behaviour 
suggests some periodic jobs running but I didn't configure any.

Any hints?

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

RE: [DISCUSS] KIP-793: Sink Connectors: Support topic-mutating SMTs for async connectors (preCommit users)

[Yash Mayya]

Apologies for the poor formatting on the quoted bits in the previous email.

On 2021/11/03 22:17:06 Diego Erdody wrote:
> Hello,
>
> I'd like to propose a small KIP to add a new field to SinkRecord in order
> to add support for topic-mutating SMTs (e.g. RegexRouter) to asynchronous
> Sink Connectors (the ones that override preCommit for internal offset
> tracking, like S3
> <
https://github.com/confluentinc/kafka-connect-storage-cloud/blob/master/kafka-connect-s3/src/main/java/io/confluent/connect/s3/S3SinkTask.java#L274
>
> ).
>
> Links:
>
> - KIP-793: Sink Connectors: Support topic-mutating SMTs for async
> connectors (preCommit users)
> 
> - PR #11464 
>
> Thanks,
>
> Diego
>

Re: Last sprint to finish line: Replace EasyMock/Powermock with Mockito

[Guozhang Wang]

Thanks everyone for the great effort!

Just one quick question regarding the Streams tests: I looked over
https://github.com/apache/kafka/pull/12492/files,
https://github.com/apache/kafka/pull/12505/files and
https://github.com/apache/kafka/pull/12465/files, but I did not find the
one covering `TaskManagerTest` that has a mixed usage of Mockito and
EasyMock. Do you have a PR covering that class and if yes, could you point
me to that one?


Guozhang

On Mon, Aug 15, 2022 at 11:48 AM Christo 
wrote:

>  Hello!
> Following Divij's example I wanted to give an update on the progress being
> made.
> With a combined effort from Yash Mayya, Matthew de Detrich and myself we
> are 3/4 through providing pull requests for moving the remaining EasyMock
> tests to Mockito (https://issues.apache.org/jira/browse/KAFKA-14133).
> Across those and other pull requests we have received insightful feedback
> from Bruno Cadonna, Chris Egerton, Dalibor Plavcic and Ismael Juma on how
> things can be improved. Thank you very much!
> Current pull requests we are trying to get to a resolution from oldest to
> newest are:
> * https://github.com/apache/kafka/pull/12409*
> https://github.com/apache/kafka/pull/12418*
> https://github.com/apache/kafka/pull/12459*
> https://github.com/apache/kafka/pull/12465
> * https://github.com/apache/kafka/pull/12473
> * https://github.com/apache/kafka/pull/12492*
> https://github.com/apache/kafka/pull/12505*
> https://github.com/apache/kafka/pull/12509
> Best,Christo 
>
> On Thursday, 4 August 2022, 18:27:17 BST, Divij Vaidya <
> divijvaidy...@gmail.com> wrote:
>
>  Hi everyone
>
> To provide you with quick updates on the progress.
>
> Open PRs (pending review):
>
>   1. Streams - https://github.com/apache/kafka/pull/12449
>   2. Streams - https://github.com/apache/kafka/pull/12465
>   3. Streams - https://github.com/apache/kafka/pull/12459
>   4. Connect - https://github.com/apache/kafka/pull/12484
>   5. Connect - https://github.com/apache/kafka/pull/12473  6. Connect -
> https://github.com/apache/kafka/pull/12409
>   7. Connect - https://github.com/apache/kafka/pull/12472
>
> Open tasks (pending an owners):
>
>   1. https://issues.apache.org/jira/browse/KAFKA-14132 (need owners for
>   separate individual tests)
>   2. https://issues.apache.org/jira/browse/KAFKA-14133
>
>
> General guidance to reduce code review churn when working on these test
> conversions:
>
>   1. Please use @RunWith(MockitoJUnitRunner.StrictStubs.class) since it
>   provides many benefits.
>   2. Please do not perform JUnit 5 migration in the same PR as Mockito
>   conversion to keep the changes few and easy to review. We will follow up
>   with a blanket JUnit5 conversion (similar to this
>   ) when Mockito migration is
>   complete.
>   3. Please use @Mock annotation to mock (Chris Egerton has added this
>   comment on various PRs, hence calling it out)
>   4. Note that @RunWith(MockitoJUnitRunner.StrictStubs.class) verifies the
>   invocation of declared stubs automatically. If the stubs are not invoked,
>   the test throws a UnnecessaryStubbingException. Note that this doesn't
> seem
>   to work for `mockStatic` and I would suggest to explicitly verify stub
>   invocations over there.
>   5. As a reference, you can use the merged PR from Chris Egerton here:
>   https://github.com/apache/kafka/pull/12409  6. Add a verification step
> in the description that the test has
>   successfully run with the command `./gradlew connect:runtime:unitTest`
> (or
>   equivalent for the module you are changing the test for). Additionally,
> you
>   can add the code coverage report using `./gradlew streams:reportCoverage
>   -PenableTestCoverage=true -Dorg.gradle.parallel=false` to verify that no
>   test assertion has been accidentally removed during the change.
>
>
> *Chris*, would you like to add anything else to the general guidance above
> which would help reduce the code review churn?
>
> --
> Divij Vaidya
>
>
>
> On Mon, Aug 1, 2022 at 6:49 PM Divij Vaidya 
> wrote:
>
> > Hi folks
> >
> > We have been trying to replace EasyMock/Powermock with Mockito
> >  for quite a while.
> > This adds complications for migrating to JDK 17 & Junit5. Significant
> > contributions have been made by various folks towards this goal and the
> > finish line is almost in sight.
> >
> > Let's join forces this week and get the task done!
> >
> > I and Christo(cc'ed) will be spending time converting the straggler tests
> > during this week.
> >
> > At this stage, we are missing a shepherd to help us wrap up this task.
> *Could
> > we please solicit some code review bandwidth from a committer for this
> week
> > to help us reach the finish line?*
> >
> > Current pending PR requests:
> > 1. KAFKA-13036: Replace EasyMock and PowerMock with Mockito for
> RocksDBMetricsRecorderTest by divijvaidya · Pull Request

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #1147

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 421165 lines...]
[2022-08-16T16:47:07.769Z] > Task :connect:api:javadoc
[2022-08-16T16:47:09.037Z] > Task :connect:api:copyDependantLibs UP-TO-DATE
[2022-08-16T16:47:09.037Z] > Task :connect:api:jar UP-TO-DATE
[2022-08-16T16:47:09.037Z] > Task :connect:json:copyDependantLibs UP-TO-DATE
[2022-08-16T16:47:09.037Z] > Task :connect:json:jar UP-TO-DATE
[2022-08-16T16:47:09.037Z] > Task 
:connect:json:generateMetadataFileForMavenJavaPublication
[2022-08-16T16:47:09.037Z] > Task 
:connect:api:generateMetadataFileForMavenJavaPublication
[2022-08-16T16:47:09.037Z] > Task :connect:api:javadocJar
[2022-08-16T16:47:09.037Z] > Task :connect:api:compileTestJava UP-TO-DATE
[2022-08-16T16:47:09.037Z] > Task :connect:api:testClasses UP-TO-DATE
[2022-08-16T16:47:09.037Z] > Task :connect:api:testJar
[2022-08-16T16:47:09.037Z] > Task 
:connect:json:publishMavenJavaPublicationToMavenLocal
[2022-08-16T16:47:09.037Z] > Task :connect:json:publishToMavenLocal
[2022-08-16T16:47:09.037Z] > Task :connect:api:testSrcJar
[2022-08-16T16:47:09.951Z] > Task 
:connect:api:publishMavenJavaPublicationToMavenLocal
[2022-08-16T16:47:09.951Z] > Task :connect:api:publishToMavenLocal
[2022-08-16T16:47:09.951Z] 
[2022-08-16T16:47:09.951Z] > Task :streams:javadoc
[2022-08-16T16:47:09.951Z] 
/home/jenkins/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/kstream/Repartitioned.java:101:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-08-16T16:47:09.951Z] 
/home/jenkins/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/kstream/Repartitioned.java:167:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-08-16T16:47:09.951Z] 
/home/jenkins/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/TopologyConfig.java:58:
 warning - Tag @link: missing '#': "org.apache.kafka.streams.StreamsBuilder()"
[2022-08-16T16:47:09.951Z] 
/home/jenkins/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/TopologyConfig.java:58:
 warning - Tag @link: can't find org.apache.kafka.streams.StreamsBuilder() in 
org.apache.kafka.streams.TopologyConfig
[2022-08-16T16:47:09.951Z] 
/home/jenkins/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/TopologyDescription.java:38:
 warning - Tag @link: reference not found: ProcessorContext#forward(Object, 
Object) forwards
[2022-08-16T16:47:09.951Z] 
/home/jenkins/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/query/Position.java:44:
 warning - Tag @link: can't find query(Query,
[2022-08-16T16:47:09.951Z]  PositionBound, boolean) in 
org.apache.kafka.streams.processor.StateStore
[2022-08-16T16:47:09.951Z] 
/home/jenkins/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:109:
 warning - Tag @link: reference not found: this#getResult()
[2022-08-16T16:47:09.951Z] 
/home/jenkins/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:116:
 warning - Tag @link: reference not found: this#getFailureReason()
[2022-08-16T16:47:09.951Z] 
/home/jenkins/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:116:
 warning - Tag @link: reference not found: this#getFailureMessage()
[2022-08-16T16:47:09.951Z] 
/home/jenkins/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:154:
 warning - Tag @link: reference not found: this#isSuccess()
[2022-08-16T16:47:09.951Z] 
/home/jenkins/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:154:
 warning - Tag @link: reference not found: this#isFailure()
[2022-08-16T16:47:09.951Z] 
/home/jenkins/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:890:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-08-16T16:47:09.951Z] 
/home/jenkins/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:919:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-08-16T16:47:09.951Z] 
/home/jenkins/workspace/Kafka_kafka_trunk/streams/src/main/java/org/apache/kafka/streams/kstream/KStream.java:939:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-08-16T16:47:09.951Z] 25 warnings
[2022-08-16T16:47:09.951Z] 
[2022-08-16T16:47:09.951Z] > Task :streams:javadocJar
[2022-08-16T16:47:10.718Z] 
[2022-08-16T16:47:10.718Z] > Task :clients:javadoc
[2022-08-16T16:47:10.718Z] 
/home/jenkins/workspace/Kafka_kafka_trunk/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/secured/OAuthBearerLoginCallbackHandler.java:147:
 warning - Tag @link: reference not found: 
[2022-08-16T16:47:12.620Z] 1 warning
[2022-08-16T16:47:12.620Z] 
[2022-08-16T16:47:1

[GitHub] [kafka-site] mimaison commented on pull request #435: MINOR:Clean up images

[GitBox]

mimaison commented on PR #435:
URL: https://github.com/apache/kafka-site/pull/435#issuecomment-1216973319

   @bbejeck I pushed new placeholders to your branch


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: Consumer Lag-Apache_kafka_JMX metrics

[Kafka Life]

Hello Experts, Any info or pointers on my query please.



On Mon, Aug 15, 2022 at 11:36 PM Kafka Life  wrote:

> Dear Kafka Experts
> we need to monitor the consumer lag in kafka clusters 2.5.1 and 2.8.0
> versions of kafka in Grafana.
>
> 1/ What is the correct path for JMX metrics to evaluate Consumer Lag in
> kafka cluster.
>
> 2/ I had thought it is FetcherLag  but it looks like it is not as per the
> link below.
>
> https://www.instaclustr.com/support/documentation/kafka/monitoring-information/fetcher-lag-metrics/#:~:text=Aggregated%20Fetcher%20Consumer%20Lag%20This%20metric%20aggregates%20lag,in%20sync%20with%20partitions%20that%20it%20is%20replicating
> .
>
> Could one of you experts please guide on which JMX i should use for
> consumer lag apart from kafka burrow or such intermediate tools
>
> Thanking you in advance
>
>

Re: Consumer Lag-Apache_kafka_JMX metrics

[Hector Geraldino (BLOOMBERG/ 919 3RD A)]

As far as I know, such metric does not exist. Strictly speaking, consumer lag 
can be defined as the difference between the last produced offset (high 
watermark) and the last committed offset by the group, but such metric has very 
little value without considering the time dimension.

It'd be tricky for the broker to report on consumer 'lag', as the concept of 
lag itself varies. You already know about  Burrow (and I recall reading about 
Uber's uGroup), and you already see that it considers a consumer lagging if it 
is not making enough progress in a sliding time window (10 mins?). But other 
tools/use cases can define lags using a different criteria (e.g. number of 
messages exceeds a threshold).

I think because of these variances, it kinda makes sense for tools like Burrow 
(and others) to be used for this purpose, instead of having the broker 
dictating when consumers are lagging.

Just my two cents

From: dev@kafka.apache.org At: 08/16/22 15:06:16 UTC-4:00To:  
us...@kafka.apache.org,  show...@gmail.com,  mmcfarl...@cavulus.com,  
dev@kafka.apache.org,  scante...@gmail.com,  ranlupov...@gmail.com,  
israele...@gmail.com
Subject: Re: Consumer Lag-Apache_kafka_JMX metrics

Hello Experts, Any info or pointers on my query please.


On Mon, Aug 15, 2022 at 11:36 PM Kafka Life  wrote:

> Dear Kafka Experts
> we need to monitor the consumer lag in kafka clusters 2.5.1 and 2.8.0
> versions of kafka in Grafana.
>
> 1/ What is the correct path for JMX metrics to evaluate Consumer Lag in
> kafka cluster.
>
> 2/ I had thought it is FetcherLag  but it looks like it is not as per the
> link below.
>
> 
https://www.instaclustr.com/support/documentation/kafka/monitoring-information/f
etcher-lag-metrics/#:~:text=Aggregated%20Fetcher%20Consumer%20Lag%20This%20metri
c%20aggregates%20lag,in%20sync%20with%20partitions%20that%20it%20is%20replicatin
g
> .
>
> Could one of you experts please guide on which JMX i should use for
> consumer lag apart from kafka burrow or such intermediate tools
>
> Thanking you in advance
>
>

Build failed in Jenkins: Kafka » Kafka Branch Builder » 3.3 #37

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 570450 lines...]
[2022-08-16T17:13:15.451Z] > Task :connect:api:testSrcJar
[2022-08-16T17:13:15.451Z] > Task 
:connect:json:publishMavenJavaPublicationToMavenLocal
[2022-08-16T17:13:15.451Z] > Task :connect:json:publishToMavenLocal
[2022-08-16T17:13:15.451Z] > Task 
:connect:api:publishMavenJavaPublicationToMavenLocal
[2022-08-16T17:13:15.451Z] > Task :connect:api:publishToMavenLocal
[2022-08-16T17:13:15.451Z] 
[2022-08-16T17:13:15.451Z] > Task :streams:javadoc
[2022-08-16T17:13:15.451Z] 
/home/jenkins/h49-shared/712657a4/workspace/Kafka_kafka_3.3/streams/src/main/java/org/apache/kafka/streams/kstream/Produced.java:84:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-08-16T17:13:15.451Z] 
/home/jenkins/h49-shared/712657a4/workspace/Kafka_kafka_3.3/streams/src/main/java/org/apache/kafka/streams/kstream/Produced.java:136:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-08-16T17:13:15.451Z] 
/home/jenkins/h49-shared/712657a4/workspace/Kafka_kafka_3.3/streams/src/main/java/org/apache/kafka/streams/kstream/Produced.java:147:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-08-16T17:13:15.451Z] 
/home/jenkins/h49-shared/712657a4/workspace/Kafka_kafka_3.3/streams/src/main/java/org/apache/kafka/streams/kstream/Repartitioned.java:101:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-08-16T17:13:15.451Z] 
/home/jenkins/h49-shared/712657a4/workspace/Kafka_kafka_3.3/streams/src/main/java/org/apache/kafka/streams/kstream/Repartitioned.java:167:
 warning - Tag @link: reference not found: DefaultPartitioner
[2022-08-16T17:13:15.451Z] 
/home/jenkins/h49-shared/712657a4/workspace/Kafka_kafka_3.3/streams/src/main/java/org/apache/kafka/streams/TopologyConfig.java:58:
 warning - Tag @link: missing '#': "org.apache.kafka.streams.StreamsBuilder()"
[2022-08-16T17:13:15.451Z] 
/home/jenkins/h49-shared/712657a4/workspace/Kafka_kafka_3.3/streams/src/main/java/org/apache/kafka/streams/TopologyConfig.java:58:
 warning - Tag @link: can't find org.apache.kafka.streams.StreamsBuilder() in 
org.apache.kafka.streams.TopologyConfig
[2022-08-16T17:13:16.375Z] 
/home/jenkins/h49-shared/712657a4/workspace/Kafka_kafka_3.3/streams/src/main/java/org/apache/kafka/streams/TopologyDescription.java:38:
 warning - Tag @link: reference not found: ProcessorContext#forward(Object, 
Object) forwards
[2022-08-16T17:13:16.375Z] 
/home/jenkins/h49-shared/712657a4/workspace/Kafka_kafka_3.3/streams/src/main/java/org/apache/kafka/streams/query/Position.java:44:
 warning - Tag @link: can't find query(Query,
[2022-08-16T17:13:16.375Z]  PositionBound, boolean) in 
org.apache.kafka.streams.processor.StateStore
[2022-08-16T17:13:16.375Z] 
/home/jenkins/h49-shared/712657a4/workspace/Kafka_kafka_3.3/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:44:
 warning - Tag @link: can't find query(Query, PositionBound, boolean) in 
org.apache.kafka.streams.processor.StateStore
[2022-08-16T17:13:16.375Z] 
/home/jenkins/h49-shared/712657a4/workspace/Kafka_kafka_3.3/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:36:
 warning - Tag @link: can't find query(Query, PositionBound, boolean) in 
org.apache.kafka.streams.processor.StateStore
[2022-08-16T17:13:16.375Z] 
/home/jenkins/h49-shared/712657a4/workspace/Kafka_kafka_3.3/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:57:
 warning - Tag @link: can't find query(Query, PositionBound, boolean) in 
org.apache.kafka.streams.processor.StateStore
[2022-08-16T17:13:16.375Z] 
/home/jenkins/h49-shared/712657a4/workspace/Kafka_kafka_3.3/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:74:
 warning - Tag @link: can't find query(Query, PositionBound, boolean) in 
org.apache.kafka.streams.processor.StateStore
[2022-08-16T17:13:16.375Z] 
/home/jenkins/h49-shared/712657a4/workspace/Kafka_kafka_3.3/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:110:
 warning - Tag @link: reference not found: this#getResult()
[2022-08-16T17:13:16.375Z] 
/home/jenkins/h49-shared/712657a4/workspace/Kafka_kafka_3.3/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:117:
 warning - Tag @link: reference not found: this#getFailureReason()
[2022-08-16T17:13:16.375Z] 
/home/jenkins/h49-shared/712657a4/workspace/Kafka_kafka_3.3/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:117:
 warning - Tag @link: reference not found: this#getFailureMessage()
[2022-08-16T17:13:16.375Z] 
/home/jenkins/h49-shared/712657a4/workspace/Kafka_kafka_3.3/streams/src/main/java/org/apache/kafka/streams/query/QueryResult.java:155:
 warning - Tag @link: reference not found: this#isSuccess()
[2022-08-16T17:13:16.375Z] 
/home/jenkins/h49-shared/712657a4/workspace/Kafka_kafka_3.3/streams/src/main/java/org/apache/kafka/streams/