[jira] [Resolved] (KAFKA-17009) Add unit test to query nonexistent replica by describeReplicaLogDirs

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-17009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-17009.

Fix Version/s: 3.9.0
   Resolution: Fixed

> Add unit test to query nonexistent replica by describeReplicaLogDirs
> 
>
> Key: KAFKA-17009
> URL: https://issues.apache.org/jira/browse/KAFKA-17009
> Project: Kafka
>  Issue Type: Test
>Reporter: Chia-Ping Tsai
>Assignee: 黃竣陽
>Priority: Minor
> Fix For: 3.9.0
>
>
> our docs[0] says that "currentReplicaLogDir will be null if the replica is 
> not found", so it means `describeReplicaLogDirs` can accept the queries for 
> nonexistent replicas. However, current UT [1] only verify the replica is not 
> found due to storage error. We should add a UT to verify it for nonexistent 
> replica
> [0] 
> https://github.com/apache/kafka/blob/391778b8d737f4af074422ffe61bc494b21e6555/clients/src/main/java/org/apache/kafka/clients/admin/DescribeReplicaLogDirsResult.java#L71
> [1] 
> https://github.com/apache/kafka/blob/trunk/clients/src/test/java/org/apache/kafka/clients/admin/KafkaAdminClientTest.java#L2356



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

apply a jira account

[周 正轲]

Hi Guys,

I want to contribute to Apache Kafka.
Would you please give me the permission as a contributor?
My JIRA ID is zzzk1.

Re: apply a jira account

[Matthias J. Sax]

Thanks for you interest to contribute to Kafka. Unfortunately, I could 
not find your account. Can you double check the Jira ID?


-Matthias

On 6/23/24 8:01 AM, 周 正轲 wrote:

Hi Guys,

I want to contribute to Apache Kafka.
Would you please give me the permission as a contributor?
My JIRA ID is zzzk1.

Re: [DISCUSS] KIP-1027 Add MockFixedKeyProcessorContext

[Shashwat Pandey]

Totally agree on the constructors to pass in key/val/ts/headers.

I was thinking about adding the method for Record as a helper method if
there was a scenario to test both a processor and fixedkeyprocessor at the
same time. Also, it keeps the same signature as the
InternalFixedKeyRecordFactory as well.


Regards,
Shashwat Pandey


On Sat, Jun 22, 2024 at 8:25 PM Matthias J. Sax  wrote:

> Thanks for the update.
>
> About the wiki account. Creating the account was done by Infra, but
> setting permissions is on us. Fixed.
>
>
> About the KIP:
>
> Should we pass-in key/value/ts/headers and mimic the constructors for
> `TestRecord` (ie, have similar createXxx(...) overloads as factory
> methods as the constructor overloads) instead of `Record` and
> `TestRecord`? Or is there some benefit I am missing to pass in
> `Record/TestRecord` ?
>
>
> -Matthias
>
>
> On 6/22/24 9:32 AM, Shashwat Pandey wrote:
> > Hi Matthias,
> >
> > That makes sense to me! I updated the code, definitely want to get your
> > perspective on whether or not we want to support the
> > `createFixedKeyRecord(Record)` method, since we already have the
> > `TestRecord` defined in the utils, it might be cleaner to just support
> the
> > `createFixedKeyRecord(TestRecord)` method.
> >
> > For reference -
> >
> https://github.com/s7pandey/kafka/commit/8ac92509d455d8175381a9b4c83900218941bf05#diff-2a3e6e23894a888e8c2fa486e2330f42b8fb28fe2216ba182e27d3d14958457b
> >
> > Also, looks like I do not have access to update the KIP, my confluence
> > account is active now (s7pandey) but I think I need some permissions on
> the
> > actual KIP page.
> >
> > Shashwat
> >
> > On Wed, Jun 12, 2024 at 8:32 PM Matthias J. Sax 
> wrote:
> >
> >> I believe the class name was picked on purpose, to make clear that it
> >> should not be used -- the problem is, that the class is in a public
> >> package and is by itself public (that's unfortunately require, given how
> >> Java works).
> >>
> >> Of course, it's also in the JavaDocs that the class is internal and
> >> should not be used, but not everyone reads the JavaDocs necessarily, so
> >> making it part of the name makes it much more explicit, what I believe
> >> is a good thing.
> >>
> >> I would consider it a fix/improvement, if we could exclude
> >> `InternalFixedKeyRecordFactory` from JavaDoc generation during the
> >> release build -- but I don't think we need a KIP for this, as I would
> >> rather consider it a bug-fix to exclude an internal class in the
> >> JavaDocs build step.
> >>
> >>
> >> -Matthias
> >>
> >> On 6/12/24 4:47 PM, Shashwat Pandey wrote:
> >>> Hi Matthias,
> >>>
> >>> I think that strategy definitely works, abstracting away changes to
> >>> FixedKeyRecord from users,  I can put that new factory class and update
> >> the
> >>> KIP.
> >>>
> >>> This might be a discussion for another KIP, but would it also make
> sense
> >> to
> >>> rename the
> >>> InternalFixedKeyRecordFactory to just FixedKeyRecordFactory also make
> >> sense?
> >>>
> >>>
> >>> Regards,
> >>> Shashwat Pandey
> >>>
> >>>
> >>> On Mon, Jun 10, 2024 at 5:07 PM Matthias J. Sax 
> >> wrote:
> >>>
>  Shaswhat,
> 
>  any updates on this KIP? -- I still think that recommending to use
>  `InternalFixedKeyRecordFactory` is not the best way to write test
> code.
>  Changing `FixedKeyRecord` constructor (as I mentioned in my last
> email)
>  might not be a good solution either.
> 
>  Maybe a cleaner way would be (so sidestep this problem), to add a new
>  public "factory class" into the test package to generate
>  FixedKeyRecords, and this factory could internally use
>  `InternalFixedKeyRecordFactory`? It looks cleaner to me from an API
> POV,
>  and if we change anything how `FixedKeyRecord` can be created, it
> would
>  become a non-user-facing / internal change to the "factory" we
> provide.
> 
> 
>  -Matthias
> 
>  On 5/22/24 12:02 AM, Matthias J. Sax wrote:
> > I was not aware of `InternalFixedKeyRecordFactory`. As the name
> > indicates, it's considered an internal class, so not sure if we
> should
> > recommend to use it in test...
> >
> > I understand why this class is required, and why it was put into a
> > public package; the way Java works, enforces this. Not sure if we
> could
> > find a better solution.
> >
> > Might be good to hear from others.
> >
> >
> > -Matthias
> >
> > On 5/21/24 3:57 PM, Shashwat Pandey wrote:
> >> Looking at the ticket and the sample code, I think it would be
> >> possible to
> >> continue using `InternalFixedKeyRecordFactory` as the avenue to
> create
> >> `FixedKeyRecord`s in tests. As long as there was a
> >> MockFixedKeyProcessorContext, I think we would be able to test
> >> FixedKeyProcessors without a Topology.
> >>
> >> I created a sample repo with the `MockFixedKeyProcessorContext` here
> >> is
> >> what I think the tests would loo

Re: [PR] adding spitha on powered-by [kafka-site]

[via GitHub]

VictorParkM commented on PR #515:
URL: https://github.com/apache/kafka-site/pull/515#issuecomment-2185440225

   Hi Matthias
   
   Nice to e-meet you.
   
   I don't fully understand what it means for a PR to be rebased. Should I
   understand it as needing to merge it again with the latest version?
   
   Also, we internally reviewed our website last week because it was said that
   it does not comply with ASF guidelines. Could you please let me know if
   this is due to the presence of the term "Apache Kafka" on our website, or
   if there are other reasons?
   
   Regards,
   
   Victor Park.
   
   
   JeongHun Park
   
   Business Development / Team Manager
   
   SPITHA INC.
   43, Daesagwan-ro, Yongsan-gu, 04401
   SEOUL, SOUTH KOREA
   M) +82.010.4321.2320
   E) ***@***.*** ***@***.***>
   
   *spitha.io *
   
   
   
   The content of this email is confidential and intended for the recipient
   specified in message only. It is strictly forbidden to share any part of
   this message with any third party, without a written consent of the sender.
   If you received this message by mistake, please reply to this message and
   follow with its deletion, so that we can ensure such a mistake does not
   occur in the future.
   
   
   2024년 6월 22일 (토) 오전 4:49, Matthias J. Sax ***@***.***>님이 작성:
   
   > @VictorParkM  -- This PR need to be
   > rebased; merged a couple of other PRs and there is merge conflicts now.
   >
   > —
   > Reply to this email directly, view it on GitHub
   > ,
   > or unsubscribe
   > 

   > .
   > You are receiving this because you were mentioned.Message ID:
   > ***@***.***>
   >
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[jira] [Created] (KAFKA-17025) KafkaThread and KafkaProducer expose method to set setUncaughtExceptionHandler

[Hongshun Wang (Jira)]

Hongshun Wang created KAFKA-17025:
-

 Summary: KafkaThread and KafkaProducer expose method to set 
setUncaughtExceptionHandler
 Key: KAFKA-17025
 URL: https://issues.apache.org/jira/browse/KAFKA-17025
 Project: Kafka
  Issue Type: Improvement
  Components: clients
Affects Versions: 3.6.2
Reporter: Hongshun Wang
 Fix For: 3.6.3


When I use KafkaProducer, OOM occurs but the KafkaProducer only log it but do 
noning:

```java

ERROR org.apache.kafka.common.utils.KafkaThread Uncaught exception in thread 
'kafka-producer-network-thread | producer-l': java.lang.Out0fMemoryError: 
Direct buffer memory

``` 

 

I try to find what happens:

1. It seems that OutOfMemoryError as a Error is not captured when 
org.apache.kafka.clients.producer.internals.Sender#run try to catch a 
Exception: 

```
@Override
public void run() {
log.debug("Starting Kafka producer I/O thread.");

// main loop, runs until close is called
while (running) {
try {
runOnce();
} catch (Exception e) {
log.error("Uncaught error in kafka producer I/O thread: ", e);
}
}

log.debug("Beginning shutdown of Kafka producer I/O thread, sending remaining 
records.");

// okay we stopped accepting requests but there may still be
// requests in the transaction manager, accumulator or waiting for 
acknowledgment,
// wait until these are completed.
while (!forceClose && ((this.accumulator.hasUndrained() ||    
this.client.inFlightRequestCount() > 0) || hasPendingTransactionalRequests())) {
try {
runOnce();
} catch (Exception e) {
log.error("Uncaught error in kafka producer I/O thread: ", e);
}
}

// Abort the transaction if any commit or abort didn't go through the 
transaction manager's queue
while (!forceClose && transactionManager != null && 
transactionManager.hasOngoingTransaction()) {
if (!transactionManager.isCompleting()) {
log.info("Aborting incomplete transaction due to shutdown");
transactionManager.beginAbort();
}
try {
runOnce();
} catch (Exception e) {
log.error("Uncaught error in kafka producer I/O thread: ", e);
}
}

if (forceClose) {
// We need to fail all the incomplete transactional requests and batches and 
wake up the threads waiting on
// the futures.
if (transactionManager != null) {
log.debug("Aborting incomplete transactional requests due to forced shutdown");
transactionManager.close();
}
log.debug("Aborting incomplete batches due to forced shutdown");
this.accumulator.abortIncompleteBatches();
}
try {
this.client.close();
} catch (Exception e) {
log.error("Failed to close network client", e);
}

log.debug("Shutdown of Kafka producer I/O thread has completed.");
}
```

2. Then KafkaThread catch uncaught exception and just log it:

```java
public KafkaThread(final String name, Runnable runnable, boolean daemon) {
super(runnable, name);
configureThread(name, daemon);
}

private void configureThread(final String name, boolean daemon) {
setDaemon(daemon);
setUncaughtExceptionHandler((t, e) -> log.error("Uncaught exception in thread 
'{}':", name, e));
}
```

 

To be honest, I don't understand why KafkaThread doing nothing but log it when 
an uncaught exception occurs? Why not exposing method to set 
setUncaughtExceptionHandler in KafkaThread or KafkaProducer so that user can 
determine what to do with uncaught exception, no matter thrown it or just 
ignore it?



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (KAFKA-16998) Fix warnings in our Github actions

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-16998.

Fix Version/s: 3.9.0
   Resolution: Fixed

> Fix warnings in our Github actions
> --
>
> Key: KAFKA-16998
> URL: https://issues.apache.org/jira/browse/KAFKA-16998
> Project: Kafka
>  Issue Type: Task
>  Components: build
>Reporter: Mickael Maison
>Assignee: Kuan Po Tseng
>Priority: Major
> Fix For: 3.9.0
>
>
> Most of our Github actions produce warnings, see 
> [https://github.com/apache/kafka/actions/runs/9572915509|https://github.com/apache/kafka/actions/runs/9572915509.]
>  for example.
> It looks like we need to bump the version we use for actions/checkout, 
> actions/setup-python, actions/upload-artifact to v4.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

回复: apply a jira account

[zhou zhengke]

Thanks for reply. I checked again, and get some useful info:

   email : madzh...@gmail.com
   Desired user name: zzzk1
   real (public) name: zhengke zhou


发件人: Matthias J. Sax 
发送时间: 2024年6月24日 05:43
收件人: dev@kafka.apache.org
主题: Re: apply a jira account

Thanks for you interest to contribute to Kafka. Unfortunately, I could
not find your account. Can you double check the Jira ID?

-Matthias

On 6/23/24 8:01 AM, 周 正轲 wrote:
> Hi Guys,
>
> I want to contribute to Apache Kafka.
> Would you please give me the permission as a contributor?
> My JIRA ID is zzzk1.

[jira] [Resolved] (KAFKA-16551) add integration test for ClusterTool

[Chia-Ping Tsai (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-16551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chia-Ping Tsai resolved KAFKA-16551.

Fix Version/s: 3.9.0
   Resolution: Fixed

> add integration test for ClusterTool
> 
>
> Key: KAFKA-16551
> URL: https://issues.apache.org/jira/browse/KAFKA-16551
> Project: Kafka
>  Issue Type: Test
>Reporter: Chia-Ping Tsai
>Assignee: PoAn Yang
>Priority: Minor
> Fix For: 3.9.0
>
>
> as title



--
This message was sent by Atlassian Jira
(v8.20.10#820010)