Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #1661

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 542257 lines...]
[2023-03-08T10:57:15.303Z] > Task :clients:compileTestJava UP-TO-DATE
[2023-03-08T10:57:15.303Z] > Task :clients:testClasses UP-TO-DATE
[2023-03-08T10:57:15.303Z] > Task 
:streams:generateMetadataFileForMavenJavaPublication
[2023-03-08T10:57:15.303Z] > Task :storage:api:compileTestJava UP-TO-DATE
[2023-03-08T10:57:15.303Z] > Task :storage:api:testClasses UP-TO-DATE
[2023-03-08T10:57:15.303Z] > Task :server-common:compileTestJava UP-TO-DATE
[2023-03-08T10:57:15.303Z] > Task :server-common:testClasses UP-TO-DATE
[2023-03-08T10:57:15.303Z] > Task :connect:json:compileTestJava UP-TO-DATE
[2023-03-08T10:57:15.303Z] > Task :connect:json:testClasses UP-TO-DATE
[2023-03-08T10:57:15.303Z] > Task :raft:compileTestJava UP-TO-DATE
[2023-03-08T10:57:15.303Z] > Task :raft:testClasses UP-TO-DATE
[2023-03-08T10:57:15.303Z] > Task :group-coordinator:compileTestJava UP-TO-DATE
[2023-03-08T10:57:15.303Z] > Task :group-coordinator:testClasses UP-TO-DATE
[2023-03-08T10:57:15.303Z] > Task :connect:json:testJar
[2023-03-08T10:57:15.303Z] > Task :connect:json:testSrcJar
[2023-03-08T10:57:15.303Z] > Task :metadata:compileTestJava UP-TO-DATE
[2023-03-08T10:57:15.303Z] > Task :metadata:testClasses UP-TO-DATE
[2023-03-08T10:57:15.303Z] > Task 
:clients:generateMetadataFileForMavenJavaPublication
[2023-03-08T10:57:19.496Z] 
[2023-03-08T10:57:19.496Z] > Task :connect:api:javadoc
[2023-03-08T10:57:19.496Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/connect/api/src/main/java/org/apache/kafka/connect/source/SourceRecord.java:44:
 warning - Tag @link: reference not found: org.apache.kafka.connect.data
[2023-03-08T10:57:21.589Z] 1 warning
[2023-03-08T10:57:21.589Z] 
[2023-03-08T10:57:21.589Z] > Task :connect:api:copyDependantLibs UP-TO-DATE
[2023-03-08T10:57:21.589Z] > Task :connect:api:jar UP-TO-DATE
[2023-03-08T10:57:21.589Z] > Task 
:connect:api:generateMetadataFileForMavenJavaPublication
[2023-03-08T10:57:21.589Z] > Task :connect:json:copyDependantLibs UP-TO-DATE
[2023-03-08T10:57:21.589Z] > Task :connect:json:jar UP-TO-DATE
[2023-03-08T10:57:21.589Z] > Task 
:connect:json:generateMetadataFileForMavenJavaPublication
[2023-03-08T10:57:21.589Z] > Task 
:connect:json:publishMavenJavaPublicationToMavenLocal
[2023-03-08T10:57:21.589Z] > Task :connect:json:publishToMavenLocal
[2023-03-08T10:57:21.589Z] > Task :connect:api:javadocJar
[2023-03-08T10:57:21.589Z] > Task :connect:api:compileTestJava UP-TO-DATE
[2023-03-08T10:57:21.589Z] > Task :connect:api:testClasses UP-TO-DATE
[2023-03-08T10:57:22.630Z] > Task :connect:api:testJar
[2023-03-08T10:57:22.630Z] > Task :connect:api:testSrcJar
[2023-03-08T10:57:22.630Z] > Task 
:connect:api:publishMavenJavaPublicationToMavenLocal
[2023-03-08T10:57:22.630Z] > Task :connect:api:publishToMavenLocal
[2023-03-08T10:57:24.886Z] > Task :streams:javadoc
[2023-03-08T10:57:24.886Z] > Task :streams:javadocJar
[2023-03-08T10:57:28.563Z] 
[2023-03-08T10:57:28.563Z] > Task :clients:javadoc
[2023-03-08T10:57:28.563Z] 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/clients/src/main/java/org/apache/kafka/common/security/oauthbearer/secured/package-info.java:21:
 warning - Tag @link: reference not found: 
org.apache.kafka.common.security.oauthbearer
[2023-03-08T10:57:29.600Z] 1 warning
[2023-03-08T10:57:30.803Z] 
[2023-03-08T10:57:30.803Z] > Task :clients:javadocJar
[2023-03-08T10:57:32.011Z] > Task :clients:srcJar
[2023-03-08T10:57:33.147Z] > Task :clients:testJar
[2023-03-08T10:57:33.147Z] > Task :clients:testSrcJar
[2023-03-08T10:57:33.147Z] > Task 
:clients:publishMavenJavaPublicationToMavenLocal
[2023-03-08T10:57:33.147Z] > Task :clients:publishToMavenLocal
[2023-03-08T10:57:45.545Z] > Task :core:compileScala
[2023-03-08T10:59:39.652Z] > Task :core:classes
[2023-03-08T10:59:39.652Z] > Task :core:compileTestJava NO-SOURCE
[2023-03-08T11:00:01.519Z] > Task :core:compileTestScala
[2023-03-08T11:01:44.156Z] > Task :core:testClasses
[2023-03-08T11:01:44.156Z] > Task :streams:compileTestJava UP-TO-DATE
[2023-03-08T11:01:44.156Z] > Task :streams:testClasses UP-TO-DATE
[2023-03-08T11:01:44.156Z] > Task :streams:testJar
[2023-03-08T11:01:44.156Z] > Task :streams:testSrcJar
[2023-03-08T11:01:44.156Z] > Task 
:streams:publishMavenJavaPublicationToMavenLocal
[2023-03-08T11:01:44.156Z] > Task :streams:publishToMavenLocal
[2023-03-08T11:01:44.156Z] 
[2023-03-08T11:01:44.156Z] Deprecated Gradle features were used in this build, 
making it incompatible with Gradle 9.0.
[2023-03-08T11:01:44.156Z] 
[2023-03-08T11:01:44.156Z] You can use '--warning-mode all' to show the 
individual deprecation warnings and determine if they come from your own 
scripts or plugins.
[2023-03-08T11:01:44.156Z] 
[2023-03-08T11:01:44.157Z] See 
https://docs.gradle.org/8.0.2/userguide/command_line_interface.html#sec:command_line_warnings
[2023-03-08T11:01:44.157Z

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #1662

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-882: Make Kafka Connect REST API request timeouts configurable

[Yash Mayya]

Hi Greg,

Thanks for the response!

1. Hm that's a fair point, and while there aren't any strict guidelines or
rules governing whether something should be a query parameter versus a
request header, I agree that it might be more idiomatic for a request
timeout value to be accepted via a custom request header. What do you
think about using a header name like "X-Request-Timeout", or maybe simply
"Request-Timeout" if we want to take into account
https://www.rfc-editor.org/rfc/rfc6648?

2. 3. 4. Adding asynchronous validations via new endpoints / new request
parameters on existing endpoints is definitely an interesting idea but I'm
not sure whether this KIP should take into consideration a hypothetical
future proposal? If there were already such a concrete proposal, I would
definitely agree that this KIP should take that into consideration - but
that isn't the case today. Furthermore, I'm not sure I understand why the
addition of request timeout configurability to the existing endpoints would
preclude the introduction of an asynchronous validation API in the future?

Thanks,
Yash

On Sat, Mar 4, 2023 at 1:13 AM Greg Harris 
wrote:

> Yash,
>
> 1.
> Currently the request parameters in the REST API are individual and pertain
> to just one endpoint.
> They also change the content of the query result, or change the action
> taken on the cluster.
> I think that a request timeout is a property of the HTTP request more than
> it is a property of the cluster query or cluster action.
> The two solutions have very similar tradeoffs, but I'm interested in
> whether one is more idiomatic and more obvious to users.
>
> 2.
> I understand that only these three endpoints are in need of increased
> timeouts at this time due to long connector validations.
> From another perspective, this change is making the API more irregular and
> someone in the future might choose to make it more regular by standardizing
> the configurable timeout functionality.
> I wouldn't (in this KIP) dismiss someone's desire to configure other
> timeouts in the future (in another KIP), and design them into a corner.
> It is acceptable to limit the scope of this change to just the three
> endpoints due to practical reasons, but I don't think that should prevent
> us from trying to ensure that this design fits in the "end goal" state of
> the Connect service.
>
> 3. 4.
> I am not suggesting an incompatible change, as the current synchronous
> behavior is still a useful API for certain situations. I think that it is
> possible to add asynchronous validations in a backwards compatible way,
> using new endpoints or other new request parameters.
> The interface could be designed such that users with connectors that exceed
> the synchronous timeouts can utilize the asynchronous API. Tooling can use
> the asynchronous API when it is available, and fall back to the synchronous
> API when it is not.
> I think that it also may be more in-line with the design of the rest of the
> REST API, where nearly every other request is asynchronous. That's why
> you're only targeting these three endpoints, they're the only ones with a
> synchronicity constraint.
> Again, I'm not necessarily saying that you must implement such an
> asynchronous validation scheme in this KIP, but we should consider if that
> is a more extensible solution. If we decided to implement configurable
> synchronous timeouts now, how would that complement an asynchronous API in
> the future?
>
> On Thu, Mar 2, 2023 at 10:00 PM Yash Mayya  wrote:
>
> > Hi Greg,
> >
> > Thanks for taking a look!
> >
> > 1. I believe Chris suggested the use of a query parameter above as we
> > already have precedents for using query parameters to configure per
> request
> > behavior in Kafka Connect (the restart connectors API and the get
> > connectors API for instance). Also, the limited choice of endpoints
> > targeted is intentional (see my reply to the next point).
> >
> > 2. I intentionally targeted just the three listed endpoints where
> > synchronous connector config validations come into the picture. This is
> > because of the legitimate cases where config validation for specific
> > connector plugins might exceed the default request timeout in edge case
> > scenarios (outlined in the KIP's motivation section). Other Connect REST
> > endpoints shouldn't be taking longer than the default 90 second request
> > timeout; if they do so, it would either be indicative of a bug in the
> > Connect framework or a cluster health issue - neither of which should be
> > covered up by manually setting a longer request timeout.
> >
> > 3. 4. I think changing the config validation behavior would be a backward
> > incompatible change and I wanted to avoid that in this particular KIP.
> > There are multiple programmatic UIs out there which rely on the current
> > synchronous config validation behavior and breaking the existing contract
> > would definitely require a larger discussion.
> >
> > Thanks,
> > Yash
> >
> > On Fri, Mar 3, 

[jira] [Resolved] (KAFKA-14781) MM2 logs misleading error during topic ACL sync when broker does not have authorizer configured

[Chris Egerton (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chris Egerton resolved KAFKA-14781.
---
Fix Version/s: 3.5.0
   Resolution: Fixed

> MM2 logs misleading error during topic ACL sync when broker does not have 
> authorizer configured
> ---
>
> Key: KAFKA-14781
> URL: https://issues.apache.org/jira/browse/KAFKA-14781
> Project: Kafka
>  Issue Type: Bug
>  Components: mirrormaker
>Reporter: Chris Egerton
>Assignee: Chris Egerton
>Priority: Major
> Fix For: 3.5.0
>
>
> When there is no broker-side authorizer configured on a Kafka cluster 
> targeted by MirrorMaker 2, users see error-level log messages like this 
> one:{{{}{}}}
> {quote}[2023-03-06 10:53:57,488] ERROR [MirrorSourceConnector|worker] 
> Scheduler for MirrorSourceConnector caught exception in scheduled task: 
> syncing topic ACLs (org.apache.kafka.connect.mirror.Scheduler:102)
> java.util.concurrent.ExecutionException: 
> org.apache.kafka.common.errors.SecurityDisabledException: No Authorizer is 
> configured on the broker
>     at 
> java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395)
>     at 
> java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1999)
>     at 
> org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:165)
>     at 
> org.apache.kafka.connect.mirror.MirrorSourceConnector.listTopicAclBindings(MirrorSourceConnector.java:456)
>     at 
> org.apache.kafka.connect.mirror.MirrorSourceConnector.syncTopicAcls(MirrorSourceConnector.java:342)
>     at org.apache.kafka.connect.mirror.Scheduler.run(Scheduler.java:93)
>     at 
> org.apache.kafka.connect.mirror.Scheduler.executeThread(Scheduler.java:112)
>     at 
> org.apache.kafka.connect.mirror.Scheduler.lambda$scheduleRepeating$0(Scheduler.java:50)
>     at 
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
>     at 
> java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
>     at 
> java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
>     at 
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>     at 
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>     at java.base/java.lang.Thread.run(Thread.java:829)
> Caused by: org.apache.kafka.common.errors.SecurityDisabledException: No 
> Authorizer is configured on the broker
> {quote}
> This can be misleading as it looks like something is wrong with MM2 or the 
> Kafka cluster. In reality, it's usually fine, since topic ACL syncing is 
> enabled by default and it's reasonable for Kafka clusters (especially in 
> testing/dev environments) to not have authorizers enabled.
> We should try to catch this specific case and downgrade the severity of the 
> log message from {{ERROR}} to either {{INFO}} or {{{}DEBUG{}}}. We may also 
> consider suggesting to users that they disable topic ACL syncing if their 
> Kafka cluster doesn't have authorization set up, but this should probably 
> only be emitted once over the lifetime of the connector in order to avoid 
> generating log spam.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [Discuss] KIP-581: Value of optional null field which has default value

[Mickael Maison]

Hi,

This KIP has been staled for a long time. Since it would be a useful
feature, I pinged Cheng about a month ago asking if he was planning to
work on it again. I've not received a reply, so I've allowed myself to
update the KIP (hopefully preserving the initial requirements) and
would like to restart a discussion.

The DISCUSS thread was split in two, you can find the other part in
https://lists.apache.org/thread/dc56k17zptzvbyc7gtscovzgzwf6yn9p

Let me know if you have any feedback.

Thanks,
Mickael

On Tue, Apr 14, 2020 at 8:28 PM Christopher Egerton  wrote:
>
> Hi Cheng,
>
> Thanks for the KIP! I really appreciate the care that was taken to ensure
> backwards compatibility for existing users, and the minimal changes to
> public interface that are suggested to address this.
>
> I have two quick requests for clarification:
>
> 1) Where is the proposed "accept.optional.null" property going to apply?
> It's hinted that it'll take effect on the JSON converter but not actually
> called out anywhere.
>
> 2) Assuming this takes effect on the JSON converter, is the intent to alter
> the semantics for both serialization and deserialization? The code snippet
> from the JSON converter that's included in the KIP comes from the
> "convertToJson" method, which is used for serialization. However, based on
> https://github.com/apache/kafka/blob/ea47a885b1fe47dfb87c1dc86db1b0e7eb8a273c/connect/json/src/main/java/org/apache/kafka/connect/json/JsonConverter.java#L712-L713
> it
> looks like the converter also inserts the default value for
> optional-but-null data during deserialization.
>
> Thanks again for the KIP!
>
> Cheers,
>
> Chris
>
> On Wed, Mar 18, 2020 at 12:00 AM Cheng Pan <379377...@qq.com> wrote:
>
> > Hi all,
> >
> > I'd like to use this thread to discuss KIP-581: Value of optional null
> > field which has default value, please see detail at:
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-581%3A+Value+of+optional+null+field+which+has+default+value
> >
> >
> > There are some previous discussion at:
> > https://github.com/apache/kafka/pull/7112
> >
> >
> > I'm a beginner for apache project, please let me know if I did any thing
> > wrong.
> >
> >
> > Best regards,
> > Cheng Pan

[jira] [Created] (KAFKA-14793) Propagate topic ids to the group coordinator

[Alexandre Dupriez (Jira)]

Alexandre Dupriez created KAFKA-14793:
-

 Summary: Propagate topic ids to the group coordinator
 Key: KAFKA-14793
 URL: https://issues.apache.org/jira/browse/KAFKA-14793
 Project: Kafka
  Issue Type: Sub-task
Reporter: Alexandre Dupriez


KAFKA-14690 introduces topic ids in the OffsetCommit API in the request layer. 
Propagation of topic ids within the group coordinator has been left out of 
scope. Whether topic ids are re-mapped internally in the group coordinator or 
the group coordinator starts to rely on {{{}TopicIdPartition{}}}.

Note that with KAFKA-14690, the offset commit response data built by the 
coordinator includes topic names only, and topic ids need to be injected 
afterwards outside of the coordinator before serializing the response.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14794) Unable to deserialize base64 JSON strings

[Jira]

José Armando García Sancio created KAFKA-14794:
--

 Summary: Unable to deserialize base64 JSON strings 
 Key: KAFKA-14794
 URL: https://issues.apache.org/jira/browse/KAFKA-14794
 Project: Kafka
  Issue Type: Bug
Reporter: José Armando García Sancio
Assignee: José Armando García Sancio


h1. Problem

The following test fails:
{code:java}
@Test
public void testBinaryNode() throws IOException {
    byte[] expected = new byte[] {5, 2, 9, 4, 1, 8, 7, 0, 3, 6};
    StringWriter writer = new StringWriter();
    ObjectMapper mapper = new ObjectMapper();

    mapper.writeTree(mapper.createGenerator(writer), new BinaryNode(expected));

    JsonNode binaryNode = mapper.readTree(writer.toString());

    assertTrue(binaryNode.isTextual(), binaryNode.toString());
    byte[] actual = MessageUtil.jsonNodeToBinary(binaryNode, "Test base64 JSON 
string");
    assertEquals(expected, actual);
}
{code}
with the following error:
{code:java}
 Gradle Test Run :clients:test > Gradle Test Executor 20 > MessageUtilTest > 
testBinaryNode() FAILED
    java.lang.RuntimeException: Test base64 JSON string: expected 
Base64-encoded binary data.
        at 
org.apache.kafka.common.protocol.MessageUtil.jsonNodeToBinary(MessageUtil.java:165)
        at 
org.apache.kafka.common.protocol.MessageUtilTest.testBinaryNode(MessageUtilTest.java:102)
{code}
The reason for the failure is because FasterXML Jackson deserializes base64 
JSON strings to a TextNode not to a BinaryNode.
h1. Solution

The method {{MessageUtil::jsonNodeToBinary}} should not assume that the input 
{{JsonNode}} is always a {{{}BinaryNode{}}}. It should also support decoding 
{{{}TextNode{}}}.

{{JsonNode::binaryValue}} is supported by both {{BinaryNode}} and 
{{{}TextNode{}}}.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14795) Provide message formatter for RemoteLogMetadata

[Ivan Yurchenko (Jira)]

Ivan Yurchenko created KAFKA-14795:
--

 Summary: Provide message formatter for RemoteLogMetadata
 Key: KAFKA-14795
 URL: https://issues.apache.org/jira/browse/KAFKA-14795
 Project: Kafka
  Issue Type: Improvement
  Components: tools
Reporter: Ivan Yurchenko
Assignee: Ivan Yurchenko


For the debugging purpose, it'd be convenient to have a message formatter that 
can display the content of the {{__remote_log_metadata}} topic.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #1663

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-911: Add source tag to MirrorSourceConnector metrics

[Chris Egerton]

Hi Mickael,

Thanks for the KIP! LGTM.

I find the rationales for the rejected alternatives convincing, and agree
with the deprecation plan and opt-in behavior for pre-4.0 releases. I also
appreciate the historical context provided in the motivation section about
why we don't already use a tag for the source cluster in MM2.

One small note: the "add.source.alias.to.metrics" property has an update
mode listed in the KIP. Is this necessary? AFAIK that concept only applies
to broker configs. Shouldn't block the KIP either way since "Read only"
doesn't make any promises it can't keep, mostly asking for my own
edification.

Cheers,

Chris

On Tue, Mar 7, 2023 at 10:15 AM Mickael Maison 
wrote:

> Hi,
>
> I created a KIP to tag the MirrorSourceConnector metrics with the
> source cluster alias. Currently they only have the target cluster
> alias.
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-911%3A+Add+source+tag+to+MirrorSourceConnector+metrics
>
> Please take a look and let me know if you have any feedback.
>
> Thanks,
> Mickael
>

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.4 #94

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-908: Add description field to connector configuration

[Chris Egerton]

Hi Mickael,

I do sympathize with the desire for a "quick fix". I think your point about
these being problematic to support sums up my hesitation here pretty well,
both with respect to the potential footgun of unintended rebalances (should
users try to do more with this field than we expect), and with making other
similar improvements (i.e., expanded metadata support in Kafka Connect)
more difficult to design correctly for us and to use effectively for users
(especially if we were to deprecate/remove the description field at a later
date).

It's also worth noting that users can already add a "description" field to
the JSON config for any connector they create; the benefit provided by this
KIP is that they're automatically prompted to do so if they're using a
UI/CLI/etc. that builds on top of the various /connector-plugins endpoints
to find the set of configuration properties that a user can/should populate
before creating a connector.

I'd welcome thoughts from you and others on whether the tradeoffs here are
worth it; right now I'm still not sure about this one.

Cheers,

Chris

On Tue, Mar 7, 2023 at 6:42 AM Mickael Maison 
wrote:

> H Chris,
>
> Thanks for taking a look.
>
> 1. Yes updating the description can potentially trigger a rebalance. I
> don't expect users to frequently update the description so I thought
> this was acceptable. I've added a note to the KIP to mention it.
>
> 2. The tags model you described could be interesting but it looks
> pretty involved with multiple new endpoints and brand new concepts.
>
> With this KIP, I really took the most basic approach and proposed the
> simplest set of changes that could get in the next release and, I
> think, immediately bring benefits. However sometimes this is not the
> best approach as a "quick fix" could end up problematic to support in
> the future. The drawbacks (new connector config field + causing a
> rebalance on update) look relatively benign to me so I thought this
> could be an acceptable proposal.
>
> Thanks,
> Mickael
>
>
>
> On Thu, Feb 23, 2023 at 2:45 PM Chris Egerton 
> wrote:
> >
> > Actually, I misspoke--a rebalance isn't triggered when an existing
> > connector's config is updated. Assuming the set of workers remains
> stable,
> > a rebalance is only necessary when a new connector is created, an
> existing
> > one is deleted, or a new set of task configs is generated.
> >
> > This weakens the point about unnecessary rebalances when a connector's
> > description is updated, but doesn't entirely address it. Spurious
> > rebalances may still be triggered if a new set of task configs is
> > generated, which for reasons outlined above, is fairly likely.
> >
> > On Thu, Feb 23, 2023 at 7:41 AM Chris Egerton  wrote:
> >
> > > Hi Mickael,
> > >
> > > Thanks for the KIP!
> > >
> > > While it's tempting to add this field to the out-of-the-box connector
> > > config def, I'm a little hesitant, for two reasons.
> > >
> > > 1. Adding this directly to the connector config could have unintended
> > > consequences on the actual data processing by the connector. Any time a
> > > connector's config is modified, the Connector object running for it is
> > > restarted with that new config. In most cases this is a trivial
> operation
> > > since we have incremental rebalancing enabled by default, the
> connector can
> > > (and probably should!) generate task configs that are functionally
> > > identical to the ones it last generated, and most (though not all)
> > > Connector classes are fairly lightweight and leave the real work to
> their
> > > Task class. However, due to KAFKA-9228 [1], it's not just common
> practice
> > > for connectors to perform transparent passthrough of most of their
> configs
> > > when generating task configs, it's actually necessary to work around a
> bug
> > > in the runtime. As a result, tweaking the description of a connector
> would
> > > be fairly likely to result in a full restart of all of its tasks, in
> > > addition to triggering two rebalances (which may not be so lightweight
> if
> > > users are still running with eager rebalancing... which, sadly, I've
> heard
> > > is still happening today).
> > >
> > > 2. The motivation section mentions some information that might go into
> the
> > > description field, such as the team that owns the connector and
> emergency
> > > contact info. It seems like this info might benefit from a little more
> > > structure if we're trying to design for programmatic access by GUIs and
> > > CLIs (which I'm assuming is the case, since I can't imagine a human
> being
> > > getting much use out of the raw output from the GET
> > > /connector-plugins/{name}/config and PUT
> > > /connector-plugins/{name}/config/validate endpoints). This might also
> make
> > > it easier to add custom validation logic around what kind of
> information is
> > > present via REST extension.
> > >
> > >
> > > With these thoughts in mind, what do you think about adding a new
> generic
> > > "tags" object to c

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #1664

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 540947 lines...]
[2023-03-08T21:07:23.511Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 178 > ZkMigrationClientTest > testCreateNewPartitions() STARTED
[2023-03-08T21:07:24.452Z] 
[2023-03-08T21:07:24.452Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 178 > ZkMigrationClientTest > testCreateNewPartitions() PASSED
[2023-03-08T21:07:24.452Z] 
[2023-03-08T21:07:24.452Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 178 > ZkMigrationClientTest > testIdempotentCreateTopics() STARTED
[2023-03-08T21:07:24.452Z] 
[2023-03-08T21:07:24.452Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 178 > ZkMigrationClientTest > testIdempotentCreateTopics() PASSED
[2023-03-08T21:07:24.452Z] 
[2023-03-08T21:07:24.452Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 178 > ZkMigrationClientTest > testWriteExistingClientQuotas() STARTED
[2023-03-08T21:07:24.452Z] 
[2023-03-08T21:07:24.452Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 178 > ZkMigrationClientTest > testWriteExistingClientQuotas() PASSED
[2023-03-08T21:07:24.452Z] 
[2023-03-08T21:07:24.452Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 178 > AllocateProducerIdsRequestTest > 
testAllocateProducersIdSentToController() > 
unit.kafka.server.AllocateProducerIdsRequestTest.testAllocateProducersIdSentToController()[1]
 STARTED
[2023-03-08T21:07:26.213Z] 
[2023-03-08T21:07:26.213Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 178 > AllocateProducerIdsRequestTest > 
testAllocateProducersIdSentToController() > 
unit.kafka.server.AllocateProducerIdsRequestTest.testAllocateProducersIdSentToController()[1]
 PASSED
[2023-03-08T21:07:26.213Z] 
[2023-03-08T21:07:26.213Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 178 > AllocateProducerIdsRequestTest > 
testAllocateProducersIdSentToNonController() > 
unit.kafka.server.AllocateProducerIdsRequestTest.testAllocateProducersIdSentToNonController()[1]
 STARTED
[2023-03-08T21:07:33.249Z] 
[2023-03-08T21:07:33.250Z] Gradle Test Run :core:integrationTest > Gradle Test 
Executor 178 > AllocateProducerIdsRequestTest > 
testAllocateProducersIdSentToNonController() > 
unit.kafka.server.AllocateProducerIdsRequestTest.testAllocateProducersIdSentToNonController()[1]
 PASSED
[2023-03-08T21:07:34.550Z] > Task :core:compileScala
[2023-03-08T21:07:36.840Z] 
[2023-03-08T21:07:36.841Z] Deprecated Gradle features were used in this build, 
making it incompatible with Gradle 9.0.
[2023-03-08T21:07:36.841Z] 
[2023-03-08T21:07:36.841Z] You can use '--warning-mode all' to show the 
individual deprecation warnings and determine if they come from your own 
scripts or plugins.
[2023-03-08T21:07:36.841Z] 
[2023-03-08T21:07:36.841Z] See 
https://docs.gradle.org/8.0.2/userguide/command_line_interface.html#sec:command_line_warnings
[2023-03-08T21:07:36.841Z] 
[2023-03-08T21:07:36.841Z] BUILD SUCCESSFUL in 2h 38m 15s
[2023-03-08T21:07:36.841Z] 224 actionable tasks: 120 executed, 104 up-to-date
[2023-03-08T21:07:36.841Z] 
[2023-03-08T21:07:36.841Z] See the profiling report at: 
file:///home/jenkins/workspace/Kafka_kafka_trunk_2/build/reports/profile/profile-2023-03-08-18-29-26.html
[2023-03-08T21:07:36.841Z] A fine-grained performance profile is available: use 
the --scan option.
[Pipeline] junit
[2023-03-08T21:07:37.874Z] Recording test results
[2023-03-08T21:07:49.384Z] [Checks API] No suitable checks publisher found.
[Pipeline] echo
[2023-03-08T21:07:49.385Z] Skipping Kafka Streams archetype test for Java 11
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // timestamps
[Pipeline] }
[Pipeline] // timeout
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[2023-03-08T21:09:29.782Z] > Task :core:classes
[2023-03-08T21:09:29.782Z] > Task :core:compileTestJava NO-SOURCE
[2023-03-08T21:09:47.872Z] > Task :core:compileTestScala
[2023-03-08T21:11:27.042Z] > Task :core:testClasses
[2023-03-08T21:11:27.042Z] > Task :streams:compileTestJava UP-TO-DATE
[2023-03-08T21:11:27.042Z] > Task :streams:testClasses UP-TO-DATE
[2023-03-08T21:11:27.042Z] > Task :streams:testJar
[2023-03-08T21:11:27.042Z] > Task :streams:testSrcJar
[2023-03-08T21:11:27.042Z] > Task 
:streams:publishMavenJavaPublicationToMavenLocal
[2023-03-08T21:11:27.042Z] > Task :streams:publishToMavenLocal
[2023-03-08T21:11:27.042Z] 
[2023-03-08T21:11:27.042Z] Deprecated Gradle features were used in this build, 
making it incompatible with Gradle 9.0.
[2023-03-08T21:11:27.042Z] 
[2023-03-08T21:11:27.042Z] You can use '--warning-mode all' to show the 
individual deprecation warnings and determine if they come from your own 
scripts or plugins.
[2023-03-08T21:11:27.042Z] 
[2023-03-08T21:11:27.042Z] See 
https://doc

Re: [DISCUSS] KIP-911: Add source tag to MirrorSourceConnector metrics

[Mickael Maison]

Hi Chris,

Thanks for taking a look at the KIP.
I've removed the mention to the update mode. It was copied from
somewhere else and I forgot to delete it.

Thanks,
Mickael

On Wed, Mar 8, 2023 at 7:26 PM Chris Egerton  wrote:
>
> Hi Mickael,
>
> Thanks for the KIP! LGTM.
>
> I find the rationales for the rejected alternatives convincing, and agree
> with the deprecation plan and opt-in behavior for pre-4.0 releases. I also
> appreciate the historical context provided in the motivation section about
> why we don't already use a tag for the source cluster in MM2.
>
> One small note: the "add.source.alias.to.metrics" property has an update
> mode listed in the KIP. Is this necessary? AFAIK that concept only applies
> to broker configs. Shouldn't block the KIP either way since "Read only"
> doesn't make any promises it can't keep, mostly asking for my own
> edification.
>
> Cheers,
>
> Chris
>
> On Tue, Mar 7, 2023 at 10:15 AM Mickael Maison 
> wrote:
>
> > Hi,
> >
> > I created a KIP to tag the MirrorSourceConnector metrics with the
> > source cluster alias. Currently they only have the target cluster
> > alias.
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-911%3A+Add+source+tag+to+MirrorSourceConnector+metrics
> >
> > Please take a look and let me know if you have any feedback.
> >
> > Thanks,
> > Mickael
> >

[jira] [Created] (KAFKA-14796) Migrate ZK ACLs to KRaft

[David Arthur (Jira)]

David Arthur created KAFKA-14796:


 Summary: Migrate ZK ACLs to KRaft
 Key: KAFKA-14796
 URL: https://issues.apache.org/jira/browse/KAFKA-14796
 Project: Kafka
  Issue Type: Sub-task
  Components: kraft
Reporter: David Arthur
Assignee: David Arthur
 Fix For: 3.5.0, 3.4.1






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14797) MM2 does not emit offset syncs when conservative translation logic exceeds positive max.offset.lag

[Greg Harris (Jira)]

Greg Harris created KAFKA-14797:
---

 Summary: MM2 does not emit offset syncs when conservative 
translation logic exceeds positive max.offset.lag
 Key: KAFKA-14797
 URL: https://issues.apache.org/jira/browse/KAFKA-14797
 Project: Kafka
  Issue Type: Bug
  Components: mirrormaker
Reporter: Greg Harris
Assignee: Greg Harris


This is a regression in MirrorMaker 2 introduced by KAFKA-12468.

Reproduction steps:
1. Set max.offset.lag to a non-zero value.
2. Set up a 1-1 replication flow which does not skip upstream offsets or have a 
concurrent producer to the target topic.
3. Produce more than max.offset.lag records to the source topic and allow 
replication to proceed.
4. Examine end offsets, checkpoints and/or target consumer group lag

Expected behavior:
Consumer group lag should be at most max.offset.lag.

Actual behavior:
Consumer group lag is significantly larger than max.offset.lag.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

max.poll.interval.ms rebalance override

[Mcs Vemuri]

Hello,
Is there a way to override a rebalance caused by max.poll.interval.ms? We have 
a case where there are multiple consumers in the same group- but some of them 
might start sooner than the rest and all consumers can be lazy pollers 
The issue is that we need to set the poll interval to a large value as the 
consumer may not always need to poll- instead, they only poll when they receive 
some other stimulus. Because of this, when the other consumer joins the group, 
rebalance takes very long 
Ideally, the rebalance timeout would have its own config(maybe 
max.rebalance.ms?) which defaults to max.poll.interval.ms but not sure if that 
was already considered and rejected due to some other reason

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #1665

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-913: add new method to provide possibility for accelerate first record's sending

[jian fu]

Hi All:

If anyone can help to give some comments or suggestions for the proposal.
Thanks in advance!

Regards
Jian

jian fu  于2023年3月6日周一 17:00写道：

> Hi Everyone:
> Nice to meet you.
>
> I created one KIP to request your review.
> KIP-913: add new method to provide possibility for accelerate first
> record's sending
> 
> The example PR:
> *https://github.com/apache/kafka/pull/13320/files
> *
>
> Thanks.
>
> Regards
> Jian
>
>

-- 
Regards  Fu.Jian
--
Cisco Communications, Inc.

Re: [DISCUSS] KIP-912: Support decreasing send's block time without worrying about metadata's fetch

[jian fu]

Hi All:

If anyone can help to give some comments or suggestions for the proposal.
Thanks in advance!

Regards
Jian

jian fu  于2023年3月6日周一 15:55写道：

> Hi Everyone:
> Nice to meet you.
>
> I created one KIP to request your review.
> KIP-912: Support decreasing send's block time without worrying about
> metadata's fetch
> 
> The example PR:
> https://github.com/apache/kafka/pull/13335/files
>
> Thanks.
>
> --
> Regards
> Jian
>
>
>

-- 
Regards  Fu.Jian
--
Cisco Communications, Inc.

Re: [DISCUSS] KIP-913: add new method to provide possibility for accelerate first record's sending

[Haruki Okada]

You can just use Producer#partitionsFor

2023年3月9日(木) 11:13 jian fu :

> Hi All:
>
> If anyone can help to give some comments or suggestions for the proposal.
> Thanks in advance!
>
> Regards
> Jian
>
> jian fu  于2023年3月6日周一 17:00写道：
>
> > Hi Everyone:
> > Nice to meet you.
> >
> > I created one KIP to request your review.
> > KIP-913: add new method to provide possibility for accelerate first
> > record's sending
> > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-913%3A+add+new+method+to+provide+possibility+for+accelerate+first+record%27s+sending
> >
> > The example PR:
> > *https://github.com/apache/kafka/pull/13320/files
> > *
> >
> > Thanks.
> >
> > Regards
> > Jian
> >
> >
>
> --
> Regards  Fu.Jian
> --
> Cisco Communications, Inc.
>


-- 

Okada Haruki
ocadar...@gmail.com

Re: max.poll.interval.ms rebalance override

[Mcs Vemuri]

I overlooked the pause resume mechanism- it should work for us I think 


On Wednesday, March 8, 2023, 4:52 PM, Mcs Vemuri  wrote:

Hello,
Is there a way to override a rebalance caused by max.poll.interval.ms? We have 
a case where there are multiple consumers in the same group- but some of them 
might start sooner than the rest and all consumers can be lazy pollers 
The issue is that we need to set the poll interval to a large value as the 
consumer may not always need to poll- instead, they only poll when they receive 
some other stimulus. Because of this, when the other consumer joins the group, 
rebalance takes very long 
Ideally, the rebalance timeout would have its own config(maybe 
max.rebalance.ms?) which defaults to max.poll.interval.ms but not sure if that 
was already considered and rejected due to some other reason

Re: [DISCUSS] KIP-913: add new method to provide possibility for accelerate first record's sending

[jian fu]

Hi Okada Haruki:

Thanks for your comment.

I can use it partitionsFor(topic) for the goal, thus there are two reasons
why I don't choose it and propose to add new dedicated method:
1) Consider how to use the method to solve the issue, We should call it in
application's startup process before any record sent. Thus, The code will
look strange to call partitionsFor(topic) . So I suggest to add one common
method such as getCluster so that you can get and print some useful
information when startup with the goal reached. It also can provide more
information self compare with partitionsFor.
2) partitionsFor(topic) using the maxBlockTimeMs as the max blocking time.
For the metadata's fetching, I will take a lot of time so that we must
set maxBlockTimeMs
to a big value (at least > time for metadata). Thus consider that the send
method is async. Most of application like to reduce the maxBlockTimeMs. It
is conflict time requirement. So we need one new blocking time as parameter
of the method. I don't want to change the existing interface.
So based on above reasons. I suggest to add new method with time control
parameter.
WDTY?

Thanks.


Haruki Okada  于2023年3月9日周四 10:19写道：

> You can just use Producer#partitionsFor
>
> 2023年3月9日(木) 11:13 jian fu :
>
> > Hi All:
> >
> > If anyone can help to give some comments or suggestions for the proposal.
> > Thanks in advance!
> >
> > Regards
> > Jian
> >
> > jian fu  于2023年3月6日周一 17:00写道：
> >
> > > Hi Everyone:
> > > Nice to meet you.
> > >
> > > I created one KIP to request your review.
> > > KIP-913: add new method to provide possibility for accelerate first
> > > record's sending
> > > <
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-913%3A+add+new+method+to+provide+possibility+for+accelerate+first+record%27s+sending
> > >
> > > The example PR:
> > > *https://github.com/apache/kafka/pull/13320/files
> > > *
> > >
> > > Thanks.
> > >
> > > Regards
> > > Jian
> > >
> > >
> >
> > --
> > Regards  Fu.Jian
> > --
> > Cisco Communications, Inc.
> >
>
>
> --
> 
> Okada Haruki
> ocadar...@gmail.com
> 
>


-- 
Regards  Fu.Jian
--
Cisco Communications, Inc.

Re: [DISCUSS] KIP-913: add new method to provide possibility for accelerate first record's sending

[jian fu]

Hi Haruki Okada:

There is another KIP912 discussion related to this one. Welcome to give
some comments/suggestions.

Thanks.

I think if the 912 is done. New method getCluster can use the new config
directly with time parameter removed.

WDTY?

[DISCUSS] KIP-912: Support decreasing send's block time without worrying
about metadata's fetch-Apache Mail Archives


jian fu  于2023年3月9日周四 11:11写道：

> Hi Okada Haruki:
>
> Thanks for your comment.
>
> I can use it partitionsFor(topic) for the goal, thus there are two reasons
> why I don't choose it and propose to add new dedicated method:
> 1) Consider how to use the method to solve the issue, We should call it in
> application's startup process before any record sent. Thus, The code will
> look strange to call partitionsFor(topic) . So I suggest to add one
> common method such as getCluster so that you can get and print some useful
> information when startup with the goal reached. It also can provide more
> information self compare with partitionsFor.
> 2) partitionsFor(topic) using the maxBlockTimeMs as the max blocking
> time. For the metadata's fetching, I will take a lot of time so that we
> must set maxBlockTimeMs to a big value (at least > time for metadata).
> Thus consider that the send method is async. Most of application like to
> reduce the maxBlockTimeMs. It is conflict time requirement. So we need
> one new blocking time as parameter of the method. I don't want to change
> the existing interface.
> So based on above reasons. I suggest to add new method with time control
> parameter.
> WDTY?
>
> Thanks.
>
>
> Haruki Okada  于2023年3月9日周四 10:19写道：
>
>> You can just use Producer#partitionsFor
>>
>> 2023年3月9日(木) 11:13 jian fu :
>>
>> > Hi All:
>> >
>> > If anyone can help to give some comments or suggestions for the
>> proposal.
>> > Thanks in advance!
>> >
>> > Regards
>> > Jian
>> >
>> > jian fu  于2023年3月6日周一 17:00写道：
>> >
>> > > Hi Everyone:
>> > > Nice to meet you.
>> > >
>> > > I created one KIP to request your review.
>> > > KIP-913: add new method to provide possibility for accelerate first
>> > > record's sending
>> > > <
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-913%3A+add+new+method+to+provide+possibility+for+accelerate+first+record%27s+sending
>> > >
>> > > The example PR:
>> > > *https://github.com/apache/kafka/pull/13320/files
>> > > *
>> > >
>> > > Thanks.
>> > >
>> > > Regards
>> > > Jian
>> > >
>> > >
>> >
>> > --
>> > Regards  Fu.Jian
>> > --
>> > Cisco Communications, Inc.
>> >
>>
>>
>> --
>> 
>> Okada Haruki
>> ocadar...@gmail.com
>> 
>>
>
>
> --
> Regards  Fu.Jian
> --
> Cisco Communications, Inc.
>
>

-- 
Regards  Fu.Jian
--
Cisco Communications, Inc.

Re: Re: [DISCUSS] KIP-842: Add richer group offset reset mechanisms

[hudeqi]

Hello, have any mates who have discussed it before seen it? Also welcome new 
mates to discuss together.

"hudeqi" <16120...@bjtu.edu.cn>写道：
> Long time no see, this issue has been discussed for a long time, now please 
> allow me to summarize this issue, and then everyone can help to see which 
> direction this issue should go in?
> 
> There are two problems to be solved by this kip:
> 1. Solve the problem that when the client configures the "auto.offset.reset" 
> to latest, the new partition data may be lost when the consumer resets the 
> offset to the latest after expanding the topic partition.
> 
> 2. In addition to the "earliest", "latest", and "none" provided by the 
> existing "auto.offset.reset", it also provides more abundant parameters, such 
> as "latest_on_start" (application startup is reset to latest, and an 
> exception is thrown if out of range occurs), "earliest_on_start" (application 
> startup is reset to earliest, and an exception is thrown if out of range 
> occurs), "nearest"(determined by "auto.offset.reset" when the program starts, 
> and choose earliest or latest according to the distance between the current 
> offset and log start offset and log end offset when out of range occurs).
> 
> According to the discussion results of the members above, it seems that there 
> are concerns about adding these additional offset reset mechanisms: 
> complexity and compatibility. In fact, these parameters do have corresponding 
> benefits. Therefore, based on the above discussion results, I have sorted out 
> two solution directions. You can help me to see which direction to follow：
> 
> 1. The first one is to follow Guozhang's suggestion: keep the three 
> parameters of "auto.offset.reset" and their meanings unchanged, reduce the 
> confusion for Kafka users, and solve the compatibility problem by the way. 
> Add these two parameters:
> a. "auto.offset.reset.on.no.initial.offse": Indicates the strategy used 
> to initialize the offset. The default value is the parameter configured by 
> "auto.offset.reset". If so, the strategy for initializing the offset remains 
> unchanged from the previous behavior, ensuring compatibility. If the 
> parameter is configured with "latest_on_start" or "earliest_on_start", then 
> the offset will be reset according to the configured semantics when 
> initializing the offset. In this way, the problem of data loss during 
> partition expansion can be solved: configure 
> "auto.offset.reset.on.no.initial.offset" to "latest_on_start", and configure 
> "auto.offset.reset" to earliest.
> b. "auto.offset.reset.on.invalid.offset": Indicates that the offset is 
> illegal or out of range occurs. The default value is the parameter configured 
> by "auto.offset.reset". If so, the processing of out of range is the same as 
> before to ensure compatibility. If "nearest" is configured, then the semantic 
> logic corresponding to "nearest" is used only for the case of out of range.
> 
> This solution ensures compatibility and ensures that the semantics of the 
> original configuration remain unchanged. Only two incremental configurations 
> are added to flexibly handle different situations.
> 
> 2. The second is to directly reduce the complexity of this problem, and 
> directly add the logic of resetting the initial offset of the newly expanded 
> partition to the earliest to "auto.offset.reset"="latest". In this way, Kafka 
> users do not need to perceive this subtle but useful change, and the 
> processing of other situations remains unchanged (without considering too 
> many rich offset processing mechanisms).
> 
> I hope you can help me with the direction of the solution to this issue, 
> thank you.
> 
> Best,
> hudeqi

Re: Re: Re: [DISCUSS] KIP-842: Add richer group offset reset mechanisms

[hudeqi]

I repost the newly changed KIP: 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-842%3A+Add+richer+group+offset+reset+mechanisms

"hudeqi" <16120...@bjtu.edu.cn>写道：
> Hello, have any mates who have discussed it before seen it? Also welcome new 
> mates to discuss together.
> 
> "hudeqi" <16120...@bjtu.edu.cn>写道：
> > Long time no see, this issue has been discussed for a long time, now please 
> > allow me to summarize this issue, and then everyone can help to see which 
> > direction this issue should go in?
> > 
> > There are two problems to be solved by this kip:
> > 1. Solve the problem that when the client configures the 
> > "auto.offset.reset" to latest, the new partition data may be lost when the 
> > consumer resets the offset to the latest after expanding the topic 
> > partition.
> > 
> > 2. In addition to the "earliest", "latest", and "none" provided by the 
> > existing "auto.offset.reset", it also provides more abundant parameters, 
> > such as "latest_on_start" (application startup is reset to latest, and an 
> > exception is thrown if out of range occurs), "earliest_on_start" 
> > (application startup is reset to earliest, and an exception is thrown if 
> > out of range occurs), "nearest"(determined by "auto.offset.reset" when the 
> > program starts, and choose earliest or latest according to the distance 
> > between the current offset and log start offset and log end offset when out 
> > of range occurs).
> > 
> > According to the discussion results of the members above, it seems that 
> > there are concerns about adding these additional offset reset mechanisms: 
> > complexity and compatibility. In fact, these parameters do have 
> > corresponding benefits. Therefore, based on the above discussion results, I 
> > have sorted out two solution directions. You can help me to see which 
> > direction to follow：
> > 
> > 1. The first one is to follow Guozhang's suggestion: keep the three 
> > parameters of "auto.offset.reset" and their meanings unchanged, reduce the 
> > confusion for Kafka users, and solve the compatibility problem by the way. 
> > Add these two parameters:
> > a. "auto.offset.reset.on.no.initial.offse": Indicates the strategy used 
> > to initialize the offset. The default value is the parameter configured by 
> > "auto.offset.reset". If so, the strategy for initializing the offset 
> > remains unchanged from the previous behavior, ensuring compatibility. If 
> > the parameter is configured with "latest_on_start" or "earliest_on_start", 
> > then the offset will be reset according to the configured semantics when 
> > initializing the offset. In this way, the problem of data loss during 
> > partition expansion can be solved: configure 
> > "auto.offset.reset.on.no.initial.offset" to "latest_on_start", and 
> > configure "auto.offset.reset" to earliest.
> > b. "auto.offset.reset.on.invalid.offset": Indicates that the offset is 
> > illegal or out of range occurs. The default value is the parameter 
> > configured by "auto.offset.reset". If so, the processing of out of range is 
> > the same as before to ensure compatibility. If "nearest" is configured, 
> > then the semantic logic corresponding to "nearest" is used only for the 
> > case of out of range.
> > 
> > This solution ensures compatibility and ensures that the semantics of the 
> > original configuration remain unchanged. Only two incremental 
> > configurations are added to flexibly handle different situations.
> > 
> > 2. The second is to directly reduce the complexity of this problem, and 
> > directly add the logic of resetting the initial offset of the newly 
> > expanded partition to the earliest to "auto.offset.reset"="latest". In this 
> > way, Kafka users do not need to perceive this subtle but useful change, and 
> > the processing of other situations remains unchanged (without considering 
> > too many rich offset processing mechanisms).
> > 
> > I hope you can help me with the direction of the solution to this issue, 
> > thank you.
> > 
> > Best,
> > hudeqi

Re: [DISCUSS] KIP-913: add new method to provide possibility for accelerate first record's sending

[Haruki Okada]

Thanks for your explanation.

> Thus, The code will look strange to call partitionsFor(topic)

Hmm is that so?
As the javadoc for `partitionsFor` states, it is simply for getting
partition metadata for the topic, so I think nothing is strange even if we
use it for "warmup" metadata.
(And if so, `getCluster` also looks strange)

> print some useful information when startup

This sounds like trying to solve a different problem from the
initial motivation (warmup metadata).
For this particular problem, we can just use Admin API's corresponding
methods.

> partitionsFor(topic) using the maxBlockTimeMs

I can understand the intention though, this is just a "block time" for the
caller thread and metadata-request is being sent individually on Sender
thread, so we can just retry calling partitionsFor until metadata-response
is eventually received and we get successful metadata even when it timed
out.

So I'm not sure if it makes sense to add a new API or timeout config (for
KIP-912), considering KafkaProducer already has many timeout parameters so
it could introduce another complexity.

2023年3月9日(木) 12:18 jian fu :

> Hi Haruki Okada:
>
> There is another KIP912 discussion related to this one. Welcome to give
> some comments/suggestions.
>
> Thanks.
>
> I think if the 912 is done. New method getCluster can use the new config
> directly with time parameter removed.
>
> WDTY?
>
> [DISCUSS] KIP-912: Support decreasing send's block time without worrying
> about metadata's fetch-Apache Mail Archives
> 
>
> jian fu  于2023年3月9日周四 11:11写道：
>
> > Hi Okada Haruki:
> >
> > Thanks for your comment.
> >
> > I can use it partitionsFor(topic) for the goal, thus there are two
> reasons
> > why I don't choose it and propose to add new dedicated method:
> > 1) Consider how to use the method to solve the issue, We should call it
> in
> > application's startup process before any record sent. Thus, The code will
> > look strange to call partitionsFor(topic) . So I suggest to add one
> > common method such as getCluster so that you can get and print some
> useful
> > information when startup with the goal reached. It also can provide more
> > information self compare with partitionsFor.
> > 2) partitionsFor(topic) using the maxBlockTimeMs as the max blocking
> > time. For the metadata's fetching, I will take a lot of time so that we
> > must set maxBlockTimeMs to a big value (at least > time for metadata).
> > Thus consider that the send method is async. Most of application like to
> > reduce the maxBlockTimeMs. It is conflict time requirement. So we need
> > one new blocking time as parameter of the method. I don't want to change
> > the existing interface.
> > So based on above reasons. I suggest to add new method with time control
> > parameter.
> > WDTY?
> >
> > Thanks.
> >
> >
> > Haruki Okada  于2023年3月9日周四 10:19写道：
> >
> >> You can just use Producer#partitionsFor
> >>
> >> 2023年3月9日(木) 11:13 jian fu :
> >>
> >> > Hi All:
> >> >
> >> > If anyone can help to give some comments or suggestions for the
> >> proposal.
> >> > Thanks in advance!
> >> >
> >> > Regards
> >> > Jian
> >> >
> >> > jian fu  于2023年3月6日周一 17:00写道：
> >> >
> >> > > Hi Everyone:
> >> > > Nice to meet you.
> >> > >
> >> > > I created one KIP to request your review.
> >> > > KIP-913: add new method to provide possibility for accelerate first
> >> > > record's sending
> >> > > <
> >> >
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-913%3A+add+new+method+to+provide+possibility+for+accelerate+first+record%27s+sending
> >> > >
> >> > > The example PR:
> >> > > *https://github.com/apache/kafka/pull/13320/files
> >> > > *
> >> > >
> >> > > Thanks.
> >> > >
> >> > > Regards
> >> > > Jian
> >> > >
> >> > >
> >> >
> >> > --
> >> > Regards  Fu.Jian
> >> > --
> >> > Cisco Communications, Inc.
> >> >
> >>
> >>
> >> --
> >> 
> >> Okada Haruki
> >> ocadar...@gmail.com
> >> 
> >>
> >
> >
> > --
> > Regards  Fu.Jian
> > --
> > Cisco Communications, Inc.
> >
> >
>
> --
> Regards  Fu.Jian
> --
> Cisco Communications, Inc.
>


-- 

Okada Haruki
ocadar...@gmail.com

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #1666

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 539852 lines...]
[2023-03-09T05:06:25.857Z] [INFO] Parameter: version, Value: 0.1
[2023-03-09T05:06:25.857Z] [INFO] Parameter: groupId, Value: streams.examples
[2023-03-09T05:06:25.857Z] [INFO] Parameter: artifactId, Value: streams.examples
[2023-03-09T05:06:25.857Z] [INFO] Project created from Archetype in dir: 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/quickstart/test-streams-archetype/streams.examples
[2023-03-09T05:06:25.857Z] [INFO] 

[2023-03-09T05:06:25.857Z] [INFO] BUILD SUCCESS
[2023-03-09T05:06:25.857Z] [INFO] 

[2023-03-09T05:06:25.857Z] [INFO] Total time:  1.386 s
[2023-03-09T05:06:25.857Z] [INFO] Finished at: 2023-03-09T05:06:25Z
[2023-03-09T05:06:25.857Z] [INFO] 

[Pipeline] dir
[2023-03-09T05:06:25.859Z] Running in 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/quickstart/test-streams-archetype/streams.examples
[Pipeline] {
[Pipeline] sh
[2023-03-09T05:06:28.180Z] + mvn compile
[2023-03-09T05:06:29.118Z] [INFO] Scanning for projects...
[2023-03-09T05:06:29.118Z] [INFO] 
[2023-03-09T05:06:29.118Z] [INFO] -< 
streams.examples:streams.examples >--
[2023-03-09T05:06:29.118Z] [INFO] Building Kafka Streams Quickstart :: Java 0.1
[2023-03-09T05:06:29.118Z] [INFO] [ jar 
]-
[2023-03-09T05:06:29.119Z] [INFO] 
[2023-03-09T05:06:29.119Z] [INFO] --- maven-resources-plugin:2.6:resources 
(default-resources) @ streams.examples ---
[2023-03-09T05:06:30.057Z] [INFO] Using 'UTF-8' encoding to copy filtered 
resources.
[2023-03-09T05:06:30.057Z] [INFO] Copying 1 resource
[2023-03-09T05:06:30.057Z] [INFO] 
[2023-03-09T05:06:30.057Z] [INFO] --- maven-compiler-plugin:3.1:compile 
(default-compile) @ streams.examples ---
[2023-03-09T05:06:30.057Z] [INFO] Changes detected - recompiling the module!
[2023-03-09T05:06:30.057Z] [INFO] Compiling 3 source files to 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/quickstart/test-streams-archetype/streams.examples/target/classes
[2023-03-09T05:06:30.997Z] [INFO] 

[2023-03-09T05:06:30.997Z] [INFO] BUILD SUCCESS
[2023-03-09T05:06:30.997Z] [INFO] 

[2023-03-09T05:06:30.997Z] [INFO] Total time:  1.987 s
[2023-03-09T05:06:30.997Z] [INFO] Finished at: 2023-03-09T05:06:30Z
[2023-03-09T05:06:30.997Z] [INFO] 

[Pipeline] }
[Pipeline] // dir
[Pipeline] }
[Pipeline] // dir
[Pipeline] }
[Pipeline] // dir
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // timestamps
[Pipeline] }
[Pipeline] // timeout
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[2023-03-09T05:07:46.345Z] > Task :core:testClasses
[2023-03-09T05:07:46.345Z] > Task :streams:compileTestJava UP-TO-DATE
[2023-03-09T05:07:46.345Z] > Task :streams:testClasses UP-TO-DATE
[2023-03-09T05:07:46.345Z] > Task :streams:testJar
[2023-03-09T05:07:46.345Z] > Task :streams:testSrcJar
[2023-03-09T05:07:46.345Z] > Task 
:streams:publishMavenJavaPublicationToMavenLocal
[2023-03-09T05:07:46.345Z] > Task :streams:publishToMavenLocal
[2023-03-09T05:07:46.345Z] 
[2023-03-09T05:07:46.345Z] Deprecated Gradle features were used in this build, 
making it incompatible with Gradle 9.0.
[2023-03-09T05:07:46.345Z] 
[2023-03-09T05:07:46.345Z] You can use '--warning-mode all' to show the 
individual deprecation warnings and determine if they come from your own 
scripts or plugins.
[2023-03-09T05:07:46.345Z] 
[2023-03-09T05:07:46.345Z] See 
https://docs.gradle.org/8.0.2/userguide/command_line_interface.html#sec:command_line_warnings
[2023-03-09T05:07:46.345Z] 
[2023-03-09T05:07:46.345Z] BUILD SUCCESSFUL in 4m 34s
[2023-03-09T05:07:46.345Z] 86 actionable tasks: 33 executed, 53 up-to-date
[Pipeline] sh
[2023-03-09T05:07:49.547Z] + grep ^version= gradle.properties
[2023-03-09T05:07:49.547Z] + cut -d= -f 2
[Pipeline] dir
[2023-03-09T05:07:54.338Z] Running in 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_trunk/streams/quickstart
[Pipeline] {
[Pipeline] sh
[2023-03-09T05:07:56.656Z] + mvn clean install -Dgpg.skip
[2023-03-09T05:07:58.414Z] [INFO] Scanning for projects...
[2023-03-09T05:07:59.353Z] [INFO] 

[2023-03-09T05:07:59.353Z] [INFO] Reactor Build Order:
[2023-03-09T05:07:59.353Z] [INFO] 
[2023-03-09T05:07:59.353Z] [INFO] Kafka Streams :: 

Re: [DISCUSS] KIP-913: add new method to provide possibility for accelerate first record's sending

[jian fu]

Hi Haruki Okada:
Thanks for commenting more ideas.
For the naming. Maybe it is personal view for which is better.
For the Admin API's approach. it can't solve the issue for Producer's
warmup and not every API can call the API with high permissions.
For the approach: retry calling partitionsFor until metadata-response. It
can meet the requirement but it is not graceful method. In local's project,
I reach the goal by java reflection And aslo I think it is not graceful
method. So I try to propose one public method.

I agree with that we don't want to involve too many timeout related
configure such as the parameter in getCluster. So maybe I can complete the
discuss of KIP912. then move back to this thread.

Anyway. thanks for your comment so that I aware that It is better for me to
not using timeout control parameter in the method.

Welcome for your more thought about it. And also we can have more
discussion about the KIP912. I want to introduce two configures to decouple
timeout control of  metadata fetching and append operation. By default, no
influence happen on the existed behavior if not set them.





Haruki Okada  于2023年3月9日周四 11:47写道：

> Thanks for your explanation.
>
> > Thus, The code will look strange to call partitionsFor(topic)
>
> Hmm is that so?
> As the javadoc for `partitionsFor` states, it is simply for getting
> partition metadata for the topic, so I think nothing is strange even if we
> use it for "warmup" metadata.
> (And if so, `getCluster` also looks strange)
>
> > print some useful information when startup
>
> This sounds like trying to solve a different problem from the
> initial motivation (warmup metadata).
> For this particular problem, we can just use Admin API's corresponding
> methods.
>
> > partitionsFor(topic) using the maxBlockTimeMs
>
> I can understand the intention though, this is just a "block time" for the
> caller thread and metadata-request is being sent individually on Sender
> thread, so we can just retry calling partitionsFor until metadata-response
> is eventually received and we get successful metadata even when it timed
> out.
>
> So I'm not sure if it makes sense to add a new API or timeout config (for
> KIP-912), considering KafkaProducer already has many timeout parameters so
> it could introduce another complexity.
>
> 2023年3月9日(木) 12:18 jian fu :
>
> > Hi Haruki Okada:
> >
> > There is another KIP912 discussion related to this one. Welcome to give
> > some comments/suggestions.
> >
> > Thanks.
> >
> > I think if the 912 is done. New method getCluster can use the new config
> > directly with time parameter removed.
> >
> > WDTY?
> >
> > [DISCUSS] KIP-912: Support decreasing send's block time without worrying
> > about metadata's fetch-Apache Mail Archives
> > 
> >
> > jian fu  于2023年3月9日周四 11:11写道：
> >
> > > Hi Okada Haruki:
> > >
> > > Thanks for your comment.
> > >
> > > I can use it partitionsFor(topic) for the goal, thus there are two
> > reasons
> > > why I don't choose it and propose to add new dedicated method:
> > > 1) Consider how to use the method to solve the issue, We should call it
> > in
> > > application's startup process before any record sent. Thus, The code
> will
> > > look strange to call partitionsFor(topic) . So I suggest to add one
> > > common method such as getCluster so that you can get and print some
> > useful
> > > information when startup with the goal reached. It also can provide
> more
> > > information self compare with partitionsFor.
> > > 2) partitionsFor(topic) using the maxBlockTimeMs as the max blocking
> > > time. For the metadata's fetching, I will take a lot of time so that we
> > > must set maxBlockTimeMs to a big value (at least > time for metadata).
> > > Thus consider that the send method is async. Most of application like
> to
> > > reduce the maxBlockTimeMs. It is conflict time requirement. So we need
> > > one new blocking time as parameter of the method. I don't want to
> change
> > > the existing interface.
> > > So based on above reasons. I suggest to add new method with time
> control
> > > parameter.
> > > WDTY?
> > >
> > > Thanks.
> > >
> > >
> > > Haruki Okada  于2023年3月9日周四 10:19写道：
> > >
> > >> You can just use Producer#partitionsFor
> > >>
> > >> 2023年3月9日(木) 11:13 jian fu :
> > >>
> > >> > Hi All:
> > >> >
> > >> > If anyone can help to give some comments or suggestions for the
> > >> proposal.
> > >> > Thanks in advance!
> > >> >
> > >> > Regards
> > >> > Jian
> > >> >
> > >> > jian fu  于2023年3月6日周一 17:00写道：
> > >> >
> > >> > > Hi Everyone:
> > >> > > Nice to meet you.
> > >> > >
> > >> > > I created one KIP to request your review.
> > >> > > KIP-913: add new method to provide possibility for accelerate
> first
> > >> > > record's sending
> > >> > > <
> > >> >
> > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-913%3A+add+new+method+to+provide+possibility+for+accelerate+first+record%27s+sending
> > >> > >
> > >> > > The ex