[DISCUSS] KIP-913: add new method to provide possibility for accelerate first record's sending

[jian fu]

Hi Everyone:
Nice to meet you.

I created one KIP to request your review.
KIP-913: add new method to provide possibility for accelerate first
record's sending

The example PR:
*https://github.com/apache/kafka/pull/13320/files
*

Thanks.

Regards
Jian

[jira] [Created] (KAFKA-14778) Kafka Streams 2.7.1 to 3.3.1 rolling upgrade with static membership triggers a rebalance

[Vinoth Rengarajan (Jira)]

Vinoth Rengarajan created KAFKA-14778:
-

 Summary: Kafka Streams 2.7.1 to 3.3.1 rolling upgrade with static 
membership triggers a rebalance
 Key: KAFKA-14778
 URL: https://issues.apache.org/jira/browse/KAFKA-14778
 Project: Kafka
  Issue Type: Bug
  Components: streams
Affects Versions: 3.3.1, 2.7.1
Reporter: Vinoth Rengarajan


Trying to upgrade Kaka Streams application from 2.7.1 to 3.3.1 with static 
membership but it triggers a rebalance

Brokers are running on Kafka 2.7.1. Enabled the static membership in the 
application. Below are the configs {*}(Stream Config & Consumer Config){*}.

Followed below steps to upgrade
 * Brokers are running on Kafka 2.7.1(tried with 3.3.1 version then also 
rebalance happens).
 * Application is running with 2.7.1 Kafka streams libraries.
 * Deployed the latest version of the application with 3.3.1 Kafka streams 
libraries, and configured the *upgrade.from* property to 2.7 (based on the 
upgrade documentation available here 
[https://kafka.apache.org/33/documentation/streams/upgrade-guide]).
 * Doing a rolling bounce with the latest changes, rebalance is being triggered 
on other instances in the cluster.

Below are logs on the instance which is being bounced, forcing a rebalance on 
others. 

*Logs:*

 
{code:java}
INFO  2023-02-27 09:52:16.805 | streams.KafkaStreams stream-client 
[kafka_upgrade.Kafka_Upgrade_Test] State transition from CREATED to REBALANCING
INFO  2023-02-27 09:52:16.946 | internals.ConsumerCoordinator [Consumer 
instanceId=kafka_upgrade.Kafka_Upgrade_Test-4, 
clientId=kafka_upgrade.Kafka_Upgrade_Test-StreamThread-4-consumer, 
groupId=kafka_upgrade.Kafka_Upgrade_Test] Notifying assignor about the new 
Assignment(partitions=[kafka_upgrade.Kafka_Upgrade_Test-version-updates-11, 
kafka_upgrade.Kafka_Upgrade_Test-version-updates-23], userDataSize=56)
INFO  2023-02-27 09:52:16.947 | internals.StreamsPartitionAssignor 
stream-thread [kafka_upgrade.Kafka_Upgrade_Test-StreamThread-3-consumer] Sent a 
version 11 subscription and got version 8 assignment back (successful version 
probing). Downgrade subscription metadata to commonly supported version 8 and 
trigger new rebalance.
INFO  2023-02-27 09:52:16.947 | internals.StreamsPartitionAssignor 
stream-thread [kafka_upgrade.Kafka_Upgrade_Test-StreamThread-2-consumer] Sent a 
version 11 subscription and got version 8 assignment back (successful version 
probing). Downgrade subscription metadata to commonly supported version 8 and 
trigger new rebalance.
INFO  2023-02-27 09:52:16.947 | internals.StreamsPartitionAssignor 
stream-thread [kafka_upgrade.Kafka_Upgrade_Test-StreamThread-4-consumer] Sent a 
version 11 subscription and got version 8 assignment back (successful version 
probing). Downgrade subscription metadata to commonly supported version 8 and 
trigger new rebalance.
INFO  2023-02-27 09:52:16.947 | internals.StreamsPartitionAssignor 
stream-thread [kafka_upgrade.Kafka_Upgrade_Test-StreamThread-1-consumer] Sent a 
version 11 subscription and got version 8 assignment back (successful version 
probing). Downgrade subscription metadata to commonly supported version 8 and 
trigger new rebalance.
INFO  2023-02-27 09:52:16.947 | internals.StreamsPartitionAssignor 
stream-thread [kafka_upgrade.Kafka_Upgrade_Test-StreamThread-2-consumer] 
Requested to schedule immediate rebalance due to version probing.
INFO  2023-02-27 09:52:16.948 | internals.StreamsPartitionAssignor 
stream-thread [kafka_upgrade.Kafka_Upgrade_Test-StreamThread-1-consumer] 
Requested to schedule immediate rebalance due to version probing.
INFO  2023-02-27 09:52:16.948 | internals.StreamsPartitionAssignor 
stream-thread [kafka_upgrade.Kafka_Upgrade_Test-StreamThread-4-consumer] 
Requested to schedule immediate rebalance due to version probing.
INFO  2023-02-27 09:52:16.948 | internals.StreamsPartitionAssignor 
stream-thread [kafka_upgrade.Kafka_Upgrade_Test-StreamThread-3-consumer] 
Requested to schedule immediate rebalance due to version probing. {code}
 

 

*Streams Config:*

 
{code:java}
acceptable.recovery.lag = 1
application.id = Kafka_Upgrade_Test
application.server =
bootstrap.servers = [broker1, broker2, broker3]
buffered.records.per.partition = 1000
built.in.metrics.version = latest
cache.max.bytes.buffering = 10485760
client.id = kafka_upgrade.Kafka_Upgrade_Test
commit.interval.ms = 3
connections.max.idle.ms = 54
default.deserialization.exception.handler = class 
org.apache.kafka.streams.errors.LogAndFailExceptionHandler
default.dsl.store = rocksDB
default.key.serde = null
default.list.key.serde.inner = null
default.list.key.serde.type = null
default.list.value.serde.inner = null
default.list.value.serde.type = null
default.production.exception.handler = class 
org.apache.kafka.streams.errors.DefaultProductionExceptionHandler
default.timestamp.extractor = class 
org.apache.kafka.streams.pro

Re: [DISCUSS] KIP-902: Upgrade Zookeeper to 3.8.1

[Christo Lolov]

Hey Luke,

Thank you for the review! My reasoning for going to ZK 3.8.1 directly is that 
if there isn’t a set date for the Kafka 4.0 release it gives us more time 
before we have to (potentially) upgrade ZK again (similar to what is mentioned 
here https://github.com/apache/kafka/pull/12620#issuecomment-1245590870). ZK 
3.7.1 also doesn’t support earlier client/server combinations different to what 
3.8.1 does.

Best,
Christo

> On 2 Mar 2023, at 07:07, Luke Chen  wrote:
> 
> Hi Christo,
> 
> Thanks for the KIP.
> The motivation of upgrading ZK makes sense.
> And thanks for the great analysis for the ZK upgrading.
> 
> One question:
> Since we are going to remove ZK in v4.0, and we don't need the latest
> feature in the "current release" ZK 3.8.1, why can't we choose the "stable
> release" (3.7.1)?
> 
> Thank you.
> Luke
> 
>> On Wed, Feb 15, 2023 at 5:47 PM Christo Lolov 
>> wrote:
>> 
>> Hello!
>> 
>> I would like to start a discussion for KIP-902: Upgrade Zookeeper to
>> 3.8.1. The Zookeeper version currently used in Kafka reached its end of
>> life in December 2022. You can find the KIP at
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=240882784
>> 
>> Thanks in advance for the reviews.
>> 
>> Christo

[jira] [Created] (KAFKA-14779) Add ACL Authorizer integration test for authorized OffsetCommits with an unknown topic

[Alexandre Dupriez (Jira)]

Alexandre Dupriez created KAFKA-14779:
-

 Summary: Add ACL Authorizer integration test for authorized 
OffsetCommits with an unknown topic
 Key: KAFKA-14779
 URL: https://issues.apache.org/jira/browse/KAFKA-14779
 Project: Kafka
  Issue Type: Sub-task
Reporter: Alexandre Dupriez


Discovered as part of [PR-13240|https://github.com/apache/kafka/pull/13240),], 
it seems the use case where a group and topic have the necessary ACLs to allow 
for offsets for that topic and consumer group to be committed, but the topic is 
unknown by the broker (either by name or id), is not covered. This purpose of 
this ticket is to add this coverage.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] KIP-902: Upgrade Zookeeper to 3.8.1

[Christo Lolov]

Hello Ismael,

Thank you for the valid points. I have updated the KIP, but do let me know if 
you believe it is still not clear enough.

Best,
Christo

> On 2 Mar 2023, at 15:21, Ismael Juma  wrote:
> 
> Thanks for the KIP. I think the following is a little confusing since it
> doesn't make it clear the the ZooKeeper deployment is separate from Kafka,
> Kafka only includes the ZooKeeper libraries. I think it would be useful to
> explain the upgrade process for someone running Apache Kafka 2.3 and
> ZooKeeper 3.4 (the hardest case) and the same for someone running Apache
> Kafka 2.4 and ZooKeeper 3.5.
> 
> Also, it's worth clarifying that we actually still test direct kafka
> upgrades from 0.8.2 to 3.4. In practice, we have distinguished "providing
> updates" versus "allowing direct upgrades from". Apache Kafka 4.0 will
> change this since you will have to upgrade to a bridge release before
> upgrading to 4.0, but that's a new development.
> 
> "Users who use Kafka clusters with Zookeeper clients older than 3.5.x won't
> be able to communicate with a Zookeeper cluster using 3.8.1. As mentioned
> in the accompanying JIRA ticket Apache Kafka has been using Zookeeper 3.5.x
> since version 2.4 so versions above and including it should be safe for
> this upgrade. It is acceptable to break compatibility with Apache Kafka
> versions prior to 2.4 as they are considered beyond their end of life and
> are not maintained. (source: Time Based Release Plan#WhatIsOurEOLPolicy)."
> 
> Ismael
> 
>> On Wed, Feb 15, 2023 at 1:47 AM Christo Lolov 
>> wrote:
>> 
>> Hello!
>> 
>> I would like to start a discussion for KIP-902: Upgrade Zookeeper to
>> 3.8.1. The Zookeeper version currently used in Kafka reached its end of
>> life in December 2022. You can find the KIP at
>> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=240882784
>> 
>> Thanks in advance for the reviews.
>> 
>> Christo

[jira] [Created] (KAFKA-14780) Make RefreshingHttpsJwksTest#testSecondaryRefreshAfterElapsedDelay deterministic

[Alexandre Dupriez (Jira)]

Alexandre Dupriez created KAFKA-14780:
-

 Summary: Make 
RefreshingHttpsJwksTest#testSecondaryRefreshAfterElapsedDelay deterministic
 Key: KAFKA-14780
 URL: https://issues.apache.org/jira/browse/KAFKA-14780
 Project: Kafka
  Issue Type: Test
Reporter: Alexandre Dupriez


The test `RefreshingHttpsJwksTest#testSecondaryRefreshAfterElapsedDelay` relies 
on the actual system clock which makes it frequently fail on my poor intellij 
setup.

 

The `RefreshingHttpsJwks` component creates and uses a scheduled executor 
service. We could expose the scheduling mechanism to be able to mock its 
behaviour. One way to do could be to use the `KafkaScheduler` which has a 
`MockScheduler` implementation which relies on `MockTime` instead of the real 
time clock.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: A query on log truncation.

[Vinoth]

Hi Luke ,
  Thanks for acknowledging my mail. Sorry for the late reply.
My query was not on keeping uncommitted records but on how teams managed
the loss of committed data in case of unclean leader election. Is there a
means to track lost data?. Is this a common problem?. I am asking based on
kip-320 which mentions committed data might be lost when unclean leader
election is enabled.

Regards,
Vinoth

On Mon, 16 Jan 2023 at 10:37, Luke Chen  wrote:

> Hi Vinoth,
>
> I'm wondering what's the use case or pain point you're trying to resolve?
> Like you said, the client will be notified the data is not successfully
> sent or propagated and handle the error, why should we keep the un-commited
> records?
> Could you elaborate more on the motivation?
>
> Thank you.
> Luke
>
> On Mon, Jan 16, 2023 at 12:33 PM Vinoth  wrote:
>
> > I was reading through about kafka , the way leader election works , log
> > truncation etc. One thing that kind of struck me was how records which
> were
> > written to log but then were not committed (It has not propagated
> > successfully through to all of the isr and and the high watermark has not
> > increased and so not committed ) ,were truncated following the
> replication
> > reconciliation logic . In case they are not committed they would not be
> > available for the consumer since the reads are  only upto to the high
> > watermark. the producer client will also be notified or will eventually
> > know if the message has not successfully propagated and it should be
> > handled thru application logic. It seems straight forward in this case.
> >
> > KIP-405
> > <
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage
> > >
> > talks about tiered storage and kafka being an important part of and an
> > entry point for data infrastructure . Else where i have read that kafka
> > also serves as way of replaying data to restore state / viewing data.
> > KIP-320
> > <
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-320%3A+Allow+fetchers+to+detect+and+handle+log+truncation
> > >
> > mentions users wanting higher availability opting for unclean leader
> > election.
> >
> > Would it be fair to assume that users might be interested in a feature or
> > at least  one that can be user enabled where a write to kafka (even with
> a
> > 0 or no acks configuration or unlcean leader election ) will remain
> written
> > until the event where clean or delete config is acted upon?.
> >
> > If this is a valid use case , i have thoughts of suggesting a kip around
> > picking up the data that is to be truncated at time of truncation and
> > replaying it as if it came through a fresh produce request. That is a
> > truncation of data will not result in the data being removed from kafka
> but
> > rather be placed differently at a different offset.
> >
> > Regards,
> > Vinoth
> >
>

[jira] [Created] (KAFKA-14781) MM2 logs misleading error during topic ACL sync when broker does not have authorizer configured

[Chris Egerton (Jira)]

Chris Egerton created KAFKA-14781:
-

 Summary: MM2 logs misleading error during topic ACL sync when 
broker does not have authorizer configured
 Key: KAFKA-14781
 URL: https://issues.apache.org/jira/browse/KAFKA-14781
 Project: Kafka
  Issue Type: Bug
  Components: mirrormaker
Reporter: Chris Egerton


When there is no broker-side authorizer configured on a Kafka cluster targeted 
by MirrorMaker 2, users see error-level log messages like this one:{{{}{}}}
{quote}[2023-03-06 10:53:57,488] ERROR [MirrorSourceConnector|worker] Scheduler 
for MirrorSourceConnector caught exception in scheduled task: syncing topic 
ACLs (org.apache.kafka.connect.mirror.Scheduler:102)
java.util.concurrent.ExecutionException: 
org.apache.kafka.common.errors.SecurityDisabledException: No Authorizer is 
configured on the broker
    at 
java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395)
    at 
java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1999)
    at 
org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:165)
    at 
org.apache.kafka.connect.mirror.MirrorSourceConnector.listTopicAclBindings(MirrorSourceConnector.java:456)
    at 
org.apache.kafka.connect.mirror.MirrorSourceConnector.syncTopicAcls(MirrorSourceConnector.java:342)
    at org.apache.kafka.connect.mirror.Scheduler.run(Scheduler.java:93)
    at 
org.apache.kafka.connect.mirror.Scheduler.executeThread(Scheduler.java:112)
    at 
org.apache.kafka.connect.mirror.Scheduler.lambda$scheduleRepeating$0(Scheduler.java:50)
    at 
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at 
java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
    at 
java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
    at 
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at 
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.apache.kafka.common.errors.SecurityDisabledException: No 
Authorizer is configured on the broker
{quote}
This can be misleading as it looks like something is wrong with MM2 or the 
Kafka cluster. In reality, it's usually fine, since topic ACL syncing is 
enabled by default and it's reasonable for Kafka clusters (especially in 
testing/dev environments) to not have authorizers enabled.

We should try to catch this specific case and downgrade the severity of the log 
message from {{ERROR}} to either {{INFO}} or {{{}DEBUG{}}}. We may also 
consider suggesting to users that they disable topic ACL syncing if their Kafka 
cluster doesn't have authorization set up, but this should probably only be 
emitted once over the lifetime of the connector in order to avoid generating 
log spam.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14782) Implementation Details Different from Documentation (delivery.timeout.ms)

[jangho kwon (Jira)]

jangho kwon created KAFKA-14782:
---

 Summary: Implementation Details Different from Documentation 
(delivery.timeout.ms)
 Key: KAFKA-14782
 URL: https://issues.apache.org/jira/browse/KAFKA-14782
 Project: Kafka
  Issue Type: Bug
  Components: clients, producer 
Reporter: jangho kwon


Hello,

I was checking the value related to {{{}delivery.timeout.ms{}}}, and I found a 
relevant document here: 
[https://cwiki.apache.org/confluence/display/KAFKA/KIP-91+Provide+Intuitive+User+Timeouts+in+The+Producer]

In the "Validation" section of the document, it is written as follows:
{quote}This configuration is backwards compatible. Throw ConfigException for 
timeouts that don't make sense. (E.g., delivery.timeout.ms < linger.ms + 
request.timeout.ms + retry.backoff.ms).
{quote}
However, I noticed that the code does not use the {{retry.backoff.ms}} value 
when checking if {{delivery.timeout.ms}} is valid.

I'm curious why this value was excluded. Is it a bug?



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] KIP-898: Modernize Connect plugin discovery

[Greg Harris]

Hi all,

This vote thread has been open over 72 hours, and has sufficient votes for
lazy majority, so I'll close the voting at this time.

+4 binding votes from Chris Egerton, Mickael Maison, John Roesler, and Bill
Bejeck.
+2 non-binding votes from myself and Federico Valeri.
No -1 binding or -1 non-binding votes.
KIP-898 has PASSED voting.

Thanks all for your support, and I'll get to work on the implementation.
Greg Harris

On Tue, Feb 28, 2023 at 7:44 AM Bill Bejeck  wrote:

> Thanks for the well-detailed KIP, Greg. It'll be a needed improvement.
>
> +1(binding)
>
> Thanks,
> Bill
>
> On Tue, Feb 28, 2023 at 9:51 AM John Roesler  wrote:
>
> > Thanks for the KIP, Greg!
> >
> > I’m +1 (binding)
> >
> > I really appreciate all the care you took in the migration and test
> > design.
> >
> > Thanks,
> > John
> >
> > On Tue, Feb 28, 2023, at 04:33, Federico Valeri wrote:
> > > +1 (non binding)
> > >
> > > Thanks
> > > Fede
> > >
> > > On Tue, Feb 28, 2023 at 10:10 AM Mickael Maison
> > >  wrote:
> > >>
> > >> +1 (binding)
> > >>
> > >> Thanks,
> > >> Mickael
> > >>
> > >> On Mon, Feb 27, 2023 at 7:42 PM Chris Egerton  >
> > wrote:
> > >> >
> > >> > +1 (binding). Thanks for the KIP!
> > >> >
> > >> > On Mon, Feb 27, 2023 at 12:51 PM Greg Harris
> > 
> > >> > wrote:
> > >> >
> > >> > > Hi,
> > >> > >
> > >> > > I'd like to call a vote for KIP-898 which aims to improve the
> > performance
> > >> > > of Connect startup by allowing discovery of plugins via the
> > ServiceLoader.
> > >> > >
> > >> > > KIP:
> > >> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-898
> > >> > > %3A+Modernize+Connect+plugin+discovery
> > >> > >
> > >> > > Discussion thread:
> > >> > > https://lists.apache.org/thread/wxh0r343w86s91py0876njzbyn5qxd8s
> > >> > >
> > >> > > Thanks!
> > >> > >
> >
>

Re: [DISCUSS] KIP-909: Allow clients to rebootstrap DNS lookup failure

[Philip Nee]

Cheers Kafka Community,

I just wanna give this thread bump, as it has been a bit quiet for the past
week. I have not updated the KIP based on Chris and Jason's feedback, as I
would also like to know more about what do people think.

Jason - Thanks for the suggestion, I think your suggestion makes a lot of
sense.

Thanks!
P

On Tue, Feb 28, 2023 at 2:45 PM Jason Gustafson 
wrote:

> Hi Philip,
>
> > Having an overall timeout also seems reasonable, but I wonder what should
> the client do after running out of the time? Should we throw a
> non-retriable exception (instead of TimeoutExceptoin to stop the client
> from retrying) and alert the user to examine the config and the DNS server?
>
> Yeah, not sure exactly. I'd probably suggest a
> `BootstrapConnectionException` or something like that with a clear message
> indicating the problem. What the user does with it is up to them, but at
> least it gives them the option to fail their application if that is what
> they prefer to do in this case. If they catch it and ignore it, I would
> expect the client to just continue retrying. Logging for bootstrap
> dns/connection failures will be helpful in any case.
>
> -Jason
>
>
>
>
>
>
> On Tue, Feb 28, 2023 at 11:47 AM Philip Nee  wrote:
>
> > Jason:
> > Thanks for the feedback.  Now giving it a second thought, I think your
> > suggestion of logging the error might make sense, as I could imagine most
> > users would just continue to retry, so it might not be necessary to throw
> > an exception anyway.
> > Having an overall timeout also seems reasonable, but I wonder what should
> > the client do after running out of the time? Should we throw a
> > non-retriable exception (instead of TimeoutExceptoin to stop the client
> > from retrying) and alert the user to examine the config and the DNS
> server?
> >
> > Chris:
> > I feel I still haven't answered your question about the pre-flight check,
> > as it seems exposing an API might be harder to push through.
> >
> > Thanks!
> > P
> >
> > On Tue, Feb 28, 2023 at 10:53 AM Jason Gustafson
> > 
> > wrote:
> >
> > > One more random thought I had just as I pushed send. We're currently
> > > treating this problem somewhat narrowly by focusing only on the DNS
> > > resolution of the bootstrap servers. Even if the servers resolve,
> there's
> > > no guarantee that they are reachable by the client. Would it make sense
> > to
> > > have a timeout which bounds the total time that the client should wait
> to
> > > connect to the bootstrap servers? Something like `
> > > bootstrap.servers.connection.timeout.ms`.
> > >
> > > -Jason
> > >
> > > On Tue, Feb 28, 2023 at 10:44 AM Jason Gustafson 
> > > wrote:
> > >
> > > > Hi Philip,
> > > >
> > > > An alternative is not to fail at all. Every other network error is
> > caught
> > > > and handled internally in the client. We see this case as different
> > > because
> > > > a DNS resolution error may imply misconfiguration. Could it also
> imply
> > > that
> > > > the DNS server is unavailable? I'm not sure why that case should be
> > > handled
> > > > differently than if the bootstrap servers themselves are unavailable.
> > > Would
> > > > it be enough to log a clear warning in the logs if the bootstrap
> > servers
> > > > could not resolve?
> > > >
> > > > On the whole, the current fail-fast approach probably does more good
> > than
> > > > bad, but it does seem somewhat inconsistent overall and my guess is
> > that
> > > > dynamic environments will become increasingly common. It would be
> nice
> > to
> > > > have a reasonable default behavior which could handle these cases
> > > > gracefully without any additional logic. In any case, it would be
> nice
> > to
> > > > see this option in the rejected alternatives at least if we do not
> take
> > > it.
> > > >
> > > > If we want to take the route of throwing an exception, then I think
> > we're
> > > > probably going to need a new configuration since I can't see what a
> > > > reasonable timeout we would use as a default. The benefit of a
> > > > configuration is that it would let us retain the current default
> > behavior
> > > > with timeout effectively set to 0 and it would also let users
> > effectively
> > > > disable the timeout by using a very large value. Otherwise, it seems
> > > like a
> > > > potential compatibility break to have a new exception type thrown at
> > some
> > > > arbitrary time without giving the user any control over it.
> > > >
> > > > Thanks,
> > > > Jason
> > > >
> > > >
> > > > On Tue, Feb 28, 2023 at 8:08 AM Chris Egerton
>  > >
> > > > wrote:
> > > >
> > > >> Hi Philip,
> > > >>
> > > >> Yeah, it's basically DNS resolution we're talking about, though
> > there's
> > > >> some additional subtlety there with the logic introduced by KIP-235
> > [1].
> > > >> Essentially it should cover any scenario that causes a client
> > > constructor
> > > >> to fail with the current logic but would not after this KIP is
> > released.
> > > >>
> > > >> We can generalize the Co

[GitHub] [kafka-site] mimaison commented on pull request #410: KAFKA-13882 Docker to preview docs locally

[via GitHub]

mimaison commented on PR #410:
URL: https://github.com/apache/kafka-site/pull/410#issuecomment-1456641170

   @mjsax My point is that when testing locally without the `.htaccess` 
changes, all pages other than localhost:8080/documentation/ fail to load. For 
example http://localhost:8080/quickstart returns a 404.
   
   I see that @qingwei91 removed the `.htaccess` changes in the last commit and 
I confirmed the issue happens again. I'm a bit scared of touching the 
`.htaccess` file as I don't want to break the real website.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] qingwei91 commented on pull request #410: KAFKA-13882 Docker to preview docs locally

[via GitHub]

qingwei91 commented on PR #410:
URL: https://github.com/apache/kafka-site/pull/410#issuecomment-1456760908

   @mimaison , really sorry, I forgotten to check quick start. I've added it 
back now, guess I need to figure out why that's required here but not in the 
actual deployment ...


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Re: Hosting Kafka Videos on ASF YouTube channel

[Bill Bejeck]

Hi Brian,

I'm bumping this thread.  I have download links for all the videos (5) now,
but I'd prefer to share them directly with you or whoever will get them
hosted on the ASF youtube channel.

Thanks!
Bill

On Thu, Feb 23, 2023 at 12:13 PM Joe Brockmeier  wrote:

> Actually adding Brian to CC now...
>
> On Thu, Feb 23, 2023 at 12:12 PM Joe Brockmeier  wrote:
>
>> Hi Bill,
>>
>> On Thu, Feb 23, 2023 at 10:48 AM Bill Bejeck  wrote:
>>
>>> It took a little time, but I have download links to share with you for
>>> the 4 Kafka Streams videos we want to host on the ASF Youtube channel, as
>>> discussed on this thread
>>> .
>>> The "What is Apache Kafka " video
>>> has been edited to remove any vendor reference.  I'll also have a download
>>> link to share for that video soon.
>>>
>>> Where is the best place to share the links with you?  I'd prefer not to
>>> share them with you directly vs. the public.
>>>
>>
>> I've stepped aside from the VP marketing role, Brian Proffitt is now
>> heading up M&P, so he can help guide you on the uploads. (Brian, lemme know
>> if you have questions or whatnot.)
>>
>> Did you mean you'd prefer *to* share them directly vs. publicly? Either
>> way, you can work out with Brian/press@ how to share and upload them.
>>
>> Thanks,
>>
>> Joe
>>
>

Re: Hosting Kafka Videos on ASF YouTube channel

[Brian Proffitt]

Yes, I was waiting for you. Just send the links directly to me, along
with any other information that may be pertinent, like title, description,
and speakers for each video.

BKP

On Mon, Mar 6, 2023 at 3:07 PM Bill Bejeck  wrote:

> Hi Brian,
>
> I'm bumping this thread.  I have download links for all the videos (5)
> now, but I'd prefer to share them directly with you or whoever will get
> them hosted on the ASF youtube channel.
>
> Thanks!
> Bill
>
> On Thu, Feb 23, 2023 at 12:13 PM Joe Brockmeier  wrote:
>
>> Actually adding Brian to CC now...
>>
>> On Thu, Feb 23, 2023 at 12:12 PM Joe Brockmeier  wrote:
>>
>>> Hi Bill,
>>>
>>> On Thu, Feb 23, 2023 at 10:48 AM Bill Bejeck  wrote:
>>>
 It took a little time, but I have download links to share with you for
 the 4 Kafka Streams videos we want to host on the ASF Youtube channel, as
 discussed on this thread
 .
 The "What is Apache Kafka " video
 has been edited to remove any vendor reference.  I'll also have a download
 link to share for that video soon.

 Where is the best place to share the links with you?  I'd prefer not to
 share them with you directly vs. the public.

>>>
>>> I've stepped aside from the VP marketing role, Brian Proffitt is now
>>> heading up M&P, so he can help guide you on the uploads. (Brian, lemme know
>>> if you have questions or whatnot.)
>>>
>>> Did you mean you'd prefer *to* share them directly vs. publicly? Either
>>> way, you can work out with Brian/press@ how to share and upload them.
>>>
>>> Thanks,
>>>
>>> Joe
>>>
>>

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #1651

[Apache Jenkins Server]

See 

Re: [VOTE] KIP-907: Add Boolean Serde to public interface

[SpacRocket]

Hello Everyone,

Thank you everyone for looking into my KIP, much appreciated. 

I’d like to inform you that I closed the voting and soon will implement the 
changes.

For the KIP, Kafka has Lazy Majority:
I’ve got +3 (binding) and +2 (non-binding) votes.
https://lists.apache.org/list?dev@kafka.apache.org:lte=1M:[VOTE] KIP-907 


Kind regards
Jakub

> On Mar 2, 2023, at 4:55 PM, Lucas Brutschy  
> wrote:
> 
> +1 (non-binding)
> 
> Thank you!
> 
> On Wed, Mar 1, 2023 at 9:03 PM Matthias J. Sax  wrote:
>> 
>> +1 (binding)
>> 
>> Thanks for the KIP!
>> 
>> On 3/1/23 10:59 AM, Walker Carlson wrote:
>>> +1 Binding
>>> 
>>> On Mon, Feb 27, 2023 at 1:46 PM Chia-Ping Tsai  wrote:
>>> 
 +1 (binding)
 
>>> 

[jira] [Created] (KAFKA-14783) Implement new STOPPED state for connectors

[Chris Egerton (Jira)]

Chris Egerton created KAFKA-14783:
-

 Summary: Implement new STOPPED state for connectors
 Key: KAFKA-14783
 URL: https://issues.apache.org/jira/browse/KAFKA-14783
 Project: Kafka
  Issue Type: Task
  Components: KafkaConnect
Reporter: Chris Egerton
Assignee: Chris Egerton


Implement the {{STOPPED}} state [described in 
KIP-875|https://cwiki.apache.org/confluence/display/KAFKA/KIP-875%3A+First-class+offsets+support+in+Kafka+Connect#KIP875:FirstclassoffsetssupportinKafkaConnect-Newtargetstate:STOPPED].



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14785) Implement connector offset read REST API

[Chris Egerton (Jira)]

Chris Egerton created KAFKA-14785:
-

 Summary: Implement connector offset read REST API
 Key: KAFKA-14785
 URL: https://issues.apache.org/jira/browse/KAFKA-14785
 Project: Kafka
  Issue Type: Sub-task
  Components: KafkaConnect
Reporter: Chris Egerton


Implement the {{GET /connector/name/offsets}} endpoint [described in 
KIP-875|https://cwiki.apache.org/confluence/display/KAFKA/KIP-875%3A+First-class+offsets+support+in+Kafka+Connect#KIP875:FirstclassoffsetssupportinKafkaConnect-Readingoffsets].



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14784) Implement connector offset reset REST API

[Chris Egerton (Jira)]

Chris Egerton created KAFKA-14784:
-

 Summary: Implement connector offset reset REST API
 Key: KAFKA-14784
 URL: https://issues.apache.org/jira/browse/KAFKA-14784
 Project: Kafka
  Issue Type: Sub-task
  Components: KafkaConnect
Reporter: Chris Egerton


Implement the {{DELETE /connectors/name/offsets}} endpoint [described in 
KIP-875|https://cwiki.apache.org/confluence/display/KAFKA/KIP-875%3A+First-class+offsets+support+in+Kafka+Connect#KIP875:FirstclassoffsetssupportinKafkaConnect-Resettingoffsets].



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-14786) Implement connector offset write/reset internal logic

[Chris Egerton (Jira)]

Chris Egerton created KAFKA-14786:
-

 Summary: Implement connector offset write/reset internal logic
 Key: KAFKA-14786
 URL: https://issues.apache.org/jira/browse/KAFKA-14786
 Project: Kafka
  Issue Type: Sub-task
  Components: KafkaConnect
Reporter: Chris Egerton


Implement the internal logic necessary for altering/resetting the offsets of 
connectors, [described in 
KIP-875|https://cwiki.apache.org/confluence/display/KAFKA/KIP-875%3A+First-class+offsets+support+in+Kafka+Connect#KIP875:FirstclassoffsetssupportinKafkaConnect-Endpointsbehavior].

This should not include any changes to public interface except the introduction 
of the new {{SourceConnector::alterOffsets}} and 
{{SinkConnector::alterOffsets}} methods (i.e., it should not expose or test any 
new REST endpoints).

Ideally, we'll separate this from KAFKA-14368, KAFKA-14784, and KAFKA-14785 by 
making all changes here target the internal Connect {{Herder}} interface, and 
have the changes for the other three rely on those new {{Herder}} methods.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[GitHub] [kafka-site] qingwei91 commented on pull request #410: KAFKA-13882 Docker to preview docs locally

[via GitHub]

qingwei91 commented on PR #410:
URL: https://github.com/apache/kafka-site/pull/410#issuecomment-1457134844

   Hi @mimaison , another way I can think of is to avoid using .htaccess in the 
docker, and inject all config required in httpd.conf instead.
   
   
https://github.com/apache/kafka-site/pull/410/commits/4d7397f17d10e64d3fadf4e8722e492bdf131fba
   
   I am not sure if this is better since we now have duplicated config
   
   I also wonder if .htaccess is actually used in the actual deployment, as it 
requires modifying httpd.conf


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

[GitHub] [kafka-site] mjsax commented on pull request #410: KAFKA-13882 Docker to preview docs locally

[via GitHub]

mjsax commented on PR #410:
URL: https://github.com/apache/kafka-site/pull/410#issuecomment-1457209098

   Ah sorry. Seems I miss-understood your comment @mimaison. -- I am not an 
expert in this domain and don't know why we need the change to `htaccess` nor 
what the actually impact is... :( 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@kafka.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #1652

[Apache Jenkins Server]

See 

[jira] [Reopened] (KAFKA-14371) quorum-state file contains empty/unused clusterId field

[Jira]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

José Armando García Sancio reopened KAFKA-14371:

  Assignee: (was: Gantigmaa Selenge)

> quorum-state file contains empty/unused clusterId field
> ---
>
> Key: KAFKA-14371
> URL: https://issues.apache.org/jira/browse/KAFKA-14371
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Ron Dagostino
>Priority: Minor
>  Labels: needs-kip
> Fix For: 3.5.0
>
>
> The KRaft controller's quorum-state file 
> `$LOG_DIR/__cluster_metadata-0/quorum-state` contains an empty clusterId 
> value.  This value is never non-empty, and it is never used after it is 
> written and then subsequently read.  This is a cosmetic issue; it would be 
> best if this value did not exist there.  The cluster ID already exists in the 
> `$LOG_DIR/meta.properties` file.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Build failed in Jenkins: Kafka » Kafka Branch Builder » trunk #1653

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 536423 lines...]
[2023-03-07T04:34:46.105Z] 
[2023-03-07T04:34:46.105Z] Gradle Test Run :connect:runtime:integrationTest > 
Gradle Test Executor 134 > 
org.apache.kafka.connect.integration.RebalanceSourceConnectorsIntegrationTest > 
testStartTwoConnectors PASSED
[2023-03-07T04:34:46.105Z] 
[2023-03-07T04:34:46.105Z] Gradle Test Run :connect:runtime:integrationTest > 
Gradle Test Executor 134 > 
org.apache.kafka.connect.integration.RebalanceSourceConnectorsIntegrationTest > 
testDeleteConnector STARTED
[2023-03-07T04:34:51.444Z] 
[2023-03-07T04:34:51.444Z] > Task 
:streams:upgrade-system-tests-0102:integrationTest
[2023-03-07T04:34:51.444Z] > Task 
:streams:upgrade-system-tests-0110:integrationTest
[2023-03-07T04:34:53.021Z] 
[2023-03-07T04:34:53.021Z] > Task :connect:runtime:integrationTest
[2023-03-07T04:34:53.021Z] 
[2023-03-07T04:34:53.021Z] Gradle Test Run :connect:runtime:integrationTest > 
Gradle Test Executor 133 > 
org.apache.kafka.connect.integration.InternalTopicsIntegrationTest > 
testCreateInternalTopicsWithDefaultSettings PASSED
[2023-03-07T04:34:53.021Z] 
[2023-03-07T04:34:53.021Z] Gradle Test Run :connect:runtime:integrationTest > 
Gradle Test Executor 133 > 
org.apache.kafka.connect.integration.InternalTopicsIntegrationTest > 
testStartWhenInternalTopicsCreatedManuallyWithCompactForBrokersDefaultCleanupPolicy
 STARTED
[2023-03-07T04:34:56.303Z] 
[2023-03-07T04:34:56.303Z] Gradle Test Run :connect:runtime:integrationTest > 
Gradle Test Executor 133 > 
org.apache.kafka.connect.integration.InternalTopicsIntegrationTest > 
testStartWhenInternalTopicsCreatedManuallyWithCompactForBrokersDefaultCleanupPolicy
 PASSED
[2023-03-07T04:34:56.303Z] 
[2023-03-07T04:34:56.304Z] Gradle Test Run :connect:runtime:integrationTest > 
Gradle Test Executor 133 > 
org.apache.kafka.connect.integration.InternalTopicsIntegrationTest > 
testFailToCreateInternalTopicsWithMoreReplicasThanBrokers STARTED
[2023-03-07T04:34:56.304Z] 
[2023-03-07T04:34:56.304Z] > Task 
:streams:upgrade-system-tests-10:integrationTest
[2023-03-07T04:34:56.304Z] > Task 
:streams:upgrade-system-tests-11:integrationTest
[2023-03-07T04:34:58.702Z] 
[2023-03-07T04:34:58.702Z] > Task :connect:runtime:integrationTest
[2023-03-07T04:34:58.702Z] 
[2023-03-07T04:34:58.702Z] Gradle Test Run :connect:runtime:integrationTest > 
Gradle Test Executor 134 > 
org.apache.kafka.connect.integration.RebalanceSourceConnectorsIntegrationTest > 
testDeleteConnector PASSED
[2023-03-07T04:34:58.702Z] 
[2023-03-07T04:34:58.702Z] Gradle Test Run :connect:runtime:integrationTest > 
Gradle Test Executor 134 > 
org.apache.kafka.connect.integration.RebalanceSourceConnectorsIntegrationTest > 
testAddingWorker STARTED
[2023-03-07T04:34:58.702Z] 
[2023-03-07T04:34:58.702Z] Gradle Test Run :connect:runtime:integrationTest > 
Gradle Test Executor 133 > 
org.apache.kafka.connect.integration.InternalTopicsIntegrationTest > 
testFailToCreateInternalTopicsWithMoreReplicasThanBrokers PASSED
[2023-03-07T04:34:58.702Z] 
[2023-03-07T04:34:58.702Z] Gradle Test Run :connect:runtime:integrationTest > 
Gradle Test Executor 133 > 
org.apache.kafka.connect.integration.RestExtensionIntegrationTest > 
testRestExtensionApi STARTED
[2023-03-07T04:35:04.053Z] 
[2023-03-07T04:35:04.053Z] > Task 
:streams:upgrade-system-tests-20:integrationTest
[2023-03-07T04:35:04.053Z] > Task 
:streams:upgrade-system-tests-21:integrationTest
[2023-03-07T04:35:07.334Z] 
[2023-03-07T04:35:07.334Z] > Task :connect:runtime:integrationTest
[2023-03-07T04:35:07.334Z] 
[2023-03-07T04:35:07.334Z] Gradle Test Run :connect:runtime:integrationTest > 
Gradle Test Executor 133 > 
org.apache.kafka.connect.integration.RestExtensionIntegrationTest > 
testRestExtensionApi PASSED
[2023-03-07T04:35:07.334Z] 
[2023-03-07T04:35:07.334Z] Gradle Test Run :connect:runtime:integrationTest > 
Gradle Test Executor 133 > 
org.apache.kafka.connect.integration.SessionedProtocolIntegrationTest > 
ensureInternalEndpointIsSecured STARTED
[2023-03-07T04:35:07.334Z] 
[2023-03-07T04:35:07.334Z] Gradle Test Run :connect:runtime:integrationTest > 
Gradle Test Executor 133 > 
org.apache.kafka.connect.integration.SessionedProtocolIntegrationTest > 
ensureInternalEndpointIsSecured SKIPPED
[2023-03-07T04:35:07.334Z] 
[2023-03-07T04:35:07.334Z] Gradle Test Run :connect:runtime:integrationTest > 
Gradle Test Executor 133 > 
org.apache.kafka.connect.integration.SourceConnectorsIntegrationTest > 
testTopicsAreCreatedWhenTopicCreationIsEnabled STARTED
[2023-03-07T04:35:10.732Z] 
[2023-03-07T04:35:10.732Z] > Task 
:streams:upgrade-system-tests-22:integrationTest
[2023-03-07T04:35:10.732Z] > Task 
:streams:upgrade-system-tests-23:integrationTest
[2023-03-07T04:35:12.312Z] 
[2023-03-07T04:35:12.312Z] > Task :connect:runtime:integrationTest
[2023-03-07T04:35:12.312Z] 
[2023-03-07T04:35:12.312Z] Gradle Test R

Re: [DISCUSS] KIP-842: Add richer group offset reset mechanisms

[hudeqi]

Long time no see, this issue has been discussed for a long time, now please 
allow me to summarize this issue, and then everyone can help to see which 
direction this issue should go in?

There are two problems to be solved by this kip:
1. Solve the problem that when the client configures the "auto.offset.reset" to 
latest, the new partition data may be lost when the consumer resets the offset 
to the latest after expanding the topic partition.

2. In addition to the "earliest", "latest", and "none" provided by the existing 
"auto.offset.reset", it also provides more abundant parameters, such as 
"latest_on_start" (application startup is reset to latest, and an exception is 
thrown if out of range occurs), "earliest_on_start" (application startup is 
reset to earliest, and an exception is thrown if out of range occurs), 
"nearest"(determined by "auto.offset.reset" when the program starts, and choose 
earliest or latest according to the distance between the current offset and log 
start offset and log end offset when out of range occurs).

According to the discussion results of the members above, it seems that there 
are concerns about adding these additional offset reset mechanisms: complexity 
and compatibility. In fact, these parameters do have corresponding benefits. 
Therefore, based on the above discussion results, I have sorted out two 
solution directions. You can help me to see which direction to follow：

1. The first one is to follow Guozhang's suggestion: keep the three parameters 
of "auto.offset.reset" and their meanings unchanged, reduce the confusion for 
Kafka users, and solve the compatibility problem by the way. Add these two 
parameters:
a. "auto.offset.reset.on.no.initial.offse": Indicates the strategy used to 
initialize the offset. The default value is the parameter configured by 
"auto.offset.reset". If so, the strategy for initializing the offset remains 
unchanged from the previous behavior, ensuring compatibility. If the parameter 
is configured with "latest_on_start" or "earliest_on_start", then the offset 
will be reset according to the configured semantics when initializing the 
offset. In this way, the problem of data loss during partition expansion can be 
solved: configure "auto.offset.reset.on.no.initial.offset" to 
"latest_on_start", and configure "auto.offset.reset" to earliest.
b. "auto.offset.reset.on.invalid.offset": Indicates that the offset is 
illegal or out of range occurs. The default value is the parameter configured 
by "auto.offset.reset". If so, the processing of out of range is the same as 
before to ensure compatibility. If "nearest" is configured, then the semantic 
logic corresponding to "nearest" is used only for the case of out of range.

This solution ensures compatibility and ensures that the semantics of the 
original configuration remain unchanged. Only two incremental configurations 
are added to flexibly handle different situations.

2. The second is to directly reduce the complexity of this problem, and 
directly add the logic of resetting the initial offset of the newly expanded 
partition to the earliest to "auto.offset.reset"="latest". In this way, Kafka 
users do not need to perceive this subtle but useful change, and the processing 
of other situations remains unchanged (without considering too many rich offset 
processing mechanisms).

I hope you can help me with the direction of the solution to this issue, thank 
you.

Best,
hudeqi

[jira] [Created] (KAFKA-14787) Supports reading ssl certificate files from hdfs

[melin (Jira)]

melin created KAFKA-14787:
-

 Summary: Supports reading ssl certificate files from hdfs
 Key: KAFKA-14787
 URL: https://issues.apache.org/jira/browse/KAFKA-14787
 Project: Kafka
  Issue Type: Bug
Reporter: melin
 Attachments: image-2023-03-07-15-27-08-077.png

spark/flink runs on hadoop yarn，Supports reading ssl certificate files from 
hdfs，
Paths.get(String) is replaced with Paths.get(uri), which registers the 
HadoopFileSystemProvider and enables NIO to identify the hdfs file path
!image-2023-03-07-15-27-08-077.png!
https://github.com/damiencarol/jsr203-hadoop



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #1654

[Apache Jenkins Server]

See