date:20231211

FW: Issue with "connections.max.idle.ms" property defined for admin client config in apache kafka version 3.3.1

2023-12-11 Thread Ankit Nigam


Hi Team,

We are using Apache Kafka 3.3.1 in our application.

Scenario - 1 --> We have created an Kafka admin client from our java 
application and have not configured the property "connections.max.idle.ms" so 
its default value which is 5 minutes is used.

In the above scenario we see a port being established every 5 mins towards 
Kafka and an old port being dropped.

9:46 :30 --> process starts.. using default property

Every 1.0s: netstat -plant | grep 9092 | grep 3455069   

   seliiuvd07797: Thu Dec  7 09:47:16 2023

(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp6   0  0 127.0.0.1:46182 127.0.0.1:9092  ESTABLISHED 
3455069/java


9:51:00 ->

Every 1.0s: netstat -plant | grep 9092 | grep 3455069   

   seliiuvd07797: Thu Dec  7 09:51:38 2023

(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp6   0  0 127.0.0.1:36866 127.0.0.1:9092  ESTABLISHED 
3455069/java

9:59:00  ->

Every 1.0s: netstat -plant | grep 9092 | grep 3455069   

   seliiuvd07797: Thu Dec  7 09:59:05 2023

(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp6   0  0 127.0.0.1:51096 127.0.0.1:9092  ESTABLISHED 
3455069/java


10:01:00 >

Every 1.0s: netstat -plant | grep 9092 | grep 3455069   

   seliiuvd07797: Thu Dec  7 10:01:40 2023

(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp6   0  0 127.0.0.1:38202 127.0.0.1:9092  ESTABLISHED 
3455069/java


10:06:32 >

Every 1.0s: netstat -plant | grep 9092 | grep 3455069   

   seliiuvd07797: Thu Dec  7 10:06:32 2023

(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp6   0  0 127.0.0.1:50450 127.0.0.1:9092  ESTABLISHED 
3455069/java



Scenario - 2 --> However, if we create a Apache Kafka admin client using the 
property "connections.max.idle.ms" to a value greater than 5 mins. for eg. 10 
mins.

Initially when our process starts (Process started at 18:19:10) we have one 
connection established with the Apache Kafka broker as shown below.

Time : 18:19:10

Every 1.0s: netstat -plant | grep 9092 | grep 2860979   

   seliiuvd07797: Wed Dec  6 18:19:35 2023

(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp6   0  0 127.0.0.1:44280 127.0.0.1:9092  ESTABLISHED 
2860979/java

After 5 minutes, we observed one more connection was established ( though we 
only created one single admin client from the code. so this connection is being 
established internally by Apache Kafka)

Time : 18:24:10

(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp6   0  0 127.0.0.1:48526 127.0.0.1:9092  ESTABLISHED 
2860979/java
tcp6   0  0 127.0.0.1:44280 127.0.0.1:9092  ESTABLISHED 
2860979/java

After further 5 minutes, i.e. 10 minutes since the process started ( 18:29 
Hrs.) , we observed that connection from the old port was dropped.

Time : 18:29:10
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp6   0  0 127.0.0.1:48526 127.0.0.1:9092  ESTABLISHED 
2860979/java

After this there were no further disconnections or new connections being 
established.

So the question basically is why Kafka admin client not getting disconnected 
every 10 minutes which was observed when the default value of 
"connections.max.idle.ms = 30" property is being used for creating admin 
client.?
Secondly why a new connection in Scenario 2 was established at the end of 5th 
minutes as shown above??


Regards,
Ankit Nigam

[ANNOUNCE] Apache Kafka 3.5.2

2023-12-11 Thread Luke Chen

The Apache Kafka community is pleased to announce the release for
Apache Kafka 3.5.2

This is a bugfix release. It contains many bug fixes including
upgrades the Snappy and Rocksdb dependencies.

All of the changes in this release can be found in the release notes:
https://www.apache.org/dist/kafka/3.5.2/RELEASE_NOTES.html


You can download the source and binary release from:
https://kafka.apache.org/downloads#3.5.2

---


Apache Kafka is a distributed streaming platform with four core APIs:


** The Producer API allows an application to publish a stream of records to
one or more Kafka topics.

** The Consumer API allows an application to subscribe to one or more
topics and process the stream of records produced to them.

** The Streams API allows an application to act as a stream processor,
consuming an input stream from one or more topics and producing an
output stream to one or more output topics, effectively transforming the
input streams to output streams.

** The Connector API allows building and running reusable producers or
consumers that connect Kafka topics to existing applications or data
systems. For example, a connector to a relational database might
capture every change to a table.


With these APIs, Kafka can be used for two broad classes of application:

** Building real-time streaming data pipelines that reliably get data
between systems or applications.

** Building real-time streaming applications that transform or react
to the streams of data.


Apache Kafka is in use at large and small companies worldwide, including
Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest, Rabobank,
Target, The New York Times, Uber, Yelp, and Zalando, among others.

A big thank you for the following contributors to this release!

A. Sophie Blee-Goldman, atu-sharm, bachmanity1, Calvin Liu, Chase
Thomas, Chris Egerton, Colin Patrick McCabe, David Arthur, Divij
Vaidya, Federico Valeri, flashmouse, Florin Akermann, Greg Harris,
hudeqi, José Armando García Sancio, Levani Kokhreidze, Lucas Brutschy,
Luke Chen, Manikumar Reddy, Matthias J. Sax, Mickael Maison, Nick
Telford, Okada Haruki, Omnia G.H Ibrahim, Robert Wagner, Rohan, Said
Boudjelda, sciclon2, Vincent Jiang, Xiaobing Fang, Yash Mayya

We welcome your help and feedback. For more information on how to
report problems, and to get involved, visit the project website at
https://kafka.apache.org/

Thank you!

Regards,
Luke

Re: [ANNOUNCE] Apache Kafka 3.5.2

2023-12-11 Thread Josep Prat

Thanks Luke for running the release!

Best!

On Mon, Dec 11, 2023 at 12:34 PM Luke Chen  wrote:

> The Apache Kafka community is pleased to announce the release for
> Apache Kafka 3.5.2
>
> This is a bugfix release. It contains many bug fixes including
> upgrades the Snappy and Rocksdb dependencies.
>
> All of the changes in this release can be found in the release notes:
> https://www.apache.org/dist/kafka/3.5.2/RELEASE_NOTES.html
>
>
> You can download the source and binary release from:
> https://kafka.apache.org/downloads#3.5.2
>
>
> ---
>
>
> Apache Kafka is a distributed streaming platform with four core APIs:
>
>
> ** The Producer API allows an application to publish a stream of records to
> one or more Kafka topics.
>
> ** The Consumer API allows an application to subscribe to one or more
> topics and process the stream of records produced to them.
>
> ** The Streams API allows an application to act as a stream processor,
> consuming an input stream from one or more topics and producing an
> output stream to one or more output topics, effectively transforming the
> input streams to output streams.
>
> ** The Connector API allows building and running reusable producers or
> consumers that connect Kafka topics to existing applications or data
> systems. For example, a connector to a relational database might
> capture every change to a table.
>
>
> With these APIs, Kafka can be used for two broad classes of application:
>
> ** Building real-time streaming data pipelines that reliably get data
> between systems or applications.
>
> ** Building real-time streaming applications that transform or react
> to the streams of data.
>
>
> Apache Kafka is in use at large and small companies worldwide, including
> Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest, Rabobank,
> Target, The New York Times, Uber, Yelp, and Zalando, among others.
>
> A big thank you for the following contributors to this release!
>
> A. Sophie Blee-Goldman, atu-sharm, bachmanity1, Calvin Liu, Chase
> Thomas, Chris Egerton, Colin Patrick McCabe, David Arthur, Divij
> Vaidya, Federico Valeri, flashmouse, Florin Akermann, Greg Harris,
> hudeqi, José Armando García Sancio, Levani Kokhreidze, Lucas Brutschy,
> Luke Chen, Manikumar Reddy, Matthias J. Sax, Mickael Maison, Nick
> Telford, Okada Haruki, Omnia G.H Ibrahim, Robert Wagner, Rohan, Said
> Boudjelda, sciclon2, Vincent Jiang, Xiaobing Fang, Yash Mayya
>
> We welcome your help and feedback. For more information on how to
> report problems, and to get involved, visit the project website at
> https://kafka.apache.org/
>
> Thank you!
>
> Regards,
> Luke
>


-- 
[image: Aiven] 

*Josep Prat*
Open Source Engineering Director, *Aiven*
josep.p...@aiven.io   |   +491715557497
aiven.io    |   
     
*Aiven Deutschland GmbH*
Alexanderufer 3-7, 10117 Berlin
Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
Amtsgericht Charlottenburg, HRB 209739 B

Re: [ANNOUNCE] Apache Kafka 3.5.2

2023-12-11 Thread Mickael Maison

Thanks for running the release, and thanks to all the contributors!

Mickael

On Mon, Dec 11, 2023 at 1:56 PM Josep Prat  wrote:
>
> Thanks Luke for running the release!
>
> Best!
>
> On Mon, Dec 11, 2023 at 12:34 PM Luke Chen  wrote:
>
> > The Apache Kafka community is pleased to announce the release for
> > Apache Kafka 3.5.2
> >
> > This is a bugfix release. It contains many bug fixes including
> > upgrades the Snappy and Rocksdb dependencies.
> >
> > All of the changes in this release can be found in the release notes:
> > https://www.apache.org/dist/kafka/3.5.2/RELEASE_NOTES.html
> >
> >
> > You can download the source and binary release from:
> > https://kafka.apache.org/downloads#3.5.2
> >
> >
> > ---
> >
> >
> > Apache Kafka is a distributed streaming platform with four core APIs:
> >
> >
> > ** The Producer API allows an application to publish a stream of records to
> > one or more Kafka topics.
> >
> > ** The Consumer API allows an application to subscribe to one or more
> > topics and process the stream of records produced to them.
> >
> > ** The Streams API allows an application to act as a stream processor,
> > consuming an input stream from one or more topics and producing an
> > output stream to one or more output topics, effectively transforming the
> > input streams to output streams.
> >
> > ** The Connector API allows building and running reusable producers or
> > consumers that connect Kafka topics to existing applications or data
> > systems. For example, a connector to a relational database might
> > capture every change to a table.
> >
> >
> > With these APIs, Kafka can be used for two broad classes of application:
> >
> > ** Building real-time streaming data pipelines that reliably get data
> > between systems or applications.
> >
> > ** Building real-time streaming applications that transform or react
> > to the streams of data.
> >
> >
> > Apache Kafka is in use at large and small companies worldwide, including
> > Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest, Rabobank,
> > Target, The New York Times, Uber, Yelp, and Zalando, among others.
> >
> > A big thank you for the following contributors to this release!
> >
> > A. Sophie Blee-Goldman, atu-sharm, bachmanity1, Calvin Liu, Chase
> > Thomas, Chris Egerton, Colin Patrick McCabe, David Arthur, Divij
> > Vaidya, Federico Valeri, flashmouse, Florin Akermann, Greg Harris,
> > hudeqi, José Armando García Sancio, Levani Kokhreidze, Lucas Brutschy,
> > Luke Chen, Manikumar Reddy, Matthias J. Sax, Mickael Maison, Nick
> > Telford, Okada Haruki, Omnia G.H Ibrahim, Robert Wagner, Rohan, Said
> > Boudjelda, sciclon2, Vincent Jiang, Xiaobing Fang, Yash Mayya
> >
> > We welcome your help and feedback. For more information on how to
> > report problems, and to get involved, visit the project website at
> > https://kafka.apache.org/
> >
> > Thank you!
> >
> > Regards,
> > Luke
> >
>
>
> --
> [image: Aiven] 
>
> *Josep Prat*
> Open Source Engineering Director, *Aiven*
> josep.p...@aiven.io   |   +491715557497
> aiven.io    |   
>      
> *Aiven Deutschland GmbH*
> Alexanderufer 3-7, 10117 Berlin
> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> Amtsgericht Charlottenburg, HRB 209739 B

Extend DefaultReplicationPolicy for MirrorSourceConnector / MirrorMaker

2023-12-11 Thread astronaut37

Hi Guys,
we have an already running Kakfa Connector Cluster.
I want to add a MirrorSourceConnector with a custom replication policy:
-
curl --request POST \
--url http://localhost:9083/connectors \
--header 'Accept: application/json' \
--header 'Content-Type: application/json' \
--data '{
"name" : "cluster-source",
"config":{
"connector.class": "org.apache.kafka.connect.mirror.MirrorSourceConnector",
...
"replication.policy.class" : "org.example.CustomReplicationPolicy",
...
}
}'
-
This leads to an error:
-
{
"error_code": 400,
"message": "Connector configuration is invalid and contains the following 1 
error(s):\nInvalid value org.example.CustomReplicationPolicy for configuration 
replication.policy.class: Class org.example.CustomReplicationPolicy could not 
be found.\nYou can also find the above list of errors at the endpoint 
`/connector-plugins/{connectorType}/config/validate`"
}
-
I added the jar which contains CustomReplicationPolicy to the /lib path but it 
won't work.

How can I use my CustomReplicationPolicy with the MirrorSourceConnector?
I'm using Kafka 2.8.
Setting up the MirrorMaker with /bin/connect-mirror-maker.sh and the jar file 
in /lib works, but unfortunately I have to configure the mirror maker 
connectors one by one in our enviroment.

BRToni

[DISCUSS] KIP-1010: Topic Partition Quota

2023-12-11 Thread Afshin Moazami

Hi folks,
I would like to propose a new feature to extend the quota management in
Kafka to support topic-partition based quotas. The following is the link to
the KIP
https://cwiki.apache.org/confluence/display/KAFKA/KIP-1010%3A+Topic+Partition+Quota


Best,
Afshin Moazami

Re: [ANNOUNCE] Apache Kafka 3.5.2

2023-12-11 Thread Dima Brodsky

Hello,

Is there a "git" issue with 3.5.2.  When I look at github I see the 3.5.2
tag.  But if I make the repo an upstream remote target I don't see 3.5.2.
Any ideas what could be up?

Thanks!
ttyl
Dima


On Mon, Dec 11, 2023 at 3:36 AM Luke Chen  wrote:

> The Apache Kafka community is pleased to announce the release for
> Apache Kafka 3.5.2
>
> This is a bugfix release. It contains many bug fixes including
> upgrades the Snappy and Rocksdb dependencies.
>
> All of the changes in this release can be found in the release notes:
> https://www.apache.org/dist/kafka/3.5.2/RELEASE_NOTES.html
>
>
> You can download the source and binary release from:
> https://kafka.apache.org/downloads#3.5.2
>
>
> ---
>
>
> Apache Kafka is a distributed streaming platform with four core APIs:
>
>
> ** The Producer API allows an application to publish a stream of records to
> one or more Kafka topics.
>
> ** The Consumer API allows an application to subscribe to one or more
> topics and process the stream of records produced to them.
>
> ** The Streams API allows an application to act as a stream processor,
> consuming an input stream from one or more topics and producing an
> output stream to one or more output topics, effectively transforming the
> input streams to output streams.
>
> ** The Connector API allows building and running reusable producers or
> consumers that connect Kafka topics to existing applications or data
> systems. For example, a connector to a relational database might
> capture every change to a table.
>
>
> With these APIs, Kafka can be used for two broad classes of application:
>
> ** Building real-time streaming data pipelines that reliably get data
> between systems or applications.
>
> ** Building real-time streaming applications that transform or react
> to the streams of data.
>
>
> Apache Kafka is in use at large and small companies worldwide, including
> Capital One, Goldman Sachs, ING, LinkedIn, Netflix, Pinterest, Rabobank,
> Target, The New York Times, Uber, Yelp, and Zalando, among others.
>
> A big thank you for the following contributors to this release!
>
> A. Sophie Blee-Goldman, atu-sharm, bachmanity1, Calvin Liu, Chase
> Thomas, Chris Egerton, Colin Patrick McCabe, David Arthur, Divij
> Vaidya, Federico Valeri, flashmouse, Florin Akermann, Greg Harris,
> hudeqi, José Armando García Sancio, Levani Kokhreidze, Lucas Brutschy,
> Luke Chen, Manikumar Reddy, Matthias J. Sax, Mickael Maison, Nick
> Telford, Okada Haruki, Omnia G.H Ibrahim, Robert Wagner, Rohan, Said
> Boudjelda, sciclon2, Vincent Jiang, Xiaobing Fang, Yash Mayya
>
> We welcome your help and feedback. For more information on how to
> report problems, and to get involved, visit the project website at
> https://kafka.apache.org/
>
> Thank you!
>
> Regards,
> Luke
>


-- 
ddbrod...@gmail.com

"The price of reliability is the pursuit of the utmost simplicity.
It is a price which the very rich find the most hard to pay."
   (Sir
Antony Hoare, 1980)

FW: Issue with "connections.max.idle.ms" property defined for admin client config in apache kafka version 3.3.1

[ANNOUNCE] Apache Kafka 3.5.2

Re: [ANNOUNCE] Apache Kafka 3.5.2

Re: [ANNOUNCE] Apache Kafka 3.5.2

Extend DefaultReplicationPolicy for MirrorSourceConnector / MirrorMaker

[DISCUSS] KIP-1010: Topic Partition Quota

Re: [ANNOUNCE] Apache Kafka 3.5.2

7 matches

Site Navigation

Mail list logo

Footer information