Re: About Kafka Java Client Producer Retry And Callback

value - use iptables between producer and broker to block the network connectivity 2023年11月12日(日) 18:10 王有胜 : Hi Community, I use Kafka Java Client to send messages asynchronously. I wonder if the producer fails to send a message, during the retry period, will the callback be executed for each ret

Re: About Kafka Java Client Producer Retry And Callback

s to impossibly large value > - use iptables between producer and broker to block the network connectivity > > 2023年11月12日(日) 18:10 王有胜 : > > > Hi Community, I use Kafka Java Client to send messages asynchronously. > > I wonder if the producer fails to send a message, durin

Re: About Kafka Java Client Producer Retry And Callback

e done by e.g.: - set acks=all and set topic's min.insync.replicas to impossibly large value - use iptables between producer and broker to block the network connectivity 2023年11月12日(日) 18:10 王有胜 : > Hi Community, I use Kafka Java Client to send messages asynchronously. > I wonder if t

About Kafka Java Client Producer Retry And Callback

Hi Community, I use Kafka Java Client to send messages asynchronously. I wonder if the producer fails to send a message, during the retry period, will the callback be executed for each retry? I debugged the source code org.apache.kafka.clients.producer.internals.Sender#canRetry and

Leftover bootstrap connection with Kafka Java Client

Dear Kafka Community we are facing an Issue with kafka-console-consumer.sh / kafka-console-producer.sh (which both use the native java client) and the bootstrap behavior in conjunction with haproxy. Behavior: After the bootsrap process the java client doesn't disconnect and just keeps

Leftover bootstrap connection with Kafka Java Client

Dear Kafka Community we are facing an Issue with kafka-console-consumer.sh / kafka-console-producer.sh (which both use the native java client) and the bootstrap behavior in conjunction with haproxy. Behavior: After the bootsrap process the java client doesn't disconnect and just keeps

Re: Transactions and `endOffsets` Java client consumer method

wrote: > > > > > > > > > > > >> Hi Chris, > > > > > >> > > > > > >> The broker does take the isolation level in the ListOffset API > > > (which > > > > is > > > > >

Re: Transactions and `endOffsets` Java client consumer method

tion level in the ListOffset API > > (which > > > is > > > > >> used for the endoffset call) in to consideration: > > > > >> > > > > >> val lastFetchableOffset = isolationLevel match { > > > > >> case Some(IsolationLevel.READ_COMMITTED)

Re: Transactions and `endOffsets` Java client consumer method

> > >> case Some(IsolationLevel.READ_COMMITTED) => > > localLog.lastStableOffset > > > >> case Some(IsolationLevel.READ_UNCOMMITTED) => > localLog.highWatermark > > > >> case None => localLog.logEndOffset > > > >> } >

Re: Transactions and `endOffsets` Java client consumer method

return the last stable offset. > > >> > > >> But this requires the broker version to be newer enough to actually > > >> know the new format of the request, is your broker on the newer > > >> version? Other than that, I cannot think of a reason explaining wh

Re: Transactions and `endOffsets` Java client consumer method

the new format of the request, is your broker on the newer > >> version? Other than that, I cannot think of a reason explaining what > >> you saw. > >> > >> > >> > >> On Wed, Mar 16, 2022 at 5:12 AM Chris Jansen < > chris.jan...@permutiv

Re: Transactions and `endOffsets` Java client consumer method

022 at 5:12 AM Chris Jansen >> wrote: >> >> > Hello all, >> > >> > I have the need to query end offsets for a particular topic using a >> > consumer that has been configured with the "READ_COMMITTED" isolation >> > level. The resp

Re: Transactions and `endOffsets` Java client consumer method

end offsets for a particular topic using a > > consumer that has been configured with the "READ_COMMITTED" isolation > > level. The response I get via the Java client > > includes offsets at the end of the log that have not yet been committed > and > > therefore ca

Re: Transactions and `endOffsets` Java client consumer method

level. The response I get via the Java client > includes offsets at the end of the log that have not yet been committed and > therefore can't be consumed. > > After digging around in the Java client code, it looks like the isolation > level is being transmitted in the API requ

Transactions and `endOffsets` Java client consumer method

Hello all, I have the need to query end offsets for a particular topic using a consumer that has been configured with the "READ_COMMITTED" isolation level. The response I get via the Java client includes offsets at the end of the log that have not yet been committed and therefor

Re: Java client 2.4.1 doesn't always handle IP changes

Checked Java DNS resolution caching: ``` sun.net.InetAddressCachePolicy.get(); sun.net.InetAddressCachePolicy.getNegative(); ``` and those return 30 and 10 respectively. So it seems fine and it shouldn't cache for too long. On 2021/05/14 12:55:00, Michał Łowicki wrote: > Hey, > > Had inciden

Java client 2.4.1 doesn't always handle IP changes

Hey, Had incident where one broker died and got later different IP address. Some clients / pods (everything lives on K8s) detected IP change and logged: [Producer clientId=producer-1] Hostname for node 1 changed from to > . (logged by org.apache.kafka.clients.ClusterConnectionState

Re: thank you ! which java-client api can has same effect about kafka-consumer-groups.sh command ?

-END-OFFSET  >   Corresponding java method  > > > -- 原始邮件 -- > 发件人: "Liam Clarke-Hutchinson" 发送时间: 2020年4月22日(星期三) 下午3:35 > 收件人: "users" > 主题: Re: thank you ! which java-client api can has same effect about > kafka-consume

?????? thank you ! which java-client api can has same effect about kafka-consumer-groups.sh command ?

i use : private static void printConsumerGroupOffsets() throws InterruptedException, ExecutionException { Properties props = new Properties(); props.setProperty("bootstrap.servers", "192.168.1.100:9081,192.168.1.100:9082,192.

Re: thank you ! which java-client api can has same effect about kafka-consumer-groups.sh command ?

Yep, looking at the source code of our app we use to track lag, we're using that method. On Wed, Apr 22, 2020 at 7:35 PM Liam Clarke-Hutchinson < liam.cla...@adscale.co.nz> wrote: > Looking at the source code, try listConsumerGroupOffsets(String > groupId, ListConsumerGroupOffsetsOptions option

Re: thank you ! which java-client api can has same effect about kafka-consumer-groups.sh command ?

Looking at the source code, try listConsumerGroupOffsets(String groupId, ListConsumerGroupOffsetsOptions options) instead? On Wed, Apr 22, 2020 at 6:40 PM 一直以来 <279377...@qq.com> wrote: > ./kafka-consumer-groups.sh --bootstrap-server localhost:9081 --describe > --group test > > > use describeCons

thank you ! which java-client api can has same effect about kafka-consumer-groups.sh command ?

./kafka-consumer-groups.sh --bootstrap-server localhost:9081 --describe --group test use describeConsumerGroups method ?? private static void print() throws InterruptedException, ExecutionException { Properties props = new Properties(); props.setPropert

Kafka Java Client not support --unavailable-partitions and --topics-with-overrides param?

Kafka Java Client not support --unavailable-partitions and --topics-with-overrides param?

Spike in Metarequest call timings in Java client

Hi There, We see spikes in the metadata request calls when a broker machine specified in the broker list goes down i.e server not reachable. Is there a property which can be set in the Java client to control the timeout when a broker is not reachable and try another one in the list. We are

Tracing instrumentation for Kafka Java client

Hi, I'm trying to implement Tracing (http://opentracing.io/) for Kafka java client. For that I need to send some metadata from producer to consumer. I use for that Key (my implementation: https://github.com/malafeev/opentracing-java-kafka). My Custom partitioner skips metadata when it calcu

Re: Java Client for Kafka Consumer(0.9) not polling the records from brokers instantly

Don't think you cannot attach diagrams in these mailling lists. On Thu, May 26, 2016 at 7:35 PM, Navneet Kumar wrote: > Hi > We are facing a issue where our Consumer component is not instantly > logging the records in DB polled from the Brokers. We have following > theOur architecture diagram is

Re: Java Client for Kafka Consumer(0.9) not polling the records from brokers instantly

Hi We are facing a issue where our Consumer component is not instantly logging the records in DB polled from the Brokers. We have following theOur architecture diagram is attached herewith. Following are the properties configured: Producer.properties bootstrap.servers=xx.xxx.xxx.140:9092,xx.xxx.x

Java Client for Kafka Consumer(0.9) not polling the records from brokers instantly

Hi We are facing a issue where our Consumer component is not instantly logging the records polled from the Brokers. We have following the below architecture as attached. Following are the properties configured: Producer.properties bootstrap.servers=xx.xxx.xxx.140:9092,xx.xxx.xxx.140:9093,xx.xxx.x

consumer (java client 2.11 0.9.0.1) poll does not return record in between

I am using 0.9.0.1 JAVA KafkaCosnumer poll API with timeout as long as 1000ms. I am connecting to a topic with 7 million messages. The message is FIX message. When I poll - I see that in between it returns blank records. Why is that happening. Is there a possibility that poll API is returning the

Re: Question on Java client consumer.seek()

Sounds reasonable to me. On 04/05/2016 05:56 PM, Mario Ricci wrote: > I found this > thread > and see that poll() must be called before seek(). > > This se

Question on Java client consumer.seek()

I found this thread and see that poll() must be called before seek(). This seems unintuitive and the error message ("No current assignment for partition pa

RE: 0.9.0.1 Kafka Java client Producer hangs while requesting metadata

: Wednesday, March 02, 2016 11:28 PM To: users@kafka.apache.org Subject: 0.9.0.1 Kafka Java client Producer hangs while requesting metadata Hi, Trying a very simple Producer with following code. "producer.send" hangs indefinitely. Attaching thread-dump snippet so that I can get some

0.9.0.1 Kafka Java client Producer hangs while requesting metadata

Hi, Trying a very simple Producer with following code. "producer.send" hangs indefinitely. Attaching thread-dump snippet so that I can get some advice if something is wrong with my code or configuration. Kafka runs in a VM and producer runs in host - it's a single-broker setup for basic testin

Re: Java Client connection errors with Kafka 0.9.0.0 when SSL is enabled

apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.java:408) > > at > > > > org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:269) > > at > > > org.apache.kafka.common.netwo

Re: Java Client connection errors with Kafka 0.9.0.0 when SSL is enabled

aChannel.java:68) > at org.apache.kafka.common.network.Selector.poll(Selector.java:281) > at kafka.network.Processor.run(SocketServer.scala:413) > at java.lang.Thread.run(Thread.java:722) > > Producer Java client code: > System.setProperty("javax.net.debug",&qu

Java Client connection errors with Kafka 0.9.0.0 when SSL is enabled

a:722) Producer Java client code: System.setProperty("javax.net.debug","ssl:handshake:verbose"); Properties props = new Properties(); props.put("bootstrap.servers", ".com:9093"); props.put(&quo

Re: Java Client connection errors with Kafka 0.9.0.0 when SSL is enabled

curity.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:845) >>> at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:758) >>> at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) >>> at >> org.apache.kafka.common.network.SslTransportLayer.handshakeUnwrap(SslTransportLayer.j

Re: Java Client connection errors with Kafka 0.9.0.0 when SSL is enabled

ansportLayer.handshakeUnwrap(SslTransportLayer.java:408) > > at > > > org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:269) > > at > > > org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:68) > &

Re: Java Client connection errors with Kafka 0.9.0.0 when SSL is enabled

sportLayer.handshake(SslTransportLayer.java:269) > at > org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:68) > at org.apache.kafka.common.network.Selector.poll(Selector.java:281) > at kafka.network.Processor.run(SocketServer.scala:413) > at java.lang.Thread.run(Thread.java

Re: Java Client connection errors with Kafka 0.9.0.0 when SSL is enabled

yer.handshakeUnwrap(SslTransportLayer.java:408) > at > org.apache.kafka.common.network.SslTransportLayer.handshake(SslTransportLayer.java:269) > at > org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:68) > at org.apache.kafka.common.network.Selector.poll(Selector.java:281) > a

Re: Java client 0.9 poll doesn't return

Looks like you might have bootstrap.servers pointed at Zookeeper. It should point to the Kafka brokers instead. The behavior of poll() currently is to block until the group's coordinator is found, but sending the wrong kind of request to Zookeeper probably results in a server-side disconnect. In th

Java client 0.9 poll doesn't return

Calling the stand alone client (org.apache.kafka" % "kafka-clients" % “0.9.0.0”) from Scala, consumer.poll never returns. I’ve tried both assign TopicPartition and subscribe and various timeouts and I’ve quintuple checked config properties. Here’s a Scala-Ide worksheet val props = loadProper

Producing messages to secured kafka from different node java client in a cluster

Hi all Trying to produce message to secured HDP2.3 kakfa broker from different node using a java producer client but facing below issue any ideas how to configure Kerberos so that I can produce messages to kafka. Thank you. [2015-11-06 07:18:57,435] INFO Closing socket connection to /10.0.2.17.

Re: Getting started with Kafka using Java client

te: > > Hi All, > > Can someone please point me to a well documented github code example for > using Kafka 's java client. So far I haven't been able to find much related > to using Java client. Most of the examples are using command prompt for > producing and consumin

Getting started with Kafka using Java client

Hi All, Can someone please point me to a well documented github code example for using Kafka 's java client. So far I haven't been able to find much related to using Java client. Most of the examples are using command prompt for producing and consuming messages. It's a bit frustrat

Exception in logs, on latest Kafka Java client - 0.8.2.1

>2015-08-20 16:55:43,242 [kafka-producer-network-thread | producer-1] DEBUG >o.apache.kafka.clients.NetworkClient - Trying to send metadata request to node >0 >2015-08-20 16:55:43,246 [kafka-producer-network-thread | producer-1] DEBUG >o.apache.kafka.clients.NetworkClient - Init connection to nod

Re: Message corruption with new Java client + snappy + broker restart

th > > org.apache.kafka.common.errors.UnknownServerException (at least when run > as > > ThreadJob). I saw this with error #3 above. > > > > org.apache.samza.SamzaException: Unable to send message from > > TaskName-Partition 6 to system kafka. > > org.apache.ka

Re: Message corruption with new Java client + snappy + broker restart

message from > TaskName-Partition 6 to system kafka. > org.apache.kafka.common.errors.UnknownServerException: The server > experienced an unexpected error when processing the request > > There seem to be two issues here: > > 1) When leadership for a topic is transferred to another broker,

Re: Consumer in Java client

day, March 30, 2015 11:06 PM > To: users@kafka.apache.org > Subject: Re: Consumer in Java client > > Hi, > > KafkaConsumer is still under development and not ready for wide use case. > Currently, it can be used to replace SimpleConsumer (low level consumer), > but can not repla

RE: Consumer in Java client

: Monday, March 30, 2015 11:06 PM To: users@kafka.apache.org Subject: Re: Consumer in Java client Hi, KafkaConsumer is still under development and not ready for wide use case. Currently, it can be used to replace SimpleConsumer (low level consumer), but can not replace ZookeeperConsumerConnector(high

Re: Consumer in Java client

Hi, KafkaConsumer is still under development and not ready for wide use case. Currently, it can be used to replace SimpleConsumer (low level consumer), but can not replace ZookeeperConsumerConnector(high level consumer). So if you need to use simple consumer, I would suggest using KafkaConsumer in

Consumer in Java client

Hi I'm new to kafka and using 0.8.2.1 kafka-clients and kafka_2.10 packages. The document says: “We are in the process of rewritting the JVM clients for Kafka. As of 0.8.2 Kafka includes a newly rewritten Java producer. The next release will include an equivalent Java consumer.” It seems the kafka

Re: can't connect to kafka from a java client.

Just for your information, I faced same issue. For me, it was origin from the hostname of Kafka node. Since the hostname of broker stored in Zookeeper was different with the hostname what I configured in source, it couldn`t fetch the meta data of brokers. If you configure the log level as debug,

Re: can't connect to kafka from a java client.

zk.connect is needed in Kafka 0.7. Since you are using Kafka 0.8, you need to set metadata.broker.list. ZK is no longer needed in the producer in 0.8. You can follow http://kafka.apache.org/documentation.html for 0.8 documentation. The latest release in maven is 0.8.0. We publish different kafka j

can't connect to kafka from a java client.

Goal: I am trying to send messages to kafka from a java cleint. And it has been a pain.. Let me describe in brief. 1. I have installed kafka on a centos VM. 2. I ran the zookeeper that comes with it, the server , the producer and the client with all of the default properties files. I sent and

Re: Use cases for new java client

aught our > interest was the new java client, which sounds like it will super useful. > > We have some client code which sits on top of SimpleConsumer which we use > for cases where we want to manage our own offsets, but also for cases where > we don't care about persistent offsets at

Use cases for new java client

Hi - Recently Mr. Kreps was kind enough to answer a bunch of questions from some of us here at Jive software. One subject that particularly caught our interest was the new java client, which sounds like it will super useful. We have some client code which sits on top of SimpleConsumer which we

Re: found that the producer called localhost: 9092 in Kafka of Java client.

%27tmyhighlevelconsumersconnecttothebrokers%3F ). Thanks, Jun On Tue, Jul 23, 2013 at 8:43 PM, yanbo.ai wrote: > Hi， > > I'm using Kafka of Java client to send events to my data center, but found > that the producer called localhost: 9092. > Actually , my broker list doesn

found that the producer called localhost: 9092 in Kafka of Java client.

Hi， I'm using Kafka of Java client to send events to my data center, but found that the producer called localhost: 9092. Actually , my broker list doesn't include localhost. why ? Thanks. Best Regards, yanbo.ai

Re: java client

We haven't really measured the scala overhead. However, my guess is that it should be small compared to the network overhead, which one has to pay for each request. Thanks, Jun On Fri, Dec 28, 2012 at 7:04 PM, S Ahmed wrote: > Since kafka is written in scala, when using a jav