> My main concern is that we don't do the migration in 0.8.3, we will be
left
with some metrics in YM format and some others in KM format (as we start
sharing client code on the broker). This is probably a worse situation to
be in.
+1. I am not sure how our servo adaptor will work if there are two
Here is the resource leak problem that we have encountered when 0.8.2 java
KafkaProducer failed in constructor. here is the code snippet of
KafkaProducer to illustrate the problem.
---
public KafkaProducer(ProducerConfig config, Serializer keySerializer,
Serializer valu
some fields haven't been
> > initialized, which presumably just means checking for null fields. (You
> > might also want to figure out if all the methods close() calls are
> > idempotent and decide whether some fields should be marked non-final and
> > cleared to nul
I submitted a patch attempt in the jira.
On Tue, Apr 14, 2015 at 10:16 AM, Steven Wu wrote:
> Thanks, Ewen and Guozhang!
>
> I will go with the try-catch option then. here is the jira. feel free to
> assign it to me. I will try to submit a patch this week.
> https://issues.
PRE-CREATION
Diff: https://reviews.apache.org/r/33242/diff/
Testing
---
Thanks,
Steven Wu
.java
b91e2c52ed0acb1faa85915097d97bafa28c413a
clients/src/test/java/org/apache/kafka/clients/producer/KafkaProducerTest.java
PRE-CREATION
Diff: https://reviews.apache.org/r/33242/diff/
Testing
---
Thanks,
Steven Wu
org/apache/kafka/clients/producer/KafkaProducer.java
b91e2c52ed0acb1faa85915097d97bafa28c413a
clients/src/test/java/org/apache/kafka/clients/producer/KafkaProducerTest.java
PRE-CREATION
Diff: https://reviews.apache.org/r/33242/diff/
Testing
---
Thanks,
Steven Wu
ducer/KafkaProducer.java
b91e2c52ed0acb1faa85915097d97bafa28c413a
clients/src/test/java/org/apache/kafka/clients/producer/KafkaProducerTest.java
PRE-CREATION
Diff: https://reviews.apache.org/r/33242/diff/
Testing
---
Thanks,
Steven Wu
zer.
Sender and Metrics classes are probably only used internally. let me know your
thoughts.
- Steven
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33242/#review80346
---
On April 16, 20
/test/java/org/apache/kafka/clients/producer/KafkaProducerTest.java
> PRE-CREATION
>
> Diff: https://reviews.apache.org/r/33242/diff/
>
>
> Testing
> ---
>
>
> Thanks,
>
> Steven Wu
>
>
/apache/kafka/clients/producer/KafkaProducerTest.java
PRE-CREATION
Diff: https://reviews.apache.org/r/33242/diff/
Testing
---
Thanks,
Steven Wu
e idea for making this less verbose and redundant: make all of these
> > classes implement Closeable so we can just write one utility method for
> > trying to close something and catching the exception.
>
> Steven Wu wrote:
> yes. I thought about it. it may break binary compa
/clients/producer/KafkaProducerTest.java
PRE-CREATION
Diff: https://reviews.apache.org/r/33242/diff/
Testing
---
Thanks,
Steven Wu
expanded scope.
- Steven
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33242/#review80642
---
On April 20, 2015, 3:30 a.m.,
-CREATION
Diff: https://reviews.apache.org/r/33242/diff/
Testing
---
Thanks,
Steven Wu
/KafkaProducerTest.java
PRE-CREATION
Diff: https://reviews.apache.org/r/33242/diff/
Testing
---
Thanks,
Steven Wu
/KafkaConsumerTest.java
PRE-CREATION
clients/src/test/java/org/apache/kafka/clients/producer/KafkaProducerTest.java
PRE-CREATION
Diff: https://reviews.apache.org/r/33242/diff/
Testing
---
Thanks,
Steven Wu
/KafkaConsumerTest.java
PRE-CREATION
clients/src/test/java/org/apache/kafka/clients/producer/KafkaProducerTest.java
PRE-CREATION
Diff: https://reviews.apache.org/r/33242/diff/
Testing
---
Thanks,
Steven Wu
/KafkaConsumerTest.java
PRE-CREATION
clients/src/test/java/org/apache/kafka/clients/producer/KafkaProducerTest.java
PRE-CREATION
Diff: https://reviews.apache.org/r/33242/diff/
Testing
---
Thanks,
Steven Wu
,
Steven Wu
test
- Steven
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33242/#review80892
---
On April 21, 2015, 5:48 a.m.,
/producer/KafkaProducerTest.java
49f1427bcbe43c773920a25aa69a71d0329296b7
clients/src/test/java/org/apache/kafka/test/MockMetricsReporter.java
6f948f240c906029a0f972bf770f288f390ea714
Diff: https://reviews.apache.org/r/33574/diff/
Testing
---
Thanks,
Steven Wu
On Tue, Apr 28, 2015 at 1:03 PM, Ewen Cheslack-Postava
wrote:
> Good point Jay. More specifically we were already implementing without the
> checked exception, we'd need to override close() in the Serializer and
> Deserializer interfaces and omit the throws clause. That definitely makes
> them so
sorry for the previous empty msg.
Jay's idea should work. basically, we override the close method in
Serializer interface.
public interface Serializer extends Closeable {
@Override
public void close();
}
On Tue, Apr 28, 2015 at 1:10 PM, Steven Wu wrote:
>
>
> On Tue, Apr
/diff/
Testing
---
Thanks,
Steven Wu
confusing. So, it seems the original code is
> probably better.
>
> Thanks,
>
> Jun
>
> On Tue, Apr 28, 2015 at 3:11 PM, Steven Wu wrote:
>
> > sorry for the previous empty msg.
> >
> > Jay's idea should work. basically, we override the close method i
6f948f240c906029a0f972bf770f288f390ea714
clients/src/test/java/org/apache/kafka/test/MockSerializer.java PRE-CREATION
Diff: https://reviews.apache.org/r/33760/diff/
Testing
---
Thanks,
Steven Wu
ache.org/r/33760/diff/
Testing
---
Thanks,
Steven Wu
/kafka/clients/producer/KafkaProducer.java
<https://reviews.apache.org/r/30158/#comment114215>
I would not use initialized flag, as long as we fix
KafkaProducer#waitOnMetadata to allow value 0 for non-blocking.
- Steven Wu
On Jan. 22, 2015, 7:04 a.m., Paul
To illustrate my point, I will use "allTopicsOwnedPartitionsCount" guage
from ZookeeperConsumerConnector as an example. It captures number of
partitions for a topic that has been assigned owner for the consumer group.
let's say that I have a topic with 9 partitions. this metrics should
normally re
re owned. This concern can be
> addressed with the new consumer.
>
> On Tue, Jan 27, 2015 at 03:20:55PM -0800, Steven Wu wrote:
> > To illustrate my point, I will use "allTopicsOwnedPartitionsCount" guage
> > from ZookeeperConsumerConnector as an example. It captur
In Netflix, we have been using route53 DNS name as bootstrap servers in AWS
env. Basically, when a kafka broker start, we add it to route53 DNS name
for the cluster. this is like the VIP that Jay suggested.
But we are also moving toward to use Eureka service registry for
bootstrapping. We are worr
Jay, we have observed CRC corruption too occasionally. I reported in an
thread and asked how should we handle some error conditions from old
high-level consumer.
On Mon, Feb 9, 2015 at 11:36 PM, Bhavesh Mistry
wrote:
> Hi Jay,
>
> 1) Sorry to get back to you so late. It is CRC check error on an
please correct me if I am missing sth here. I am not understanding how
would throttle work without cooperation/back-off from producer. new Java
producer supports non-blocking API. why would delayed response be able to
slow down producer? producer will continue to fire async sends.
On Mon, Mar 16,
I think I can answer my own question. delayed response will cause the
producer buffer to be full, which then result in either thread blocking or
message drop.
On Mon, Mar 16, 2015 at 11:24 PM, Steven Wu wrote:
> please correct me if I am missing sth here. I am not understanding how
>
I am chatting to myself :)
On Mon, Mar 16, 2015 at 11:29 PM, Steven Wu wrote:
> I think I can answer my own question. delayed response will cause the
> producer buffer to be full, which then result in either thread blocking or
> message drop.
>
> On Mon, Mar 16, 2015 at 11:24
; > One additional point -- I think "malicious clients" shouldn't be our
> target
> > here, they can do a lot worse than what's been addressed in this thread.
> > But I do agree that any proposal should have a clear explanation of how
> > existing clients
the limits of the server and experience
> backpressure. Quotas just allow you to set that same limit at something
> lower than 100% of all resources on the server, which is useful for a
> shared cluster.
>
> -Jay
>
> On Mon, Mar 16, 2015 at 11:34 PM, Steven Wu wrote:
>
> > wait. w
of those instances
> added up across all their topics. So actually the effect would be the same
> either way but throttling gives the producer the choice of either blocking
> or dropping.
>
> -Jay
>
> On Tue, Mar 17, 2015 at 10:08 AM, Steven Wu wrote:
>
> > Jay,
> >
> > > > 2. Metrics Migration and usage in quotas
> > >> > > > > - The metrics library in clients has a notion of quotas that
> we
> > >> > should
> > >> > > > > reuse. For that to happen, we need to migrate the server to
&
> if we can produce the segment with Parquet, which is the native format
in a data lake, the consumer application (e.g., Spark jobs for ingestion)
can directly dump the segments as raw byte buffer into the data lake
without unwrapping each record individually and then writing to the Parquet
file o
It may be too late to change the Producer API now.
I always find "ListenableFuture" is very nice/usable. It essentially adds
callback to Future. It's a lot easier to chain/combine ListenableFuture.
http://docs.guava-libraries.googlecode.com/git/javadoc/com/google/common/util/concurrent/ListenableF
I am checking out the source code of 0.8.2 producer code. I have two
questions and hope to get some clarifications.
1) Sender thread/Runnable has this run loop. what if the in-memory queue is
mostly empty (i.e. producer has very few msgs to send out)? will this
become a simple tight loop that just
roducer. The write path for the
> data is it is written to the internal queue/buffer and the sender grabs
> data from that for ready connections and writes to them. This slightly
> complex ready/send api is required to allow back-pressure in the producer
> to work.
>
> -Jay
>
44 matches
Mail list logo
