Hi,
I'm looking for information on where the stream transformations are
applied - the server(broker) or the client?
Would it be possible for clients to share the topology?
--
Warm regards
Roshan
closer to
async mode.
-roshan
On 4/24/15 11:42 AM, "Navneet Gupta (Tech - BLR)"
wrote:
>Hi,
>
>I ran some tests on our cluster by sending message from multiple clients
>(machines). Each machine had about 40-100 threads per producer.
>
>I thought of trying out h
Can we use the new 0.8.2 producer perf tool against a 0.8.1 broker ?
-roshan
On 4/24/15 1:19 PM, "Jay Kreps" wrote:
>Do make sure if you are at all performance sensitive you are using the new
>producer api we released in 0.8.2.
>
>-Jay
>
>On Fri, Apr 24, 2015 at 1
Jay,
Its not evident how to switch between sync and async modes using this
new 'org.apache.kafka.clients.tools.ProducerPerformance'
AFAIKT it measures in async mode by default.
-roshan
On 4/24/15 3:23 PM, "Jay Kreps" wrote:
>That should work. I recommend using the
On 4/27/15 11:15 AM, "Jay Kreps" wrote:
>
>The new producer is pretty new still so I suspect there is a fair amount
>of
>low-hanging performance work for anyone who wanted to take a shot at it.
I do see something. Will discuss it on a different email thread.
Been evaluating the perf of old and new Produce APIs for reliable high volume
streaming data movement. I do see one area of improvement that the new API
could use for synchronous clients.
AFAIKT, the new API does not support batched synchronous transfers. To do
synchronous send, one needs to do
all threads (correct me if I am wrong).
-roshan
On 4/27/15 1:36 PM, "Joel Koshy" wrote:
>This sounds like flush:
>https://cwiki.apache.org/confluence/display/KAFKA/KIP-8+-+Add+a+flush+meth
>od+to+the+producer+API
>
>which was recently implemented in trunk.
>
>
ersist
to accommodateŠ as opposed start/end position of each log message in the
file.
-roshan
On 4/27/15 2:07 PM, "Joel Koshy" wrote:
>As long as you retain the returned futures somewhere, you can always
>iterate over the futures after the flush completes and check for
&
On 4/27/15 2:59 PM, "Gwen Shapira" wrote:
>@Roshan - if the data was already written to Kafka, your approach will
>generate LOTS of duplicates. I'm not convinced its ideal.
Only if the delivery failure rate is very high (i.e. short lived but very
frequent). This b
ubled with doubling of event size). The perf
tool was not using the Producer.send(list<> ) api though. I saw great
improvement when I changed it to use the Producer.send(list<> ). Sometimes
it easily exceeded the throughput async mode.
-roshan
On 4/27/15 10:32 PM, "Ewen Ch
@Ewen
No I did not use compression in my measurements.
or a batch going to a
partition.
For simplicity, streaming clients may not want to deal explicitly with
partitions (and get exposed to repartitioning & leader change type issues)
-roshan
On 4/30/15 2:07 PM, "Gwen Shapira" wrote:
>Why do we think atomicity is expected, if t
Hi,
Is it possible to set up a topic with only a size limit (retention.bytes)
and not a time limit (retention.ms)?
Roshan
Louro (Hortonworks)
- [20m] – Rethinking the Storm 2.0 Worker - Roshan Naik (Hortonworks)
- [57m] – Storm in Retail Context: Catalog data processing using
Kafka, Storm & Microservices - Karthik Deivasigamani (WalMart Labs)
- [1h: 54m:45sec] – Schema Regi
14 matches
Mail list logo
