I see that it is safe to keep it this off by default due to some concerns.
Eventually, for settings such as this whose 'preferred' value is platform
specific (or based on other criteria), it might be worth considering
having a default value that is not a constant but an 'auto' value .. When
kafka
Based on recent suggestion by Joel, I am experimenting with using flush() to
simulate batched-sync behavior.
The essence of my single threaded producer code is :
for (int i = 0; i < numRecords;) {
// 1- Send a batch
for(int batchCounter=0; batchCounter f = producer.send(rec
t;Joel
>
>On Wed, Apr 29, 2015 at 12:58:43AM +, Roshan Naik wrote:
>> Based on recent suggestion by Joel, I am experimenting with using
>>flush() to simulate batched-sync behavior.
>> The essence of my single threaded producer code is :
>>
>> for
@Jay,
My bad. I mistook the batch.size to be number of messages instead of bytes.
Below are revised measurements based on computing the batch.size in bytes .
@Jun,
With explicit flush()... linger should not impact. Isn't it ?
@Wang,
Larger batches are not necessarily giving better numbe
For some reason the HTML formatting is being dropped from my email..
Making it harder to read the measurements table.
On 4/29/15 8:32 PM, "Roshan Naik" wrote:
>
>@Jay,
>My bad. I mistook the batch.size to be number of messages instead of
>bytes. Below are revised
Thanks @Jay for suggesting changes to batch.size and linger.ms. I tried
them out. It appears one can do better than the default batch.size for
this synchronous batch mode with flush().
These new measurements are giving more "rational" numbers which with I can
reason and infer some thumb rules (fo
Louro (Hortonworks)
- [20m] – Rethinking the Storm 2.0 Worker - Roshan Naik (Hortonworks)
- [57m] – Storm in Retail Context: Catalog data processing using
Kafka, Storm & Microservices - Karthik Deivasigamani (WalMart Labs)
- [1h: 54m:45sec] – Schema Regi
Here is a case of data duplication that should be avoidable.
It is observed when leadership of partition changes from the current leader
back to preferred leader.
Steps to reproduce:
- Using 3 broker setup.
- Create topic with 1 partition, replication factor=3, ISR count=2 and
leader.imbalance
My initial thoughts:
Although it is kind of discussed very broadly, I did struggle a bit to properly
grasp the value add this adds over the alternative approaches that are
available today (or need a little work to accomplish) in specific use cases. I
feel its better to take specific common use
Thanks Jay and Ewen for the response.
>@Jay
>
> 3. This has a built in notion of parallelism throughout.
It was not obvious how it will look like or differ from existing systemsŠ
since all of existing ones do parallelize data movement.
@Ewen,
>Import: Flume is just one of many similar syste
10 matches
Mail list logo
