Ok, definitely a dev box problem (network for sure). I moved the process
from my dev box to the mesos cluster and the delay between puts is now
60ms.
On Fri, Jun 10, 2016 at 10:32 AM, Barry Kaplan wrote:
> Hmm, well CPU is pretty much zero. Heap is barely used. I even made the
> task put method
Hmm, well CPU is pretty much zero. Heap is barely used. I even made the
task put method be a noop other than to log time-since-last call. No
change. With yourkit I see that ES has a thread that is sleeping, but it's
in a monitor thread pool and clearly not blocking kafka. Anyway, I even
removed all
Barry,
It might help to know whether you're hitting a (single threaded) CPU limit
or if the bottleneck is elsewhere. Also, how large on average are the
messages you are consuming? There's nothing that'll force batching like
you're talking about. You can tweak any consumer settings via worker-level
I am running a connect consumer that receives JSON records and indexes into
elasticsearch. The consumer is pushing out 300 messages/s into the a topic
with a single partition. The connect job is configured with 1 task. (This
is all for testing).
What I see is that push is called about every 10s wi
