Hi Chen,
Yes, theoretically, it's possible and it'll certainly increase parallelism.
You should be careful about auto-commit when you implement this, Say you
have auto-commit turned on, and do a time-consuming operation for certain
types of messages in your process method (like - a long computati
I’m not sure you can do this for the purposes of consuming messages faster, but
I’m interested to hear if I’m mistaken.
You could spawn threads to do other work within a job though.
> On Nov 18, 2015, at 11:40 PM, Chen Song wrote:
>
> Thanks Navina
>
> So theoretically I can create a threa
Thanks Navina
So theoretically I can create a thread pool within a container. I know it
is very hacky but it should increase parallelism.
Chen
On Mon, Nov 16, 2015 at 5:49 PM, Navina Ramesh wrote:
> Hi Chen,
> Samza container is still single threaded. In case of yarn based deployment,
> Samza
Hi Chen,
Samza container is still single threaded. In case of yarn based deployment,
Samza uses this config value to verify that the cluster has sufficient
capacity to support running your job.
Apart from this verification, I don't believe we utilize this config value.
If you set it to > 1, it won
According to the documentation, each Samza container is single threaded.
Why giving yarn.container.cpu.cores as a config option and what is the
implication
to set this to a value > 1?
--
Chen Song
