Re: Runtime Execution Model

2015-09-29 Thread Lukas Steiblys
0.10.0 yet as I see a lot of changes in that code. Lukas -Original Message- From: Yan Fang Sent: Monday, September 14, 2015 11:08 AM To: dev@samza.apache.org Subject: Re: Runtime Execution Model Hi Bruno, AFAIK, there is no existing JobFactory that brings as many threads as the

Re: Runtime Execution Model

2015-09-16 Thread Lukas Steiblys
ng Sent: Wednesday, September 16, 2015 3:45 PM To: dev@samza.apache.org Subject: Re: Runtime Execution Model -- Hi Lukas, I want to learn more from your production environment. How do you use ProcessJobFactory in Docker containers? Do you use one ProcessJobFactory process all the tasks, or s

Re: Runtime Execution Model

2015-09-16 Thread Yi Pan
as > > -Original Message- From: Yan Fang > Sent: Wednesday, September 16, 2015 3:45 PM > > To: dev@samza.apache.org > Subject: Re: Runtime Execution Model > > -- Hi Lukas, > > I want to learn more from your production environment. How do you use > ProcessJobFac

Re: Runtime Execution Model

2015-09-16 Thread Yi Pan
@doubledutch.me> > > > > wrote: > > > > > > > > > Hi Yan, > > > > > > > > > > We use Samza in a production environment using ProcessJobFactory in > > > > Docker > > > > > containers because it greatly

Re: Runtime Execution Model

2015-09-16 Thread Lukas Steiblys
ickly. Lukas -Original Message- From: Yan Fang Sent: Wednesday, September 16, 2015 3:45 PM To: dev@samza.apache.org Subject: Re: Runtime Execution Model -- Hi Lukas, I want to learn more from your production environment. How do you use ProcessJobFactory in Docker containers? Do you us

Re: Runtime Execution Model

2015-09-16 Thread Yan Fang
y > > > > multithreaded? I will look into doing that myself, but I think it > might > > > be > > > > useful to implement this for everyone. I am sure there are plenty of > > > cases > > > > where people do not want to use YARN, but want mor

Re: Runtime Execution Model

2015-09-15 Thread Yi Pan
lement this for everyone. I am sure there are plenty of > > cases > > > where people do not want to use YARN, but want more parallelism in > their > > > tasks. > > > > > > Lukas > > > > > > -Original Message- From: Yan Fang > > > Sent:

Re: Runtime Execution Model

2015-09-15 Thread Hannes Stockner
; > tasks. > > > > Lukas > > > > -Original Message----- From: Yan Fang > > Sent: Monday, September 14, 2015 11:08 AM > > To: dev@samza.apache.org > > Subject: Re: Runtime Execution Model > > > > > > Hi Bruno, > > > >

Re: Runtime Execution Model

2015-09-15 Thread Bruno Bonacci
r everyone. I am sure there are plenty of cases > where people do not want to use YARN, but want more parallelism in their > tasks. > > Lukas > > -Original Message- From: Yan Fang > Sent: Monday, September 14, 2015 11:08 AM > To: dev@samza.apache.org > Subject:

Re: Runtime Execution Model

2015-09-15 Thread Lukas Steiblys
@samza.apache.org Subject: Re: Runtime Execution Model Hi Bruno, AFAIK, there is no existing JobFactory that brings as many threads as the partition number. But I think nothing stops you to implement this: you can get the partition information from the JobCoordinator, and then bring as many threads

Re: Runtime Execution Model

2015-09-14 Thread bruno . bonacci
Many thanks Yi Bye Bruno > On 14 Sep 2015, at 23:49, Yi Pan wrote: > > Hi, Bruno, > > The number of partitions consumed by a single task is also configurable via > the partition assignment policies (job.systemstreampartition. > grouper.factory). By default, there are two partition assignment p

Re: Runtime Execution Model

2015-09-14 Thread Yi Pan
Hi, Bruno, The number of partitions consumed by a single task is also configurable via the partition assignment policies (job.systemstreampartition. grouper.factory). By default, there are two partition assignment policies implemented: org.apache.samza.container.grouper.stream.GroupByPartitionFact

Re: Runtime Execution Model

2015-09-14 Thread bruno . bonacci
Hi Yi, Does a single task consume from a single partition or it consumes from more/all partitions? Thanks Bruno > On 14 Sep 2015, at 23:22, Yi Pan wrote: > > Hi, Bruno, > > The number of containers are configurable in YarnJobFactory via > yarn.container.count. > Each container is a single th

Re: Runtime Execution Model

2015-09-14 Thread Yi Pan
Hi, Bruno, The number of containers are configurable in YarnJobFactory via yarn.container.count. Each container is a single threaded model and you can run multiple tasks in a single container. At maximum, you can have as many containers as the number of tasks in this config to achieve 1 task / thr

Re: Runtime Execution Model

2015-09-14 Thread Bruno Bonacci
Thanks Yan for writing me back, That's ok for ThreadJobFactory and ProcessJobFactory but what about the YarnJobFactory? How many task/executors will be spawning? Bruno On Mon, Sep 14, 2015 at 7:08 PM, Yan Fang wrote: > Hi Bruno, > > AFAIK, there is no existing JobFactory that brings as many t

Re: Runtime Execution Model

2015-09-14 Thread Yan Fang
Hi Bruno, AFAIK, there is no existing JobFactory that brings as many threads as the partition number. But I think nothing stops you to implement this: you can get the partition information from the JobCoordinator, and then bring as many threads as the partition/task number. Since the two local fa

Runtime Execution Model

2015-09-12 Thread Bruno Bonacci
Hi, I'm looking for additional documentation on the different RUNTIME EXECUTION MODELS of the different `job.factory.class`. I'm particularly interested on how each factory (ThreadJobFactory, ProcessJobFactory and YarnJobFactory) will create tasks consume and process messages out of Kafka and the