Good information. Thanks for the update, Lukas! On Tue, Jun 2, 2015 at 12:17 AM, Lukas Steiblys <lu...@doubledutch.me> wrote:
> Hi Michael, > > So far none of our jobs have crashed yet and everything is working just > fine, but I can already see a major memory saving (my estimate is at least > 33% less memory is needed) with ThreadJobFactory. > > One downside was that the logger wouldn't respect the lib/log4j.xml > configuration file anymore, but that's probably some error on my part. > > Lukas > > On Sat, May 30, 2015 at 2:41 PM, Michael Ravits <michaelr...@gmail.com> > wrote: > > > Hi Lukas, > > > > We use similar docker based deployment. > > Would be glad if you could post here if ThreadJobFactory worked well for > > you. > > > > Thanks, > > Michael > > > > On Sat, May 30, 2015 at 12:22 AM, Lukas Steiblys <lu...@doubledutch.me> > > wrote: > > > > > Yes, I think switching to ThreadJobFactory is a good solution. I think > > the > > > reasons why I switched to ProcessJobFactory earlier no longer hold > true. > > > > > > Thanks. > > > > > > Lukas > > > > > > -----Original Message----- From: Yi Pan > > > Sent: Friday, May 29, 2015 2:05 PM > > > > > > To: dev@samza.apache.org > > > Subject: Re: ProcessJobFactory parent process > > > > > > Hi, Lukas, > > > > > > Yes. That's exactly part of the feature to allow > > > health-check/failure-detection of containers. Another short-term option > > is > > > trying to use ThreadJobFactory, which has the JobCoordinator and > > containers > > > in the same process. Does that work for your use case? > > > > > > -Yi > > > > > > On Fri, May 29, 2015 at 12:59 PM, Lukas Steiblys <lu...@doubledutch.me > > > > > wrote: > > > > > > Yes, I'm talking about the child process crashing. I'd like the parent > > to > > >> die as well if the child crashes so Docker can understand that the > > process > > >> failed and restart the container. > > >> > > >> Lukas > > >> > > >> -----Original Message----- From: Yi Pan > > >> Sent: Friday, May 29, 2015 12:47 PM > > >> To: dev@samza.apache.org > > >> Subject: Re: ProcessJobFactory parent process > > >> > > >> > > >> Hi, Lukas, > > >> > > >> I assume that when you say "the job crashes", you were referring to > the > > >> child process running the container, not the parent process? If yes, > we > > >> were actually talking about adding container > > >> health-check/failure-detection > > >> in the JobCoordinator. SAMZA-680 would be the good place to start > these > > >> kind of discussion. > > >> > > >> Thanks! > > >> > > >> -Yi > > >> > > >> On Fri, May 29, 2015 at 8:39 AM, Lukas Steiblys <lu...@doubledutch.me > > > > >> wrote: > > >> > > >> Hi Yan, > > >> > > >>> > > >>> The memory usage is not very high, but I'm trying to cut the usage > any > > >>> way > > >>> I can. > > >>> > > >>> The bigger problem is when the job crashes and the parent process > stays > > >>> active preventing an auto restart by the Docker supervisor. > > >>> > > >>> Lukas > > >>> > > >>> On Thursday, May 28, 2015, Yan Fang <yanfang...@gmail.com> wrote: > > >>> > > >>> > Hi Lukas, > > >>> > > > >>> > The parent process is used to manage the lifecycle of the actual > > >>> process. I > > >>> > am curious how much memory the parent process takes? > > >>> > > > >>> > Thanks, > > >>> > > > >>> > Fang, Yan > > >>> > yanfang...@gmail.com <javascript:;> > > >>> > > > >>> > On Thu, May 28, 2015 at 2:30 PM, Lukas Steiblys < > > lu...@doubledutch.me > > >>> > <javascript:;>> > > >>> > wrote: > > >>> > > > >>> > > Hello, > > >>> > > > > >>> > > I’m running Samza tasks using ProcessJobFactory and after I start > > the > > >>> > job, > > >>> > > the initial process spawns a new process that is the actual > process > > >>> where > > >>> > > the code is run. The problem is that the parent process stays > > active > > >>> even > > >>> > > after the job is started and that messes with the way I deploy > > Samza > > >>> (in > > >>> > > Docker containers) and consumes memory while not doing anything. > > >>> > > > > >>> > > My question: is it possible to kill the parent process while > still > > >>> > leaving > > >>> > > the Samza tasks to process messages? > > >>> > > > > >>> > > Lukas > > >>> > > > >>> > > >>> > > >>> > > >> > > > > > >
