Hi Mohammad, Thanks for the response. We are using Oozie 2.3.2. When I reported this, there were no coordinator jobs running but we do see similar issues when the coordinator jobs are running as well. I will try with the params you mentioned and get back.
BTW, We also see these errors in the log - 2013-02-21 22:33:08,250 WARN CallableQueueService:528 - USER[-] GROUP[-] max concurrency for callable exceeded 2013-02-21 22:33:08,248 WARN SignalCommand:528 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] SignalCommand lock was not acquired - failed 0000161-130130163537664-oozie-oozi-W 2013-02-21 22:33:08,248 WARN SignalCommand:528 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] SignalCommand lock was not acquired - failed 0000161-130130163537664-oozie-oozi-W 2013-02-21 22:33:08,248 WARN SignalCommand:528 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[-] ACTION[-] SignalCommand lock was not acquired - failed 0000161-130130163537664-oozie-oozi-W Looks like some old job (from a few days back) is still lingering. Could this have an effect as well? Thanks, Siva On Fri, Feb 22, 2013 at 12:19 AM, Mohammad Islam <[email protected]> wrote: > Hi, > What version of Oozie are you using? Do you have any coordinator job > running? > > You could bumped up the value for this two properties: > > oozie.service.CallableQueueService.threads = 150 > oozie.service.CallableQueueService.callable.concurrency = 60 > > Please let us know how does it go. > > Regards, > Mohammad > > > > > > ________________________________ > From: Siva Subramanian <[email protected]> > To: [email protected] > Sent: Thursday, February 21, 2013 11:41 PM > Subject: Callable Queue is Full > > Hey, > > We have a complex workflow with about lots of parellel actions (around 300 > - a mix of MR, FS & java actions). Frequently we notice that, even though > the MR job corresponding to the action (this mostly happens with Java > actions) finishes in less than 10 seconds, the corresponding oozie action > takes more than 10 minutes to finish. In the oozie log, I see these > warnings: > > 2013-02-21 22:33:08,259 INFO CallbackServlet:525 - USER[-] GROUP[-] > TOKEN[-] APP[-] JOB[0000229-130215175004882-oozie-oozi-W] > ACTION[0000229-130215175004882-oozie-oozi-W@move_data] callback for action > [0000229-130215175004882-oozie-oozi-W@move_data] > 2013-02-21 22:33:08,259 WARN CallableQueueService:528 - USER[-] GROUP[-] > queue if full, ignoring queuing for [callback,2] > 2013-02-21 22:33:08,259 WARN CallableQueueService:528 - USER[-] GROUP[-] > Could not queue callable > 2013-02-21 22:33:08,260 WARN DagEngine:528 - USER[-] GROUP[-] TOKEN[-] > APP[-] JOB[0000229-130215175004882-oozie-oozi-W] > ACTION[0000229-130215175004882-oozie-oozi-W@move_data] queue is full or > system is in SAFEMODE, ignoring callback > > > This particular job was started at 22:32:44 and the MR job ended in 10 > seconds. > > We have the following config: > oozie.service.CallableQueueService.queue.size = 10000 > oozie.service.CallableQueueService.threads = 10 > oozie.service.CallableQueueService.callable.concurrency = 3 > > What parameter(s) can we tune to scale? > > Thanks in advance, > Siva > -- *Siva Subramanian* *Engineering Analytics*
