Hey, We have a complex workflow with about lots of parellel actions (around 300 - a mix of MR, FS & java actions). Frequently we notice that, even though the MR job corresponding to the action (this mostly happens with Java actions) finishes in less than 10 seconds, the corresponding oozie action takes more than 10 minutes to finish. In the oozie log, I see these warnings:
2013-02-21 22:33:08,259 INFO CallbackServlet:525 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000229-130215175004882-oozie-oozi-W] ACTION[0000229-130215175004882-oozie-oozi-W@move_data] callback for action [0000229-130215175004882-oozie-oozi-W@move_data] 2013-02-21 22:33:08,259 WARN CallableQueueService:528 - USER[-] GROUP[-] queue if full, ignoring queuing for [callback,2] 2013-02-21 22:33:08,259 WARN CallableQueueService:528 - USER[-] GROUP[-] Could not queue callable 2013-02-21 22:33:08,260 WARN DagEngine:528 - USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000229-130215175004882-oozie-oozi-W] ACTION[0000229-130215175004882-oozie-oozi-W@move_data] queue is full or system is in SAFEMODE, ignoring callback This particular job was started at 22:32:44 and the MR job ended in 10 seconds. We have the following config: oozie.service.CallableQueueService.queue.size = 10000 oozie.service.CallableQueueService.threads = 10 oozie.service.CallableQueueService.callable.concurrency = 3 What parameter(s) can we tune to scale? Thanks in advance, Siva
