Yes, I have tried using separate Yarn queues for this, but I have my doubts.
Here is what I am trying to do: I have 2 Flink Jobs JobA : Regular Job running every x minutes. JobB : User requested adhoc job Tried1. Sol A : Have 2 queues on Yarn with 95 : 5 resource distribution Sol B: Have parallelism control the concurrent execution i.e. divide total task slots in 95:5 ratio In both the above solutions I would be dedicating a chunk of resources to Job B. I want Job A to run will all my resources and only if Job B is submitted, it should get some resources to process and not throw back NoResourceException. I hope I am able to explain my problem :) VG, VIkram On Wed, Apr 27, 2016 at 2:13 PM, Flavio Pompermaier <pomperma...@okkam.it> wrote: > That would be definitely interesting but I think that at the moment the > only way to achieve that is to exploit YARN for that.. > An integration with some job-workflow engine (like Apache Oozie and Apache > Falcon) would also be very useful! I tried to wrote on Apache Oozie mailing > list if there is any interest in integrating with apache Flink but I didn't > receive any feedback yet :( > > Best, > Flavio > > On Wed, Apr 27, 2016 at 1:00 PM, Vikram Saxena <vikram....@gmail.com> > wrote: > >> Hi >> >> I am reading and learning about Flink and I have tried to implement some >> Flink Jobs. >> >> In my application I have 2 Flink Jobs which I want to run in parallel. >> Of course, as I understand I can have the task slots divided so that each >> one can run concurrently. >> >> But, is there a possibility for scheduling jobs only to be processed when >> the JobManager has resources free(task slots) and not throw back >> (NoResourceAvailableException) ?? >> >> The reason I ask this is because I have 2 Flink Jobs, one which would be >> always running in short intervals and other which would be in ad hoc basis. >> >> I do not want to have my task slots divided just for this ad hoc job, >> So I wanted to submit this adhoc job and the Job Manager can schedule it >> whenever it has free task slots ? >> >> >> Cheers, >> Vikram >> > >
