Re: Change NN and JobTracker dynamically during runtime

Hi Dipesh, I'd go for a load balancer option that supports calling another service on deciding who's next. Let the LB address be provided inside job.properties. Have you seen *Varnish *? It's a well configurable LB option that can allegedly *call

Re: Change NN and JobTracker dynamically during runtime

@Andreas Hope you understood my use case above. I would appreciate if you please shed some more light and share about using load balancer to route jobs and keeping this load balancer outside. I would like to know how you are suggesting enable this load balancer in RM or NN or you are suggesting to

Re: Change NN and JobTracker dynamically during runtime

You mean I have to set env variables for each job/workflow execution and then it will be picked up by Oozie. And I should set them in my service (the service which is finding the best cluster?). For example let say I have 3 cluster: - When a job is sent via Oozie/Hue/Zepellin/Livy etc. - they are

Re: Change NN and JobTracker dynamically during runtime

Hi Dipesh, during workflow / job submission you can define variables inside job.properties coming e.g. from env vars that are used in workflow.xml. So much for the flexibility. Can you tell me a use case where runtime routing to different JT / NN instances via Oozie (and not e.g. coming from a lo

Re: Change NN and JobTracker dynamically during runtime

Hi Alex, The idea is to call this external service which will find the best cluster and inform the caller. So today this caller is Oozie, tomorrow it will be Zeppelin or any other application. How can I provide multiple JT and NN addresses in job.properties? You mean during job/workflow creation?

Re: Change NN and JobTracker dynamically during runtime

Hi Dipesh, seems like a bad idea to programmatically change job-tracker or name-node properties - it's just not the task of Oozie to determine what are the exact JT or NN instances Oozie should use. Instead, I'd rather setup a load balancer for JT and another one for NN, and provide those address