You too!
On Fri, Aug 26, 2016 at 4:15 PM, Niels Basjes <ni...@basjes.nl> wrote:
> Thanks!
> I'm going to work with this next week.
>
> Have a nice weekend.
>
> Niels
>
> On Fri, Aug 26, 2016 at 2:49 PM, Maximilian Michels <m...@apache.org> wrote:
>>
>> It is a bit more involved as I thought. We could simply the API further:
>>
>> import org.apache.flink.client.program.PackagedProgram;
>> import org.apache.flink.configuration.Configuration;
>> import org.apache.flink.configuration.GlobalConfiguration;
>> import org.apache.hadoop.fs.Path;
>>
>> import java.io.File;
>> import java.net.URL;
>> import java.util.Collections;
>>
>> public class Niels {
>>
>> public static void main(String[] args) throws Exception {
>>
>> final String basePath = "/Users/max/Dev/flink/build-target/";
>> final String confDir = basePath + "/conf";
>> final String flinkJarPath = basePath +
>> "/lib/flink-dist_2.10-1.2-SNAPSHOT.jar";
>>
>> final PackagedProgram packagedProgram =
>> new PackagedProgram(
>> new File(basePath + "/examples/streaming/WordCount.jar"),
>> // We need the Flink jar here because we want to lookup the
>> main method
>> // which might contains dependencies to Flink which are not in
>> the user jar
>> Collections.singletonList(new URL("file://" + flinkJarPath)));
>>
>> final YarnClusterDescriptor descriptor = new
>> YarnClusterDescriptor();
>>
>> Configuration configuration =
>> GlobalConfiguration.loadConfiguration(confDir);
>> descriptor.setFlinkConfiguration(configuration);
>>
>> descriptor.setConfigurationDirectory(confDir);
>> descriptor.setConfigurationFilePath(new Path(confDir +
>> "/flink-conf.yaml"));
>>
>> descriptor.setLocalJarPath(new Path(flinkJarPath));
>> descriptor.setTaskManagerCount(2);
>> descriptor.setName("Testing the YarnClusterClient");
>>
>> final YarnClusterClient client = descriptor.deploy();
>> client.run(packagedProgram, 2);
>> client.shutdown();
>> }
>> }
>>
>>
>> On Thu, Aug 25, 2016 at 5:06 PM, Niels Basjes <ni...@basjes.nl> wrote:
>>>
>>> Sounds good.
>>> Is there a basic example somewhere I can have a look at?
>>>
>>> Niels
>>>
>>> On Thu, Aug 25, 2016 at 2:55 PM, Maximilian Michels <m...@apache.org>
>>> wrote:
>>>>
>>>> Hi Niels,
>>>>
>>>> If you're using 1.1.1, then you can instantiate the
>>>> YarnClusterDescriptor and supply it with the Flink jar and
>>>> configuration and subsequently call `deploy()` on it to receive a
>>>> ClusterClient for Yarn which you can submit programs using the
>>>> `run(PackagedProgram program, String args)` method. You can also
>>>> cancel jobs or shutdown the cluster from the ClusterClient.
>>>>
>>>> Cheers,
>>>> Max
>>>>
>>>> On Thu, Aug 25, 2016 at 10:24 AM, Niels Basjes <ni...@basjes.nl> wrote:
>>>> > Hi,
>>>> >
>>>> > We have a situation where we need to start a flink batch job on a yarn
>>>> > cluster the moment an event arrives over a queue.
>>>> > These events occur at a very low rate (like once or twice a week).
>>>> >
>>>> > The idea we have is to run an application that listens to the queue
>>>> > and
>>>> > executes the batch when it receives a message.
>>>> >
>>>> > We found that if we start this using 'flink run -m yarn-cluster ..."
>>>> > the
>>>> > moment we run this the jobmanager in yarn is started and the resources
>>>> > for
>>>> > these batches is claimed immediately.
>>>> >
>>>> > What is the recommended way to only claim these resources when we
>>>> > actually
>>>> > have a job to run?
>>>> > Can we 'manually' start and stop the jobmanager in yarn in some way
>>>> > from our
>>>> > java code?
>>>> >
>>>> > --
>>>> > Best regards / Met vriendelijke groeten,
>>>> >
>>>> > Niels Basjes
>>>
>>>
>>>
>>>
>>> --
>>> Best regards / Met vriendelijke groeten,
>>>
>>> Niels Basjes
>>
>>
>
>
>
> --
> Best regards / Met vriendelijke groeten,
>
> Niels Basjes
