It is a bit more involved as I thought. We could simply the API further:
import org.apache.flink.client.program.PackagedProgram;
import org.apache.flink.configuration.Configuration;
import org.apache.flink.configuration.GlobalConfiguration;
import org.apache.hadoop.fs.Path;
import java.io.File;
import java.net.URL;
import java.util.Collections;
public class Niels {
public static void main(String[] args) throws Exception {
final String basePath = "/Users/max/Dev/flink/build-target/";
final String confDir = basePath + "/conf";
final String flinkJarPath = basePath +
"/lib/flink-dist_2.10-1.2-SNAPSHOT.jar";
final PackagedProgram packagedProgram =
new PackagedProgram(
new File(basePath + "/examples/streaming/WordCount.jar"),
// We need the Flink jar here because we want to lookup
the main method
// which might contains dependencies to Flink which are
not in the user jar
Collections.singletonList(new URL("file://" + flinkJarPath)));
final YarnClusterDescriptor descriptor = new YarnClusterDescriptor();
Configuration configuration =
GlobalConfiguration.loadConfiguration(confDir);
descriptor.setFlinkConfiguration(configuration);
descriptor.setConfigurationDirectory(confDir);
descriptor.setConfigurationFilePath(new Path(confDir +
"/flink-conf.yaml"));
descriptor.setLocalJarPath(new Path(flinkJarPath));
descriptor.setTaskManagerCount(2);
descriptor.setName("Testing the YarnClusterClient");
final YarnClusterClient client = descriptor.deploy();
client.run(packagedProgram, 2);
client.shutdown();
}
}
On Thu, Aug 25, 2016 at 5:06 PM, Niels Basjes <ni...@basjes.nl> wrote:
> Sounds good.
> Is there a basic example somewhere I can have a look at?
>
> Niels
>
> On Thu, Aug 25, 2016 at 2:55 PM, Maximilian Michels <m...@apache.org>
> wrote:
>
>> Hi Niels,
>>
>> If you're using 1.1.1, then you can instantiate the
>> YarnClusterDescriptor and supply it with the Flink jar and
>> configuration and subsequently call `deploy()` on it to receive a
>> ClusterClient for Yarn which you can submit programs using the
>> `run(PackagedProgram program, String args)` method. You can also
>> cancel jobs or shutdown the cluster from the ClusterClient.
>>
>> Cheers,
>> Max
>>
>> On Thu, Aug 25, 2016 at 10:24 AM, Niels Basjes <ni...@basjes.nl> wrote:
>> > Hi,
>> >
>> > We have a situation where we need to start a flink batch job on a yarn
>> > cluster the moment an event arrives over a queue.
>> > These events occur at a very low rate (like once or twice a week).
>> >
>> > The idea we have is to run an application that listens to the queue and
>> > executes the batch when it receives a message.
>> >
>> > We found that if we start this using 'flink run -m yarn-cluster ..." the
>> > moment we run this the jobmanager in yarn is started and the resources
>> for
>> > these batches is claimed immediately.
>> >
>> > What is the recommended way to only claim these resources when we
>> actually
>> > have a job to run?
>> > Can we 'manually' start and stop the jobmanager in yarn in some way
>> from our
>> > java code?
>> >
>> > --
>> > Best regards / Met vriendelijke groeten,
>> >
>> > Niels Basjes
>>
>
>
>
> --
> Best regards / Met vriendelijke groeten,
>
> Niels Basjes
>
