bash-4.1$ hadoop fs -chmod 777 * chmod: `config.sh': No such file or directory chmod: `flink': No such file or directory chmod: `flink.bat': No such file or directory chmod: `jobmanager.sh': No such file or directory chmod: `pyflink2.sh': No such file or directory chmod: `pyflink3.sh': No such file or directory chmod: `start-cluster.sh': No such file or directory chmod: `start-cluster-streaming.sh': No such file or directory chmod: `start-local.bat': No such file or directory chmod: `start-local.sh': No such file or directory chmod: `start-local-streaming.sh': No such file or directory chmod: `start-scala-shell.sh': No such file or directory chmod: `start-webclient.sh': No such file or directory chmod: `stop-cluster.sh': No such file or directory chmod: `stop-local.sh': No such file or directory chmod: `stop-webclient.sh': No such file or directory chmod: `taskmanager.sh': No such file or directory chmod: `webclient.sh': No such file or directory chmod: `yarn-session.sh': No such file or directory
2015-06-04 17:08 GMT+02:00 Pa Rö <paul.roewer1...@googlemail.com>: > [cloudera@quickstart bin]$ sudo su yarn > bash-4.1$ hadoop fs -chmod 777 > -chmod: Not enough arguments: expected 2 but got 1 > Usage: hadoop fs [generic options] -chmod [-R] <MODE[,MODE]... | > OCTALMODE> PATH... > bash-4.1$ > > you understand? > > 2015-06-04 17:04 GMT+02:00 Robert Metzger <rmetz...@apache.org>: > >> It looks like the user "yarn" which is running Flink doesn't have >> permission to access the files. >> >> Can you do "sudo su yarn" to become the "yarn" user. Then, you can do >> "hadoop fs -chmod 777" to make the files accessible for everyone. >> >> >> On Thu, Jun 4, 2015 at 4:59 PM, Pa Rö <paul.roewer1...@googlemail.com> >> wrote: >> >>> okay, it's work, i get a exception: >>> >>> [cloudera@quickstart Desktop]$ cd flink-0.9-SNAPSHOT/bin/ >>> [cloudera@quickstart bin]$ flink run /home/cloudera/Desktop/ma-flink.jar >>> bash: flink: command not found >>> [cloudera@quickstart bin]$ ./flink run >>> /home/cloudera/Desktop/ma-flink.jar >>> log4j:WARN No appenders could be found for logger >>> (org.apache.hadoop.metrics2.lib.MutableMetricsFactory). >>> log4j:WARN Please initialize the log4j system properly. >>> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig >>> for more info. >>> Found YARN properties file >>> /home/cloudera/Desktop/flink-0.9-SNAPSHOT/bin/../conf/.yarn-properties >>> Using JobManager address from YARN properties quickstart.cloudera/ >>> 127.0.0.1:53874 >>> java.io.IOException: Mkdirs failed to create /user/cloudera/outputs >>> at >>> org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:438) >>> at >>> org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:424) >>> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:905) >>> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:886) >>> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:783) >>> at >>> mgm.tp.bigdata.ma_commons.commons.Seeding.randomSeeding(Seeding.java:21) >>> at >>> mgm.tp.bigdata.ma_flink.FlinkMain.getCentroidDataSet(FlinkMain.java:178) >>> at mgm.tp.bigdata.ma_flink.FlinkMain.main(FlinkMain.java:47) >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>> at >>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >>> at >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>> at java.lang.reflect.Method.invoke(Method.java:606) >>> at >>> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:437) >>> at >>> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:353) >>> at org.apache.flink.client.program.Client.run(Client.java:315) >>> at >>> org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:584) >>> at org.apache.flink.client.CliFrontend.run(CliFrontend.java:290) >>> at >>> org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:880) >>> at org.apache.flink.client.CliFrontend.main(CliFrontend.java:922) >>> org.apache.flink.client.program.ProgramInvocationException: The program >>> execution failed: Failed to submit job 934743a5c49c6d5e31c9e8201452e36d >>> (KMeans Flink) >>> at org.apache.flink.client.program.Client.run(Client.java:412) >>> at org.apache.flink.client.program.Client.run(Client.java:355) >>> at org.apache.flink.client.program.Client.run(Client.java:348) >>> at >>> org.apache.flink.client.program.ContextEnvironment.execute(ContextEnvironment.java:63) >>> at mgm.tp.bigdata.ma_flink.FlinkMain.main(FlinkMain.java:70) >>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>> at >>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >>> at >>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>> at java.lang.reflect.Method.invoke(Method.java:606) >>> at >>> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:437) >>> at >>> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:353) >>> at org.apache.flink.client.program.Client.run(Client.java:315) >>> at >>> org.apache.flink.client.CliFrontend.executeProgram(CliFrontend.java:584) >>> at org.apache.flink.client.CliFrontend.run(CliFrontend.java:290) >>> at >>> org.apache.flink.client.CliFrontend.parseParameters(CliFrontend.java:880) >>> at org.apache.flink.client.CliFrontend.main(CliFrontend.java:922) >>> Caused by: org.apache.flink.runtime.client.JobExecutionException: Failed >>> to submit job 934743a5c49c6d5e31c9e8201452e36d (KMeans Flink) >>> at org.apache.flink.runtime.jobmanager.JobManager.org >>> $apache$flink$runtime$jobmanager$JobManager$$submitJob(JobManager.scala:595) >>> at >>> org.apache.flink.runtime.jobmanager.JobManager$$anonfun$receiveWithLogMessages$1.applyOrElse(JobManager.scala:192) >>> at >>> scala.runtime.AbstractPartialFunction$mcVL$sp.apply$mcVL$sp(AbstractPartialFunction.scala:33) >>> at >>> scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:33) >>> at >>> scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:25) >>> at >>> org.apache.flink.yarn.ApplicationMasterActor$$anonfun$receiveYarnMessages$1.applyOrElse(ApplicationMasterActor.scala:99) >>> at scala.PartialFunction$OrElse.apply(PartialFunction.scala:162) >>> at >>> org.apache.flink.runtime.ActorLogMessages$$anon$1.apply(ActorLogMessages.scala:36) >>> at >>> org.apache.flink.runtime.ActorLogMessages$$anon$1.apply(ActorLogMessages.scala:29) >>> at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:118) >>> at >>> org.apache.flink.runtime.ActorLogMessages$$anon$1.applyOrElse(ActorLogMessages.scala:29) >>> at akka.actor.Actor$class.aroundReceive(Actor.scala:465) >>> at >>> org.apache.flink.runtime.jobmanager.JobManager.aroundReceive(JobManager.scala:94) >>> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516) >>> at akka.actor.ActorCell.invoke(ActorCell.scala:487) >>> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:254) >>> at akka.dispatch.Mailbox.run(Mailbox.scala:221) >>> at akka.dispatch.Mailbox.exec(Mailbox.scala:231) >>> at >>> scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) >>> at >>> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) >>> at >>> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) >>> at >>> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) >>> Caused by: org.apache.flink.runtime.JobException: Creating the input >>> splits caused an error: File /user/cloudera/outputs/seed-1 does not exist >>> or the user running Flink ('yarn') has insufficient permissions to access >>> it. >>> at >>> org.apache.flink.runtime.executiongraph.ExecutionJobVertex.<init>(ExecutionJobVertex.java:162) >>> at >>> org.apache.flink.runtime.executiongraph.ExecutionGraph.attachJobGraph(ExecutionGraph.java:471) >>> at org.apache.flink.runtime.jobmanager.JobManager.org >>> $apache$flink$runtime$jobmanager$JobManager$$submitJob(JobManager.scala:535) >>> ... 21 more >>> Caused by: java.io.FileNotFoundException: File >>> /user/cloudera/outputs/seed-1 does not exist or the user running Flink >>> ('yarn') has insufficient permissions to access it. >>> at >>> org.apache.flink.core.fs.local.LocalFileSystem.getFileStatus(LocalFileSystem.java:106) >>> at >>> org.apache.flink.api.common.io.FileInputFormat.createInputSplits(FileInputFormat.java:390) >>> at >>> org.apache.flink.api.common.io.FileInputFormat.createInputSplits(FileInputFormat.java:51) >>> at >>> org.apache.flink.runtime.executiongraph.ExecutionJobVertex.<init>(ExecutionJobVertex.java:146) >>> ... 23 more >>> >>> >>> how i must set the files in the hdfs? >>> quickstart.cloudera:50075/home/cloudera/output? >>> >>> 2015-06-04 16:51 GMT+02:00 Robert Metzger <rmetz...@apache.org>: >>> >>>> Once you've started the YARN session, you can submit a Flink job with >>>> "./bin/flink run <pathToYourJar>". >>>> >>>> The jar file of your job doesn't need to be in HDFS. It has to be in >>>> the local file system and flink will send it to all machines. >>>> >>>> On Thu, Jun 4, 2015 at 4:48 PM, Pa Rö <paul.roewer1...@googlemail.com> >>>> wrote: >>>> >>>>> okay, now it run on my hadoop. >>>>> how i can start my flink job? and where must the jar file save, at >>>>> hdfs or as local file? >>>>> >>>>> 2015-06-04 16:31 GMT+02:00 Robert Metzger <rmetz...@apache.org>: >>>>> >>>>>> Yes, you have to run these commands in the command line of the >>>>>> Cloudera VM. >>>>>> >>>>>> On Thu, Jun 4, 2015 at 4:28 PM, Pa Rö <paul.roewer1...@googlemail.com >>>>>> > wrote: >>>>>> >>>>>>> you mean run this command on terminal/shell and not define a hue job? >>>>>>> >>>>>>> 2015-06-04 16:25 GMT+02:00 Robert Metzger <rmetz...@apache.org>: >>>>>>> >>>>>>>> It should be certainly possible to run Flink on a cloudera live VM >>>>>>>> >>>>>>>> I think these are the commands you need to execute: >>>>>>>> >>>>>>>> wget >>>>>>>> http://stratosphere-bin.s3-website-us-east-1.amazonaws.com/flink-0.9-SNAPSHOT-bin-hadoop2.tgz >>>>>>>> tar xvzf flink-0.9-SNAPSHOT-bin-hadoop2.tgz >>>>>>>> cd flink-0.9-SNAPSHOT/ >>>>>>>> *export HADOOP_CONF_DIR=/usr/lib/hadoop/etc/hadoop/* >>>>>>>> ./bin/yarn-session.sh -n 1 -jm 1024 -tm 1024 >>>>>>>> >>>>>>>> If that is not working for you, please post the exact error message >>>>>>>> you are getting and I can help you to get it to run. >>>>>>>> >>>>>>>> >>>>>>>> On Thu, Jun 4, 2015 at 4:18 PM, Pa Rö < >>>>>>>> paul.roewer1...@googlemail.com> wrote: >>>>>>>> >>>>>>>>> hi robert, >>>>>>>>> >>>>>>>>> i think the problem is the hue api, >>>>>>>>> i had the same problem with spark submit script, >>>>>>>>> but on the new hue release, they have a spark submit api. >>>>>>>>> >>>>>>>>> i asked the group for the same problem with spark, no reply. >>>>>>>>> >>>>>>>>> i want test my app on local cluster, before i run it on the big >>>>>>>>> cluster, >>>>>>>>> for that i use cloudera live. maybe it give an other way to test >>>>>>>>> flink on a local cluster vm? >>>>>>>>> >>>>>>>>> 2015-06-04 16:12 GMT+02:00 Robert Metzger <rmetz...@apache.org>: >>>>>>>>> >>>>>>>>>> Hi Paul, >>>>>>>>>> >>>>>>>>>> why did running Flink from the regular scripts not work for you? >>>>>>>>>> >>>>>>>>>> I'm not an expert on Hue, I would recommend asking in the Hue >>>>>>>>>> user forum / mailing list: >>>>>>>>>> https://groups.google.com/a/cloudera.org/forum/#!forum/hue-user. >>>>>>>>>> >>>>>>>>>> On Thu, Jun 4, 2015 at 4:09 PM, Pa Rö < >>>>>>>>>> paul.roewer1...@googlemail.com> wrote: >>>>>>>>>> >>>>>>>>>>> thanks, >>>>>>>>>>> now i want run my app on cloudera live vm single node, >>>>>>>>>>> how i can define my flink job with hue? >>>>>>>>>>> i try to run the flink script in the hdfs, it's not work. >>>>>>>>>>> >>>>>>>>>>> best regards, >>>>>>>>>>> paul >>>>>>>>>>> >>>>>>>>>>> 2015-06-02 14:50 GMT+02:00 Robert Metzger <rmetz...@apache.org>: >>>>>>>>>>> >>>>>>>>>>>> I would recommend using HDFS. >>>>>>>>>>>> For that, you need to specify the paths like this: >>>>>>>>>>>> hdfs:///path/to/data. >>>>>>>>>>>> >>>>>>>>>>>> On Tue, Jun 2, 2015 at 2:48 PM, Pa Rö < >>>>>>>>>>>> paul.roewer1...@googlemail.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> nice, >>>>>>>>>>>>> >>>>>>>>>>>>> which file system i must use for the cluster? java.io or >>>>>>>>>>>>> hadoop.fs or flink? >>>>>>>>>>>>> >>>>>>>>>>>>> 2015-06-02 14:29 GMT+02:00 Robert Metzger <rmetz...@apache.org >>>>>>>>>>>>> >: >>>>>>>>>>>>> >>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>> you can start Flink on YARN on the Cloudera distribution. >>>>>>>>>>>>>> >>>>>>>>>>>>>> See here for more: >>>>>>>>>>>>>> http://ci.apache.org/projects/flink/flink-docs-master/setup/yarn_setup.html >>>>>>>>>>>>>> >>>>>>>>>>>>>> These are the commands you need to execute >>>>>>>>>>>>>> >>>>>>>>>>>>>> wget >>>>>>>>>>>>>> http://stratosphere-bin.s3-website-us-east-1.amazonaws.com/flink-0.9-SNAPSHOT-bin-hadoop2.tgz >>>>>>>>>>>>>> tar xvzf flink-0.9-SNAPSHOT-bin-hadoop2.tgzcd flink-0.9-SNAPSHOT/ >>>>>>>>>>>>>> ./bin/yarn-session.sh -n 4 -jm 1024 -tm 4096 >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Tue, Jun 2, 2015 at 2:03 PM, Pa Rö < >>>>>>>>>>>>>> paul.roewer1...@googlemail.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> hi community, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> i want test my flink k-means on a hadoop cluster. i use the >>>>>>>>>>>>>>> cloudera live distribution. how i can run flink on this >>>>>>>>>>>>>>> cluster? maybe only >>>>>>>>>>>>>>> the java dependencies are engouth? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> best regards, >>>>>>>>>>>>>>> paul >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
