You're the man! When I included the 'statusdir' param I get the following
output in stderr.
Exception in thread "main" java.io.FileNotFoundException:
/tmp/camus_non_avro.properties (No such file or directory)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(FileInputStream.java:146)
at com.linkedin.camus.etl.kafka.CamusJob.run(CamusJob.java:592)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
at com.linkedin.camus.etl.kafka.CamusJob.main(CamusJob.java:562)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:187)
It seems Camus is looking at the local filesystem instead of HDFS for the
properties file. Thanks so much for the help!
On Mon, Dec 30, 2013 at 3:53 PM, Eugene Koifman <ekoif...@hortonworks.com>wrote:
> It looks like in 0.11 it writes to stderr (limited logging anyway).
>
> Perhaps you can try adding '*statusdir*' param to your REST call and see
> if anything useful is written to that directory.
>
>
> On Mon, Dec 30, 2013 at 2:22 PM, Jonathan Hodges <hodg...@gmail.com>wrote:
>
>> I don't see 'TrivialExecService' output in the jobtracker or tasktracker
>> logs. We are using hive 0.11 though so maybe not set to DEBUG?
>>
>>
>> On Mon, Dec 30, 2013 at 2:11 PM, Eugene Koifman <ekoif...@hortonworks.com
>> > wrote:
>>
>>> Is there any output from TrivialExecService class in any hadoop logs?
>>> (it's DEBUG level log4j output in hive 0.12).
>>> It should print the command that TempletonControllerJob's launcher task
>>> (LaunchMapper) is trying to launch
>>>
>>>
>>> On Mon, Dec 30, 2013 at 12:55 PM, Jonathan Hodges <hodg...@gmail.com>wrote:
>>>
>>>> I didn't try that before, but I just did.
>>>>
>>>> curl -s -d user.name=hadoop \
>>>>
>>>> > -d
>>>> jar=/tmp/camus-non-avro-consumer-1.0-SNAPSHOT-jar-with-dependencies.jar \
>>>>
>>>> > -d class=com.linkedin.camus.etl.kafka.CamusJob \
>>>>
>>>> > -d arg=-P \
>>>>
>>>> > -d arg=/tmp/camus_non_avro.properties \
>>>>
>>>> > '
>>>> http://internal-daalt-hcatalog-1507773817.us-east-1.elb.amazonaws.com/templeton/v1/mapreduce/jar
>>>> '
>>>>
>>>> {"id":"job_201312212124_0166"}
>>>>
>>>> DEBUG | 30 Dec 2013 20:33:43,157 | org.apache.hcatalog.templeton.Server
>>>> | queued job job_201312212124_0166 in 300 ms
>>>>
>>>> I still the same behavior with just the TempletonControllerJob getting
>>>> kicked off and ending successfully without the Camus job starting. I
>>>> didn't see any errors in the jobtracker or tasktracker logs. It just seems
>>>> to silently fail and I can't figure out why.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Mon, Dec 30, 2013 at 12:35 PM, Eugene Koifman <
>>>> ekoif...@hortonworks.com> wrote:
>>>>
>>>>> have you tried adding
>>>>> -d arg=-P
>>>>> before
>>>>> -d arg=/tmp/....properites
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Dec 30, 2013 at 11:14 AM, Jonathan Hodges
>>>>> <hodg...@gmail.com>wrote:
>>>>>
>>>>>> Sorry accidentally hit send before adding the lines from webhcat.log
>>>>>>
>>>>>> DEBUG | 30 Dec 2013 19:08:01,042 |
>>>>>> org.apache.hcatalog.templeton.Server | queued job job_201312212124_0161
>>>>>> in
>>>>>> 267 ms
>>>>>>
>>>>>> DEBUG | 30 Dec 2013 19:08:38,880 |
>>>>>> org.apache.hcatalog.templeton.tool.HDFSStorage | Couldn't find
>>>>>> /templeton-hadoop/jobs/job_201312212124_0161/notified: File does not
>>>>>> exist:
>>>>>> /templeton-hadoop/jobs/job_201312212124_0161/notified
>>>>>>
>>>>>> DEBUG | 30 Dec 2013 19:08:38,881 |
>>>>>> org.apache.hcatalog.templeton.tool.HDFSStorage | Couldn't find
>>>>>> /templeton-hadoop/jobs/job_201312212124_0161/callback: File does not
>>>>>> exist:
>>>>>> /templeton-hadoop/jobs/job_201312212124_0161/callback
>>>>>>
>>>>>>
>>>>>> Any ideas?
>>>>>>
>>>>>>
>>>>>> On Mon, Dec 30, 2013 at 12:13 PM, Jonathan Hodges
>>>>>> <hodg...@gmail.com>wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I am trying to kick off a mapreduce job via WebHCat. The following
>>>>>>> is the hadoop jar command.
>>>>>>>
>>>>>>> hadoop jar
>>>>>>> /home/hadoop/camus-non-avro-consumer-1.0-SNAPSHOT-jar-with-dependencies.jar
>>>>>>> com.linkedin.camus.etl.kafka.CamusJob -P
>>>>>>> /home/hadoop/camus_non_avro.properties
>>>>>>>
>>>>>>> As you can see there is an application specific parameter '-P' which
>>>>>>> designates the properties file location. How do I pass this to WebHCat?
>>>>>>>
>>>>>>> Referring to the docs (
>>>>>>> https://cwiki.apache.org/confluence/display/Hive/WebHCat+Reference+MapReduceJar)
>>>>>>> I came up with the following.
>>>>>>>
>>>>>>> curl -s -d user.name=hadoop \
>>>>>>> -d
>>>>>>> jar=/tmp/camus-non-avro-consumer-1.0-SNAPSHOT-jar-with-dependencies.jar
>>>>>>> \
>>>>>>> -d class=com.linkedin.camus.etl.kafka.CamusJob \
>>>>>>> -d arg=/tmp/camus_non_avro.properties \
>>>>>>> '
>>>>>>> http://internal-daalt-hcatalog-1507773817.us-east-1.elb.amazonaws.com/templeton/v1/mapreduce/jar
>>>>>>> '
>>>>>>>
>>>>>>> This command gets the following response from WebHCat
>>>>>>> {"id":"job_201312212124_0161"}
>>>>>>>
>>>>>>> However I only see TempletonControllerJob in the jobtracker UI. I
>>>>>>> don't see the Camus jobs that will show up if executed at the
>>>>>>> command-line.
>>>>>>>
>>>>>>> The following are the only things showing in webhcat.log
>>>>>>>
>>>>>>>
>>>>>>> The jar and properties files are in the /tmp directory on HDFS.
>>>>>>>
>>>>>>> hadoop fs -ls /tmp
>>>>>>> -rw-r--r-- 2 hadoop supergroup 41456481 2013-12-27 17:45
>>>>>>> /tmp/camus-non-avro-consumer-1.0-SNAPSHOT-jar-with-dependencies.jar
>>>>>>> -rw-r--r-- 2 hadoop supergroup 2605 2013-12-27 17:45
>>>>>>> /tmp/camus_non_avro.properties
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> CONFIDENTIALITY NOTICE
>>>>> NOTICE: This message is intended for the use of the individual or
>>>>> entity to which it is addressed and may contain information that is
>>>>> confidential, privileged and exempt from disclosure under applicable law.
>>>>> If the reader of this message is not the intended recipient, you are
>>>>> hereby
>>>>> notified that any printing, copying, dissemination, distribution,
>>>>> disclosure or forwarding of this communication is strictly prohibited. If
>>>>> you have received this communication in error, please contact the sender
>>>>> immediately and delete it from your system. Thank You.
>>>>
>>>>
>>>>
>>>
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity
>>> to which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender immediately
>>> and delete it from your system. Thank You.
>>>
>>
>>
>
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity
> to which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
