Hi Team,
Now i've changed my code and reading configuration from hbase-site.xml
file(this file is in classpath). When i run this program using : mvn
exec:java
-Dexec.mainClass="com.cisco.ana.accessavailability.AccessAvailability". It
is working fine. But when i run this program from spark-submit i'm getting
below exception
Please find below exception :
spark-submit command not able to found the HbaseConfiguration. How to
resolve this issue?
rajesh@rajesh-VirtualBox:~/Downloads/spark-1.0.0$ ./bin/spark-submit
--master local --class com.cisco.ana.accessavailability.AccessAvailability
--jars
/home/rajesh/Downloads/MISC/ANA_Access/target/ANA_Access-0.0.1-SNAPSHOT.jar,
/home/rajesh/hbase-0.96.1.1-hadoop2/lib/hbase-client-0.96.1.1-hadoop2.jar,
/home/rajesh/hbase-0.96.1.1-hadoop2/lib/hbase-common-0.96.1.1-hadoop2.jar,
/home/rajesh/hbase-0.96.1.1-hadoop2/lib/hbase-hadoop2-compat-0.96.1.1-hadoop2.jar,
/home/rajesh/hbase-0.96.1.1-hadoop2/lib/hbase-it-0.96.1.1-hadoop2.jar,
/home/rajesh/hbase-0.96.1.1-hadoop2/lib/hbase-protocol-0.96.1.1-hadoop2.jar,
/home/rajesh/hbase-0.96.1.1-hadoop2/lib/hbase-server-0.96.1.1-hadoop2.jar,
/home/rajesh/hbase-0.96.1.1-hadoop2/lib/htrace-core-2.01.jar,
/home/rajesh/Downloads/spark-1.0.0/assembly/target/scala-2.10/spark-assembly-1.0.0-hadoop2.2.0.jar
Warning: Local jar
/home/rajesh/hbase-0.96.1.1-hadoop2/lib/hbase-client-0.96.1.1-hadoop2.jar,
does not exist, skipping.
Before
*Exception in thread "main" java.lang.NoClassDefFoundError:
org/apache/hadoop/hbase/HBaseConfiguration*
at
com.cisco.ana.accessavailability.AccessAvailability.main(AccessAvailability.java:80)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:622)
at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:292)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:55)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException:
org.apache.hadoop.hbase.HBaseConfiguration
at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
at java.lang.ClassLoader.loadClass(ClassLoader.java:323)
at java.lang.ClassLoader.loadClass(ClassLoader.java:268)
... 8 more
Please find below code :
public class AccessAvailability {
public static void main(String[] args) throws Exception {
System.out.println(" Before");
Configuration configuration = HBaseConfiguration.create();
System.out.println(" After");
SparkConf s = new SparkConf().setMaster("local");
JavaStreamingContext ssc = new
JavaStreamingContext(master,"AccessAvailability", new Duration(40000),
sparkHome, "");
JavaDStream<String> lines_2 = ssc.textFileStream(hdfsfolderpath);
}
}
Regards,
Rajesh
On Wed, Jul 16, 2014 at 5:39 AM, Krishna Sankar <ksanka...@gmail.com> wrote:
> Good catch. I thought the largest port number is 65535.
> Cheers
> <k/>
>
>
> On Tue, Jul 15, 2014 at 4:33 PM, Spark DevUser <spark.devu...@gmail.com>
> wrote:
>
>> Are you able to launch *hbase shell* and run some commands (list,
>> describe, scan, etc)? Seems *configuration.set("hbase.**master",
>> "localhost:600000")* is wrong.
>>
>>
>> On Tue, Jul 15, 2014 at 3:00 PM, Tathagata Das <
>> tathagata.das1...@gmail.com> wrote:
>>
>>> Also, it helps if you post us logs, stacktraces, exceptions, etc.
>>>
>>> TD
>>>
>>>
>>> On Tue, Jul 15, 2014 at 10:07 AM, Jerry Lam <chiling...@gmail.com>
>>> wrote:
>>>
>>>> Hi Rajesh,
>>>>
>>>> I have a feeling that this is not directly related to spark but I might
>>>> be wrong. The reason why is that when you do:
>>>>
>>>> Configuration configuration = HBaseConfiguration.create();
>>>>
>>>> by default, it reads the configuration files hbase-site.xml in your
>>>> classpath and ... (I don't remember all the configuration files hbase has).
>>>>
>>>> I noticed that you overwrote some configuration settings in the code
>>>> but I'm not if you have other configurations that might have conflicted
>>>> with those.
>>>>
>>>> Could you try the following, remove anything that is spark specific
>>>> leaving only hbase related codes. uber jar it and run it just like any
>>>> other simple java program. If you still have connection issues, then at
>>>> least you know the problem is from the configurations.
>>>>
>>>> HTH,
>>>>
>>>> Jerry
>>>>
>>>>
>>>> On Tue, Jul 15, 2014 at 12:10 PM, Krishna Sankar <ksanka...@gmail.com>
>>>> wrote:
>>>>
>>>>> One vector to check is the HBase libraries in the --jars as in :
>>>>> spark-submit --class <your class> --master <master url> --jars
>>>>> hbase-client-0.98.3-hadoop2.jar,commons-csv-1.0-SNAPSHOT.jar,hbase-common-0.98.3-hadoop2.jar,hbase-hadoop2-compat-0.98.3-hadoop2.jar,hbase-it-0.98.3-hadoop2.jar,hbase-protocol-0.98.3-hadoop2.jar,hbase-server-0.98.3-hadoop2.jar,htrace-core-2.04.jar,spark-assembly-1.0.0-hadoop2.2.0.jar
>>>>> badwclient.jar
>>>>> This worked for us.
>>>>> Cheers
>>>>> <k/>
>>>>>
>>>>>
>>>>> On Tue, Jul 15, 2014 at 6:47 AM, Madabhattula Rajesh Kumar <
>>>>> mrajaf...@gmail.com> wrote:
>>>>>
>>>>>> Hi Team,
>>>>>>
>>>>>> Could you please help me to resolve the issue.
>>>>>>
>>>>>> *Issue *: I'm not able to connect HBase from Spark-submit. Below is
>>>>>> my code. When i execute below program in standalone, i'm able to connect
>>>>>> to Hbase and doing the operation.
>>>>>>
>>>>>> When i execute below program using spark submit ( ./bin/spark-submit
>>>>>> ) command, i'm not able to connect to hbase. Am i missing any thing?
>>>>>>
>>>>>>
>>>>>> import java.util.HashMap;
>>>>>> import java.util.List;
>>>>>> import java.util.Map;
>>>>>> import java.util.Properties;
>>>>>>
>>>>>> import org.apache.hadoop.conf.Configuration;
>>>>>> import org.apache.hadoop.hbase.HBaseConfiguration;
>>>>>> import org.apache.hadoop.hbase.client.Put;
>>>>>> import org.apache.log4j.Logger;
>>>>>> import org.apache.spark.SparkConf;
>>>>>> import org.apache.spark.api.java.JavaRDD;
>>>>>> import org.apache.spark.api.java.function.Function;
>>>>>> import org.apache.spark.streaming.Duration;
>>>>>> import org.apache.spark.streaming.api.java.JavaDStream;
>>>>>> import org.apache.spark.streaming.api.java.JavaStreamingContext;
>>>>>> import org.apache.hadoop.hbase.HTableDescriptor;
>>>>>> import org.apache.hadoop.hbase.client.HBaseAdmin;
>>>>>>
>>>>>> public class Test {
>>>>>>
>>>>>>
>>>>>> public static void main(String[] args) throws Exception {
>>>>>>
>>>>>> JavaStreamingContext ssc = new
>>>>>> JavaStreamingContext("local","Test", new Duration(40000), sparkHome, "");
>>>>>>
>>>>>> JavaDStream<String> lines_2 =
>>>>>> ssc.textFileStream(hdfsfolderpath);
>>>>>>
>>>>>> Configuration configuration = HBaseConfiguration.create();
>>>>>> configuration.set("hbase.zookeeper.property.clientPort",
>>>>>> "2181");
>>>>>> configuration.set("hbase.zookeeper.quorum", "localhost");
>>>>>> configuration.set("hbase.master", "localhost:600000");
>>>>>>
>>>>>> HBaseAdmin hBaseAdmin = new HBaseAdmin(configuration);
>>>>>>
>>>>>> if (hBaseAdmin.tableExists("HABSE_TABLE")) {
>>>>>> System.out.println(" ANA_DATA table exists ......");
>>>>>> }
>>>>>>
>>>>>> System.out.println(" HELLO HELLO HELLO ");
>>>>>>
>>>>>> ssc.start();
>>>>>> ssc.awaitTermination();
>>>>>>
>>>>>> }
>>>>>> }
>>>>>>
>>>>>> Thank you for your help and support.
>>>>>>
>>>>>> Regards,
>>>>>> Rajesh
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
